I am using libcurl with pthreads to do some POST requests to a single website. This website may change over time. I want this program to be as fast as possible. So one thing came to my mind was to first find the website IP address, then pass this IP instead of the website URLs to threads.
At first, do you think this is a good thing to do? As it removes the DNS lookup time in each connection (thread).
And how can I do this in libcurl? Is there any function to just find the IP of a hostname? Or is there any other simple way to do this in C language?
This is the problem that the DNS query system in your OS is supposed to solve.
The performance improvement you want by not doing an additional lookup is countered by the fact that the IP can change over time. Any queries should be cached presumably respecting some record timeout rules set by the DNS entry. Windows and OS X cache requests automatically out of the box as far as I can tell. On my debian box I found I had to enable the systemd-resolved service for this feature. Not knowing your needs, is probably the most sane general solution to what you're describing.
Doing this with getaddrinfo() and caching it yourself for some period of time within the bounds of your program (even to disk) is a valid solution if you want absolute control and don't want to hardcode it somewhere (in the app or something like the hosts file).
Related
I'm trying to come up with a way to sync a sqlite database between two computers.
If this were on a machine with a public IP this would not be difficult but I'm trying to find a way to make this work for ANY two devices, and most computers don't have a static IP.
What are some of the ways I can tackle this problem?
Assuming you just need to find the peers IP address...
Broadcast a Udp packet onto Lan, if machines are same lan segment. You can also try using admin scoped multicast, but mileage will vary according to the network setup and gear in use.
If trying to find two machines across the internet (assuming you can solve the NAT address translation issue), you need to bounce off a node that will hold info for you. Eg write a packet to dweet.io or sparkfun or other website that will hold store and forward data. You can also read twitter feeds etc, basically you need a known reference point to both talk too. Look into how malware create command and control networks for some ideas. Or search for rendezvous servers and protocols.
If the address range is small, probe all possible addresses. But be careful as you might trigger anti virus or ISP action
If wanting more browser based, look at webrtc, not quite what you are after but some of the techniques for discovery might be interesting.
If you have access, you can play with your DNS records. Essentially this is a variation of (2).
There are more options too, but that get more special purpose or become a bit too stelthy for general use. Also see how mesh networks are formed.
Is there any way to query the system's date/time via USB without installing anything on the host computer (maybe just drivers)?
Background of the original problem
To avoid the XY problem, let me explain a bit what I'm trying to do.
To be able to calculate a TOTP token for 2FA (e.g. like Google Authenticator app does) you need a real-time clock to get the date and time.
There's this USB device called SC4-HSM that I would like to use to calculate the tokens, however it doesn't have a clock and according to the designer, adding one would be too expensive (needs a battery, etc).
Possible solution to the original problem
This device is going to be used with a computer which already has an RTC of course. Thus I had the idea of querying the system for a date/time which would solve the issue.
(Note: I know that a USB device can be connected to all sorts of hosts and not all hosts will have an RTC, but since this only needs to work with a computer, I thought this shouldn't be an issue)
My first thought was that there might be some USB device class that had date/time needs, so I could register the device as that type and then I would be able to query the values.
After going through the device class codes list (Internet Archive) nothing jumped at me as needing date/time. The closest ones I could think of were:
Content Security (PDF)
Personal Healthcare
Smart Card Class (PDF)
I skimmed the device class documents in the USB Implementers Forum but there's nothing in there even remotely related to date or time.
Current problem
Since the USB specs seemed like a dead-end I thought that maybe there was a way to write a very simple USB driver that can be auto-loaded when the device is plugged in to a computer and then we can use the driver to return the date/time when the device asks for it (unless I'm misunderstanding something).
I am now looking through USB development docs like Michael Opdenacker's Linux USB drivers course, I tried the Linux USB Project which seems dead. Skimmed through Driver Development for Windows NT just to get an idea, however I am still not able to figure out if this is possible or not, and how hard it would be.
I'm a complete beginner at this and maybe this is something out of my skill level, but I would like to figure out if will I need weird hacks and workarounds or is there a much more straightforward way to do this?
There seems to be little information about it or I'm just searching the wrong places.
Any ideas/or pointers on either solving the original problem or the current one?
system time is not necessarily the general time i.e. the 'atomic' time you get from a NTP server
the most obvious solution is to use autorun, this is also possible on linux but normally autorun is blocked so the user explicitely has to activate it
https://askubuntu.com/questions/642511/how-to-autorun-files-and-scripts-in-ubuntu-when-inserting-a-usb-stick-like-autor
the linux command to get the time is date or hwclock or if the computer is connected to the net it may be possible to contact a NTP server (if the firewall does not block this)
then your autorun program has to send the data to the SC4-HSM. i do not know what USB classes the SC4-HSM implements if it implements CDC ACM (virtual COM port) this is easy:
Unable to sync computer time to Arduino via USB
(something like echo "T$(($(date +%s)+60*60*$TZ_adjust))" >/dev/tty.usbmodemfa131)
maybe it is possible to access system time over the USB drivers, i do not know this right now
One of my stand-alone java applications (no sources available) picks random-available port to listen on.
At this stage I assume it uses getaddrinfo system call to obtain addresses to bind against.
Since I'm maintaining hundreds of various servers with assigned ports, the black app sometimes kicks in and pick one of 'the assigned' ports, which cause my small servers to fail on startup...
I'm wondering is there a way to restrict number of ports proposed by the OS?
Would be mostly interested in system config solutions,
but if there are no other solutions I'm also able to hack bind()/getaddrinfo (this would require some hits as well ... )
thanks
You must be able to control it from proc entries - For example, here is a system wide setting :
/proc/sys/net/ipv4/ip_local_port_range
You can modify them. Or there may be utilities available for the same purpose.
If OS-wide change is not what you had in mind, configure the JVM's Java Security Manager so that SecurityManager.CheckListen(NNN) throws SecurityException for any of the port numbers you want to reserve.
Take a look on:
http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap6sec70.html
It's the solution for my problem, than I could limit port ranges
I need to discover all network neighbors in Linux(they are running Linux too) and I need to get theirs IP addresses(3rd layer). Any ideas how to do that?
Btw, I need to do that in C, not in shell
Many thanks in advance!
What you should do is, have the neighbours run a daemon which responds (with a unicast response to the sender) to UDP multicasts.
Then send a UDP multicast with a TTL of 1 (so it will not be routed) and listen to see who responds. You will only receive responses from the neighbours which are running the agent.
Another possibility is to use an existing protocol which already does this, for example, mDNS.
There is no guaranteed way to do this if the machines in question aren't co-operating.
The best you can do is to scan likely addresses and probe each one to see if you can get a response - that probe could be anything from a simple ICMP echo request (a ping) up to a sophisticated malformed packet that attempts to elicit a response from the remote host.
The level of sophistication required, and whether it will work at all, depends entirely on how heavily firewalled etc the host in question is.
As a commenter has already observed, there are entire programs like nmap dedicated to attempting to discover this information, which gives some idea of how non-trivial this can be.
At the other extreme, if the hosts are co-operating, then a simple broadcast ICMP echo request might be enough.
If your segment uses reasonably decent switch, you can discover the link-layer neighbours by inspecting the forwarding database of one of the switches. You should be able to obtain this fairly automatically via SNMP, check your switch's documentation.
Once you have a list of link neighbours, you can try and find out their IP addresses, but remember that they may have many or none at all. For this you'd need some sort of reverse-ARP. Perhaps your router maintains a list of MAC-to-IP associations and you can query it (again SNMP would be the most convenient solution).
Let's suppose you deploy a network-attached appliances (small form factor PCs) in the field. You want to allow these to call home after being powered on, then be identified and activated by end users.
Our current plan involves the user entering the MAC address into an activation page on our web site. Later our software (running on the box) will read the address from the interface and transmit this in a "call home" packet. If it matches, the server response with customer information and the box is activated.
We like this approach because it's easy to access, and usually printed on external labels (FCC requirement?).
Any problems to watch out for? (The hardware in use is small form factor so all NICs, etc are embedded and would be very hard to change. Customers don't normally have direct acccess to the OS in any way).
I know Microsoft does some crazy fuzzy-hashing function for Windows activation using PCI device IDs, memory size, etc. But that seems overkill for our needs.
--
#Neall Basically, calling into our server, for purposes of this discussion you could call us the manufacturer.
Neall is correct, we're just using the address as a constant. We will read it and transmit it within another packet (let's say HTTP POST), not depending on getting it somehow from Ethernet frames.
I don't think that the well-known spoofability of MAC addresses is an issue in this case. I think tweakt is just wanting to use them for initial identification. The device can read its own MAC address, and the installer can (as long as it's printed on a label) read the same number and know, "OK - this is the box that I put at location A."
tweakt - would these boxes be calling into the manufacturer's server, or the server of the company/person using them (or are those the same thing in this case)?
I don't think there's anything magic about what you're doing here - couldn't what you're doing be described as:
"At production we burn a unique number into each of our devices which is both readable by the end user (it's on the label) and accessible to the internal processor. Our users have to enter this number into our website along with their credit-card details, and the box subsequently contacts to the website for permission to operate"
"Coincidentally we also use this number as the MAC address for network packets as we have to uniquely assign that during production anyway, so it saved us duplicating this bit of work"
I would say the two obvious hazards are:
People hack around with your device and change this address to one which someone else has already activated. Whether this is likely to happen depends on some relationship between how hard it is and how expensive whatever they get to steal is. You might want to think about how easily they can take a firmware upgrade file and get the code out of it.
Someone uses a combination of firewall/router rules and a bit of custom software to generate a server which replicates the operation of your 'auth server' and grants permission to the device to proceed. You could make this harder with some combination of hashing/PKE as part of the protocol.
As ever, some tedious, expensive one-off hack is largely irrelevant, what you don't want is a class-break which can be distributed over the internet to every thieving dweep.
The MAC address is as unique as a serial number printed on a manual/sticker.
Microsoft does hashing to prevent MAC address spoofing, and to allow a bit more privacy.
With the only MAC approach, you can easily match a device to a customer by only being in the same subnet. The hash prevents that, by being opaque to what criteria are used and no way to reverse engineer individual parts.
(see password hashing)
From a security perspective, I know that it is possible to spoof a MAC, though I am not entirely sure how difficult it is or what it entails.
Otherwise, if the customers don't have easy access to the hardware or the OS, you should be fairly safe doing this... probably best to put a warning sticker on saying that messing with anything will disrupt communication to the server.