I'm working with a third party webservice who requires that all calls to their service are made from whitelisted IP addresses. That is, I must give them IP addresses from which I will be making calls to their service.
Problem is I'm using Google Appengine. Is there any way to get a static IP address when making outgoing http requests from Appengine? Failing that - is there a block of IP addresses that all requests will come from? I could get the entire bloc whitelisted. If this exists, how likely is it to change?
I know I could setup a simple Amazon EC2 instance to use as a proxy (will ask another question for how to do this specifically) but just wanted to make sure there was no other way.
I had the same problem a couple of weeks ago connecting via Urlfetch from Google App Engine to the Stack Exchange API (The team has promptly fixed the problem whitelisting all the GAE IPs).
The range of IP addresses that urlfetch connections may come from, can be found by performing the following DNS lookup:
dig -t TXT _netblocks.google.com #ns1.google.com
Last I checked this wasn't possible. You can get the current IP address dynamically, but it isn't predictable.
Please note: _netblocks.google.com is apparently not accurate. Currently I have noticed that GAE connects from addresses not listed when you dig _netblocks, for example from 8.35.201.166.
This range is not listed in _netblocks, _netblocks2 or _netblocks3.
Current dig output:
ip4:216.239.32.0/19
ip4:64.233.160.0/19
ip4:66.249.80.0/20
ip4:72.14.192.0/18
ip4:209.85.128.0/17
ip4:66.102.0.0/20
ip4:74.125.0.0/16
ip4:64.18.0.0/20
ip4:207.126.144.0/20
ip4:173.194.0.0/16
Related
I've a simple webserver bound on 0.0.0.0:3000 on my machine which works as intended on local networks. By local network I mean to say, if my mobile or any other device is on the same network, it can access local webserver by going to the IP assigned to my machine and adding port 3000 to it. Eg 192.168.1.4:3000.
Now I've to expose it to the internet but not through some sort of 3rd party application like ngrok, localtunnel or browserSync. I know that these applications work perfectly, but since I've my own pet project of controlling home appliances, I don't want to rely on availability of 3rd party services. So the current state is, I cannot control it through the internet. Keeping in mind I don't have a static IP otherwise this would've been easier.
I already have a vps and a domain name assigned to it. I can send my currently allocated ip address (since it is dynamic), by using getifaddrs, to my server and keep track of it. But how do I expose my local server to the internet through it? Those 3rd party applications assign some sort of subdomains to each exposed server, and I'll be able to assign subdomains too, but I'm still not getting any way to expose the local webserver. Any help would be appreciated, thankyou :)
Step one, you need to expose your webserver at your internet access router.
Typically this requires you to configure port forwarding for (in your case) port 3000.
With this done, any client could access your service via (current external dynamic ip):3000
Step two, you need to dynamically map a fixed DNS name to your current dynamic ip. There are of course third party services (such as DynDNS) that would help you map yourfavoritename.dyndns.org to that ever-changing ip address.
If you want to do the latter without 3rd party, you need to have some static (web) server somewhere and could proceed as follows:
Clients visit http://www.yourstaticserver.example/ and that server redirects them to (current dynamic ip):3000.
Of course, for this to happen, your static server needs to know the dynamic ip and needs no find out about changes to it.
To this end, you could have your internal server contact the static server on a regular interval (such as once a minute), say, have it access http://www.yourstaticserver.example/some-secret-special-page and the static server always stores the REMOTE_ADDR of such a request (preferably with some authorization!) for its future redirections.
Actually, there is a step zero before step one: Be aware that exposing your server to the Internet means that you expose your server to the Internet. So I hope you have invested enough thought into security.
I am calling a third party web service from app engine. This particular service is picky. I ran into an issue where calls would work fine for a while, then stop working, then start working again. I realized that if I manually stopped all instances in the admin console, that the calls would work again.
I setup a proxy to route the calls through that so I could see the headers and all detail. I think I have tracked the issue down to the following. After an instance has been up for a while (the app usually just needs 1 to 3 instances right now) app engine will start using the IP address of the destination as the value for the host header instead of the hostname. Well the service doesn't like that. Whether it should care is another matter.
So my question is, why does app engine use the ip address for the host header eventually instead of the hostname? And, of course, is there anything I can do about it? I know that I cannot set the host header, but maybe there is something else that can be done.
Thanks for any insight.
First, thank you for finding this behavior. We have had intermittent issues with urlfetch for a long time, and will try to detect if this is the issue.
One thing you could try is to target a specific instance/module:
http://instance.version.module.app-id.appspot.com
and cycle through the instances. If you just target the module, it will kill the instance after some inactivity. So, perhaps that would not trigger the GAE DNS shortcut.
Another trick would be to add a fake, random, query string after your url: ?foo=D7hfka67h. Perhaps that would prevent GAE from recognizing the repeat url, and trying to shortcut the DNS.
For software I am developing, I need to have a list of external IP addresses that Google App Engine uses for urlFetch requests. Getting a complete list is proving difficult.
The accepted wisdom as detailed on SO is to use the output from...
dig -t txt _netblocks.google.com
...but unfortunately this list is incomplete. For example, my urlFetch requests currently emerge from unlisted addresses:
US based App: 8.35.201.x
EU based App: 8.35.200.x
Is there any way to get an actively updated list of external IP addresses used by urlFetch?
The official advice is https://developers.google.com/appengine/kb/general#static-ip. Note that relying on Static IP addresses is bad, for all the reasons listed on that page.
So, the answer is:
dig txt _cloud-netblocks.googleusercontent.com +short
"v=spf1 include:_cloud-netblocks1.googleusercontent.com include:_cloud-netblocks2.googleusercontent.com include:_cloud-netblocks3.googleusercontent.com ?all"
I want to host an SMS application on GAE and all my traffic will come
from a SMS GATEWAY with a single IP address.
Is that fine(I'm expecting 500 dynamic requests/sec) ?
Will there be any problems like unusual traffic errors or any other issues ?
EDITED
More info:
My users send queries through SMS which will be routed to my app from SMS GATEWAY(single IP address).
My app processes those queries and reply back through SMS (again through SMS GATEWAY).
I can reply using URL FETCH(not a problem) but what i'm worried about is if I receive some 500 dynamic requests/sec from single IP address , GAE might block them thinking them as Dos(denial of service) attack .
GAE either asks the user to enter a captcha at https://www.google.com/accounts/DisplayUnlockCaptcha or redirects to sorry.google.com and displays an error message itseems if it receives unusual traffic from single IP . But my users access app only through SMS.
Please look at this production issue filed.
It would be technically doable - your app can detect the user's IP via the REMOTE_ADDR environment variable, and if it's the one you want, show them the actual page (showing them a 403 otherwise). Your second question is a bit trickier to answer - your App Engine app could handle it assuming you wrote it in a scalable manner (not a trivial assumption!) and if you can afford the amount of traffic you're trying to throw at it.
You're right to be concerned that getting that level of traffic from a single IP might set off some form of DoS protection - it shouldn't, but it's impossible to rule it out. If it were to happen, you could file a production issue, and we'd take care of it.
I am using Nokogiri to screen scrape a few websites.
My website is hosted on US servers, and so when it fetches the website, the website responses as if the user was a US user. I need the website to responds as if I was an Australian user, even though the server is located in the US.
When running locally it works fine because locally it responds as if it was Australian.
How can I read the site using Nokogiri as if I was from another country?
You have to run your requests through a proxy in Australia.
This doesn't have anything to do with Nokogiri - it applies no matter how you're trying to scrape a page. HTTP travels over TCP, which is a bi-directional protocol so you can't spoof its IP address. If you tried to spoof the IP address of a TCP packet, you would never get your response back.
You can configure Tor to always use exit nodes from a specific country. Please, however, do not use this method if this will put the Tor network under some serious strain (i.e. fetching the pages continually); in this case please consider buying an (Austrlia-based) annonimizing service (or simply a proxy).