I have an application that consists of a Windows Phone client sending HTTP requests to a Python server hosted in Google App Engine. In the GAE log, I see that I often receive multiple identical requests from the same client within a few milliseconds (see below). I never saw this behavior when testing the client in my development environment. Nonetheless I realize that this is probably error in my code, but my question is:
Can any part of the infrastructure (the mobile network, the internet, the google app engine itself) cause requests to be duplicated?
And if so, a follow on question is: are there best practices to minimize this?
No, HTTP requests are not be duplicated by the underlying infrastructure. At least they should not be.
What probably happened is that you see mobile app requests which are made in native cod and they do not use cookies and have same user agent string. The same IP is because mobile networks internally use NAT, hiding multiple (possibly thousands) clients behind a single IP address.
It's really not possible for the network to duplicate HTTP requests. It can duplicate IP datagrams with misconfigured routing, but the TCP layer filters duplicate IP datagrams so that the end to end connection only sees one TCP stream. App engine might reuse TCP ports without the standard time to wait for the previous TCP connection to die for performance purposes, but I still don't think duplicate packets would survive from the three-way TCP handshake used to initialize connections.
Related
I am reading about dos.xml for configure Java Application against DOS attack.
I did not understand how this configuration works.
I know about DOS and DDOS, but in this case it is confused for me.
When I configure Blacks of IP or subnets, will all requests be blocked? I think that GAE should block only if it receives too many requests of the same IP
What is the best way to prevent too many requests (evil requests ) in my login page? I can not have IP whitelist, because the app is available for any places in the world.
What is the difference between a black IP in two.xml and block IP in App Engine firewall?
Thanks !
When I configure Blacks of IP or subnets, will all requests be blocked? I think that GAE should block only if it receives too many requests of the same IP
Yes, that's how it's suppose to work: the IP address will be blocked only when AppEngine thinks they're abusing your resources (whatever that means). See more here:
Do not use this service for security because some requests from blacklisted users can still get through to your application. This service is designed for quantitative abuse prevention, such as preventing DoS attacks, only.
Unfortunately there is no clear algorithm described how it's handling this so it's probably best not to rely on it.
What is the best way to prevent too many requests (evil requests ) in my login page? I can not have IP whitelist, because the app is available for any places in the world.
You would have to implement some kind of rate limiting (by IP addresses or by accounts being accessed) yourself. Shouldn't be very difficult by incrementing some memcache key's value on each authentication attempt and setting it to expire in N minutes (or whatever approach that works best for you).
What is the difference between a black IP in two.xml and block IP in App Engine firewall?
A couple of differences between dos.yaml and AppEngine Firewall that comes to mind:
AppEngine Firewall can completely block access from a given IP address while the addresses listed in dos.yaml may still reach your app
AppEngine Firewall allows up to 1,000 rules while dos.yaml can only handle up to a 100
With AppEngine Firewall you can set up a whitelist (i.e. give access to your app only to, say, your company's network and no one else)
Also, judging from the wording AppEngine docs are using - it seems dos.yaml isn't the recommended approach so you would likely need to implement something yourself.
From looking at App Engine's XMPP docs at https://cloud.google.com/appengine/docs/go/xmpp/
It seems that Google only offers a client to be hosted on appengine, but not the XMPP server itself. For that, one needs to use a different host (such as GTalk)
Am I understanding this correctly?
If so- does that mean I must host my own xmpp server (ejabbered) if I want on-the-fly session-based clients?
If that is also true- then is there a mechanism in ejabbered for lightweight session-based clients? (i.e. many to be quickly created and destroyed- only needs to respond to presence requests for the duration of the session, will never be used again)
Is there a stable, scalable host out there that can do this for me so I don't need to worry about the ejabbered server going down (whether it's hosted on AWS,GCE, etc. the beauty of AppEngine so far was I never had to worry about that)
Thanks!
To address your questions:
Am I understanding this correctly?
You use to have to use Google Talk XMPP service, but I guess, yes, now you need to deploy your own server.
If so- does that mean I must host my own xmpp server (ejabbered) if I want on-the-fly session-based clients?
Not sure what "on-the-fly" session means, but yes, it seems you need your own server.
If that is also true- then is there a mechanism in ejabbered for lightweight session-based clients? (i.e. many to be quickly created and destroyed- only needs to respond to presence requests for the duration of the session, will never be used again)
Not sure what you mean by this. XMPP is a connected protocol. It means the session is linked to having a TCP connection opened. That said, ejabberd SaaS allows to maintain the session for a while if you do lose the connection (designed for mobile). You can simply reattach to it.
And by the way: In XMPP, you do not "respond to presence request". This is the reverse: Your presence is broadcasted to your contact (roster)
Is there a stable, scalable host out there that can do this for me so I don't need to worry about the ejabbered server going down (whether it's hosted on AWS,GCE, etc. the beauty of AppEngine so far was I never had to worry about that)
The easiest (and arguably cheapest) way to use ejabberd is to use ejabberd SaaS, managed by ProcessOne, developer of ejabberd (I am developer of ejabberd and founder of ProcessOne).
I am trying to generate lot of requests from my angularjs app to a backend rest server. I am using http service to make the calls. I want to control the concurrency of requests going to server.
I know that browser itself throttles the number of connetions per server.
Now my questions are:
How can I control the number of connections chrome opens to a server? How to do it using angularjs?
How does the angularjs http service works? Does it opens and close the http connection every time I make a http call? If yes then how can I create persistent connections in angularjs?
XMLHttpRequest is used under the hood. Since it uses HTTP, most of the answers to your questions are dependent on the headers you use and the server you are communicating with. For instance, whether or not connections persist depend on whether or not Keep-Alive is specified and the server supports it.
As for the number of connections, this is generally limited by the browser on a per domain basis. You won't be able to make use of more than the browser can allow, but you can probably write your own code to throttle it down if you so desire.
This question has good information about connection limits for the various browsers.
Do you happen to know common application using unix socket api doesn't work on computer connected to internet router? For example, assume that there is a computer that is running a simple web server using socket in C. when a web browser in another remoter computer send a request, the web server cannot send a response to the request since its port is closed by the internet router(?) (Of course, there might exist another reasons).
However, the common applications by a competent developers works well. For example, utorrent client receives a request for some data from peers and responds to the request well, although a computer that is running utorrent is connected to the internet router. Does utorrent adjust router configurations using some system calls? If not, how does it upload the some data?
So my question is that
how does common application using socket API accomplish to forward its port, with the connection to the internet router?
How my program in C accomplish to forward its port with computer connected to the internet router?
Thank you in advance.
If you're connected the internet through a NAT router, in most cases any unsolicited connection into the router from the WAN will be refused. What you need is to tell the router (in some way) that unsolicited traffic coming in on a specific port number or range is to be accepted, and forwarded on towards a specific local IP address. This can either be done manually in your router's configuration, or if your router supports UPnP you can use that protocol to configure port mapping for the traversal of the router.
They don't. To send and receive data on the connections your program has started it's not needed. Port forwarding needs to be done by hand by the machine administrator and is only required to receive new connections.
What are the usual patterns for bidirectional communication between a client and a server in a wlan environment. How is it possible for the server to push data to a mobile client over wlan after a connection has been established.
Lets say I have a webservice running on a server and the moblie cients in the wlan can use this webservice. Now the question is how can the server invoke methods at the client, or directly send data to the client. How is this handled usually?
I would apriciate some links to read about this topic.
Is this a common problem or is it not that easy to solve?
Cheers
HTTP server push (also known as HTTP streaming) is a mechanism for sending data from a web server to a web browser. HTTP server push can be achieved through several mechanisms.
More at http://en.wikipedia.org/wiki/Push_technology#HTTP_server_push
In web development, Comet is a neologism to describe a web application model in which a long-held HTTP request allows a web server to push data to a browser, without the browser explicitly requesting it. Comet is an umbrella term for multiple techniques for achieving this interaction. All these methods rely on features included by default in browsers, such as JavaScript, rather than on non-default plugins.
More at http://en.wikipedia.org/wiki/Comet_(programming)
Also there is a recent IETF draft on
Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTP
https://datatracker.ietf.org/doc/html/draft-loreto-http-bidirectional-01