How to implement an efficient map tile engine on Google App Engine? - google-app-engine

Question

I am trying to implement a map tile engine on Google App Engine.
The map data is stored in the database, the datastore (big table). The problem is that 20 requests might come in at approximately the same time to draw 20 tiles based upon the same set of rows in the database.
So 20 requests come in, if I write the code to read from the database for each request, then I will be doing 20 read's, which are the same, from the database, one read for each tile image output. Since each read is the same query, it doesn't make sense to do the same query 20 times. In fact, this is very inefficient.
Can anyone suggest a better way to do this?
If I use the memcache, I need to put the data into memcache, but there are 20 requests coming in at the same time for the data, then if I do a nieve implementation then 20 processes will be writing to memcache, since they are all going at the same time in parallel.
I am programming in Google Go version 1 beta on Google App Engine, I refer to the Python doc's here since they are more complete.
References:
Google datastore http://code.google.com/appengine/docs/python/datastore/overview.html
Leaflet JS I am using for showing map tiles http://leaflet.cloudmade.com/
To clarify.
I generate the tile images from data in the database, that is, I query the database for the data (this is not the tile image), then I draw the data into a image and render the image as a JPEG. As GAE is efficient for drawing images on the server side http://blog.golang.org/2011/12/from-zero-to-go-launching-on-google.html

Answer 1

I don't know about how Google App Engine does it, but MySQL has a query cache so that if the same query gets asked twice in a row, then it uses the results from the first to answer the second. Google is smart about things, so hopefully they do that as well. (You might be able to figure out if they are by timing it.)
One thing you might need to make sure of is that the queries are exactly the same, not just returning the same results. For example, you don't want query1 to be
SELECT lat, lng FROM mytable WHERE tileX=1 AND tileY=1
and query2 to be
SELECT lat, lng FROM mytable WHERE tileX=1 AND tileY=2
I make tiles with gazillions of polygons, and when I did timing and optimization, I found to my surprise that it was faster to return ALL values and weed out the ones I didn't want in PHP than it was to stick in a WHERE clause to the SQL. I think that partly it was because the WHERE clause was different for every tile so the MySQL server couldn't cache effectively.

Answer 2

Organize tile entities so that you can find them via key instead of querying for them, i.e. using get() instead of query(). If you identify a tile based on several criteria, then create a natural ID by combining the criteria. E.g. if you would find a tile based on vertical and horizontal position inside an image then you'd do: naturalID = imageID + verticalID + horizontalID (you can also add separators for better viewing).
Once you have your own unique IDs you can use it to save the tile in Memcache.
If your tiles are immutable (= once created, their content does not change), than you can also cache them inside instance in a global map.
Edit: removed Objectify reference as I just realized you use python.
Edit2: added point 3.

Answer 3

A few things comes to mind:
How do you query for the tiles? you should be able to get the tiles using Key.get() which is way more efficient than query
Try to reduce the number of requests, using levels should reduce the number to around 4 request to retrieve the map.