Asynchronously pull data from datastore and draw map - google-app-engine

Question

I have a Spring MVC 3 app (that uses JSP) running on Google App Engine and saving information on the Datastore. I'm using the Google Maps API v3 to project some of the data on maps by drawing shapes, colouring etc. My database will potentionally hold millions of entries.
I was wondering what the best way is to keep pulling data from the datastore and project them on the map until there are no more database entries left to project. I need to do this to avoid hitting the 30 seconds limit (and getting a DeadlineExceededException) but also for good user experience.
Is it worth using GWT?
Any advice would be great.
Thanks!

Answer 1

You could use a cursor similar to the pagination technique described here:
Pagination in Google App Engine with Java
When your page with the map loads, have it make an AJAX request with a blank cursor parameter. The request handler would fetch a small number of entities, then return a response containing them and a cursor (if there are entities remaining).
From the client javascript, after displaying the items on the map, if there is a cursor in the response start a new request with the cursor as an argument. In the request handler if a cursor is provided, use it when making the query.
This will set up a continuous loop of AJAX requests until all items have been fetched and displayed on the map.
Update:
You could write a service which returns JSON something like this:
{
items:
[
{ lat: 1.23, lon: 3.45, abc = 'def' },
{ lat: 2.34, lon: 4.56, abc = 'ghi' }
],
cursor: '1234abcd'
}
So, it contains an array of items (with lat/lon and whatever other info you need per item), as well as a cursor (which would be null when the last entity has been fetched).
Then, on the client side I would recommend using jQuery's ajax function to make the ajax calls, something like this:
$(document).ready(function()
{
// first you may need to initialise the map - then start fetching items
fetchItems(null);
});
function fetchItems(cursor)
{
// build the url to request the items - include the cursor as an argument
// if one is specified
var url = "/path/getitems";
if (cursor != null)
url += "?cursor=" + cursor;
// start the ajax request
$.ajax({
url: url,
dataType: 'json',
success: function(response)
{
// now handle the response - first loop over the items
for (i in response.items)
{
var item = response.items[i];
// add something to the map using item.lat, item.lon, etc
}
// if there is a cursor in the response then there are more items,
// so start fetching them
if (response.cursor != null)
fetchItems(response.cursor);
}});
}