Could use some help with this please.
Django and PostgreSQL is the backend for a React frontend project of mine. The frontend needs to fetch and perform calculations on a large number of rows in a table... a million + rows is possible.
Right now, fetching and doing those calculations is implemented, but it takes way too long, and that's locally. When deployed, it doesn't even work because it's not in batches, it's just one big GET request which times out after 30 seconds.
I've set up splicing on the frontend for dealing with POST/PUT related requests to get around that time out issue, but how would I go about doing a GET request in batches? I've been looking into iterator and cursor stuff to try and deal with things in chunks, but I am also confused - if a single GET request is performed on the frontend, is it even possible to send multiple responses? I have seen conflicting information on that.
Some guidance on the best way to address this would be appreciated. If it helps, I am primarily working with a class based way of doing things, Model, Serializer, ModelViewSet.
Here's what I'm working on currently to attempt to do batch get-requests, but not sure if I'm on the right page.
def batch_get(self, request):
count = UploadedShares.objects.all().count()
chunk_size = 500
for i in range(0, count, chunk_size):
shares = UploadedShares.objects.all()[i:i + chunk_size]
serialized_shares = serializers.serialize('json', shares)
return JsonResponse(serialized_shares, safe=False)
Thanks.
Related
I have a Python application deployed on Google Cloud Platform. There is a Google Cloud Datastore in the background, with two Kinds. I use NDB to pull the data into the application.
class AttEvent(ndb.Model):
event = ndb.StringProperty()
matchdate = ndb.DateTimeProperty()
class MainPage(webapp2.RequestHandler):
def get(self):
query = AttEvent.query().order(AttEvent.matchdate)
for q in query.fetch():
try:
# application code
One of the Kinds (AtEvent in the code above) is causing me trouble. The app will deploy and work as expected for hours / days, but then intermittently stop returning data. Debugging shows the q object is legitimate object of the type AttEvent, but for each of the items in the values collection, it says "(Object has no fields)". When the application code attempts to reference a property of the model (i.e. q.event), it fails.
The query will suddenly start working again, minutes / hours later, even if I take no action. I can't see any pattern or apparent cause. Obviously this isn't ideal from a user perspective.
The Kind that is causing trouble is static data and only actually contains 3 entities. The other Kind is transactional, contains thousands of records, but has never exhibited the same behaviour.
The intermittent nature of the fault leads me to believe this is something to do with caching, but I am fairly new to Python and GCP, so I am not exactly sure. I've tried doing a context.clear_cache() before the query, but it has no effect.
Am I missing something obvious?
I don't know why this is happening, but I have a possible work around. Since the data is static and the entities seem to be small, you could store them in instance memory instead of querying for them every time you need them.
Store the entities in a module level variable like this:
att_entities = AttEvent.query().order(AttEvent.matchdate).fetch()
class AttEvent(ndb.Model):
event = ndb.StringProperty()
matchdate = ndb.DateTimeProperty()
class MainPage(webapp2.RequestHandler):
def get(self):
for q in att_entities:
try:
# application code
You would get the entities only when a new instance is launched so as long as it works the first time you are all set. As a bonus, it will make the get call faster since you don't need to retrieve the data from the data store.
You might need to add extra logic to cause att_entities to be updated as needed.
My problem is I'm making too many API requests, which I want to cut down if possible. Below I'll describe the situation:
I have three pages, all linked using ngRoute. Like this:
Page A: Teams (list of teams)
URL: "/teams"
Page B: Team Details (list of players)
URL: "/teams/team-details"
Page C: Player Details (list of player stats)
URL: "/teams/team-details/player-details"
Page A is populated by pulling an array of the teams from an API very easily using a simple $resource.query() request, and using ng-repeat to iterate through them.
Page B is populated by calling an html template and populating specific fields with values from a separate API request to the /team-details endpoint, taking the team_id value from the clicked element on Page A.
Page C (as with page B) takes a player_id from the clicked player on Page B and calls the /player-details endpoint using that value. This is yet another separate request.
This all works fine, but as you can imagine, a single user could quite easily rack up in excess of 100 API requests within an hour.
I have a request limit of 1000/hour, so if a mere 10 users are online at the same time, it could easily exceed my limit and shut down my API.
If I could access the API as one single master endpoint that outputted all data and subdata in one set, then that would solve my problem, but since I need to request separate endpoints I can't see how to do this.
Is there a better way to approach this? Or are these excessive API requests the only way?
Any help would be appreciated.
As far as I can see, Your model looks suitable for the application and meets how an API-driven application should work...
However, One potential cut-down you could make is to cache some of the results locally. i.e. store a local version of some of the data that is unlikely to change within a session. For example, If the number of teams is unlikely to change, then store the results of 1 API request locally and use that instead of recalling data from your API.
Following on from this route, you could choose to only update certain data after a certain time period. So, if a user has looked at some team-details then refuse to update this data for the next 10-20minutes. However, this does again depend how time-sensitive your data is.
I am currently using ajax to do autocomplete emails and would like to find out what is the best way to do this without too much read operations. Thanks!
The best way to do these kind of operations is use the following approach
Use full text search:
https://cloud.google.com/appengine/docs/java/search/
When creating a document to search on, you could tokenize the email id. for example if you have foobar#baz.com. you could tokenize it to f, fo, foo, foobar .... and save it into a textfield.
then use index.search to query for the results.
then every successful lookup can be cached for say 2 hours ( you can change it as per your requirement ).
Anytime you update the model add/update/remove entries then delete the memcache entries/flush the memcache, preferably using the datastore callbacks.
https://cloud.google.com/appengine/docs/java/datastore/callbacks
please note that the tokenize + adding a document could to be processed in task queue to fit into the "gae way of doing things"
Also as a footnote, you could try implementing client side caching mechanism using http cache control + etags. I have not implemented such a solution so others could pitch in how their experience was implementing such a solution.
https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=en
I'm making JMeter load tests for an ASP.NET web application, and the tests should post some data to the server. Specifically, they should post grades for all the pupils in a class. However, the tests are supposed to be general, so that they can be run towards different schools with a small change in the configuration.
However, this creates a problem when the grades are posted, since the number of parameters in the post request (pupils in the class) can vary from run to run, or even from thread to thread. Currently I only know how to pass parameters through the HTTP request form as shown below:
However, in the next thread there could be a saveModel.PupilOrderAndBehaviours[2] or even up to 30. I have all of this information available directly from csv files. That is, I can tell JMeter ahead of time how many pupils will be in each class, and what grades each of them should receive, so I do not need to read this out from previous responses or anything like that.
Is there a way, potentially using BeanShell, I can configure JMeter to do this correctly?
It can be done with Beanshell Preprocessor.
int count = 10;
for(int i=1;i<=count;i++)
{
sampler.addArgument("Parameter" + i, "Value" + i);
}
It adds 10 parameters as given below # run time.
Please refer to this site.
http://theworkaholic.blogspot.com/2010/03/dynamic-parameters-in-jmeter.html
I am doing a NDB query, which seems to only be fetching a partial object. For the Model, I've turned off caching, in case that was it. However, a number of properties are coming back with None, when I can see them filled in the Datastore Viewer.
This is with the local development server ( and deployed), and the query is being done by a Backend process.
Note: Clearing the memcache did not help.
NOTE: If I cause the backend to restart, it will start pulling down the correct data.
Basically:
Backend starts querying for instances of a Model every X seconds
Frontend causes a change to an instance of the Model
Backend continues to see the original version of the instance until restarted
Backend code is pretty simple:
while 1:
time.sleep(2)
q = None
res = None
q = core.Agent.query()
res = q.fetch(10)
for a in res:
logging.error("%s" % a.to_dict())
Frontend changes some properties (and it shows in the viewer) but the backend will only show old values. It also seems like a Filter will filter based on correct values, but fetch() returns old stuff.
You need to clear the context cache at the top of the loop, e.g.
while 1:
ndb.get_context().clear_cache()
<rest of your code>