I have a task queue with several tasks. If I delete a particular task from Admin Console, it disappears from the task queue but GAE doesnt terminate it. The task is still being executed in the background.
Is this a common behavior ?
Yeah, I see the same behavior. Seems you can only delete pending tasks from the admin console. Once they've started they continue to run until they finish or hit an exception (could be as long as 10 minutes with the new update).
I've noticed they don't stop on version upgrades either, which is a little weird if you aren't expecting it... if the task takes a long time you end up with handlers running in two versions of the app simultaneously. It makes sense though.
Related
I'm working with Go on App Engine and I'm trying to build an API that needs to perform a long-running background task - in this case it needs to parse and chunk a big file out to task queues. I'd like it to return a 200 and close the user connection immediately and let the process keep running in the background until it's complete (this could take 5-10 minutes). Task queues alone don't really work for my use case because parsing the initial file can take more than the time limit for an API request.
At first I tried a Go routine as a solution for this problem. This failed because my app engine context expired as soon as the parent function closed the user connection. (I suppose I could try writing a go routine that doesn't require a context, but then I'd lose logging and I'd need to fetch the entire remote file and pass it to the go routine.)
Looking through the docs, it looks like App Engine used to have functionality to support exactly what I want to do: [runtime.RunInBackground], but that functionality is now deprecated and the replacement isn't obvious.
Is there a "right" or recommended way to do background processing now?
I suppose I could put a link to my big file into a task queue, but if I understand correctly, even functions called through task queues have to complete execution within a specified amount of time (is it 90 seconds?) I need to be able to run longer than that.
Thanks for any help.
try using:
appengine.BackgroundContext()
it should be long-lived but will only work on GAE Flex
I have a script that, using Remote API, iterates through all entities for a few models. Let's say two models, called FooModel with about 200 entities, and BarModel with about 1200 entities. Each has 15 StringPropertys.
for model in [FooModel, BarModel]:
print 'Downloading {}'.format(model.__name__)
new_items_iter = model.query().iter()
new_items = [i.to_dict() for i in new_items_iter]
print new_items
When I run this in my console, it hangs for a while after printing 'Downloading BarModel'. It hangs until I hit ctrl+C, at which point it prints the downloaded list of items.
When this is run in a Jenkins job, there's no one to press ctrl+C, so it just runs continuously (last night it ran for 6 hours before something, presumably Jenkins, killed it). Datastore activity logs reveal that the datastore was taking 5.5 API calls per second for the entire 6 hours, racking up a few dollars in GAE usage charges in the meantime.
Why is this happening? What's with the weird behavior of ctrl+C? Why is the iterator not finishing?
This is a known issue currently being tracked on the Google App Engine public issue tracker under Issue 12908. The issue was forwarded to the engineering team and progress on this issue will be discussed on said thread. Should this be affecting you, please star the issue to receive updates.
In short, the issue appears to be with the remote_api script. When querying entities of a given kind, it will hang when fetching 1001 + batch_size entities when the batch_size is specified. This does not happen in production outside of the remote_api.
Possible workarounds
Using the remote_api
One could limit the number of entities fetched per script execution using the limit argument for queries. This may be somewhat tedious but the script could simply be executed repeatedly from another script to essentially have the same effect.
Using admin URLs
For repeated operations, it may be worthwhile to build a web UI accessible only to admins. This can be done with the help of the users module as shown here. This is not really practical for a one-time task but far more robust for regular maintenance tasks. As this does not use the remote_api at all, one would not encounter this bug.
I am not an IT person but I have to play one at work sometimes. I was about to run diagnostics on our EHR which I do weekly, I always stop the ADS and restart it before running the diagnostics. When I went to stop it this time I got an error box that said operation complete which I've gotten before but the box to restart it never came back, I tried to close it and it said not responding. After waiting 10-15 minutes I used the task manager to end ADS. Now when I try to restart it it says it can't because there is another instance running. When I look in the task manager under service the ADS says stopping and has said that for at least 30 minutes now.
Any help would be greatly appreciated!
Thank you
I set up a staging envierment to a web app I created about 3 weeks ago,
and I tried to transfer the data from the production envierment that was
already set up using the Copy data to another app in the datastore admin.
The data was indeed copied to my staging envierment. The problem was that
the copy jobs are still running, 3 weeks after they were fired! (It took the data
about 3 hours to transfer to my staging evnierment. )
I tried to cancel the jobs using the abort option, with no luck.
As for now, 7 out of the 14 jobs are listed as completed, and the others are
listed as active. My /_ah/mapreduce/controller_callback handler is bombarded with
3.1 posts per second, and I think it got to a point it is harming my site performance, not to mention costing me money...
How do I get the tasks to abort?
You can purge your task queues from the normal datastore admin task queues section. That will force the jobs to stop.
You can clean up the mapreduce jobs by deleting the entities they store in the datastore to keep track of their progress - they are called "mr_progress" or something like that.
I had a similar situation where DataStore Admin tasks would not die. The tasks were from a copy and / or backup operation. Aborting the jobs didn't do anything for me, either. Even after going into the task queue and purging the queue, the tasks would reappear. I did all of the following to keep the tasks dead (not sure if all steps are necessary!):
Paused the task queue (click task Queues, and then the queue in question).
Purged the queue (clicking purge repeatedly)
Went into the datastore viewer and deleted all DatastoreAdmin entries with Active status, and MapReduce items. I'm not sure if this is necessary.
Changed the queue.yaml file to include a task_age_limit of 10s and task_retry_limit of 3 under the retry_parameters.
I originally was trying to do a large backup and copy to another app at the same time, during which the destination app ran into a quota. Not sure if this caused the issues. Riley Lark's answer got me started.
One other addition: when I went into the task queue and clicked the task name to expand the details, nothing ever loaded into the "Headers", "Raw Payload", etc. tabs. They just said "loading..."
I know how to set up a job to alert when it's running.
But I'm writing a job which is meant to run many times a day, and I don't want to be bombarded by emails, but rather I'd like a solution where I get an alert when the job hasn't been executed for X minutes.
This can be acheived by setting the job to alert on execution, and then setting up some process which checks for these alerts, and warns when no such alert is seen for X minutes.
I'm wondering if anyone's already implemented such a thing (or equivalent).
Supporting multiple jobs with different X values would be great.
The danger of this approach is this: suppose you set this up. One day you receive no emails. What does this mean?
It could mean
the supposed-to-be-running job is running successfully (and silently), and so the absence-of-running monitor job has nothing to say
or alternatively
the supposed-to-be-running job is NOT running successfully, but the absence-of-running monitor job has ALSO failed
or even
your server has caught fire, and can't send any emails even if it wants to
Don't seek to avoid receiving success messages - instead devise a strategy for coping with them. Because the only way to know that a job is running successfully is getting a message which says precisely this.