I was trying out AppEngine TaskQueues with Python. I ran some code and it created a task on the default queue, I ran the code a few more times and it created a few more tasks as expected.
I went into the Task Queue management section and manually deleted one of the tasks. All the other tasks completed ok.
I then removed the code that created the task from my AppEngine code.
But now the task I deleted keeps getting called every 20 minutes or so, is there something I can do to stop this task? It doesn't show up in any of the task queues.
I have tried disable/re-enabling the application, uploading a clean queue.yaml file.
Any Ideas?
Thanks
If you are sure this is happening, please file a a bug with the label Component-TaskQueue and include your app ID.
Did you use the deferred modul? If so, remove the builtins: deferred from your app.yaml. Then you should make the url the task calls avaiable again and return a http-200 status code. (For the deferred-modul this is /_ah/queue/deferred). This should stop the GAE from retrying.
Related
Maybe you can help me with my problem
I start spark job on google-dataproc through API. This job writes results on the google data storage.
When it will be finished I want to get a callback to my application.
Do you know any way to get it? I don't want to track job status through API each time.
Thanks in advance!
I'll agree that it would be nice if there was to either wait for or get a callback for when operations such as VM creation, cluster creation, job completion, etc finish. Out of curiosity, are you using one of the api clients (like google-cloud-java), or are you using the REST API directly?
In the mean time, there are a couple of workarounds that come to mind:
1) Google Cloud Storage (GCS) callbacks
GCS can trigger callbacks (either Cloud Functions or PubSub notifications) when you create files. You can create an file at the end of your Spark job, which will then trigger a notification. Or, just add a trigger for when you put an output file on GCS.
If you're modifying the job anyway, you could also just have the Spark job call back directly to your application when it's done.
2) Use the gcloud command line tool (probably not the best choice for web servers)
gcloud already waits for jobs to complete. You can either use gcloud dataproc jobs submit spark ... to submit and wait for a new job to finish, or gcloud dataproc jobs wait <jobid> to wait for an in-progress job to finish.
That being said, if you're purely looking for a callback for choosing whether to run another job, consider using Apache Airflow + Cloud Composer.
In general, the more you tell us about what you're trying to accomplish, we can help you better :)
I have a flexible app engine app in which I am running a set of integration tests upon request. I get a sys.exit(1) after 30 seconds every time I run it. I cannot use Task Queue or Deferred library since this is a Flexible(Not Standard) app engine project. Any ideas on how to extend this 30 second deadline?
I also tried to change from auto scaling to manual scaling without any luck :|
In your Flexible environment you can use pub/sub library to create background tasks. You need to create a worker service which listens to a particular queue and you can add tasks in a queue , when the task is ready it will be thrown to worker service and be taken care of by it. Here is the reference you can use to solve it, https://cloud.google.com/python/getting-started/using-pub-sub. Hope it will help :)
I have configured Google App Engine to record exception with ereporter.
The cron job is configured to run every 59 minutes. The cron.yaml is as follows
cron:
- description: Daily exception report
url: /_ereporter?sender=xxx.xxx#gmail.com # The sender must be an app admin.
schedule: every 59 minutes
How to do I run this immediately.
What I am trying to do here is simulate a 500 HTTP error and see the stack trace delivered immediately via the cron job.
Just go to the URL from your browser.
You can't using cron. Cron is a scheduling system, you could get it to run every minute.
Alternately you could wrap your entire handler in a try/except block and try to catch everything. (You can do this for some DeadlineExceededErrors for instance) then fire off a task which invokes ereporter handler, and then re-raise the Exception.
However in many cases Google infrastructure can be the cause of the Error 500 and you won't be able to catch the error. To be honest you are only likely to be able to cause an email sent for a subset of all possible Error 500's. The most reliable way probably be to have a process continuously monitor the logs, and email from there.
Mind you email isn't consider reliable or fast so a 1 min cron cycle is probably fast enough.
I came across this thread as I was trying to do this as well. A (hacky) solution I found was to add a curl command at the end of my cloudbuild.yaml file that triggers the file immediately per this thread. Hope this helps!
Make a curl request in Cloud Build CI/CD pipeline
I have a Google App Engine servlet that is cron configured to run once a week. Since it will take more than 1 minute of execution time, it launches a task (i.e. another servlet task/clear) on the application's default push task queue.
Now what I'm observing is this: if the task causes an exception (e.g. NullPointerException inside its second servlet), this gets translated into HTTP status 500 (i.e. HttpURLConnection.HTTP_INTERNAL_ERROR) and Google App Engine apparently reacts by immediately relaunching the same task again. It announces this by printing:
Web hook at http://127.0.0.1:8888/task/clear returned status code 500. Rescheduling..
I can see how this can sometimes be a feature, but in my scenario it's inappropriate. Can I request that Google App Engine should not do such automatic rescheduling, or am I expected to use other status codes to indicate error conditions that would not cause rescheduling by its rules? Or is this something that happens only on the dev. server?
BTW, I am currently also running other tasks (with different frequencies) on the same task queue, so throttling reschedules on the level of task queue configuration would be inconvenient (so I hope there is another/better option too.)
As per https://developers.google.com/appengine/docs/java/taskqueue/overview-push#Java_Task_execution - the task must return a response code between 200 and 299.
You can either return the correct value, set the taskRetryLimit in RetryOptions or check the header X-AppEngine-TaskExecutionCount when task launches to check how many times it has been launched and act accordingly.
I think I've found a solution: in the Java API, there is a method RetryOptions#taskRetryLimit, which serves my case.
I am unable to update my frontends nor my backends. I get the error message 'Version is not ready'. This bug has persisted for coming up to 24 hours now. I have a task perpetually running in a queue. My best guess is that this task is stopping the update. I am unable to delete the task as it is perpetually running, nor can I delete the queue as I am unable to upload a new queue.yaml definition. The same task previously failed due to a maximum recursion error as I had a synchronous RPC within an asynchronous tasklet.
I'm pretty sure the fix will require someone from the GAE side forcibly resetting the task queue. Thus, this question would be more suitably directed to the GAE team with details about my app in a less public forum. Though, from what I can see, they do not allow direct support questions and suggest posting the question here. My follow up question, then, is when you have a GAE issue that requires action from the GAE team - how do you get hold of them (other than paying US$500/month for a premium support account)?
EDIT:
The task is/was meant to be running on a backend instance. I intended to shutdown all backend and frontend instances via the console assuming that they would cancel the task and restart themselves. But I found that only one frontend instance was running - no backends. After shutting down that frontend instance, the dashboard has reported that I have 0 instances running, yet the website is still serving and the task remains perpetually running.
EDIT:
Disabling the app stopped the task from running. After reenabling the app, I was able to update it. Though I am left with a ghost task in my queue.
If you have a stuck task queue job, I'd try disabling the queue and killing the instance running that job. If that doesn't work, I'd try disabling the app temporarily.