I am running an optimization program in GAE. The program runs on my laptop/eclipse and could potentially take up to 6-10 minutes to run. It seems that GAE has a timeout of 60 secs and throws a 500 Error.
How do you increase the memory requirement for GAE? How would you increase the timeout requirement to more than 10 min? Is there something that I can do in GAE settings in Eclipse or do I have to get in touch with Google.
I second #PatrickGray's answer.
Related note: the actual execution on GAE can often take longer than on the development server, so don't use the local execution time as a reference.
For increasing the memory requirement - you can configure the module's instance class as needed (with cost implications, of course).
I was running in to this error in Google App Engine before. You can't increase the timeout time, it's a hard limit. I recommend the deferred library or using a Task Queue. Here's how I solved my problem (in Python):
How do I return data from a deferred task in Google App Engine
I assume that you don't need an automatic scaling instance for this program. Instances with basic or manual scaling give you as much time as you need to complete your tasks.
Related
I have a backend for an iOS app that I've built on App Engine and I'm looking to do potentially long running background tasks to add records to my Cloud SQL database. Is this possible without Compute Engine? I've seen Cloud Tasks can do asynchronous work and you can set the dispatchDeadline to basically anything you want, but I've also read in the documentation
For App Engine tasks, 0 indicates that the request has the default deadline. The default deadline depends on the scaling type of the service: 10 minutes for standard apps with automatic scaling, 24 hours for standard apps with manual and basic scaling, and 60 minutes for flex apps. If the request deadline is set, it must be in the interval [15 seconds, 24 hours 15 seconds]. Regardless of the task's dispatchDeadline, the app handler will not run for longer than than the service's timeout. We recommend setting the dispatchDeadline to at most a few seconds more than the app handler's timeout. For more information see Timeouts.
I don't particularly need the App Engine instance to care if the task completes or not... so I'm not sure why the recommendation is at most a few seconds more than the app handler's timeout ... can anyone shed any light on this? What am I missing? Adding a Compute Engine for these relatively simple tasks that will take at most a few ours to complete seems like a lot of overhead and I don't want this to dictate which scaling options I choose.
Thanks for your time.
The recommendation is only for logging purpose. If your task timeout is shorter that your app timeout, you never know if there is an error from your app, because you don't have the return.
If you have longer timeout on Cloud Task, you can catch and trace in Cloud Task logs the app return code and thus gracefully track the errors.
App Engine with a basic scaling mode is a great solution.
You have 9H free per days (B instance type)
App Engine scale to 0 automatically after a period of inactivity (that you can define in the basic scaling: idle_timeout parameter)
You have a regional available service. Not a zonal like a compute engine, or you need to have 9 computes engine, to cover the regional High Availability (3 per zone, over 3 zones)
You don't have server to manage: no update, no patching, no network/ip/firewall rule...
If you ask me about the overhead, I will anser Compute Engine and not App Engine (even if you need few configuration)
I would like to try out dropwizard-metrics + graphite.
For order to this to work out i need to run a job regular (e.g. each 5:th second) that sends metrics from the instance to the graphite server.
Is this even possible?
The java documentation and python documentation for Cron in App Engine says that the minimal interval in App Engine is configurable in 'minutes'. Thus the simple answer would be: No you cannot schedule a job every 5 seconds.
However...
Knowing that tasks queue tasks can run up to 10 Minutes (see deadlines) you could manually schedule a task every (let's say) 5 minutes and handle the 5 second interval yourself in your servlet (or whatever it is called in python).
I'm just saying it is possible to use suche short intervals. You should really avoid a crutch like that. This kind of behaviour will eat through your quota and make your app expensive really fast.
Edit:
Since the title of the question asks for running jobs in specific instances:
As Dmitry pointed out and as documented here it is possible to address specific instances when using manual or basic scaling with modules. Instances are anonymous when using automatic scaling and thus cannot be addressed. It seems this feature is only documented and available for app engine modules.
I developed an application for client that uses Play framework 1.x and runs on GAE. The app works great, but sometimes is crazy slow. It takes around 30 seconds to load simple page but sometimes it runs faster - no code change whatsoever.
Are there any way to identify why it's running slow? I tried to contact support but I couldnt find any telephone number or email. Also there is no response on official google group.
How would you approach this problem? Currently my customer is very angry because of slow loading time, but switching to other provider is last option at the moment.
Use GAE Appstats to profile your remote procedure calls. All of the RPCs are slow (Google Cloud Storage, Google Cloud SQL, ...), so if you can reduce the amount of RPCs or can use some caching datastructures, use them -> your application will be much faster. But you can see with appstats which parts are slow and if they need attention :) .
For example, I've created a Google Cloud Storage cache for my application and decreased execution time from 2 minutes to under 30 seconds. The RPCs are a bottleneck in the GAE.
Google does not usually provide a contact support for a lot of services. The issue described about google app engine slowness is probably caused by a cold start. Google app engine front-end instances sleep after about 15 minutes. You could write a cron job to ping instances every 14 minutes to keep the nodes up.
Combining some answers and adding a few things to check:
Debug using app stats. Look for "staircase" situations and RPC calls. Maybe something in your app is triggering RPC calls at certain points that don't happen in your logic all the time.
Tweak your instance settings. Add some permanent/resident instances and see if that makes a difference. If you are spinning up new instances, things will be slow, for probably around the time frame (30 seconds or more) you describe. It will seem random. It's not just how many instances, but what combinations of the sliders you are using (you can actually hurt yourself with too little/many).
Look at your app itself. Are you doing lots of memory allocations in the JVM? Allocating/freeing memory is inherently a slow operation and can cause freezes. Are you sure your freezing is not a JVM issue? Try replicating the problem locally and tweak the JVM xmx and xms settings and see if you find similar behavior. Also profile your application locally for memory/performance issues. You can cut down on allocations using pooling, DI containers, etc.
Are you running any sort of cron jobs/processing on your front-end servers? Try to move as much as you can to background tasks such as sending emails. The intervals may seem random, but it can be a result of things happening depending on your job settings. 9 am every day may not mean what you think depending on the cron/task options. A corollary - move things to back-end servers and pull queues.
It's tough to give you a good answer without more information. The best someone here can do is give you a starting point, which pretty much every answer here already has.
By making at least one instance permanent, you get a great improvement in the first use. It takes about 15 sec. to load the application in the instance, which is why you experience long request times, when nobody has been using the application for a while
I've encountered this problem for a second time now, and I'm wondering if there is any solution to this. I'm running an application on Google App Engine that relies on frequent communication with a website through HTTP JSON RPC. It appears that GAE has a tendency to randomly display a message like this in the logs:
"This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application."
And reset all variables stored in RAM without warning. The same process happens over and over no matter how many times I set the variables again or upload newer code to GAE, although incrementing the app version number seems to solve the problem.
How can I get more information on this behaviour, how to avoid it and prevent data loss of my Golang applications on Google App Engine?
EDIT:
The variables stored in RAM are small classes of strings, bytes, bools and pointers. Nothing too complicated or big.
Google App Engine seems to "start a new process" in matter of seconds of heavier use, which shouldn't be long enough time for the application to be shut down for not being used. The timespan between application being uploaded to GAE, having its variable set and a new process being created is less than a minute.
Do you realize that GAE is a cloud hosting solution that automatically manages instances based on the load? This is it's main feature and reason people are using it.
When load increases, GAE creates a new instance, which , of course, has all RAM variables empty.
The solution is not to expect variables to be available or store them to permanent storage at the end of request (session, memcache, datastore) and load them if not present at the beginnig of request.
You can read about GAE instances in their documentation here, check out the performance section:
http://code.google.com/appengine/kb/java.html
In your case of having small data available, if its static then you can load it into memory on startup of a new instance. If it's dynamic data, you should be saving it to the database using their api.
My recommendation for keeping a GAE instance alive, either pay for the Always-On service or follow my recommendations for using a cron here:
http://rwyland.blogspot.com/2012/02/keeping-google-app-engine-gae-instances.html
I use what I call a "prime schedule" of a 3, 7, 11 minute cron job.
You should consider using Backends if you want long running instances with resident memory.
I am using Task Queue in GAE for performing some background work for my application. I have come to know that there is a 10 minute time limit for a particular task. My concern is how do I test this thing in my local environment. I tried thread sleep but it didn't throw any exception as mentioned in google app engine docs. Also is this time limit is measured by CPU time or the actual time.
Thanks.
The time is measured in wall clock time. The development server doesn't enforce time limits, although it's unclear why you'd want to test it because it's unlikely your tests will perform the same as they will in production, so trying to guess how much you'll be able to accomplish in 10 minutes on the production servers by seeing how much you can accomplish in 10 minutes on the development server will fail horribly.
For your development server, start a timer when a task is initiated. keep checking in your code if you reached 10 mins wall clock time. When you reach, throw a DeadlineExceededError. It would be better to have the try and except statements in the class handlers which call a particular function of your code.