When I deploy my project to App Engine from Eclipse, the time it takes for "Veryfing availability" to finish varies a lot. Sometimes it just takes a couple of seconds, but mostly it looks like this:
Verifying availability:
Will check again in 1 seconds.
Will check again in 2 seconds.
Will check again in 4 seconds.
Will check again in 8 seconds.
Will check again in 16 seconds.
Will check again in 32 seconds.
Will check again in 60 seconds.
Will check again in 60 seconds.
Will check again in 60 seconds.
Will check again in 1 seconds.
Will check again in 2 seconds.
Will check again in 4 seconds.
Will check again in 8 seconds.
Closing update: new version is ready to start serving.
What's happening during this process, and is there anything I can do to make it go away? It's a little frustrating when I only do small changes and have to wait for ages to test it.
That is some App Engine delay over which we have no control. Just have to live with it. It is some race to deploy on the servers where the GAE traffic cops are trying to get everyone's apps deployed, updated, and readied. Can be frustrating sometimes, but it's part of the deal.
From this answer (from the Java GAE):
The delay is probably caused by an unacknowledged temporary server
condition. It has happened previously and usually improves after a few
hours.
This resource suggest the same.
From personal experience it indeed improves after a few hours.
Related
We have a performance issue with an AngularJS website hosted on IIS. This issue only affects our users connected via VPN (working from home).
The problem: regularly, a page that usually takes one or two seconds to load can take over 10 seconds.
This issue first appeared to be random, but we were able to reproduce it in a test environment and found out that the problem seems to arise on a very regular basis (every 10-15 minutes).
What we did: using a tool (ThousandEyes), we send every minute the same simple GET request via 12 clients to the Test server. We can see in the IIS logs that this request is processed in less than 50ms most of the time. However, every 15 minutes or so, the same request takes more than 5 seconds to process at least for 1 client. Example below: the calls done every minutes by client #1 takes more than 5 sec at 21:12, 21:13, 21:14, then 21:28, 21:29, then 21:45:
The graph below shows the mean response times for the 12 clients (peak every 10-15 minutes):
For both the test and the production environments, this issue only affect users connected via VPN (but not all the users connected via VPN are affected at the same time).
Any idea what can cause this behavior ?
All suggestions and questions are welcome.
Notes:
Session State. InProcess. I tried Not Enabled and State Server but we still have the same results.
Maximum Worker Process. 1. I tried 2, no change.
Test server usage. As far as I can tell, nothing special happen every 15 minutes on the server (no special events).
Test server configuration: 2 Xeon proc #2.6GHz, 8 GB RAM, 20 GB disk space, Windonws 2016.
Test server load: almost nothing beside these 12 requests every minute from the 12 test clients.
This issue cost us a lot of time. We finally found out that a VPN server was misconfigured.
Rebuilding this server was the solution.
I have an script in App engine that gets called every 10min. I am the only user.
The script pulls data from a web source does light processing and returns an image. It takes several minutes to run the first time. The source gets updated every 10min, so the next time my script runs, (10min later), it returns in a few seconds.
I'm using over 30 instance hours a day which is over the 28 free hours.
I read somewhere that every time an instance starts, it uses a minimum of 15min. (so 144x15=36hrs)
Therefore, am I better off trying to keep the instance running 24hrs (using up 24hrs) and limiting to one instance max? Perhaps setting idle_timeout to 10min. Another potential way to save would be to somehow pause my script during late night/early morning hours.
This is where you likely found the 15 minutes minimum statement and it refers to when the accrual of instance hours ends. If you look at that documentation you will see that it depends on what type of scaling you are using.
Here are my tought in all options you mentioned on your question for staying in the free tier:
Use only one instance. The problem with this approach is that giving a higher load to a single instance may increase other cost such as vCPU and memory since all the processing will be done there, also, it's likely that it's going to take longer to run the script.
Pause the script on low activity hours, this will be the ideal if the acumulated load when turning it back on it not too big.
Setting idle_timeout to 10min. This won't work if it runs every 10 min on all instances, since your app runs every 10min and app engine will only stop charging you every 15min idle, which will never happen, unless it does not run every 10min for every instance, if that is the case, it could be worth trying.
So, summing it up, all 3 options have their pros and cons, I would suggest that you test all of them and see what options suits your needs better.
Hope this helped.
I have experienced an strange problem since two weeks ago. I have a system running in GAE Python (server side) with 100 restaurantes and 1000 users working without problem, but suddenly, since two weeks ago, every day at rush hour, the tasks in the task queue had experienced a long delay in their start, two weeks before was only 1 or 2 seconds, now is 15 to 60 seconds, impacting the user experience and usability. I have to modify all the code with taskqueues and replace them with calls to urlfetch request async without waiting for the rpc (testing in some customers with success). The worst is that when adding task to the queues they are causing a dealine exceeded 123 error at rush hour (more than 100 QPS), losing between 50 to 1000 request every day (from 300.000 a day without problems). The task and my proceses are very fast, they last only from 50ms to 3 seconds, not more, but I get a lot of them with 60000ms and more in the "LIMBO", never executed and get cancelled without even starting (i have logging.debug message at the very beggining of every task/process that never get executed). I have 2 idle instances and all the settings to increase the instances without restriction when pending latency is more than 500ms. The start time of my instances is only 1 second, there are no special processes in the booting. I have 6 modules, separated modules for the task and the problem is affecting the module that call the task.add to add the task to the bucket(not the module that executes the task). I made all the changes proposed in this forum and google documentation to avoid datastore contention, I also deactivated the logs, I use a lot the memcache, I changed the F1 instances to F2, ant this error continues. And it APPEARED TWO WEEKS AGO. I have 1 year and a half runing my app, without problems, and suddenly this problem appeared.
Has anyone experienced the same problem and in this case, do you have a reccomendation? Please note that my code was working fine during a year and this problem arises since two weeks ago, the number of users is growing but not so much, two weeks ago were 850 users and now 1.000, so I think is not a problem of scale. My processes are veru efficient and quick. I'm having 3 years programming in GAE Python and 30 years experience in TI, for me this is very strange and could be related to platform changes.
This my module.yaml standard config:
runtime: python27
api_version: 1
instance_class: F2
threadsafe: true
automatic_scaling:
min_idle_instances: 2
max_idle_instances: automatic
min_pending_latency: 10ms
max_pending_latency: 500ms
max_concurrent_requests: 20
This is the taskqueue config (I have 10 queues each with 10 restaurantes)
- name: TaskRegOr00
rate: 10/s
bucket_size: 100
This seems to be a temporary issue with GAE, see this incident status https://status.cloud.google.com/incident/appengine/15024?_ga=1.267668750.1284093861.1444800865
I have a simple online ordering application I have built. It probably handles 25 hours a week, most of those on Mondays and Tuesday.
Looking at the dashboard I see:
Billing Status: Free - Settings Quotas reset every 24 hours. Next reset: 7 hrs
Resource Usage
Frontend Instance Hours 16% 4.53 of 28.00 Instance Hours
4.53 hours seems insanely high for the number of users I have.
Some of my pages make calls to a filemaker database stored on another service and have latencies like:
URI Reqs MCycles Latencies
/profile 50 74 1241 ms
/order 49 130 3157 ms
my authentication pages also have high latencies as they call out to third parties:
/auth/google/callback 9 51 2399 ms
I still don't see how they could add up to 4.53 hours though?
Can anyone explain?
You're charged 15 minutes every time an instance is spins up.
If you have few requests, but they are spaced out, your instance would shut down, and you'll incur the 15 minute charge the next time the instance spins up.
You could easily rack up 4.5 instance hours with 18 HTTP requests.
In addition to the previous answer, I thought to add a bit more about your billing which might have you confused. Google gives you 28 hours of free instance time for each 24 hour billing period.
Ideally you always have one instance running so that calls to your app never have to wait for an instance to spin up. One instance can handle a pretty decent volume of calls each minute, so a lot can be accomplished with those free 28 hours.
You have a lot of zero instance time (consumed less than 5 instance hours in seventeen hours of potential billing.) You need to worry more about getting this higher not lower because undoubtedly most of the calls to your app currently are waiting for both spin-up latency plus actual execution latency. If you are running a Go app, spin-up is likely not an issue. Python, likely a small-to-moderate issue, Java...
So think instead about keeping your instance alive, and consume 100% of your free instance quota. Alternatively, be sure to use Go, or Python (with good design). Do not use Java.
I am using Task Queue in GAE for performing some background work for my application. I have come to know that there is a 10 minute time limit for a particular task. My concern is how do I test this thing in my local environment. I tried thread sleep but it didn't throw any exception as mentioned in google app engine docs. Also is this time limit is measured by CPU time or the actual time.
Thanks.
The time is measured in wall clock time. The development server doesn't enforce time limits, although it's unclear why you'd want to test it because it's unlikely your tests will perform the same as they will in production, so trying to guess how much you'll be able to accomplish in 10 minutes on the production servers by seeing how much you can accomplish in 10 minutes on the development server will fail horribly.
For your development server, start a timer when a task is initiated. keep checking in your code if you reached 10 mins wall clock time. When you reach, throw a DeadlineExceededError. It would be better to have the try and except statements in the class handlers which call a particular function of your code.