Almost a week ago I migrated my (paid) app from Master/Slave to HRD. Since that time my app has been restricted to 100 emails/day with a warning indicating "Resource is currently experiencing a short-term quota limit". I know it mentions a limitation until the first successful billing so I was hoping the limit would disappear once that happened - but alas it has not! I have also filled out the "request additional resources" form hoping that might help.
Anyone encountered this problem migrating from master/slave? Any suggestions of who I can contact or how I can recover from this limit? The migration process was relatively smooth - except for this problem which has become a significant impact to my customers.
I am having the same problem right now. I just finished my HRD migration yesterday and only today realized that my Mail API requests are failing because the mail quota on my new HRD app is only at 100 messages per day. I did not have this limitation before, and I find it pretty disappointing that such a triviality is causing my users trouble despite a successful migration. I submitted a quota increase request and hoping I don't also have to wait a week before it's applied.
Anyone who is waiting until the last minute to migrate their app to HRD be warned: make sure you apply for a mail quota increase on your new app because your old setting will not carry over, even if you have billing enabled on both apps.
Related
We have seen a sudden latency increase in our application on Google App Engine latency within the past few hours. The logs show that requests fail with message "Request was aborted after waiting too long to attempt to service your request.", with no stack-trace or any other relevant information. Users get an empty page with message "Rate exceeded.". No changes have been done to the application that correlate to this spike in latency.
The application is therefore down, with no information from app engine that can help point to the source of the latency.
We have filed a issue in the issuer tracker, no luck in getting response yet.
Does anyone have ideas on what we could do to deal with this kind of situation?
Update
The problem went away after 3 hours as suddenly as it came, and without any intervention on our part. Since there is consensus on min_idle_instances, we have decided to leave all the setting as they have always been so that we can see if this ever happens again. If it does happen, we will have an opportunity to test this by making the suggested changes, and post an update here.
Here is a screen shot for the entire incident:
The comment that #Parth Mehta added is useful and it made me think of what could be causing your issues.
I'm thinking that maybe your increased latency is due to not having idle instances ready for the requests as they increase and come in, so when requests increase a bit is taken until the new instances are ready and there might be your latency cause.
Setting enough min_idle_instances might alleviate the 500's as they would be warm and ready for the requests.
If this doesn't solve your issue I would recommend creating a case with GCP Support and we will surely be able to assist you more.
Try it and let us know.
I am using GAE for my product development. Now it is MVP version and in development phase. So the user traffic is still small.
But today (14:00 27th Feb 2017 UTC+7) when I saw the statistics I cannot believe my eyes. So huge numbers of requests and bandwidth. I am attaching here:
Abnormal resource usage
My request log and the App Engine Dashboard show the same traffic as usual (very small). So I think the Resource usage numbers is not correct.
Dashboard and Request log
Please explain me where the huge number of requests come from?
Here is the answer I got from Google Cloud Support. I am sharing below, in case someone get the same problem as I had:
The number of requests you see on your App Engine Dashboard is not 100% reliable as it sometimes reflect the only the projected usage not the exact usage. Your exact bill will only be reflected on your transaction history and it takes 24-48 hours for it to refresh to be able to display your usage for February 27, 2017. We can wait for 24-48 hours to validate that your transaction history will be the one to show your charges and usage.
And here is the final resource usage statistics
We have a PHP application running on GAE. It connects to Cloud Datastore using the Google PHP library (v0.6.7).
Google introduced in the last days a new version of App Engine, v1.9.0 (not oficially released), which apparently was running fine, just as 1.8.9 was. However, we have been experiencing some issues related to Cloud Datastore. Sometimes, all the operations regarding to entities updating are just ignored. All the queries used to retrieve information work perfectly, however if we want to create a new entity of update any property, no action is performed. I have been checking for some errors in the response returned by the Cloud Api, but there is no errors or warnings at all.
This situation happened for the first time the 31st of January, and it is also happening today. It started to fail at 3am (GMT +1) and according to the instance log, at the same time the latency times of all the requests suffered an important increase (from 1-3 secs to 5-10 secs). The first time after a few hours the system started to work properly again, but now this problem is lasting much more.
Has anyone experienced anything similar?
Thank you for the report, we're investigating the issue now.
Update: We've addressed the issue. Please join the Google Cloud Datastore downtime notify mailing list for future updates.
https://groups.google.com/forum/?fromgroups=#!topic/gcd-downtime-notify/sNXCFJYFNQU
For future reports about production issues, please refer to the Contact support section of our documentation.
I was in a place without Internet access for 3 weeks and just came back to find out that one of my apps since January 18 started to reach a quota limit (Datastore Read Operations) after around the 18 hours.
I don't see any increase in traffic from either users or crawlers.
This is the error in the logs:
"The API call datastore_v3.RunQuery() required more quota than is available."
It seems very strange since this application has been running for some years and I'm memcaching most the datastore requests.
Please help - This is affecting my bottom line!
Thanks.
I found a subset of pages in the site that had got a sudden interest from several crawlers and some of the requests that those pages made to the Datastore were not being memcached, so that was it...problem solved.
Thanks.
I migrated a project to HRD using the migration tool.
This seems to have various side effects. Some might be related to unanchored entities or queries. The app doesn't send the notification emails it is supposed to which may have something to do with Billing needed to be re-enabled and Google waiting seven days before applying mail quota.
Anyways, I am wondering if it is possible to roll back to master/slave? Disable or delete the HRD app and re-enable the old one?
As a way for me to buy time, prepare better for this migration and try again at a later time.
I'm afraid that you can't go back because Master/Slave Datatsore is deprecated since April 4, 2012. You could contact the support, but I would suggest you to spend some time and try to fix it on HRD.