I have a simple online ordering application I have built. It probably handles 25 hours a week, most of those on Mondays and Tuesday.
Looking at the dashboard I see:
Billing Status: Free - Settings Quotas reset every 24 hours. Next reset: 7 hrs
Resource Usage
Frontend Instance Hours 16% 4.53 of 28.00 Instance Hours
4.53 hours seems insanely high for the number of users I have.
Some of my pages make calls to a filemaker database stored on another service and have latencies like:
URI Reqs MCycles Latencies
/profile 50 74 1241 ms
/order 49 130 3157 ms
my authentication pages also have high latencies as they call out to third parties:
/auth/google/callback 9 51 2399 ms
I still don't see how they could add up to 4.53 hours though?
Can anyone explain?
You're charged 15 minutes every time an instance is spins up.
If you have few requests, but they are spaced out, your instance would shut down, and you'll incur the 15 minute charge the next time the instance spins up.
You could easily rack up 4.5 instance hours with 18 HTTP requests.
In addition to the previous answer, I thought to add a bit more about your billing which might have you confused. Google gives you 28 hours of free instance time for each 24 hour billing period.
Ideally you always have one instance running so that calls to your app never have to wait for an instance to spin up. One instance can handle a pretty decent volume of calls each minute, so a lot can be accomplished with those free 28 hours.
You have a lot of zero instance time (consumed less than 5 instance hours in seventeen hours of potential billing.) You need to worry more about getting this higher not lower because undoubtedly most of the calls to your app currently are waiting for both spin-up latency plus actual execution latency. If you are running a Go app, spin-up is likely not an issue. Python, likely a small-to-moderate issue, Java...
So think instead about keeping your instance alive, and consume 100% of your free instance quota. Alternatively, be sure to use Go, or Python (with good design). Do not use Java.
Related
I host a basic web app on Google Cloud Platform, and I've noticed my costs creeping up over the last couple of months. It's really accelerated over the last 30 days (fortunately, on a tiny base - I'm still ticking along at under $2 a day). I haven't added any new functionality or clients in months so this was a bit surprising.
My first instinct was an increase in traffic. I couldn't see anything like that in the App Engine dashboard, but I put in a heap of optimizations and dramatically decreased QPS just in case. No change.
The number of instances hasn't moved around much either - this looks like the most likely culprit but it's still just flat, not growing.
My next guess was that data was accumulating in Datastore (even though the cost chart is filtered to App Engine only, I figured a fuller datastore -> a slower datastore -> more instance time in GAE). There's no chart for this, annoyingly, but I determined the data store size was more or less flat (I have a blunt instrument TTL job that runs daily) and culled it by dropping my retention threshold by 20% just to be safe.
These optimizations were on the 17th, but my cost hasn't moved at all. I considered forex fluctuations (I'm billed in Aussie dollars, all my charges are for frontend instances in Japan) but they haven't been anywhere near big enough to explain this.
Any ideas what's going on? I've clicked through all the graphs and reports in billing but can't reconcile the ~100% growth in cost with a flat or dropping qps, instance count and database size.
Yes! I've seen the same thing on a simple App Engine website running Python 3.7! I've had a ticket open since April 29th and they're not helpful. I saw a step change in frontend instance hours on March 24th with no corresponding increase in traffic. I have screenshots that are really telling but I can't upload them since I don't have 10 reputation points.
There's no corresponding increase in traffic, either in the cloud console or in Google analytics.
What's worse, each day the daily estimate shows I'm be under the 28 hour quota. For example, I took a screenshot that showed after 15 hours I was on pace for 24.352 frontend instance hours for the day (I didn't take one at the end of the quota day since it resets at 3AM)
When I woke up the next morning the billing report showed I was charged $0.00 for frontend instance hours for the previous day, but 3 hours later it shot up to $0.48, which means I used 38.6 frontend instance hours worth.
Somehow, the estimated cost calculation was off by 14 hours. Why have the estimate at all if it has an error that large? When I looked at the minute-by-minute billed instance hours for the hours after taking the screenshot through the end of the quota-day, there's nothing that indicates I would have used 23 additional hours from the time I took the screenshot to the time of the quota reset.
This behavior has been happening every day since March 24th for me with no explanation from Google besides "it looks like you exceeded your instances..." I wish I could share the screenshots so you can compare what you're seeing.
Situation:
My project are mostly automated tasks.
My GAE (standard environment) app has 40 crons job like this, all run on default module (frontend):
- description: My cron job Nth
url: /mycronjob_n/ ###### Please note n is the nth cron job.
schedule: every 1 minutes
Each of cron jobs
#app.route('/mycronjob_n/')
def mycronjob_n():
for i in (0,100):
pram = prams[i]
options = TaskRetryOptions(task_retry_limit=0,task_age_limit=0)
deferred.defer(mytask,pram)
Where mytask is
def mytask(pram):
#Do some loops, read and write datastore, call api, which I guesss taking less than 30 seconds.
return 'Task finish'
Problem:
As title of the question, i am running out of RAM. Frontend instance hours are increasing to 100 hours.
My wrong thought?
defer task runs on background because it is not something that user sends request when visit the website. Therefore, they will not be considered as a request.
I break my cronjobs_n into small different tasks because i think it can help to reduce the running time each cronjobs_n so that REDUCE instance's ram consumption.
My question: (purpose: keep the frontend/backend instance hours as low as possible, and I accept latency)
Is defer task counted as request?
How many request do I have in 1 mintues?
40 request of mycronjob_n
or
40 requests of mycronjob_n x 100 mytask = 4000
If 3-4 instances can not handle 4000 requests, why doesnt GAE add 10 to 20 F1 instances more and then shut down if idle? I set autoscale in app.yaml. I dont see the meaning of autoscale of GAE here as advertised.
What is the best way to optimize my app?
If defer task is counted as request, it is meaningless to slit mycronjob_n into different small tasks, right? I mean, my current method is as same as:
#app.route('/mycronjob_n/')
def mycronjob_n():
for i in (0,100):
pram = prams[i]
options = TaskRetryOptions(task_retry_limit=0,task_age_limit=0)
mytask(pram) #Call function mytask
Here, will my app has 40 requests per minute, each request runs for 100 x 30s = 3000s? So will this approach also return out of memory?
Should I create a backend service running on F1 instance and put all cron jobs on that backend service? I heard that a request can run for 24 hours.
If I change default service instance from F1 to F2,F3, will I still get 28 hours free? I heard free tier apply to F1 only. And will my backend service get 9 hours free if it runs on B2 instead of B1?
My regret:
- I am quite regret that I choose GAE for this project. I choosed it because it has free tier. But I realized that free tier is just for hobby/testing purpose. If I run a real app, the cost will increase very fast that it make me think GAE is expensive. The datastore reading/writing are so expensive even though I tried my best to optimize them. The frontend hours are also always high. I am paying 40 usd per month for GAE. With 40 usd per month, maybe I can get better server if I choose Heroku, Digital Ocean? Do you think so?
Yes, task queue requests (deferred included) are also requests, they just can run longer than user requests. And they need instances to serve them, which count as instance hours. Since you have at least one cron job running every minute - you won't have any 15 minute idle interval allowing your instances to shut down - so you'll need at least one instance running at all times. If you use any instance class other than F1/B1 - you'll exceed the free instance hours quota. See Standard environment instances billing.
You seem to be under the impression that the number of requests is what's driving your costs up. It's not, at least not directly. The culprit is most likely the number of instances running.
If 3-4 instances can not handle 4000 requests, why doesnt GAE add 10
to 20 F1 instances more and then shut down if idle?
Most likely GAE does exactly that - spawns several instances. But you keep pumping requests every minute, they don't reach an idle state long enough, so they don't shut down. Which drives your instance hours up.
There are 2 things you can do about it:
stagger your deferred tasks so they don't hit need to be handled at the same time. Fewer instance (maybe even a single one?) may be necessary to handle them in such case. See Combine cron jobs to reduce number of instances and Preventing Google App Engine Cron jobs from creating multiple instances (and thus burning through all my instance hours)
tune your app's scaling configuration (the range is limited though). See Scaling elements.
You should also carefully read How Instances are Managed.
Yes, you only pay for exceeds the free quota, regardless of the instance class. Billing is in F1/B1 units anyways - from the above billing link:
Important: When you are billed for instance hours, you will not see any instance classes in your billing line items. Instead, you will
see the appropriate multiple of instance hours. For example, if you
use an F4 instance for one hour, you do not see "F4" listed, but you
see billing for four instance hours at the F1 rate.
About the RAM usage, splitting the cron job in multiple tasks isn't necessarily helping, see App Engine Deferred: Tracking Down Memory Leaks
Finally, cost comparing GAE with Heroku, Digital Ocean isn't an apples-to-apples comparison: GAE is PaaS, not IaaS, it's IMHO expected to be more expensive. Choosing one or the other is really up to you.
When I deploy my project to App Engine from Eclipse, the time it takes for "Veryfing availability" to finish varies a lot. Sometimes it just takes a couple of seconds, but mostly it looks like this:
Verifying availability:
Will check again in 1 seconds.
Will check again in 2 seconds.
Will check again in 4 seconds.
Will check again in 8 seconds.
Will check again in 16 seconds.
Will check again in 32 seconds.
Will check again in 60 seconds.
Will check again in 60 seconds.
Will check again in 60 seconds.
Will check again in 1 seconds.
Will check again in 2 seconds.
Will check again in 4 seconds.
Will check again in 8 seconds.
Closing update: new version is ready to start serving.
What's happening during this process, and is there anything I can do to make it go away? It's a little frustrating when I only do small changes and have to wait for ages to test it.
That is some App Engine delay over which we have no control. Just have to live with it. It is some race to deploy on the servers where the GAE traffic cops are trying to get everyone's apps deployed, updated, and readied. Can be frustrating sometimes, but it's part of the deal.
From this answer (from the Java GAE):
The delay is probably caused by an unacknowledged temporary server
condition. It has happened previously and usually improves after a few
hours.
This resource suggest the same.
From personal experience it indeed improves after a few hours.
AppEngine kills my JVM instance really quickly. It does not live longer than 30 seconds idle. Subsequent request will create new instance but this roundtrip takes 8-10 seconds. Is there potential problem (bug) in my application ? There is no record in logs/admin logs which would indicate any problem or reason of shutdown. Development server works normaly. Is there any chance to find out why the instance is shutdown so quickly ?
AppEngine especially on free account can kill the instance anytime. Instance can live couple of seconds if idle or couple of minutes, noone knows how long. Should you need resident instance (to prevent long instance start-up), switch to paid version. Even if it is paid, you can still run it for free if you keep your instance hours within free quota. This means if you have single F1 instance, it will consume 24instance-hours per day. As per today quotas, you have free 28instance-hours per day so you have 4 instance-hours spare for app redeployments (every redeployment costs 1/4 of instance-hour or 15 minutes). Having F2 instance will consume quota 2x faster e.g. it will take 14 hours to consume your 28hour free quota. Next 10 hours will be billed to your credit card as per price list.
I need to send out emails at a rate exceeding App Engine's free email quota (8 emails/minute). I'm planning to use a TaskQueue to queue the emails, but I wondered: is there already a library or Python module I could use to automate this? It seems like the kind of problem someone might have run into before.
If it's an option, why not just enable billing? It'll jump the max rate from 8 recipients/minute to 5,100 recipients/minute.
The first 2000 recipients is free each day, as long as you aren't going over the daily free quotas my understanding is that it will not cost you anything (and if you need to email more than 2000 people per day you're going to have to enable billing anyways).
The deferred library is designed for exactly this sort of thing. Simply use deferred.defer(message.send), and make sure the queue you're using has the appropriate execution rate.
its cheaper to just pay for it for a year than to engineer a workaround.
Easiest way in my opinion would be to use a queue, ex Amazon SQS, and pull 8 records per minute, in a cron job running every minute.
Considering it was pushed into the queue, then taken out, I am working out the math that it is an extremely cheap service.
See below, 0.000002 is the rate for 2 requests. (Add and View)
8 requests per minute, 60 minutes in an hour, and 24 hours in a day. Take into account 30 days in the average month, you are still under $1.
0.000002 * 8 * 60 * 24 * 30 = $0.6912
This might not be exactly what you were looking for, but it should be a pretty simple solution.
EDIT:
See here, a python SQS & S3 Lib (sqs is all that you should be looking for).
http://pypi.python.org/pypi/Python-Amazon/0.5
I'm not familiar with any canned solutions to this problem, but it should be something very easy to solve. Write the emails to a datastore table, with an auto_add_now date field to record the order in which they entered. Your cron job that runs every minute pulls the eight oldest records off, mails them and deletes them.
Certainly, if you can solve this is a reasonably generic manner, you can be the person who solves this problem for everyone with a nice open source module.