So, i'm a bit confused how youtube API queries impact on quota. They all lay out on a "composition" aspec of requisitions:
The Quota calculator show the cost of various resources.
For example search.list = 100 and videos.list = 1.
One aspect of it is not clear to me though. How it calculates on a 'multi-results' single request?
query
quota doubt
/search?maxResults=10
is it one 100 quota, or ten 100 (1000) quotas?
/videos?id=A,B
is it one quota, or two quotas?
/video?part=A,B
is this adding two quotas? (each video??) Since no ?part= returns only id related data
/...?fields=A,B(C)
is fields query impacting the request quota anyhow?
I first thought it was really straightforward: 1 call, 1 quota "package". And that seemed to be supported by this calculator's quote:
If your application calls a method, such as search.list, that returns multiple pages of results, each request to retrieve an additional page of results incurs the estimated quota cost.
But while developing a simple video list, my daily quota blew up pretty damn fast. So not sure anymore.
Every time you call the method in question you incure the quota cost
For example:
search.list 100
WHen you call search.list it costs 100 if you call it again to get the next page of results it will cost you another 100 points.
Ig we check this one where you are trying to get back two videos.
/videos?id=A,B
The same is true it will be a single request to the server so the quota cost will be one.
Fields does not effect it. Its just the request you make. Batching will also not save you from quota cost. If you batch these requests you will be charged for each of the quests within the back.
Intro to YouTube API and cost based quota for beginners
A lot of this information is on the Quota cost page.
The table below shows the quota cost for calling each API method. All API requests, including invalid requests, incur a quota cost of at least one point.
If your application calls a method, such as search.list, that returns multiple pages of results, each request to retrieve an additional page of results incurs the estimated quota cost.
Related
I'd like to make a GAE app multi-tenant to cater to different clients (companies), database namespaces seems like a GAE endorsed solution. Is there a meaningful way to split GAE fees among client/namespaces? GAE costs for app are mainly depends on user activities - backend instances up time, because new instances are created or (after 15 min delay) terminated proportionally to the server load, not total volume of data user has or created. Ideally the way the fees are split should be meaningful and could be explained to the clients.
I guess the most fair fee splitting solution is just create a new app for a new client, so all costs reported separately, yet total cost will grow up, I expect few apps running on same instances will use server resources more economically.
Every app engine request is logged with a rough estimated cost measurement. It is possible to log the namespace/client associated with every request and query the logs to add up the estimated instance costs for that namespace. Note that the estimated cost field is deprecated and may be inaccurate. It is mostly useful as a rough guide to the proportion of instance cost associated with each client.
As far as datastore pricing goes, the cloud console will tell you how much data has been stored in each namespace, and you can calculate costs from that. For reads/writes, we have set up a logging system to help us track reads and writes per namespace (i.e. every request tracks the number of datastore reads and writes it does in each namespace and logs these numbers at the end of the request).
The bottom line is that with some investments into infrastructure and logging, it is possible to roughly track costs per namespace. But no, App Engine does not make this easy, and it may be impossible to calculate very accurate cost estimates.
I'm experimenting with the Search API with appengine, and am consistently running into the "short term burst quota" described in this SO post: quotas on appengine search api for java.
In our use case, we need to delete all documents from an Index, and re-populate the index. We've attacked this by:
Looping through the list and deleting documents
Adding documents through a task queue (1/s) throughput rate
I'm still bumping against this burst limit and I'm wondering if I might have to put a sleep when I delete the documents?
This burst rate is severely limiting us (since we are building these indexes on the fly based on other criteria) and am curious if anyone has any more insight.
You should limit the usage using a queue and not sleep (never ever use sleep in AppEngine).
You can request an increase on your quota.
I believe that there is a 100 calls per minute quota as well as 20k per day (although the per minute quota may now have been removed).
The best way around this that I have found is with cursors and taskqueues to spread the load.
Does anyone have any number on Google App Engine (free quota) in terms of total number of request and unique visitors it allows per day?
Maybe someone who has live production code can tell us this?
Rough number is enough, just to get the idea.
I can not get this information from the pricing model.
Thanks
I had this question when I first started using App Engine, but it's impossible to answer with the information in your question.
You must have an estimate on the individual API quota usages, then calculate based on that.
You might be able to simplify it by trying to figure out which API quota you're likely to hit first, and then figuring out the number of requests you can serve before that quota runs out. ie:
Storing photos or other large data for users? You'll probably hit the blobstore quota first. Daily/unique visitor counts probably won't matter.
Serving lots of photos or large data? You'll probably hit the bandwidth quota first.
Need to start a channel for every view? You'll probably hit the channel quota first and get 100 views per day.
Need to send an email for every view? You'll probably hit the mail quota first.
Need to query the datastore a lot? You'll probably hit the datastore limit first.
The datastore limit is the hardest to calculate. You get 50k read and 50k write ops. Most likely you'd read more than write.
If you need 2 read ops per page, you might could do 25k views per day.
If you need 2 read ops per page, but you're smart and you memcache them, and memcache is effective 80% of the time, you could get 125k views per day.
If you need 500 read ops per page and you can't cache it, you can do 100 views per day. That's provided you don't run out of one of the other quotas.
Do your own math.
The quotas and rates (for free and paid apps) are listed on https://developers.google.com/appengine/docs/quotas.
The logs tell me how much a certain request costs
api_cpu_ms=278 cpm_usd=0.009244
Is this cost still accurate with the new billing model?
Can I deduce anything about the number of datastore operations the request used?
Is there any way to know the exact number of read, write and small datastore operations used during a single request?
I was told about 2 months ago that they do not reflect the new billing model, and I haven't seen any indications that they've been fixed to reflect the new billing model.
You can get the number of operations per request by enabling Appstats in your application.
I'm having some trouble with the google app engine datastore. Ever since the new pricing model was introduced, the cost of running my app has increased massively.
The culprit appears to be "Datastore small operations", which come in at more than 20 Million ops per day!
Has anyone had this problem, I don't think I'm doing an excessive amount of key lookups, and I only have 5000 users, with roughly 10 - 20 requests per minute.
Thanks in advance!
Edit
Ok got some stats, these are after abut 3 hours. Here is what I am seeing in my dashboard, in the billing section:
And here are some of the stats:
Obviously there are quite a lot of calls to datastore.get. I am starting to think that it is my design that is causing the problem. Those gets correspond to accounts. Every user has an account, but an account can be one of two types, for this I use composition. So each account entity has a link to its sub account entity.
As a result when I do a search for nearby users it involves fetching the accounts using the query, and then doing a get on each account to get its sub account. The top request in the stats picture is a call that gets 100 accounts, and then has to do a get on each one. I would have thought that this was a very light query, but I guess not. And I am still confused by the number of datastore small ops being recorded in my dashboard.
Definitely use appstats as Drew suggests; regardless of what library you're using, it will tell you what operations your handlers are doing. The most likely culprits are keys-only queries and count operations.
My advice would be to use AppStats (Python / Java) to profile your traffic and figure out which handler is generating the most datastore ops. If you post the code here we can potentially suggest optimizations.
Don't scan your datastore, use get(key) or get_by_id(id) or get_by_key_name(keyname) as much as you can.
Do you have lots of ReferenceProperty properties in your models? Accessing them will trigger db.get for each property unless you prefetch them. This would trigger 101 db.get requests.
class Foo(db.Model):
user = db.ReferenceProperty(User)
foos = Foo.all().fetch(100)
for f in foos:
print f.user.name # this triggers db.get(parent=f, key=f.user)