How do I set a cost limit in Google Developers Console - google-app-engine

Some functions in the Google Developers Console, like the Analytics API, are free until you reach a quota. Other functions, like Google Cloud Storage, create costs from the first click.
When I upload a file under https://console.developers.google.com/ > Storage > Cloud Storage > Storage Browser and I make this file publicly available, I pay about $0.12 per GB traffic.
But theoretically the traffic to this link could explode, e.g. because of sudden popularity. Therefore I would like to set something like a daily or monthly cost limit.
Q: How do I protect myself from overly high costs in the Google Developers Console?

You cannot. I asked Google about this, here's their response, from May 7 2016:
(GCE = Google cloud engine. No spending limits.
GAE = Google app engine — yes it has spending limits.)
... you are eligible for support on ... only ...
... [various helpful links] ...
That been said, at the moment there is no a feature that allows you to
configure a limited budget on GCE. This feature is certainly available
for GAE [1]. As you mentioned in your comments, you either can totally
shut down your VMs (will depend on your use case) or set the VMs to
send you alerts if they reach a certain traffic limit [2].
Sincerely,
Someone's first name
Technical Solutions Representative
Google Cloud Platform
[1] https://cloud.google.com/appengine/docs/quotas
[2] https://cloud.google.com/monitoring/support/notification-options
#wmdry, you wrote: "traffic to this link could explode" — I'm afraid of this too. That's why I asked Google about this. And I'm planning to avoid Google's CDN because of this, and use another CDN provider instead, which has spending limits. Because, unlike Nginx, I don't see any way for me to rate limit / throttle Google's CDN.
I do plan to use GCE (Google Cloud Engine) though. Therefore, right now I'm reading about how to rate limit my Nginx server. Because if I just configure Nginx correctly, then those $0.12 / GB you mentioned, cannot possible explode to ... like $10k in a month? What if Google sends a $10k bill when I'm back from an a few week's vacation, just because of my hobby project and a few people downloading a 1 MB movie over and over again forever (because: evil). Hmm, & the bigger & faster my servers, the higher the risk.
I hope Google will add spending limits, because I did want to use Google's CDN.
Update 2020: Apparently this does bite people from time to time — look here:
"Burnt $72k testing Firebase and Cloud Run and almost went bankrupt", Dec 08, 2020, https://news.ycombinator.com/item?id=25372336,
In that case, they could contact Google and in the end didn't need to pay.

As of July 2017 you can set budgets that send notifications via email but do not cap spending:
To set an alert-only budget, which will not cap spending:
Go to the Cloud Platform Console.
Open the console left side menu and click Billing
If you have more than one billing account, click the billing account name.
On the left, click Budgets & alerts.
Official help page: https://support.google.com/cloud/answer/6293540?hl=en

I found that Google's documentation now provides two methods to actually limit the cost of a GCP project. It involves the following setup:
Create a Cloud Function that checks the cost against the budget, and carries out a certain action if the cost exceeds the budget. Google's Documentation provides a sample code snip that can either shutdown all VM instances in a Project or disable the billing for a project. Shutting down all VMs would stop all VM-related cost but you get to keep your data (and still have to pay for the storage). Disabling the billing for a project would effectively zap all cost-related activities and you could lose data. You can name the Cloud Function "budget-enforcer".
The Google code snip as provided above has a hard coded ZONE variable. Remember to change it to match your zone!
Create a Service Account to run the Cloud Function "budget-enforcer". For shutting down VMs, the Service Account would need role "Compute Instance Admin (v1)". For disabling billing on a project, the Service Account would need role "Project Billing Manager".
Set a Topic for the Cloud Function (I call mine "proj-name-stop-vm" and "proj-name-disable-bill").
Set up a budget alert as usual, and connect it to one of the Pub/Sub topic above.
Please be noted that Google's documentation did mention that there could be a delay between the cost exceeds a budget and the function is triggered, so you should build in a buffer if you have an absolute hard cost limit. I use 90% of the budget as the trigger line for shutting down my instances.

The API usage can be limited with a hard limit:
Depending on the API, you can explicitly cap requests in a variety of
ways, including: requests per day, requests per 100 seconds, and
requests per 100 seconds per user. You might want to limit the
billable usage by setting caps. For example, to prevent getting billed
for usage beyond the free courtesy usage limits, you can set requests
per day caps
Source

You can combine budget pub/sub alerts with a cloud function that can disable billing on your entire account if a threshold is met.
Full Tutorial Here:
https://www.youtube.com/watch?v=KiTg8RPpGG4
GitHub Repo Here: https://github.com/aioverlords/Google-Cloud-Platform-Killswitch
To Disable Billing
const _disableBillingForProject = async projectName => {
const res = await billing.updateBillingInfo({
name: projectName,
resource: {
billingAccountName: ''
}, // Disable billing
});
console.log(res);
console.log("Billing Disabled");
return `Billing disabled: ${JSON.stringify(res.data)}`;
};

Simply go to the developer console:
https://console.developers.google.com/project
Select your project.
Select "billings & settings"
Enable billing.
Then go to Compute/AppEngine/Settings and set a daily budget.

Go to Google Cloud console, and then to Billing / Budgets and Alerts and create a new budget for one or all your projects. You can select which services should be included in the limit and set a monthly amount that should not be exceeded.

Related

Which Google Cloud product to use to execute a very long process Google Cloud VideoIntelligence Analysis

I have been using Google Cloud Video Intelligence annotation functionality with Google App Engine Flex. When I try to use VideoIntelligence with a two hour video, it takes 60 minutes for the AnnotateVideo function to respond.
gs_video_path ='gs://'+bucket_name+'/'+videodata.video.path+videodata.video.name
print(gs_video_path)
video_client = videointelligence.VideoIntelligenceServiceClient()
features = [videointelligence.enums.Feature.OBJECT_TRACKING]
operation = video_client.annotate_video(gs_video_path, features=features)
Currently, the only place I can execute this is on Google App Engine Flex. Yet, Google App Engine Flex keeps an instance idle all the time, it is very similar to running a VM in terms of cost.
Google App Engine has a 540 seconds timeout and same wise Google Cloud Run has a timeout of 900 seconds and Google Cloud Functions has a max timeout of 600 seconds as far as I understand.
In these circumstances, which Google Cloud product should I use for a one hour process that should take place while avoiding having an idle instance when there is no usage.
(Please don't respond quoting GKE or other VM based solutions, no idle instance solutions are accepted)
Cloud Run’s 900 second timeout is subject to change soon to satisfy your needs (up to an hour). There's a feature in the works. I’ll update here once it's available in beta, stay tuned.
#ahmetb-todo
You can specify output_uri in the original request. This will write the final result to your GCS bucket. Then you don't have to wait for the long running operation to finish on your VM. The initial request will only take a few seconds so you can use Google Cloud Function.
When the operation finishes an hour later, you process the output json files by setting up a trigger on your output GCS bucket.
I don't think Google has a service that matches your needs.
Probably you should implement some custom workflow, like:
From "short living" environment, like Function, CloudRun or AppEngine do following:
Put an event for your long-running task to PubSub
Use Compute Engine API to start a VM
When VM starts, it's startup script should get latest item from the PubSub and start your long-running task
When task is done, VM terminates itself using ComputeEngine API or calls a Function that invokes shutdown

Does Google App Engine support log sampling?

By log sampling I mean that not all the logs are saved to Stackdriver but just a sample of them. Useful for high traffic applications.
I've searched but the only mention I see is here in the context of App Engine Flex and OpenAPI.
There is also an option to sample logs at query time here.
Log sampling is also mentioned as an option for Trace clients here but that doesn't seem applicable to App Engine.
Stackdriver does not have built in support for sampling logs but you can specify a log exclusion filter which will allow you to remove log entries based on a criteria.
If you choose to exclude everything and specify a sampling criteria I think you can achieve this:
The link below explains this in a bit more detail:
https://cloud.google.com/logging/docs/exclusions

How to increase the Socket Connect Count quota for app engine

There's a hard limit on the socket connect count for GAE https://cloud.google.com/appengine/quotas. The number 3M per day seems relatively low for any largish scale project. I'm calling into Google Drive APIs (https://developers.google.com/drive/api/v3/reference/) from java and hitting against the limit for my project. Is there a workaround for this?
You can ask for a quota increase request to Google but keep in mind that your account must not be a free trial to do it and that in some regions this is not possible. In your request you must to explain the reason why you need the quota increase.
Once you submit the request, it will be take until 3 business days to apply the change.
You can take a look at this link Request a quota increase to know how to request a quota increase. You only must to replace ‘GPU’ for ‘Socket receive count per day’

Using 1 intance of google-app-engine to monitor external service

I planning to create a NodeJS program, that work 24/7, that ping and make requests to an external server (outside of google cloud) every minute. Just to see that it the external services are are live.
If there is any error it will notify me by SMS & Email.
I don't need any front-end for this app, and no one needs to connect to it. Just simple NodeJS program.
The monitoring and configuration will be by texts files.
Now the questions:
It looks like it will cost me just $1.64. It sounds very cheap. Am I missing something?
It needs to work around the clock, I will request it to start it once, and it need to continue working, (by using setInterval). Is it will be aborted?
What it is exactly mean buy 1 instance. What an instance can do? Only respond to one request or what?
I tried to search in Google: appengine timeout, but didn't found anything that helps.
Free Quota
If you write your application in Python, PHP, Go or Java it can fit in free usage quota:
https://cloud.google.com/appengine/docs/quotas
So there will be absolutely no costs to run it on Google App Engine platform.
There are limit of 657,000 UrlFetch API Calls per day (more than 450 calls per minute in 24/7 mode) for free apps. 4GB traffic may also be sufficient for this kind of work.
Keep in mind there is no SMS sending services provided by Google App Engine and you will need to spend additional UrlFetch API calls to use external SMS services.
Email sending is also limited to 100 Emails per day (or 5000 Emails to admin address), so try not so send repeated notifications about same monitored server every minute, or you'll deplete your Email quote in 1.5 hours.
Scheduled Tasks
There is no way to run single process indefinitely without interruption on App Engine. But you don't have to!
You'll need to encapsulate all the work you're planning to execute in every iteration into single task and then schedule it to run every minute with Cron. See this documentation for Python: https://cloud.google.com/appengine/docs/python/config/cron
It is recommended to have some configuration page where you can set some internal configuration or see monitoring statistics, at least manage flag to temporarily pause tasks execution without redeploying your app.

"Over quota" when using GCS json-api from App Engine

I am using Go on App Engine. In most cases, I use the file api to access GCS, which works great, except that deletes don't work so to delete files I use the JSON-API (specifically, the google-go-api-client). To authenticate, I use app engine service accounts. We are sometimes seeing an error come back of "Over quota:" with nothing after the colon. Since we are a paid app, what quota could this be? Is there a burst limit (e.g. no more than X requests in a single minute)? Is there any places where any such applicable quotas are documented?
The caching mechanism is broken for goauth2 and serviceaccount tokens. You can see the issue I created here for more detail: https://code.google.com/p/goauth2/issues/detail?id=28
I came across a "over quota" issue myself when requesting more than 60 service accounts a minute. I opened a ticket with AppEngine support (I pay for the silver package) and got this undocumented information out of them.
You can apply the patch yourself in your $GOPATH/src/code.google.com/p/goauth2/appengine/serviceaccount/cache.go file. This fixed the issue you described for my team.
Even i had found same problem and found two reasons:-
1.Daily budget
2.Logs retention
Solution:
for problem 1 increase the daily budget
for problem 2 increase the retention from 1 to higher GB
![enter image description here][1]

Resources