I'm trying to use the Google Monitoring API to retrieve metrics about my cloud usage. I'm using the Google Client Library for Python.
The API advertises the ability to access over 900 Stackdriver Monitoring Metrics. I am interested in accessing some Google App Engine metrics, such as Instance count, total memory, etc. The Google API Metrics page has a list of all the metrics I should be able to access.
I've followed the guides on the Google Client Library page , but my script making the API calls is not printing the metrics, it is just printing the metric descriptions.
How do I use the Google Monitoring API to access the metrics, rather than the descriptions?
My Code:
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
...
response = monitor.projects().metricDescriptors().get(name='projects/{my-project-name}/metricDescriptors/appengine.googleapis.com/system/instance_count').execute()
print(json.dumps(response, sort_keys=True, indent=4))
My Output
I expect to see the actual instance count. How can I achieve this?
For anyone reading this, I figured out the problem. I was assuming the values would come from the 'metric descriptors' class in the api, but that was a poor assumption.
For values, you need to use a 'timeSeries' call. For this call, you need to specify the project you want to monitor, start time, end time, and a filter (the metric you want, such as cpu, memory, etc.)
So, to retrieve the app engine project memory, the above code becomes
request = monitor.projects().timeSeries().list(name='projects/my-appengine-project',
interval_startTime='2016-05-02T15:01:23.045123456Z',
interval_endTime='2016-06-02T15:01:23.045123456Z',
filter='metric.type="appengine.googleapis.com/system/memory/usage"')
response = request.execute()
This example has the start time and end time to cover a month of data.
Related
On my appspot website, I use a third party API to query a large amount of data. The user then downloads the data in CSV. I know how to generate a csv and download it. The problem is that because the file is huge, I get the DeadlineExceededError.
I have tried tried increasing the fetch deadline to 60 (urlfetch.set_default_fetch_deadline(60)). It doesn't seem reasonable to increase it any further.
What is the appropriate way to tackle this problem on Google App Engine? Is this something where I have to use Task Queue?
Thanks.
DeadlineExceededError means that your incoming request took longer than 60 secs, not your UrlFetch call.
Deploy the code to generate the CSV file into a different module that you setup with basic or manual scaling. The URL to download your CSV will become http://module.domain.com
Requests can run indefinitely on modules with basic or manual scaling.
Alternately, consider creating a file dynamically in Google Cloud Storage (GCS) with your CSV content. At that point, the file resides in GCS and you have the ability to generate a URL from which they can download the file directly. There are also other options for different auth methods.
You can see documentation on doing this at
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/
and
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/functions
Important note: do not use the Files API (which was a common way of dynamically create files in blobstore/gcs) as it has been depracated. Use the above referenced Google Cloud Storage Client API instead.
Of course, you can delete the generated files after they've been successfully downloaded and/or you could run a cron job to expire links/files after a certain time period.
Depending on your specific use case, this might be a more effective path.
Say that in a Google App Engine application (Java) some requests take a very long time to complete; perhaps some even time out after 30 seconds. Does the GAE Console (Dashboard, Monitoring or similar) provide any way to list the URLs (or any other request properties, such as API method calls) associated with the long-running requests?
https://cloud.google.com/appengine/docs/python/tools/appstats
The Python SDK includes the Appstats library used for profiling the
RPC (Remote Procedure Call) performance of your application. An App
Engine RPC is a roundtrip network call between your application and an
App Engine Service API. For example, all of these API calls are RPC
calls:
Datastore calls such as ndb.get_multi(), ndb.put_multi(), or
ndb.gql(). Memcache calls such as memcache.get(), or
memcache.get_multi(). URL Fetch calls such as urlfetch.fetch(). Mail
calls such as mail.send().
Actually, the old Dashboard (https://appengine.google.com/dashboard) provides the info I wanted in the Current Load box (bottom left), in the Avg Latency (last hr) column.
Our hosting provider uses an app called AWStats for to give us data about visits/visitors etc. However when I compare that to my google analytics data the numbers are far off. For example AWStats says we had about 5000 visitors but GA says about 1500. How can I uncover the source of the disparity?
A few ideas:
Improper implementation for one or both analytics services
Different definition of what a visitor is for both analytics services. For example, does GA say that you have 1500 unique visitors? or 1500 visits?
Have you tried using a program like HTTPFox to look at the requests getting sent? Are there duplicate requests?
There is search support (experimental) for python and Java, and eventually Go also may supported. Till then, how can I do minimal search on my records?
Through the mailing list, I got an idea about proxying the search request to a python backend. I am still evaluating GAE, and not used backends yet. To setup the search with a python backed, do I have to send all the request (from Go) to data store through this backend? How practical is it, and disadvantages? Any tutorial on this.
thanks.
You could make a RESTful Python app that with a few handlers and your Go app would make urlfetches to the Python app. Then you can run the Python app as either a backend or a frontend (with a different version than your Go app). The first handler would receive a key as input, would fetch that entity from the datastore, and then would store the relevant info in the search index. The second handler would receive a query, do a search against the index, and return the results. You would need a handler for removing documents from the search index and any other operations you want.
Instead of the first handler receiving a key and fetching from the datastore you could also just send it the entity data in the fetch.
You could also use a service like IndexDen for now (especially if you don't have many entities to index):
http://indexden.com/
When making urlfetches keep in mind the quotas currently apply even when requesting URLs from your own app. There are two issues in the tracker requesting to have these quotas removed/increased when communicating with your own apps but there is no guarantee that will happen. See here:
http://code.google.com/p/googleappengine/issues/detail?id=8051
http://code.google.com/p/googleappengine/issues/detail?id=8052
There is full text search coming for the Go runtime very very very soon.
i tried to use the custom search api ( http://code.google.com/intl/de-DE/apis/websearch/docs ) with java. it works perfectly on eclipse on my local machine.
when i try to do the same from google app engine the reply is: {"responseData": null, "responseDetails": "Quota Exceeded. Please see http://code.google.com/apis/websearch", "responseStatus": 403}
i do not understand. isn't it possible to call search api from GAE apps?
If you look at the very top of that page you linked to, they note that the API has been deprecated and the number of search queries you can make is limited.
However, if you absolutely NEED to use that API instead of the Custom Search API as Google suggests, there are a few troubleshooting steps you can take:
1) Check that your API key is unique to the project, and the limited number of queries you're allowed isn't being consumed by some other application.
2) Google does (did?) hostname filtering so that one computer doesn't use up all the API requests. You may be able to move the queries to Javascript instead of Java -- essentially move the request from the server to the client.
3) Try using a named backend (Java Backends)