Simple Task: keep track of web traffic (hits) so that I can graph the number of hits per day for the last 30 days.
Current Datastore Model (2 fields): 1) Website ID 2) Timestamp of Hit
Problem: I'm using Google App Engine's datastore and don't have the ability to do a group-by or count.
Can anyone offer a simple way to structure my Google Datastore database to achieve this task? By returning all of the hits and then grouping them in my code seems like a performance hog. Any ideas?
I would use sharding counters for this specific task; have a look to the last example of this documentation.
Related
Im a little confused about this because the docs say I can use stackdriver for "Request logs and application logs for App Engine applications" so does that mean like web requests? Does that mean like millions of web requests?
Stackdriver's pricing is per resource so does that mean I can log all of my web servers web request logs (which would be HUGE) for no extra cost (meaning I would not be charged by the volume of storage the logs use)?
Does stackdriver use GCP cloud storage as a backend and do I have to pay for the storage? It just looks like I can get hundreds of gigabytes of log aggregation for virtually no money just want to make sure Im understanding this.
I bring up ELK because elastic just partnered with google so it must not do everything elasticsearch does (for almost no money) otherwise it would be a competitor?
Things definitely seem to be moving quickly at Google's cloud division and documentation does seem to suffer a bit.
Having said that, the document you linked to also details the limitations -
The request and application logs for your app are collected by a Cloud
Logging agent and are kept for a maximum of 90 days, up to a maximum
size of 1GB. If you want to store your logs for a longer period or
store a larger size than 1GB, you can export your logs to Cloud
Storage. You can also export your logs to BigQuery and Pub/Sub for
further processing.
It should work out of the box for small to medium sized projects. The built in log viewer is also pretty basic.
From your description, it sounds like you may have specific needs, so you should not assume this will be free. You should factor in costs for Cloud Storage for the logs you want to retain and BigQuery depending on your needs to crunch the logs.
My application is currently on app engine server. My application writes the records(for logging and reporting) continuously.
Scenario: Views count in the website. When we open the website it hits the server to add the record with time and type of view. Showing these counts in the users dashboard.
Seems these requests are huge now. For now 40/sec. Google App Engine writes are going heavy and cost is increasing like anything.
Is there any way to reduce this or any other db to log the views?
Google App Engine's Datastore is NOT suitable for such a requirement where you have to continuously write to datastore and read less often.
You need to offload this task to a third party service (either you write one or use existing one)
Better option for user tracking and analytics is Google Analytics (Although you wont be directly able to show the hit counters on website using analytics).
If you want to show your user page hit count use a page hit counter: https://www.google.com/search?q=hit+counter
In this case you should avoid Datastore.
For this kind of analytics it's best to do the following:
Dump data to GAE log (yes, this sounds counter-intuitive, but it's actually advice from google engineers). GAE log is persistent and is guaranteed to not loose data you write to it.
Periodically parse the log for your data and then export it to BigQuery.
BigQuery has a quite powerful query language so it's capable of doing complex analytics reports.
Luckily this was already done before: see the Mache framework. Also see related video.
Note: there is now a new BigQuery feature called streaming inserts, which could potentially replace the cumbersome middle step (files on Cloud Storage) used in Mache.
I would like to do this for a web app, with a large volume of requests ongoing (hundreds per day). I didn't see anything like this on the offical Google Plus API.
Maybe this will do it?
http://www.tomanthony.co.uk/blog/google_plus_one_button_seo_count_api/
I want to add analytics and reports to my application on Google App Engine. Can you tell me how I can do that on google's "database"? Are there any oss libraries/tools available?
Thank you
Look at the new Map Reduce API (it doesn't contain reduce yet, but that's what Google calls it anyway).
This makes it easy to map over all the entities of a type in your datastore and count the things you're interested in.
Christopher O'Donnell has an article about using the Mapper API for analytics. He creates time slices (hour, day, month) ahead of time and maps over those to store the data for each time slice.
Is it possible to get a data source URL of Google Spreadsheets for appengine datastore entities? I want to use the google visualization query objects to query my datastore. Or how I an expose my datastore with a datasource URL.
And for a Google visualization based project which one is better between Google Spreadsheet and GAE big table. Since Google Spreadsheet has very good query options and a nice harmonics with google visualization. One can get a direct DataTable from a data source URL. To do the same thing needs a good amount of task with GAE big table. Please share your experience in this area.
There's nothing built in to do this. You'll need to write your own code that returns your data in a format GViz supports.