There is a feature in Google Cloud Platform that allows reporting resource usages. This is cool because it supports two things I will make use of:
Time range (usage_start_time, usage_end_time).
Labels (labels_key, labels_value).
Usage data (usage_amount and more fields).
And, by doing this, I can make use of plain SQL queries over the dumped data (which goes to BigQuery) to report what I need.
But is there a way I can do this without using BigQuery? Like doing a real-time query? Concretely, solve this requirement:
Take a time range.
Take the service type I need (e.g. stating "Google Cloud Run Deployment").
Take a custom label (let's say I have a label named "customer" and its value "my-mother").
Ask the usage amounts and corresponding pricing values for each resource of that service type and labels criteria.
Weight-summarize the end value (the value would be expressed in US dollars).
Is there a way I can do that with some direct GCP Billing API and not by using SQL on BigQuery dumps? (this means: An existing function or class in the billing library or somewhere else - not sure but one that allows me to ask this query).
You can export billing data to Pub/Sub, Cloud Storage, or BigQuery. The Billing APIs do not provide query abilities.
Is there a way I can do that with some direct GCP Billing API and not
by using SQL on BigQuery dumps?
If you want those features without using BigQuery then you will need to create an export sink and load the data into a system that supports queries.
The BigQuery dataset exports are the source of truth for your resource consumption. This is similar to CUR of AWS.
If you'd like to access the GCP billing data over an API - the best way would be to connect your billing dataset to a cloud cost management platform and use their APIs to query your cost by day, week, month or services, or by labels.
You can check out Economize - generous free tier, connects with your GCP billing dataset and you can use the API to query your data.
Related
Firstly I'm new to development and currently I have a problem with server data filling up rapidly. I'm looking at solutions such as watcher programs to help me detect when the server data is reaching the limit but I wanted to know if cloud solutions could help in this regard. Additionally I also wanted to know if companies such as Snowflake can help to handle fast growing data and in what way can a developer use it or will it be too costly to use this approach from an enterprise point of view.
I have tried to look up the documentations of Snowflake but I am unable to reach any conclusions as to whether it can help me. I could just see articles about storage and that they store data by compressing it but I wanted more clarity on this solution.
Snowflake stores the data using Cloud Storege Services (AWS S3, Google Cloud Storage, or Microsoft Azure), so you can't fill the server data in normal conditions (never heard that S3 is full on any region).
Check the pricing page to see if it will be costly for you (or not):
https://www.snowflake.com/pricing/
Are there any cloud hosting solutions for geospatial data? I am currently writing a directory style app where businesses can sign up and then users can find nearby ones.
I am considering Google App Engine for this, but from what I can tell the GeoModel code is quite expensive (up to tens of thousands of dollars a year) to run since Google updated the pricing of App Engine. It doesn't seem like App Engine's database is really suited to this kind of query (though the SQL solution may be an answer).
I was hoping to find a service where I could send off a HTTP request to add data (a business' id, name and icon url) to a database, and then another one to find a list of businesses that are nearby to a given point. A service is preferable as this is work done for a client and we would like the solution to be managed with as little interaction from us or the client needed as possible.
EDIT:
I just found cartodb.com which uses PostgreSQL and is reasonably priced. Are the any other alternatives?
The App Engine Search API (currently in Experimental) supports GeoPoints and geosearch, and is great for exactly the kind of query that you describe.
See the Google Developers Academy (GDA) App Engine Search API classes for a bit more info and an example as well.
http://www.iriscouch.com/ is a cloud-based host for CouchDB and they support the geocouch extensions for CouchDB to store geoJSON data and perform spatial queries.
We have decided to go with cartodb.com because it looks like they have a good price to ease of use ratio.
You mentioned going with CartoDB, which is a good choice with a nice UI.
Just adding, if you were just looking for a scalable backend, you could use StormDB. It is a cloud hosted SQL database with geospatial extensions. You data is automatically distributed amongst multiple nodes for write, read, and parallel query scalability.
I am building an application that is an enterprise management system using gae. I have built several applications using gae and the datastore, but never one that will require a high volume of users entering transactions along with the need for administrative and management reporting. My biggest fear is that when I need to create cross-tab and other detailed reports (or business intelligence reporting and data manipulation) I will be facing a mountain of problems with gae's datastore querying and data pull limits. Is it really just architectural preference or are there quantitative concerns here?
In the past I have built systems using C++/c#/Java against an Oracle/MySql/MSSql (with a caching layer sprinkled in for some added performance on complex or frequently accessed db results).
I keep reading that we are to throw away the old mentality of relational data and move to the new world of the big McHashTable in the sky... but new isnt always better... Any insight or experience on the above would be helpful.
From the Cloud SQL FAQ:
Should I use Google Cloud SQL or the App Engine Datastore?
This depends on the requirements of the application. Datastore provides NoSQL key-value > storage that is highly scalable, but does not support the complex queries offered by a SQL database. Cloud SQL supports complex queries and ACID transactions, but this means the database acts as a ‘fixed pipe’ and performance is less scalable. Many applications use both types of storage.
If you need a lot of writes (~XXX per/s) to db entity w/ distributed keys, that's where the Google App Engine datastore really shine.
If you need support for complex and random user crafted queries, that's where Google Cloud SQL is more convenient.
What is scare me more in GAE datastore is index number limitation. For example if you need search by some field or sorting - you need +1 index. Totally you can have 200 indexes. If you have entity with 10 searchable fields and you can sort by any field - there will be about 100 combunations. So you need 100 indexes. I have developed few small projects for gae - and this is success stories. But when big one come - this is not for gae.
About cache - you can do it with gae, but they distributed cache works very slow. I prefer to create private single instance of permanent backend with RESTfull API that holds cached values in memory. Frontend instances call this API to get/set values.
Maybe it is posible to build complex system with gae, but this will be a set of small applications/services.
I want to use the Google App Engine to store my data and then query/display/ edit it using Google Spreadsheets as the user interface, with multiple concurrent users having their own view of the data. The problem I have now is that if I put everyone's data on the same Google Spreadsheet that everyone accesses, we can't each do sorting / filtering at the same time.
Is there a way to do this, and is it a good idea to build a simple system this way? I'll eventually need to query a series of Google Word Processor documents as well.
Can someone point me in the right direction on this or suggest other options?
I would ask what the advantage of doing something like this is as opposed to say hosting your application on Google App Engine and building a javascript front end with grids to help sort/filter and view data.
Anyway to answer your questions, you can build your interface over Google Spreadsheets using Google App Scripts. This will allow you to do things like authenticate your user, query, update and display data. If you want to merely display data it turns out that Google Spreadsheets has some built-in functions to do that.
Regarding consistency you should read up on GAE's Datastore as well as its features like transactions. The datastore is not an RDBMS, but is an object database which stores objects against keys. Again something to consider if you are planning to do a lot of data analysis and computation (summations, aggregations).
Overall I would recommend doing a rough design of your system without fixing on particular technologies (like GAE, and Google Spreadsheets). Once you identify what your key goals are for your application, then you can figure out which technologies and resources would make the most sense within your budget.
For a contact management system web app that allows tennants to upload lists of contact records (having varying field structures) and then displays these back to multiple users (within tennant) one at a time is there a good p/saas database solution to handle this
-it would need to allow uploading lists with custom fields (20K records per list)
-allow updating of fields changed when users edit them (user may update 60 records a minute)
-would need to allow running queries against the lists to determine next record to display (this part utilizes set fields)
Obviously a scalable, easy to use, hassle free as possible design is the aim here.
Will it be easier than developing a local database design?
(Prefer not to use a full paas would like to keep the application tier seperate.)
If your queries are always against the set fields and not the custom fields, then any database would do assuming you keep the "custom" fields in blob format (xml, for example).
If that is the case then your local database design is pretty simple. If you host on amazon ec2, then you can use either of their saas database solutions (mysql or SimpleDB even).
-Dave