Ii it possible to load own (my traced website with GA) analytics raw data using a GAP account, and now make deeper analysis with BigQuery?
If you contact you GAP re-seller then he/she should enable the BigQuery integration for your account and you will not need do anything. Otherwise if you have raw analytics data you can upload it into BigQuery yourself (but you will need tohandle tha update process if this is not a once off thing)
Related
I use these following GCP products together for a CRM system:
Cloud SQL
App Engine
Bigquery
Once a week an external application exports data from Bigquery in this way:
The external application makes a request to Appengine with a token.
AppEngine retrieves permissions for this token from Cloud SQL, makes some additional computation to obtain a list of allowed IDs.
Appengine runs a Bigquery's query filtered with these ids. Something like that: SELECT * FROM table WHERE id IN(ids)
Appengine responds to the external application with a unmodified result of query in JSON.
The problem is that the export is not very often, but amount of data can be large and I dont want to load AppEngine with this data. What other GCP products are useful in this case? Remember I need to retrieve permissions from Appengine and CloudSQL.
Unclear whether the JSON is just directly from BigQuery query results, or you do additional processing in the application to render/format it. I'm assuming direct results.
An option that comes to mind is to leverage cloud storage. You can use the signed url feature to provide a time-limited link to your (potential large) results without exposing public access.
This, coupled with BigQuery's ability to export results to GCS (either via an export job, or using the newer EXPORT DATA SQL statement allows you to run a query and deliver results directly to GCS.
With this, you could simply redirect the user to the signed URL at the end of your current flow. There's additional features that are complementary here, such as using GCS data lifecycle features to age out and remove files automatically so you don't need to concern yourself with slow accumulation of results.
I have an express API application running on GAE. It is my understanding that every time someone makes a GET request to it, it creates a log. In the Logs Explorer (Operations -> Logging -> Logs Explorer), I can filter to view only GET requests from a certain source by querying:
protoPayload.method="GET"
protoPayload.referrer="https://fakewebsite.com/"
In the top-right of the Logs Explorer, I can also select a time range of 1 day to view logs from the past 24 hours.
I want to be able to see how many GET requests the app receives from a given referrer every day. Is this possible? And is there functionality to display the daily logs in a bar chart (say, to easily visualize how many logs I get every day over the period of a week)?
You can achieve this but not directly in the Cloud Logging service. You have started well: you have created a custom filter.
Then, create a sink and save the logs to BigQuery.
So now, you can perform a query to count the GET every day, and you can build a datastudio dashboard to visualise your logs.
If only the count is needed on daily basis you can create a sink to stream data directly into the BigQuery. As the data needs to be segregated on daily basis while creating a sink , a better option would be to use a partition table which can help you in two ways:
You would have a new table everyday
Based on your usage although BigQuery provide a free tier , this data is not
needed in near future storing it this way will reduce your cost and querying cost
BigQuery comes with data studio , as soon as you query on table you'll have the option to explore the result in Data studio and generate reports as needed.
I have some logs in Cloud Logging. I did not create metrics before the logs came in, so I do not have any metrics containing this data. I'd like to visualize this data in a chart. Is this possible?
I see from this answer (Can't display data with with log-based metric) and the docs:
The data for logs-based metrics comes from log entries received after the metrics are created. The metrics are not populated with data from log entries that are already in Logging.
that Metrics only contain log entries from after the Metric was created. Therefore, it seems impossible to chart historical data using a Metric that was created after the data.
The only ways I've found to create a chart are using Metrics Explorer and the Monitoring dashboard. Both of these ultimately require a Metric with the data, which I am not able to create. Are there any other ways to chart data that don't require a Metric? If not, does this mean it's impossible to chart historical data with Cloud Logging/Monitoring?
what actions/connectors one could use to parse a excel file and map columuns from a excel table to and external store, in this case i wish to make a record in Dynamics 365 operations using OData entity.
Thanks
Why exactly are the obvious RapidStart Services no option for you?
If you definitely need to use OData, I suggest building a publishable OData page via WebServices. You then can implement a parser in any environment/language you prefer and submit the new record via a RESTful Webservice-Call (which nearly every framework should allow) to this very same page, in order to submit the record to your productive environment.
We're using Cloudant as the remote database for our app. The database contains documents for each user of the app. When the app launches, we need to query the database for all the documents belonging to a user. What we found is the CDTDatastore API only allows pulling the entire database and storing it inside the app then performing the query in the local copy. The initial pulling to the local datastore takes about 10 seconds and I imagine will take longer when adding more users.
Is there a way I can save only part of the remote database to the local datastore? Or, are we using the wrong service for our app?
You can use a server side replication filter function; you'll need to add information about your filter to the pull replicator. However replication will have a performance hit when using the function.
That being said a common pattern is to use one database per user, however this has other trade offs and it is something you should read up on. There is some information on the one database per user pattern here.