For App Engine (Python, Standard environment), I have created a log-export (v2) in the same project as the app. Destination of the sink is a dataset in Google BigQuery.
I can perform some simple queries in BigQuery:
SELECT
severity,
timestamp AS Time,
protoPayload.host AS Host,
protoPayload.status AS Status,
protoPayload.resource AS Path,
httpRequest.status,
httpRequest.requestMethod,
httpRequest.userAgent,
httpRequest.remoteIp
FROM
[MY_PROJECT:MYLOGS.appengine_googleapis_com_request_log_20170214]
LIMIT
10
While httpRequest.status will be shown in the results with values (and the same for all the other selected fields), other fields of httpRequest are shown with null, e.g.: requestMethod, userAgent, remoteIp.
On the Cloud Log web page, I can see these log entries and these values are existing, but it seems that they are not exported to BigQuery.
When I try to filter by the request method to be GET, e.g.:
SELECT
severity,
timestamp AS Time,
protoPayload.host AS Host,
protoPayload.status AS Status,
protoPayload.resource AS Path,
httpRequest.status,
httpRequest.requestMethod,
httpRequest.userAgent,
httpRequest.remoteIp
FROM
[MY_PROJECT:MYLOGS.appengine_googleapis_com_request_log_20170214]
WHERE
httpRequest.requestMethod = 'GET'
LIMIT
10
This query will return zero records.
Any idea, why some fields are not shown in queries and cannot be used for filters in BigQuery?
Related
Am I crazy or are all the HealthCloud field permissions just unavailable to be queried via the rest API Soql calls?
If I issue this query in the developer console:
SELECT Id, SobjectType, Field, PermissionsEdit, PermissionsRead FROM FieldPermissions where SObjectType = 'PlanBenefitItem'
It returns exactly what I expect it to return - data about the fields and permissions. Issuing it via the API returns no records at all. I've checked all the object permission stuff because there's some weird rules about showing and not showing row data, but all of that seems fine and I wouldn't expect any return in the dev console if it weren't the case.
Is there something I'm missing here?
we are working on a datawarehouse project and need to pull data from our customer´s shopify database.
We want to analyse inventory stock quantity and stock value, also do market analysis, traffic analysis, top landing pages, conversions, revenues and others.
Shopify offers REST APIs for data pulling but not all our requirements can be fulfilled.
Our question are wether
there is a plan we can order, like shopify partner plus or AdminAPI that allows data pulling all data fields?
are there experiences in data pulling from shopify, deeper than the published REST API queries?
I am new to shopify requirements, that´s why my questions.
I am not sure that if it can solve your problem. Currently, our company uses an online tool called Acho Studio to retrieve Shopify data. It's like ETL tools that allow you to connect to Shopify and export data to somewhere. The difference is that the tool hosts a sever, so you can store data on it. Also, you can write SQL queries or apply some actions to do analysis.
This is the codes snippet i am using
# Prepare headers
headers = {}
headers["Content-Type"] = "application/json"
headers["X-Shopify-Access-Token"] = f"{access_token}"
# Define the shopify url
shop_host = f'https://{hostName}.myshopify.com'
orderEndpoint = f'{order_endpoint}'
url = f'{shop_host}/{orderEndpoint}'
params = {'status': 'any',
'limit': 250,
'created_at_min': '2022-01-01', 'created_at_max' : '2022-06-20'}
response = session.get(url=url, headers=headers, params=params)
orders = response.json()['orders'] # order data in json format
while 'next' in response.links:
try:
session = requests.Session()
response = requests.get(response.links['next']['url'], headers=headers)
orders.extend(response.json()['orders'])
except KeyError as e:
if response.links == 'next':
print('go next page')
else:
print('No more pages')
I am able to pull the Order data and other data such as refund, .etc
UPDATED:
I do not have an entry in the index.yaml, expecting zig-zag to work fine on equalities (which it appears to in the datastore UI) and here is the actual query I'm making:
UploadBlock.query(
UploadBlock.time_slices == timeslice, # timeslice is each value in the repeated property
UploadBlock.company == block.company,
UploadBlock.aggregator_serial == block.aggregator_serial,
UploadBlock.instrument_serial == block.instrument_serial)\
.fetch(10, keys_only=True)
I've cross-posted this because I didn't read the initial support page first, so this is also on the google forum at https://groups.google.com/forum/#!topic/google-appengine/-7n8Y_tzCgM
I have a datastore query I make as an integral part of my system. It's an equality filter on four properties. When I make this query for one set of criteria, the query just hangs. It does not hang for this set of criteria in the datastore viewer, so I can see that there are only two results to return. I can fetch both of these entities by id without problem, but the query always hangs, both in remote shell and in production. The query is running fine for a multitude of different criteria, it appears to just be this one set. I have attempted replacing both entities so that the criteria still matches, but they have different ids and I get the same result.
For some more specific information, the four properties are defined as follows:
company = ndb.KeyProperty(indexed=True)
aggregator_serial = ndb.StringProperty(indexed=True)
instrument_serial = ndb.StringProperty(indexed=True)
time_slices = ndb.IntegerProperty(indexed=True, repeated=True)
I'm running gcloud versions
Google Cloud SDK 207.0.0
app-engine-java 1.9.64
app-engine-python 1.9.71
app-engine-python-extras 1.9.69
beta 2018.06.22
bq 2.0.34
core 2018.06.22
gcloud
gsutil 4.32
Any help on what I should try for next steps would be greatly appreciated.
Thanks in advance!
My Google App Engine logs are being exported to BigQuery via the standard streaming export tool. I'd like to query "show me all log lines for requests in which any log line contains a string".
This query gives me the request ids I'm interested in:
SELECT protoPayload.requestId AS reqId
FROM TABLE_QUERY(logs, 'true')
WHERE protoPayload.line.logMessage contains 'INTERNAL_SERVICE_ERROR'
...and this lets me query for the related lines:
SELECT
metadata.timestamp AS Time,
protoPayload.host AS Host,
protoPayload.status AS Status,
protoPayload.resource AS Path,
protoPayload.line.logMessage
FROM
TABLE_QUERY(logs, 'true')
WHERE
protoPayload.requestId in ("requestid1", "requestid2", "etc")
ORDER BY time
However, I'm having trouble combining the two into a single query. BQ doesn't seem to allow subselects in the WHERE clause and I get confusing error messages when I try to do a traditional self-join with named tables. What's the secret?
To select lines where at least one of logMessage contains given string, you can use OMIT IF construct
SELECT
metadata.timestamp AS Time,
protoPayload.host AS Host,
protoPayload.status AS Status,
protoPayload.resource AS Path,
protoPayload.line.logMessage
FROM
TABLE_QUERY(logs, 'true')
OMIT RECORD IF
EVERY(NOT (protoPayload.line.logMessage contains 'INTERNAL_SERVICE_ERROR'))
ORDER BY time
I would like to rewrite the example from the GAE djangoforms article to be show most up to date after submitting a form (e.g. when updating or adding a new entry) on Google App Engine using the High Replication Datastore.
The main recurring query in this article is:
query = db.GqlQuery("SELECT * FROM Item ORDER BY name")
which we will translate to:
query = Item.all().order('name') // datastore request
This query I would like to get the latest updated data from the high replication datastore after submitting the form (only in these occasions, I assume I can redirect to a specific urls after submission which just uses the query for the latest data and in all other cases I would not do this).
validating the form storing the results happens like:
data = ItemForm(data=self.request.POST)
if data.is_valid():
# Save the data, and redirect to the view page
entity = data.save(commit=False)
entity.added_by = users.get_current_user()
entity.put() // datastore request
and getting the latest entry from the datastore for populating a form (for editing) happens like:
id = int(self.request.get('id'))
item = Item.get(db.Key.from_path('Item', id)) // datastore request
data = ItemForm(data=self.request.POST, instance=item)
So how do I add entity groups/ancestor keys to these datastore queries to reflect the latest data after form submission. Please note, I don't want all queries to have the latest data, when populating a form (for editing) and after submitting a form.
Who can help me with practical code examples?
If it is in the same block, you have reference of the current intance.
Then once you put() it, you can get its id by:
if data.is_valid():
entity = data.save(commit=False)
entity.added_by = users.get_current_user()
entity.put()
id= entity.key().id() #this gives you inserted data id