I have an app which stores user data in GCP Datastore.
I have made a cron job that is scheduled to export the data in Datastore using the instructions given here.
Now, I want to get the results and finished time when this job fails.
(Not cron but job moved by cron)
How do I get the result detail of the export Cloud Datastore？
The scheduled datastore exports (and imports) can be long-running operations, so they have support for asynchronous execution and progress monitoring.
Basically when you launch the operation you get an operation id:
For example, when you start an export, the Cloud Datastore service
creates a long-running operation to track the export status. Here's
the output from the start of an export:
{
"name": "projects/[YOUR_PROJECT_ID]/operations/ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI",
"metadata": {
"#type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata",
"common": {
"startTime": "2017-05-25T23:54:39.583780Z",
"operationType": "EXPORT_ENTITIES"
},
"progressEntities": {},
"progressBytes": {},
"entityFilter": {
"namespaceIds": [
""
]
},
"outputUrlPrefix": "gs://[YOUR_BUCKET_NAME]"
}
}
The value of the name field is the ID of a long-running operation.
You can then use the operation ID with projects.operations.get in the operation Admin API to obtain its completion time and result URL:
This example output shows a recently completed export operation.
Operations are accessible for a few days after completion:
{
"operations": [
{
"name": "projects/[YOUR_PROJECT_ID]/operations/ASAyMDAwOTEzBxp0bHVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKKhI",
"metadata": {
"#type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesMetadata",
"common": {
"startTime": "2017-12-05T23:01:39.583780Z",
"endTime": "2017-12-05T23:54:58.474750Z",
"operationType": "EXPORT_ENTITIES"
},
"progressEntities": {
"workCompleted": "21933027",
"workEstimated": "21898182"
},
"progressBytes": {
"workCompleted": "12421451292",
"workEstimated": "9759724245"
},
"entityFilter": {
"namespaceIds": [
""
]
},
"outputUrlPrefix": "gs://[YOUR_BUCKET_NAME]"
},
"done": true,
"response": {
"#type": "type.googleapis.com/google.datastore.admin.v1.ExportEntitiesResponse",
"outputUrl": "gs://[YOUR_BUCKET_NAME]/2017-05-25T23:54:39_76544/2017-05-25T23:54:39_76544.overall_export_metadata"
}
}
]
}
I'm not entirely certain how you'd distinguish just the failed jobs, though, I imagine something different would show in the response field.
Related
Could really use some help here. I have a GAE NodeJS app in the standard environment. Until a few days ago (09/23) it was running just fine, it would respond to requests as expected, etc.
Today, the app responds with 403's when I try to make any request to my appspot url. I'm 100% certain this is not a code issue, as if I deploy the same code to GAE in another project, it works fine. Furthermore, the only firewall rule is a wildcard to allow all traffic.
Edit: adding the only relevant-looking log entry I see from the project:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "address#domain.com"
},
"requestMetadata": {
"callerIp": "x.x.x.x",
"requestAttributes": {
"time": "2021-09-23T15:04:05.198927Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "appengine.googleapis.com",
"methodName": "google.appengine.v1.Services.UpdateService",
"authorizationInfo": [
{
"resource": "apps/my-google-cloud-project-id/services/default",
"permission": "appengine.services.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "apps/my-google-cloud-project-id/services/default",
"serviceData": {
"#type": "type.googleapis.com/google.appengine.v1.AuditData",
"updateService": {
"request": {
"name": "apps/my-google-cloud-project-id/services/default",
"service": {
"networkSettings": {
"ingressTrafficAllowed": "INGRESS_TRAFFIC_ALLOWED_INTERNAL_AND_LB"
}
},
"updateMask": "networkSettings"
}
}
},
"resourceLocation": {
"currentLocations": [
"us-east1"
]
}
},
"insertId": "an-id",
"resource": {
"type": "gae_app",
"labels": {
"project_id": "my-google-cloud-project-id",
"zone": "",
"module_id": "default",
"version_id": ""
}
},
"timestamp": "2021-09-23T15:04:05.131761Z",
"severity": "NOTICE",
"logName": "projects/my-google-cloud-project-id/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "some-operation-uuid",
"producer": "appengine.googleapis.com/admin",
"first": true
},
"receiveTimestamp": "2021-09-23T15:04:05.495890906Z"
}
I don't recall making this change, and I'm not sure what the ingressTrafficAllowed value was before.
Somehow the ingress setting on the GAE service got changed. I believe that issue was fixed by going to GCP console > App Engine > Services > select affected service(s) -> Edit ingress setting from the top, and select the appropriate value.
I say I believe this fixed the issue as I was still getting 403's on my appspot url after doing this, and ultimately I ended up deleting and re-creating the project from scratch, which got everything working again. Clearly there was some misconfiguration somewhere in my project, but GCP does not make it easy to diagnose what the issue might be.
Created Google Smart Home Action.
Implemented device with:
a. deviceType = action.devices.types.SETTOP
b. deviceTrait = action.devices.traits.Channel
Device is successfully discovered and added to Google Home App's Homegraph.
User sends command: "Ok Google, change to ESPN"
Receives the following json in fulfillment URL:
{
"requestId": "[RequestId GUID]",
"inputs": [{
"intent": "action.devices.EXECUTE",
"payload": {
"commands": [{
"devices": [{
"id": "[SettopBox device Id]"
}],
"execution": [{
"command": "action.devices.commands.selectChannel",
"params": {
"channelCode": "espn",
"channelName": "ESPN",
"channelNumber": "206"
}
}]
}]
}
}]
}
Questions:
How does Google Smart Home determine the "channelNumber" value for "ESPN"? The user's command was "Ok Google, change to "ESPN". This does not contain any information about the channel number.
If a provider was set automatically, is there a setting in Google Home or Google Assistant to change this provider?
The number of a channel for the Channel trait is provided in the SYNC request along with any relevant labels.
{
"availableChannels": [
{
"key": "ktvu2",
"names": [
"Fox",
"KTVU"
],
"number": "2"
},
{
"key": "abc1",
"names": [
"ABC",
"ABC East"
],
"number": "4-11"
}
]
}
As shown in the snippet, the channel number comes from the service. This may be up to the developer of the integration how these numbers may be determined, whether from a cable provider or over-the-air. The field is optional, so a service without channel numbers may still work by saying its name.
I implement google smart home actions for plug device,
I can control on/off plug with no problems, but about implement
response offline status for Execute and Query, it seems google assistant never get status, it always said "OK, turn on/off plug", My response JSON log from cloud function log as below:
Execute:
{
"requestId": "847886417406301663",
"payload": {
"commands": {
"ids": [
"T90197200015"
],
"status": "ERROR",
"errorCode": "deviceOffline",
"online": false
}
}
}
Query:
{
"requestId": "11887439270473779795",
"payload": {
"devices": {
"T90197200015": {
"errorCode": "deviceOffline",
"status": "ERROR",
"online": false
}
}
}
}
Is the format correct, why google assistant didn't get offline status?
BR,
Jack
Yes that format is correct. The device generally may be offline, sending the status: "ERROR" and errorCode are more contextually applied to the specific request and response.
In a microservice environment where requests span multiple services including eventgrid i'd like to configure an end-to-end logging with correlationid.
Inspired by this blog https://toonvanhoutte.wordpress.com/2018/08/05/end-to-end-correlation-across-logic-apps/
How can i configure the EventGrid triggers clientTrackingId with my correlationnr from Events data payload?
Checkout my definition below which does not work.
If i substitute "#{coalesce(json(triggerBody().Data)?.CorrelationNr, guid())}" with a string value or even "#parameters('$connections')['azureeventgrid']['connectionId']" it works like a charm.
"triggers": {
"When_a_resource_event_occurs": {
"correlation": {
"clientTrackingId": "#{coalesce(json(triggerBody().Data)?.CorrelationNr, guid())}"
},
"inputs": {
"body": {
"properties": {
"destination": {
"endpointType": "webhook",
"properties": {
"endpointUrl": "#{listCallbackUrl()}"
}
},
"filter": {
"includedEventTypes": [
"webhook.sp.updated"
]
},
"topic": "/subscriptions/xxxx/resourceGroups/xxx/providers/Microsoft.EventGrid/topics/WebHookManager"
}
},
"host": {
"connection": {
"name": "#parameters('$connections')['azureeventgrid']['connectionId']"
}
},
"path": "/subscriptions/#{encodeURIComponent('xxx')}/providers/#{encodeURIComponent('Microsoft.EventGrid.Topics')}/resource/eventSubscriptions",
"queries": {
"x-ms-api-version": "2017-06-15-preview"
}
},
"splitOn": "#triggerBody()",
"type": "ApiConnectionWebhook"
}
}
Logic App does not trigger. No Error message.
Please check the description about clientTrackingId, and your logic app no runs history is because your triggerBody() doesn't have CorrelationNr with the definition you show.
Actually your Event Grid trigger has detected the event, it just couldn't run with the logic. You could go to the EVALUATION and check the trigger history. It's because the value is null, then it won't run.
If you use HTTP request trigger, you could set the x-my-custom-correlation-id header. or set any key-value in the json body, then set the clientTrackingId with like #{coalesce(json(triggerBody())['keyname'], guid())}.
And if you are using some trigger without header, you have to point the value with string or other parameter like you said the connectionid or the parameter value you custom like below.
So the point is the clientTrackingId must be set before it runs and value could be obatined.
Hi, I am creating a chatbot. I developed a IBM cloud function(action) in IBM.
This is the action code..
{
"context": {
"my_creds": {
"user": "ssssssssssssssssss",
"password": "sssssssssssssssssssssss"
}
},
"output": {
"generic": [
{
"values": [
{
"text": ""
}
],
"response_type": "text",
"selection_policy": "sequential"
}
]
},
"actions": [
{
"name": "ssssssssssss/user-detail",
"type": "server",
"parameters": {
"name": "<?input.text?>",
"lastname": "<?input.text?>"
},
"credentials": "$my_creds",
"result_variable": "$my_result"
}
]
}
Now my action user detail is giving response when i am invoking the code.
But when i am checking the output with my chatbot I am getting execution of cloud functions action took too long.
There is currently a 5 second limitation on processing time for a cloud function being called from a dialog node. If your process will need longer than this, you'll need to do it client side through your application layer.