I have a google cloud dataflow job, triggering at certain time interval. I need to trigger a mail after dataflow job completed with status of the job to certain mail id.
Thanks,
Damodar
You can do
PipelineResult result = pipeline.run();
PipelineResult.State status = result.waitUntilFinish();
// send email
You can also include the metrics of the finished job in your email returned by result.metrics().
Related
Currently, I have a streaming job which is firing a batch job when it receives a specific trigger.
I want to follow that fired batch job and when it finishes, want to insert an entry to a database like elastic search or so.
Any ideas, how we can achieve this? How we can listen to that job?
FLINK provides some REST APIs to query job status, you could use this one to query batch job state: https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/rest_api.html#jobs-jobid. While tasks are running, their status will be reported to JM. Through this API, you can get the job state based on the response from the request.
I'm trying to monitor the availability of my flink jobs using Prometheus alerts.
I have tried with the flink_jobmanager_job_uptime/downtime metrics but they don't seem to fit since they just stop being emmited after the job has failed/finished.
I have already been pointed out to the numRunningJobs metric in order to alert of a missing job. I don't want to use this solution since I would have to update my prometheus config each time i want to deploy a new job.
Has anyone managed to create this alert of a Flink failed job using Prometheus?
Prometheus has an absent() function that will return 1 if the metric don't exist. So, you can just set the alert expression to something like
absent(flink_jobmanager_job_uptime) == 1
I have an app which stores user data in GCP Datastore. Since this data is very important, I have made a cron job that is scheduled to export the data in the datastore using the instructions given here: https://cloud.google.com/datastore/docs/schedule-export
Now, I want to get notifications when this cron job fails. I have tried the error reporting service from Stackdriver (https://console.cloud.google.com/errors?time=PT1H&filter&order=COUNT_DESC) to do this but it doesn't notify me when the job fails (yes, I intentionally made it fail to test it). The problem is Stackdriver regards cron job failure as mere warnings instead of errors. Click here to see Stackdriver Logs Screenshot.
How to get notifications when the cron job fails?
One way can be to get the Stackdriver Logs using stackdriver logging entries.list API (https://cloud.google.com/logging/docs/reference/v2/rest/) and then use it in a cron job which will notify me when any log has severity: warning or error or critical, but this process is very tedious.
You can try to catch the exceptions and send yourself an email to get notified of a task failure.
Just a thought, My first job will be triggered by a Cloud Function by any file arrival event. I will capture it's job ID in Cloud Function itself. Once I get the job ID, I will pass that ID to App engine code and that code will monitor the completion of job with that particular ID. Once the App engine code identifies the 'success' status of that job then it will trigger another job which was dependant on the successful completion status for prior job.
Any idea whether it can be made possible or not? If yes, then any help with code samples will be highly appreciated.
I'm developing a voting application with GAE, which involves sending email to each voter. In my initial tests, I went over the per-minute email quota, and this exception was raised:
OverQuotaError: The API call mail.Send() required more quota than is available.
I was able to solve this short term by enabling billing, which greatly increases the per minute email quota, but what is the right way to prevent such an exception from being raised in the future? If my app becomes wildly successful and I exceed the larger quota, it would be a big problem to have this exception raised.
I don't want to put the call to send emails in a try, except block, since this is being done after processing a form, and I don't want the user to wait around for the response to the POST.
Is this a good use case for a task queue? If so, would I put a request to send a batch of emails in the task queue or would each request to send an email go in the task queue? The former seems better in that processing the POST would be faster. Regardless of which way I do it, would I add a delay between sending each email to ensure they are not sent to fast and I go over quota?
yes, ideally suited to the task queue as you can limit the rate at which your emails are sent out by changing the properties in the queue.yaml
one email per task would be best, so if the task fails and is retried it will only retry the failed one not all of the batch
yes. use a task queue. if a task is sending a email, you can decide how many tasks should run per minute. and if a task failed it will retry to execute.