Currently, I have a streaming job which is firing a batch job when it receives a specific trigger.
I want to follow that fired batch job and when it finishes, want to insert an entry to a database like elastic search or so.
Any ideas, how we can achieve this? How we can listen to that job?
FLINK provides some REST APIs to query job status, you could use this one to query batch job state: https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/rest_api.html#jobs-jobid. While tasks are running, their status will be reported to JM. Through this API, you can get the job state based on the response from the request.
Related
I am new in the field of Azure. I am testing the functionality called "Send, receive, an batch process messages in Azure Logic Apps". This is the link of the documentation:
Batches in Logic Apps
I could do everything what in that tutorial exists and it works. I created a batch called "Test" (This is the Batch name). My question is: Is there a monitor in Azure portal where I can see which messages were created in that batch from the "batch sender" and therefore see the current status of these messages?.
In other words I would like to see which message was already processed by the "batch receiver" and which ones still remain to be processed. I would like to know if I can monitor this batch that I created.
I would like to see which message was already processed by the "batch receiver"
As of now there is no way we can alone monitor the batch process but Here is one of the workarounds for this, You can add a compose connector and add all the parameters as per the requirement.
Here is my logic app workflow
So each time this logic app gets triggered by the batch process we can check all the properties that we have mentioned for that particular message.
Here is the sample output:
You can check the processed message by the "batch receiver" from the run history of the logic app.
and therefore see the current status of these messages?.
You can monitor this from Azure Monitor logs. which gives number runs that are running, succeeded and failed.
and which ones still remain to be processed.
This depends on from where you are trying to send the messages
You can follow this document for Monitor logic apps by using Azure Monitor logs
I'm trying to monitor the availability of my flink jobs using Prometheus alerts.
I have tried with the flink_jobmanager_job_uptime/downtime metrics but they don't seem to fit since they just stop being emmited after the job has failed/finished.
I have already been pointed out to the numRunningJobs metric in order to alert of a missing job. I don't want to use this solution since I would have to update my prometheus config each time i want to deploy a new job.
Has anyone managed to create this alert of a Flink failed job using Prometheus?
Prometheus has an absent() function that will return 1 if the metric don't exist. So, you can just set the alert expression to something like
absent(flink_jobmanager_job_uptime) == 1
Maybe you can help me with my problem
I start spark job on google-dataproc through API. This job writes results on the google data storage.
When it will be finished I want to get a callback to my application.
Do you know any way to get it? I don't want to track job status through API each time.
Thanks in advance!
I'll agree that it would be nice if there was to either wait for or get a callback for when operations such as VM creation, cluster creation, job completion, etc finish. Out of curiosity, are you using one of the api clients (like google-cloud-java), or are you using the REST API directly?
In the mean time, there are a couple of workarounds that come to mind:
1) Google Cloud Storage (GCS) callbacks
GCS can trigger callbacks (either Cloud Functions or PubSub notifications) when you create files. You can create an file at the end of your Spark job, which will then trigger a notification. Or, just add a trigger for when you put an output file on GCS.
If you're modifying the job anyway, you could also just have the Spark job call back directly to your application when it's done.
2) Use the gcloud command line tool (probably not the best choice for web servers)
gcloud already waits for jobs to complete. You can either use gcloud dataproc jobs submit spark ... to submit and wait for a new job to finish, or gcloud dataproc jobs wait <jobid> to wait for an in-progress job to finish.
That being said, if you're purely looking for a callback for choosing whether to run another job, consider using Apache Airflow + Cloud Composer.
In general, the more you tell us about what you're trying to accomplish, we can help you better :)
Just a thought, My first job will be triggered by a Cloud Function by any file arrival event. I will capture it's job ID in Cloud Function itself. Once I get the job ID, I will pass that ID to App engine code and that code will monitor the completion of job with that particular ID. Once the App engine code identifies the 'success' status of that job then it will trigger another job which was dependant on the successful completion status for prior job.
Any idea whether it can be made possible or not? If yes, then any help with code samples will be highly appreciated.
I have configured Google App Engine to record exception with ereporter.
The cron job is configured to run every 59 minutes. The cron.yaml is as follows
cron:
- description: Daily exception report
url: /_ereporter?sender=xxx.xxx#gmail.com # The sender must be an app admin.
schedule: every 59 minutes
How to do I run this immediately.
What I am trying to do here is simulate a 500 HTTP error and see the stack trace delivered immediately via the cron job.
Just go to the URL from your browser.
You can't using cron. Cron is a scheduling system, you could get it to run every minute.
Alternately you could wrap your entire handler in a try/except block and try to catch everything. (You can do this for some DeadlineExceededErrors for instance) then fire off a task which invokes ereporter handler, and then re-raise the Exception.
However in many cases Google infrastructure can be the cause of the Error 500 and you won't be able to catch the error. To be honest you are only likely to be able to cause an email sent for a subset of all possible Error 500's. The most reliable way probably be to have a process continuously monitor the logs, and email from there.
Mind you email isn't consider reliable or fast so a 1 min cron cycle is probably fast enough.
I came across this thread as I was trying to do this as well. A (hacky) solution I found was to add a curl command at the end of my cloudbuild.yaml file that triggers the file immediately per this thread. Hope this helps!
Make a curl request in Cloud Build CI/CD pipeline