Dynamic toggling Flink job name in Datadog metric - apache-flink

Question

I have several Apache Flink jobs that are monitored in Datadog.
Problem I'm trying to solve is that I would like to have a single dashboard, and toggle the Flink job names (and environment) so I don't have to build multiple dashboard with the exact same context, but just a different job identifier/name. For instance, this is the JSON for a sample Memory Usage widget that I have:
{
"viz": "timeseries",
"requests": [
{
"style": {
"palette": "dog_classic",
"type": "solid",
"width": "normal"
},
"type": "line",
"formulas": [
{
"alias": "Heap Used",
"formula": "query1"
},
{
"alias": "Heap Max",
"formula": "query2"
},
{
"alias": "Non-Heap Used",
"formula": "query3"
},
{
"alias": "Non-Heap Max",
"formula": "query4"
},
{
"alias": "heap % of max",
"formula": "query1 / query2 * 100"
},
{
"alias": "non-heap % of max",
"formula": "query3 / query4 * 100"
}
],
"queries": [
{
"query": "avg:the_flink_job_name.jobmanager.Status.JVM.Memory.Heap.Used.value{env:$environment.value}",
"data_source": "metrics",
"name": "query1"
},
{
"query": "avg:the_flink_job_name.jobmanager.Status.JVM.Memory.Heap.Max.value{env:$environment.value}",
"data_source": "metrics",
"name": "query2"
},
{
"query": "avg:the_flink_job_name.jobmanager.Status.JVM.Memory.NonHeap.Used.value{env:$environment.value}",
"data_source": "metrics",
"name": "query3"
},
{
"query": "avg:the_flink_job_name.jobmanager.Status.JVM.Memory.NonHeap.Max.value{env:$environment.value}",
"data_source": "metrics",
"name": "query4"
}
],
"response_format": "timeseries"
}
]
}
What I'm trying to achieve is to be able to provide the_flink_job_name as a variable, in the same way env is provided.
Is there a way to accomplish this?

Answer 1

We solved this by setting a global tag per job in the Datadog reporter with the job name: https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/metric_reporters/#datadog
metrics.reporter.dghttp.tags: flink_job:<job_name>
A bit manual, but should allow you to achieve what you are looking for. We had to do it like this since it seemed there was no other way to create reusable dashboards using template variables.