Need a help. Does any one of you have an example Prometheus query for following metrics. These are gaugue metrics, I am not sure, which operator to use for these metrics.
flink_taskmanager_job_task_numRecordsInPerSecond
flink_jobmanager_job_fullRestarts/flink_jobmanager_job_fullRestarts
flink_taskmanager_job_task_isBackPressured
flink_jobmanager_job_numberOfFailedCheckpoints
flink_jobmanager_job_lastCheckpointDuration
thanks.
I think it would be good if you look some official Prometheus API examples and Grafana docs if you use it.
Here the query that I use :
sum by(job_name)(flink_jobmanager_job_totalNumberOfCheckpoints{job_name=~"myJobName_.+"})
Related
I am working on sending custom app metrics to prometheus via the Prometheus Flink Metrics Reporter. The metrics are correctly created since I am able to accurately see them in the flink dashboard. I configured the prometheus metrics reporter similar to found here. When I curl to the prometheus endpoint (curl http://localhost:9090/api/v1/metrics), I am only able to see the cluster metrics and not the custom metrics I am creating. I suspect this issue has to do with how I configured the Prometheus Flink Metrics Reporter since when I try to visit http://localhost:9090, there is no UI and just a list of the cluster metrics mentioned above.
flink job code to create metrics(visible in Flink UI):
this.anomalyCounter = getRuntimeContext.getMetricGroup.addGroup("metric1").counter("counter")
flink-conf.yaml:
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: 9090
promethus.yml:
scrape_configs:
- job_name: 'flink'
static_configs:
- targets: ['localhost:9090']
Is there anything I am missing in the configuration? Why are my cluster metrics reaching prometheus and not my custom ones?
Hi #sarvad123 probably (depending on your Flink version) you should add flink-metrics-prometheus-{version}.jar in the /lib folder.
I've seen similar issues based on a bug in the 1.13.6 Flink we were using. The reporter was blowing up and thus you got no custom metrics. This has been fixed in 1.16 version we are using now and we can view both custom and rocksdb metrics. For what it's worth the 1.13.6 version had lots of issues that apparently made the Flink UI pretty useless for data reporting. 1.16 is much more stable and reports things quite well.
Is it possible to use PyFlink with python machine learning libraries such as LightGBM for a streaming application? Is there any good example for this?
There is no complete example but you can take a loot at Getting Started with Flink Python and then take a look at how Python UDFs can be used: UDFs in the Table API.
My application uses hystrix as circuit breaker. I want to export hystrix metrics data to InfluxDB(or other storage service). I didn't find any docuemnts taking about how to read these data.
Thanks!
I found this blog very useful regarding this subject. http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html .
This talks about exporting data to graphite, I am sure it can be extended to InfluxDB as well.
If you want to custom write hystrix metrics data, you can see here
Is there any concept for aggregation in Kafka like the concept of camel-aggregation .
If yes, Could anyone provide a detailed description of what it is and how to use it .
I am not familiar with Apache Camel, but you can use Kafka Streams to aggregate messaged:
I refer to the docs for now: http://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-dsl-aggregating
Also check out the examples: https://github.com/confluentinc/kafka-streams-examples/tree/5.3.1-post/
Hello Friends,
I want to know by any way we can use solr data import handler with cassandra.
If I can get any reference site or example would be good help.
Thanks
I'd have a look at datastax's page on cassandra integration with solr. Also look at this Github repository, its a library for cassandra and solr.
That gitHub library is old , only Datastax Enterprise uses the integration of Cassandra with Solr, but its not free.
You can see Stargate-core soluton for cassandra but it uses lucene
Another one is Stratio Cassandra again it used Lucene
Hope this helps
Regards
Asit