I'm looking for a metric to verify NTP time synchronization of a host with remote NTP servers located within the organization through Prometheus node-exporter.
In the below documentation, I see node_ntp_stratum metric can be used to identify the synchronization against the stratum value.
https://github.com/prometheus/node_exporter/blob/master/docs/TIME.md#node_ntp_stratum
But I don't see the NTP related metrics available in the list of metrics exposed through node_exporter in my server.
curl -X GET http://127.0.0.1:19100/metrics | grep -i 'ntp_'
node_exporter Version - 0.17.0
Prometheus Version - 2.10.0
I was able to get NTP related metrics by enabling --collector.ntp as part of node_exporter service script.
node-exp 26698 1 1 17:00 ? 00:01:21 /usr/local/bin/node_exporter --collector.systemd --collector.ntp --collector.textfile.directory=/var/lib/node_exporter --web.listen-address=0.0.0.0:19100
Related
We're using opsgenie and Nagios for monitoring and incident notifications. I'm looking to set priority of incidents based on the environment they come from (staging/dev P3 or lower, production P2 or higher).
We're using the Nagios Opsgenie plugin but I don't see anywhere that I can configure the priority of alerts by default. Is this an option with Nagios to Opsgenie integrations?
I am using Thruk Version 2.46.3 and pulling the availability report for one of the service group(last Month Report). It shows 100% for one the device which is not correct as the device was down for 8hrs 50min, but if I drill down the device it shows the correct percentage(98.813%) which is the correct result.
I tried to pull the availability report from the backend server for the same but there i am getting the correct result. We are using Nagios® Core™ Version 4.4.6.
Could you please let us know how to overcome this issue?
I am working on sending custom app metrics to prometheus via the Prometheus Flink Metrics Reporter. The metrics are correctly created since I am able to accurately see them in the flink dashboard. I configured the prometheus metrics reporter similar to found here. When I curl to the prometheus endpoint (curl http://localhost:9090/api/v1/metrics), I am only able to see the cluster metrics and not the custom metrics I am creating. I suspect this issue has to do with how I configured the Prometheus Flink Metrics Reporter since when I try to visit http://localhost:9090, there is no UI and just a list of the cluster metrics mentioned above.
flink job code to create metrics(visible in Flink UI):
this.anomalyCounter = getRuntimeContext.getMetricGroup.addGroup("metric1").counter("counter")
flink-conf.yaml:
metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: 9090
promethus.yml:
scrape_configs:
- job_name: 'flink'
static_configs:
- targets: ['localhost:9090']
Is there anything I am missing in the configuration? Why are my cluster metrics reaching prometheus and not my custom ones?
Hi #sarvad123 probably (depending on your Flink version) you should add flink-metrics-prometheus-{version}.jar in the /lib folder.
I've seen similar issues based on a bug in the 1.13.6 Flink we were using. The reporter was blowing up and thus you got no custom metrics. This has been fixed in 1.16 version we are using now and we can view both custom and rocksdb metrics. For what it's worth the 1.13.6 version had lots of issues that apparently made the Flink UI pretty useless for data reporting. 1.16 is much more stable and reports things quite well.
There is a requirement where we get a stream of data from Kafka Stream and our objective is to push this data to SOLR.
We did some reading but we could find there are lot of Kafka Connect solutions available in the market, but the problem is we do not know which is the best solution and how to achieve.
The options are:
Use Solr connector to connect with Kafka.
Use Apache Storm as it directly provides support for integrating with Solr.
There is no much documentation or in depth information provided for the above mentioned options.
Will anyone be kind enough to let me know
How we can use a Solr connector and integrate with Kafka stream without using Confluent?
Solr-Kafka Connector: https://github.com/MSurendra/kafka-connect-solr
Also, With regard to Apache Storm,
will it be possible for Apache Storm to accept the Kafka Stream and push it to Solr, though we would need some sanitization of data before pushing it to Solr?
I am avoiding Storm here, because the question is mostly about Kafka Connect
CAVEAT - This Solr Connector in the question is using Kakfa 0.9.0.1 dependencies, therefore, it is very unlikely to work with the newest Kafka API's.
This connector is untested by me. Follow at your own risk
The following is an excerpt from Confluent's documentation on using community connectors, with some emphasis and adaptations. In other words, written for Kafka Connects not included in Confluent Platform.
1) Clone the GitHub repo for the connector
$ git clone https://github.com/MSurendra/kafka-connect-solr
2) Build the jar with maven
Change into the newly cloned repo, and checkout the version you want. (This Solr connector has no releases like the Confluent ones).
You will typically want to checkout a released version.
$ cd kafka-connect-solr; mvn package
From here, see Installing Plugins
3) Locate the connector’s uber JAR or plugin directory
We copy the resulting Maven output in the target directory into one of the directories on the Kafka Connect worker’s plugin path (the plugin.path property).
For example, if the plugin path includes the /usr/local/share/kafka/plugins directory, we can use one of the following techniques to make the connector available as a plugin.
As mentioned in the Confluent docs, the export CLASSPATH=<some path>/kafka-connect-solr-1.0.jar option would work, though plugin.path will be the way moving forward (Kafka 1.0+)
You should know which option to use based on the result of mvn package
Option 1) A single, uber JAR file
With this Solr Connector, we get a single file named kafka-connect-solr-1.0.jar.
We copy that file into the /usr/local/share/kafka/plugins directory:
$ cp target/kafka-connect-solr-1.0.jar /usr/local/share/kafka/plugins/
Option 2) A directory of dependencies
(This does not apply to the Solr Connector)
If the connector’s JARs are collected into a subdirectory of the build’s target directories, we can copy all of these JARs into a plugin directory within the /usr/local/share/kafka/plugins, for example
$ mkdir -p /usr/local/share/kafka/plugins/kafka-connect-solr
$ cp target/kafka-connect-solr-1.0.0/share/java/kafka-connect-solr/* /usr/local/share/kafka/plugins/kafka-connect-solr/
Note
Be sure to install the plugin on all of the machines where you’re running Kafka Connect distributed worker processes. It is important that every connector you use is available on all workers, since Kafka Connect will distribute the connector tasks to any of the worker
4) Running Kafka Connect
If you have properly set plugin.path or did export CLASSPATH, then you can use connect-standalone or connect-distributed with the appropriate config file for that Connect project.
Regarding,
we would need some sanitization of data before pushing it to Solr
You would need to do that with a separate process like Kafka Streams, Storm, or other process prior to Kafka Connect. Write your transformed output to a secondary topic. Or write your own Kafka Connect Transform process. Kafka Connect has very limited transformations out of the box.
Also worth mentioning - JSON seems to be the only supported Kafka message format for this Solr connector
I have been able to run JMX with Flink with the following configuration applied to the flink-conf.yaml file of all nodes in the cluster:
metrics.reporters: jmx
metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 9020-9022
env.java.opts: -Dcom.sun.management.jmxremote -
Dcom.sun.management.jmxremote.port=9999 -
Dcom.sun.management.jmxremote.authenticate=false -
Dcom.sun.management.jmxremote.ssl=false
When I run JConsole and listen on ports master-IP:9999/slave-IP:9020, I am able to see the system metrics like CPU, memory etc.
How can I access the task metrics and their respective graphs like bytesRead, latency etc. which are collected for each subtask and shown on the GUI.
you can go to mbeans tab on jconsole and there you will see various dropdowns on RHS in the name of job and tasks. Let me know if you have any issues.