We have a secured Kafka cluster in our production environment (using TLS encryption, certificate based client authentication and ACL's). I am trying to figure out how to configure the Flink Kafka connector with the appropriate settings so it can connect securely to our Kafka cluster. Is this possible with Flink? Do i have to pass the security configuration stuff via the properties?
Our Flink cluster is running on Kubernetes (1.14.2) and using the latest Flink stable release (v1.8) with the integrated kafka connector.
After some fiddling with the docs I have got it working by myself. I am now providing the required Java keystores at deployment time when running helm (We deploy the whole Flink stuff through helm charts). The keystores will be base64 encoded and saved as a Kubernetes secret. The taskmanager pods mount the secret at the given location.
I can now pass the location of the keystore / truststore and their passwords on the command-line as params when running the Flink job. These params are finally used to configure the Kafka client via properties.
Related
SQL Server database is in local system
Kafka is configured in confluent cloud
Operating system of the local machine is Windows 10
Is there any way without installing Apache Kafka in the local machine to stream data out from SQL Server database to Kafka topic on confluent cloud.
You don't need to install Apache Kafka locally in order to stream data from your database.
Since you are using Confluent Cloud it is easiest to use one of the two available managed connectors for getting data from SQL Server. Since you're using a local machine you're going to have to make sure you can expose it on a fixed IP and port (the connector is "pull" from Confluent Cloud and not a "push" from your local machine, if that makes sense).
Debezium connector
JDBC connector
Which connector you use depends on factors covered briefly here and in more detail here.
happy new am experimenting with Kafka for the first time, i set up an AWS MSK Kafka cluster successfully, next step in my experiment is the using kafka-connect and in particular i want to use the salesforce-bulk-source connector, according to the documentation its say that this connector is a proprietary connector and it requires a confluent license as part of the configuration as well as a topic where to store here the confluent platform configuration, my question, does this connector work with an MSK cluster or would it require my cluster to be on the confluent platform?
You can use the connector regardless of your Kafka cluster. You will need to run your own Kafka Connect worker, configured to use your Kafka cluster.
As you have noted, it is a proprietary connector and therefore:
You can use it with a 30-day free trial
or under the Developer Licence indefinitely if you have just a single Kafka broker
Here's an example of a Docker Compose to run Kafka Connect docker image.
Edit: You cannot use Confluent connectors that require a commercial license with MSK. You need a valid Confluent subscription.
I have a App Engine Application written in Flask Python 3.7
My usecase is to get information from Composer Metadata DB (dag runs, dag success, dag failures etc) from Composer metadata DB and show as a dashboard inside App Engine Application (few charts).
Homework Done so far -
I was able to run sql queries on Composer metadata after logging in to one of the worker nodes (as worker nodes already have Cloud SQL proxy running which connects to Cloud SQL running in other container). This was done after creating a Compute engine under same VPC as of Composer and then doing ssh from Compute engine to one of the worker nodes.
Now the question is how to connect to Composer metadata DB that is under VPC from App Engine application using Cloud SQL Proxy ?
I would look at Serverless VPC connector, although it designed mainly for App Engine and Cloud Functions, you may consider implement this connector on a Flask app side that gives you opportunity to unify network between App engine and Compute engine nodes parties, thus you would be able to reach Cloud SQL proxy as well.
The setup is fairly much simple, though you just have to attach connector to the specific VPC network and region in the particular GCP project. The IP addresses pool must be in CIDR /28 range, reserved for a connector usage.
We plan to use Appengine for our Node.JS REST-API which will serve content for our customers. The data will come from a MOngoDB Cluster (or Replicaset), which will also be hosted on Google Cloud Services. Some of the contents might be cached using the Memcache of AppEngine.
Now my question. Is it neccessary to setup a kind of VLAN or similar, to connect to the DB server? How can we assure that the DB server is connected with a good bandwidth / latency to the AppEngine instances? Or is this no matter of concern at all?
Another question, does Google Cloud have a kind of internal DNS, which can be used? So for example that we have a domainname like "ourmongodb.internal" which can be used in connection string of our REST API? Or how is the most recent approach to connect to our DB server instance?
If you use app engine flex you can setup your nodejs app engine server and the GCE VMs that run the mogoDB cluster in the same network and than connect them both in the internal IP space.
I am using Hawt.io to monitor my Apache Camel java app, but I have found that if I let it run I cannot use jvisualvm (and similar tools) to profile my app. It seems Hawt.io is using the connector that Camel exposes so jvisualvm cannot connect to the jvm.
What can I do to have Hawt.io and still be able to use jvisualvm?
Thank you!
Can JConsole connect to your JVM too?
Hawtio just connects to Jolokia on the server side; which just exposes JMX over HTTP/JSON; so the hawtio console doesn't itself change JMX in any way.
Incidentally are you using Local connector to connect from the hawtio web app into a separate JVM? Only that approach does add a jolokia agent dynamically to the JVM you are trying to connect to. If thats whats causing the issue - don't use it - and just configure your own Jolokia agent in your JVM?