I am new to apache kafka and apache spark. I want to integrate the kafka with my angularjs code. Basically I want to make sure that, when a user click on any link or searches anything on my website, then the those searches and clicks should be triggered as an event and send it to the kafka data pipe for the use of analytics.
My question is how can I integrate frontend code which is in angular.js to apache kafka?
Can I send the searches and click stream data directly to apache spark using kafka pipeline or do I need to send those data to kafka and apache spark will do polling to kafka server and receive the data in batches?
I don't think (just cannot find at glance) there is Kafka client for front-end JavaScript. I cannot actually imagine stable setup when millions of producers (each client's browser) writing to the same Kafka topic.
What you need to do in Angular, is to call your server side function to log your events in Kafka.
Server side code may be written in a bunch of languages, including JavaScript for node.js.
Please take a look for available clients at Kafka Documentation
Update 2019: There are several projects implementing REST over HTTP(s) proxy for producer and consumer interfaces. For example Kafka Rest project (source). Never tried these by myself though.
Related
I'm implementing a Django web service, which is about to have different platform apps,
Reactjs for computers, a swift app for ios, and Kotlin for android devices. the protocol is rest API and perhaps a chat feature included then Django channels are used as well. The data format is JSON. For deployment, I intend to use docker which includes Django, celery, and ReactJS app. And the database is on another separate server which is PostgreSQL. I was thinking to collect some user activity data and some history logs to show the user itself what she/he has done so far. After hours of searching, I came up with Kafka! unfortunately, I have no idea how can I use Kafka and integrate these stuff together and how can I deploy these things. I wish there was a system schema for this specific kind of system that shows what is what and where is what?
Kafka will only integrate your database and Django, with some effort, and ideally a separate Kafka Connect service.
From React (or other clients), you'll need to query some Django API routes which will then query your database. Kafka won't help with your frontend, and isn't really what is exposing the history/activity you're interested in displaying. In other words, you could simply write that to the database, and skip Kafka entirely.
Essentially, you're following the CQRS design pattern if you properly separate Kafka writes from end user / UI reads.
shows what's what and what's where!
Unclear what this means, but data lineage and metadata tools are a whole separate thing. For example, LinkedIn DataHub collects information such as this
I have Apache Camel application deployed on two servers and they consume from JMS endpoint. I want to make sure that only one camel route is consuming from jms endpoint at a time. The only option that I can use for clustering is using database as a lock store. Does Apache Camel provide such a feature?
I think the easiest way is to consume from a topic and not a queue.
On connection, use the same subscriptionName. Only the first connection will be allowed as far i know.
I have large data stored in Postres database and I need to send the data to the client via a REST API using Django. The requirement is to send the data in chunks and to not load the entire content into memory at once. I understand that there is a StreamingHttpResponse class in Django which I will explore. But are there any other better options? I've heard about Kafka and Spark for streaming applications but the tutorials I've checked about these two tend to involve streaming live data like interacting with Twitter data, etc. But is it possible to stream data from database using any of these two? If yes, how do I then integrate it with REST so that clients can interact with it? Any leads would be appreciated. Thanks.
You could use debezium or apache-kafka-connect to bulk load your database into Kafka.
Once the data is there, you can either put a Kafka consumer within your Django application or outside of it and make REST requests as messages are consumed. Spark isn't completely necessary, and shouldn't be used within Django
I am trying to create a web application with Apache Flink as a backend. Here flink need to talk to application layer(generally UI), so that the results from flink can be sent to frontend.
In Short i am looking for flink counterparts of projects like:
Spark Job Server https://github.com/spark-jobserver/spark-jobserver
Mist https://github.com/Hydrospheredata/mist
The (deceptively titled) Monitoring Rest API detailed here seems like the way to go. Even though it says "Monitoring" it actually can upload, start, and stop jobs. In fact, based on the documentation it seems like it should do everything the web UI does.
Can anyone give me a sample source code where i can use restful jax-rs web service as an interface to message broker using active mq. The requirement is traffic comes to application through jax-rs webservice and the message is transferred to active mq which is processed asynchronously and the consumer on active mq inserts data into db. Can anyone please provide sample code, that would be great
The question is very fuzzy. There are multiple concerns to take into account when doing this kind of interfaces as jax-rs is a http interface (synchronous, non transcational, non persistent, non guaranteed delivery) while activemq (jms) is the other way around, asynchronous, transactional and persistent.
I suggest you take a look at Apache Camel which is a lightweight integration framework that works really good with ActiveMQ. It supports JAX-RS as well. There are multiple code examples over at the Camel website and connecting rest with activemq is rather easy given you have your case fully designed.