A service A will run in EKS with replicated instances across several pods. This service does many things, among this it's meant to subscribe to a salesforce Streaming API that implements pub/sub for publishing messages.
The salesforce streaming API follows the Bayeaux protocol. The Bayeaux protocol is implemented with CometD which is just a long-polling over http strategy.
Long story short, this pub/sub pattern is based on a long-lived request from the subscriber to the publisher server. Publisher only responds when there's an update and a new long-lived request is sent right away.
I'm worried about they way AWS EKS would deal with this type of subscription, specially considering there will be multiple replica instances of service A running at all times. I need to guarantee each message is processed only once.
Maybe the inherent load-balancing in EKS will handle this? After googling around and reading for half a day I haven't found a concrete answer, and I don't yet have access to the EKS resources to set up a test.
My question:
Are my worries justified for this EKS to external long-polling pub/sub justified? Do I need to add additional AWS elements into the picture?
Related
I'm fairly new to things that aren't strictly front end, so after reading the Google pub/sub docs and doing a few searches its not clear to me whether using it with react is possible.
My use case is I (hypothetically) have tens of thousands of people on my webpage at a time that all need to be told at the same time that some external event occurred (the message would be very small).
I know Google Firestore has a listener feature but based on this specification it would not be within the free tier usage anymore. I've seen libraries that allow Google Pub/Sub to be used with IOT devices so I'm confused on why I can't find any resources on using it in the browser.
Creating a Cloud Pub/Sub subscriber in the frontend would be an anti-pattern for several reasons. First of all, the quota limits only allow 10,000 subscriptions per topic. Since you say you have tens of thousands of people on the web page at a time, you would not be able to create enough subscriptions for this case. Additionally, subscriptions created when users come to the website would not be able to get any notifications from before the time the subscription was created; Cloud Pub/Sub only guarantees delivery of messages published after the subscription was successfully created. Finally, you'd have the issue of security and authentication. In order to start a subscriber from the client, you'd need to pass it credentials that it could use. If you use separate credentials for each webpage viewer, then you'd have to create these credentials on the fly and revoke them when the user disappears. If you use the same credentials across all of the subscribers, then one subscriber could intercept the feed of another subscriber.
Overall, Cloud Pub/Sub is designed for the torrents use case: fewer feeds with a lot of data that has to be processed by fewer subscribers. What you are talking about is the trickles use case: a small number of messages that need to be distributed among a large number of subscribers with individual ACLs. Firebase Cloud Messaging is the product designed for this latter case.
While it is true that Cloud Pub/Sub is on the path for Google Cloud IoT, it is used on the publish side: many devices send their events to a topic that can be processed by subscribers. Note that these messages from devices don't come directly into Cloud Pub/Sub; they go through a Cloud IoT server and that server is what publishes the messages to Cloud Pub/Sub. Device authentication is done via Cloud IoT and not via permissions on Cloud Pub/Sub topics. The delivery of messages to IoT devices is not done with Cloud Pub/Sub.
My project has microserver architecture working in Google Cloud. I'm thinking about moving from container with RabbitMQ to PubSub engine.
The question is: is it possible to receive messages one by one? My code is written on Go and docs says
The callback is invoked concurrently by multiple goroutines,
maximizing throughput.
But how many goroutines can be invoked? How can I set the max allowed? E.g. one of my workers works with third-party API allowing only one connection per IP, so I can have only one task in time for this worker.
Correct solution is App Engine Task Pull Queues
https://cloud.google.com/appengine/docs/standard/go/taskqueue/overview-pull
Continuous replication of a Cloudant database can result in quite a high number of HTTP requests consuming your HTTP usage. See here for more info.
Question: What are the main alternative options available for replicating my database?
One option is to setup is to write a small cron type service (e.g. in AWS or Bluemix) that uses the Cloudant HTTP Replication API to create one off replications at set intervals (e.g. hourly, or whatever period the business deem acceptable).
I'm looking to move over a messaging system that we have over to the google app engine environment but I have a few questions that I'm hoping someone can help me with.
Our current message environment uses rabbit mq to process messages and then uses about 10 consumers that connect to the que to send the messages. This works well for us as having 10 consumer instances to process the messages dramatically increases delivery rates.
I understand that the app engine doesn't support rabbit mq so I was wondering what would be the best alternative to achieve the same result. I see that you can run tasks in the background which is great but this would only act as one instance, which will slow down the delivery rates.
Are there any other options?
I never use rabbitmq before, but your requirement looks like quite fit the usage of taskqueue and pipeline on app engine.
TaskQueue provide the ability to setup consumers and setup their process rate.
https://developers.google.com/appengine/docs/python/taskqueue/
With the Task Queue API, applications can perform work outside of a user request, initiated by a user request. If an app needs to execute some background work, it can use the Task Queue API to organize that work into small, discrete units, called tasks. The app adds tasks to task queues to be executed later.
The piepline is based on taskqueue and provide more feature on control the flow.
https://code.google.com/p/appengine-pipeline/
The Google App Engine Pipeline API connects together complex, time-consuming workflows (including human tasks). The goals are flexibility, workflow reuse, and testability. A primary use-case of the API is connecting together various App Engine MapReduces into a computational pipeline.
Is it possible to route HTTP traffic between google app engine applications without going through the public internet?
For example, if I'm running a Web Service API on one application and want to build a second application on top of it without traffic going through the internet - for performance reasons.
Between separate apps running on different domains? I suspect not.
But you can use backends to do different work behind the scenes:
Backends are special App Engine instances that have no request deadlines, higher memory and CPU limits, and persistent state across requests. They are started automatically by App Engine and can run continously for long periods. Each backend instance has a unique URL to use for requests, and you can load-balance requests across multiple instances.
When I look at the logs between the backend and the front end instances I see IPs like
0.1.0.3
So yes, those communication paths are internal. I'd hazard a guess that as so much of the internet is google you could say requests between different apps might not travel on the public internet.
Logs indicate low latency communication between front and back ends, not under any particular load however. Your milage may vary.
Backends in Python