I created an App Engine Service to transcode video files as well as images. Video files can be large and thus will take longer to process. Cloud Tasks seemed like a great fit but my front-end clients needs to monitor tasks during execution.
For example, if a user uploads a video file, I would like to keep the client informed on the progress of their upload. I can't really see anywhere in the docs that show how to request this information from an actively executing task (or an API I can send these updates to?). My current implementation uses web sockets to relay this information, however, this doesn't seem scalable if lots of clients start uploading videos. My thought is to store task state in a NoSQL db and return the DB task ID to the client when a video file is uploaded. Then, I would just poll the DB for updates.
Is this correct or is there a better approach?
The approach come from different perspectives, but my suggestion its to move the less possible pieces of the board.
If you have developed a nodeJS application to perform your video operations, complete them with a notification system, or tool like Service Worker. https://www.youtube.com/watch?v=HlYFW2zaYQM
Meanwhile, you can use PubSub and make a the complete ring about notifications/events:
"Ingest events at any scale"
Data ingestion is the foundation for analytics and machine learning, whether you are building stream, batch, or unified pipelines. Cloud Pub/Sub provides a simple and reliable staging location for your event data on its journey towards processing, storage, and analysis.
Related
I'm working with a Django web app deployed on Google App Engine flexible environment.
I'm streaming my data while processing requests in my views using bigquery.Client(). But I think it is not the best way to do it. Do I need to delegate this process outside of the view (using pub/sub, tasks, cloud functions etc.? If so, give me a suitable architecture: which GCP product should I use, how to connect, and what to read.
Based on your comment, I could recommend you Cloud Run;
Cloud Run is a serverless container based product. You write a webserver (that handle your POST request), wrap it in a container and deploy it on Cloud Run.
With a brand new feature, named always on the CPU is not throttled after the response sent (the normal behavior). With always on, you keep the full CPU up to the Cloud Run instances off load (usually after 15 minutes, but can be quicker).
The benefit of the feature is the capacity to return immediately the response to the client, and then to continue to process, asynchronously, your data to store in BigQuery (in streaming mode).
I'm building an app on google cloud that takes users' audio recording. Users would record 1 or more audio clip and upload them to the the backend, which would process the clips, run Machine Learning prediction model I built and return an integer back to the user for each of the audio that are uploaded. Processing and predicting 1 piece of audio takes about 10 seconds. Users can upload 20 audio at a time.
What I have so far is:
HTML, Javascript, css on the client side. The upload functionality is async using fetch and return a promise
The backend is running Google AppEngine (python3.7), Firebase Authentication, Google CLoud storage and cloud logging
The processing and the prediction is running on Google Cloud Function.
My question is as follow:
Since, it might take up to 200-300 seconds for the processing to complete, how should I be handling the task once users hit the upload button? Is simple request-response enough?
I have investigated the following:
Google Cloud tasks. This seems inappropriate, because client actually needs to know when the processing is done. There is really no call back when the task is done
Google Cloud PubSub. There is a call back for when the job is done (subscribe), but it's server side. This seems more appropriate for server to server communication, instead of client-server.
What is the appropriate piece of tech to use in this case?
There is to way to improve the user experience.
Firstly, on the processing, you can perform parallel processing. All the prediction should be handled by the same Cloud Functions. In App Engine, you should have a multi-thread processing which invoke your CLoud Functions for only one audio clip, and do that 20 time in parallel. I don't know how to achieve that with async in Python, but I know that you can
Then, if you implement that, you will wait all the end of all the audio clip processing to send a response to your users. the total should be between 15 - 20 seconds.
If you use Cloud Run, you can use streaming (or partial HTTP response). Therefore, you could send a partial response when you get a response from your CLoud Functions, whoever the audio clip (the 3rd can be finished before the 1st one).
As you note, Cloud Pub/Sub is more appropriate for server-to-server communication since there is currently a limit of 10,000 subscriptions per-topic/per-project (this may change in the future).
Firebase Cloud Messaging may be a more appropriate solution for client-server messaging.
I want to send live updates to a phone to my app, approximately 1KB/sec. It is acceptable, if for reasons of power management, the updates will come about once per minute in batches, since Google advises that what drains the battery is leaving the radio on, and that a radio drops power after 15sec. What is the appropriate tool to use?
Websockets/stomp. That's what I am already using for the browser client. However my worry is that it will keep the radio on, and devastate the battery.
Google Cloud Messaging. Google promises that it's battery efficient, however how live is it? Who controls the frequency of pushes? Is it designed for a lot of data? Does it have to put notifications up for every push which would be highly undesirable to me since it's a push every second.
Amazon SNS. I already use Amazon AWS, so it may fit the profile well. How good is it with power management?
Something else that I don't know of
There is now https://firebase.google.com/docs/cloud-messaging/
Firebase Cloud Messaging (FCM) is a cross-platform messaging solution
that lets you reliably deliver messages and notifications at no cost.
and
For use cases such as instant messaging, a message can transfer a
payload of up to 4KB to a client app.
FCM is the new version of GCM under the Firebase brand. (read more about FCM v GCM and limits and pricing)
As you write
the updates will come about once per minute in batches
the server could send a notification if a new batch is ready, instead of every second.
Instead of transferring the payload with the notification, the app could use a HTTP(S) connection to get the payload. This would allow to download all updates in one batch with one HTTP request, which might be more efficient (less overhead).
I'm looking to move over a messaging system that we have over to the google app engine environment but I have a few questions that I'm hoping someone can help me with.
Our current message environment uses rabbit mq to process messages and then uses about 10 consumers that connect to the que to send the messages. This works well for us as having 10 consumer instances to process the messages dramatically increases delivery rates.
I understand that the app engine doesn't support rabbit mq so I was wondering what would be the best alternative to achieve the same result. I see that you can run tasks in the background which is great but this would only act as one instance, which will slow down the delivery rates.
Are there any other options?
I never use rabbitmq before, but your requirement looks like quite fit the usage of taskqueue and pipeline on app engine.
TaskQueue provide the ability to setup consumers and setup their process rate.
https://developers.google.com/appengine/docs/python/taskqueue/
With the Task Queue API, applications can perform work outside of a user request, initiated by a user request. If an app needs to execute some background work, it can use the Task Queue API to organize that work into small, discrete units, called tasks. The app adds tasks to task queues to be executed later.
The piepline is based on taskqueue and provide more feature on control the flow.
https://code.google.com/p/appengine-pipeline/
The Google App Engine Pipeline API connects together complex, time-consuming workflows (including human tasks). The goals are flexibility, workflow reuse, and testability. A primary use-case of the API is connecting together various App Engine MapReduces into a computational pipeline.
I am trying to use Google App Engine as a mediator between the mobile platform and a popular cloud storage service. The mobile app tells app engine what parts of a particular file it wants from the cloud storage, app engine should then fetch that file data, processes it and extracts the requested parts to send back to the mobile app. Yes it has to be set up this way, the mobile os is unable to read files of this particular format, but app engine can, and this particular cloud storage is integrated with a required desktop software.
The issue: processing the file and extracting the data exceeds the 60 second response limit and the Task Queue cannot return data back to the originally requesting mobile app. in most cases, the data would be ready to return in 1-3 minutes. I realize that the Channel Api could allow me to receive real-time messages via a web view as to when the data is ready, but this api is very expensive since I would need to allow for thousands of connections a day and each user has to have their own channel per the docs. Should I look in to polling (outside the channel api)? What design models, methods or even other services should I look in to (I have been using gae because of its ease of use, automatic scaling and security; I'm a one man show).
The product relies on a capability that only exists in Java to process the data. Thanks.
You could return a transaction id to the client, and then let the client periodically ping your server with that id to see if the long process is complete.
Appengine 'Backend' instances do not have the 60 seconds limit. You can see the comparison between normal frontend instance and backend instance here: https://developers.google.com/appengine/docs/java/backends/