I'm running a backend app on App Engine (still on the free plan), and it supports client mobile apps in a Firebase Realtime Database setup. When a client makes a change to the database, I need my backend to review that change, and potentially calculate some output.
I could have my App Engine instance sit awake and listen on Firebase ports all the time, waiting for change anywhere in the database, but That would keep my instance awake 24/7 and won't support load balancing.
Before I switched to Firebase, my clients would manually wake up the backend by sending a REST request of the change they want to perform. Now, that Firebase allows the clients to make changes directly, I was hoping they won't need to issue a manual request. I could continue to produce a request from the client, but that solution won't be robust, as it would fail to inform the server if for some reason the request didn't come through, and the user switched off the client before it succeeded to send the request. Firebase has its own mechanism to retain changes, but my request would need a similar mechanism. I'm hoping there's an easier solution than that.
Is there a way to have Firebase produce a request automatically and wake up my App Engine when the db is changed?
look at the new (beta) firebase cloud functions. with that, you can have node.js code run, pre-process and call your appengine on database events.
https://firebase.google.com/docs/functions/
Firebase currently does not have support for webhooks.
Have a look to https://github.com/holic/firebase-webhooks
From Listening to real-time events from a web browser:
Posting events back to App Engine
App Engine does not currently support bidirectional streaming HTTP
connections. If a client needs to update the server, it must send an
explicit HTTP request.
The alternative doesn't quite help you as it would not fit in the free quota. But here it is anyways. From Configuring the App Engine backend to use manual scaling:
To use Firebase with App Engine standard environment, you must use
manual scaling. This is because Firebase uses background threads to
listen for changes and App Engine standard environment allows
long-lived background threads only on manually scaled backend
instances.
Related
I'm working with a Django web app deployed on Google App Engine flexible environment.
I'm streaming my data while processing requests in my views using bigquery.Client(). But I think it is not the best way to do it. Do I need to delegate this process outside of the view (using pub/sub, tasks, cloud functions etc.? If so, give me a suitable architecture: which GCP product should I use, how to connect, and what to read.
Based on your comment, I could recommend you Cloud Run;
Cloud Run is a serverless container based product. You write a webserver (that handle your POST request), wrap it in a container and deploy it on Cloud Run.
With a brand new feature, named always on the CPU is not throttled after the response sent (the normal behavior). With always on, you keep the full CPU up to the Cloud Run instances off load (usually after 15 minutes, but can be quicker).
The benefit of the feature is the capacity to return immediately the response to the client, and then to continue to process, asynchronously, your data to store in BigQuery (in streaming mode).
I have an API made with NodeJS (NodeJS v10 + Express v4.16 + Node-Fetch v2.3) and into this API, I have one endpoint that need to consume content from a third-party API/Service via HTTP Request (POST)
The problem is: This third-party API only accepts requests coming from Brazil
In the past, my API was hosted on Digital Ocean, but with this rule I have migrated to GCP (since DO doesn't have hosts in Brazil) and created my App Engine Application under region southamerica-east1 (Sao Paulo/Brazil according with this document)
And yeah... It works on my machine ¯|_(ツ)_/¯
What's happening: Sometimes the requests runs Ok, working fine, but after some version updates (I'm using CI/CD to make de deployment) the requests goes down.
The Question: Exist a way to control my application to only use the hosted region to make the outgoing requests??
PS* I'm not using flex env, purposely to prevent auto-scale (and cost elevation). (I don't know if I'm right about it because I'm new on GCP)
The IPs of Google Cloud Platform share the same geolocation (US) so I would say that it's expected for the requests to fail. You can have a look at this and this questions for more info and potential workarounds.
We are developing an app that uses the Gmail API to synchronize the e-mails of our users. We are relying on watch to get change notifications through a PubSub, as recommended in the documentation.
Everything is okay, and we are receiving the notifications correctly.
However, as many software companies, we do have a staging environment to test our new features. We have a staging Google OAuth2 client with different Client ID / Client Secret to authenticate to Google, and a staging PubSub topic/subscription to receive notifications.
If I connect my Gmail account on the staging environment, everything works fine. I receive the notifications in staging. If I connect after that the same Gmail account on the production environment, I receive the notifications in production ; but the staging notifications stops coming. The same happens the other way round.
I thought that by using a different client and a different PubSub, we could get the notifications in both environments. It doesn't seem the case. Maybe Google limits the subscription per Google Cloud project?
Does anyone have already met this limitation or have more information about this?
Best regards,
François Voron
I am trying to use Google App Engine as a mediator between the mobile platform and a popular cloud storage service. The mobile app tells app engine what parts of a particular file it wants from the cloud storage, app engine should then fetch that file data, processes it and extracts the requested parts to send back to the mobile app. Yes it has to be set up this way, the mobile os is unable to read files of this particular format, but app engine can, and this particular cloud storage is integrated with a required desktop software.
The issue: processing the file and extracting the data exceeds the 60 second response limit and the Task Queue cannot return data back to the originally requesting mobile app. in most cases, the data would be ready to return in 1-3 minutes. I realize that the Channel Api could allow me to receive real-time messages via a web view as to when the data is ready, but this api is very expensive since I would need to allow for thousands of connections a day and each user has to have their own channel per the docs. Should I look in to polling (outside the channel api)? What design models, methods or even other services should I look in to (I have been using gae because of its ease of use, automatic scaling and security; I'm a one man show).
The product relies on a capability that only exists in Java to process the data. Thanks.
You could return a transaction id to the client, and then let the client periodically ping your server with that id to see if the long process is complete.
Appengine 'Backend' instances do not have the 60 seconds limit. You can see the comparison between normal frontend instance and backend instance here: https://developers.google.com/appengine/docs/java/backends/
The Twitter streaming api says that we should open a HTTP request and parse updates as they come in. I was under the impression that Google's urlfetch cannot keep the http request open past 10 seconds.
I considered having a cron job that polled my Twitter account every few seconds, but I think Google AppEngine only allows cron jobs once a minute. However, my application needs near-realtime access to my twitter #replies (preferably only a 10 second or less lag).
Are there any method for receiving real-time updates from Twitter?
Thanks!
Unfortunately, you can't use the urlfetch API for 'hanging gets'. All the data will be returned when the request terminates, so even if you could hold it open arbitrarily long, it wouldn't do you much good.
Have you considered using Gnip? They provide a push-based 'web hooks' notification system for many public feeds, including Twitter's public timeline.
I'm curious.
Wouldn't you want this to be polling twitter on the client side? Are you polling your public feed? If so, I would decentralize the work to the clients rather than the server...
It may be possible to use Google Compute Engine https://developers.google.com/compute/ to maintain unrestricted hanging GET connections, then call a webhook in your AppEngine app to deliver the data from your compute engine VM to where it needs to be in AppEngine.