I am trying to create a prototype using Firebase on AppEngine, and I keep hitting the 9-hour quota as described here: https://cloud.google.com/solutions/mobile/firebase-app-engine-android-studio#configuring_the_app_engine_backend_to_use_manual_scaling
The doc states the following:
To use Firebase with App Engine standard environment, you must use manual scaling. This is because Firebase uses background threads to listen for changes and App Engine standard environment allows long-lived background threads only on manually scaled backend instances.
I am not using the Firebase realtime database features, so does it still need to spawn threads to listen for changes? Is there a way to configure Firebase to not need these background threads, and therefore, not need manual scaling?
To quote the doc you linked:
To use Firebase with App Engine standard environment, you must use manual scaling.
So to be able to interact with the Firebase Database from app engine, your instance must be set to manual scaling.
If you don't interact with the database, you might be able to use auto-scaling. But that depends on your use-case.
Related
Previously the Firebase Admin SDK would not run on App Engine with automatic scaling. Probably it will create some threads and the Thread class or something like that is not whitelisted.
Now that Google has given us support for Java 8 on App Engine (and removed the whitelist), would this problem be gone? Will the Firebase Admin SDK work on App Engine with automatic scaling?
Firebase Admin SDK still won't work in the AppEngine Java 8 runtime without manual scaling, for several reasons:
The threads support available in the Java 8 runtime comes with some limitations. For instance, any thread not created using the AppEngine's ThreadManager interface cannot call AppEngine APIs. This means these threads cannot make outgoing network calls.
The database client code in the SDK spawns long-running threads. The only way to do that in the AppEngine environment is via background thread support. The SDK specifically looks for this feature when deployed in AppEngine. I've tried to run the SDK with this requirement relaxed, but it doesn't work. It seems request-scoped threads cannot outlive the request even in the new Java 8 runtime.
However, you should be able to use the FirebaseAuth interface (used for custom token minting, ID token verification and user management) in the AppEngine, without manual scaling. This part of the SDK supports running on request-scoped threads.
I am currently experimenting with the Google App Engine flexible environment, especially the feature allowing you to build custom runtimes by providing a Dockerfile.
Docker provides a really nice feature called docker-compose for defining and running multi-container Docker applications.
Now the question is, is there any way one can use the power of docker-compose within GAE? If the answer is no, what would be the best approach for deploying a multi-container application (for instance Nginx + PHP-FPM + RabbitMQ + Elasticsearch + Redis + MongoDB, ...) within GAE flexible environment using Docker?
It is not possible at this time to use docker-compose to have multiple application containers within a single App Engine instance. This does seem however to be by design.
Scaling application components independently
If you would like to have multiple application containers, you would need to deploy them as separate App Engine services. There would still only be a single application container per service instance but there could be multiple instances of each service. This would grant you the flexibility you seek of scaling each application component independently. In addition, if the application in a container were to hang, it could not affect other services as they would reside in different VMs.
An added benefit of deploying each component as a separate service is that one need not use the flexible environment for every service. For some very small tasks such as API backends or serving relatively slow-changing web content, the standard environment may suffice and may be less expensive at low resource levels.
Communication between components
Since one of your comments mentions getting instance IPs, I thought you might find inter-service communication useful. I'm not certain for what reason you wish to use VM instance IPs but a typical use case might be to communicate between instances or services. To do this without instance IPs, your best bet is to issue HTTP request from one service to another simply using the appropriate url. If you have a service called web and one called api, the web service can issue a request to api.mycustomdomain.com where your application is hosted and the api service will receive a request with the X-Appengine-Inbound-Appid header specified with your project ID. This can serve as a way a identifying the request as coming from your own application.
Multicontainer application using Docker
You mention many examples of applications including NGinx, PHP-FPM, RabbitMQ, etc.. With App Engine using custom runtimes, you can deploy any container to handle traffic as long as it responds to requests from port 8080. Keep in mind that the primary purpose of the application is to serve responses. The instances should be designed to start up and shut down quickly to be horizontally scalable. They should not be used to store any application data. That should remain outside of App Engine using tools like Cloud SQL, Cloud Datastore, BigQuery or your own Redis instance running on Compute Engine.
I hope this clarifies a few things and answers your questions.
You can follow following steps to create a container with docker-compose file in Google App Engine.
Follow link
You can build your custom image using docker-compose file
docker-compose build
Create a tag for local build
docker tag [SOURCE_IMAGE] [HOSTNAME]/[PROJECT-ID]/[IMAGE]
Push image to google registry
docker push [HOSTNAME]/[PROJECT-ID]/[IMAGE]
deploy Container
gcloud app deploy --image-url=[HOSTNAME]/[PROJECT-ID]/[IMAGE]
please add auth for docker commands to run.
i wanted to create a multiple instances of my application in either google cloud or EC2. I have two queries regarding this
1.How to achieve this?
Can we create a virtual instances by using zookeeper?
Google App Engine instances are started automatically, as your traffic raises. You may also have always on instances or backends instaces. Just read the docs: http://code.google.com/intl/pt-BR/appengine/docs/adminconsole/instances.html
Google App Engine is not adequate to use with Zookeper. Since Java code runs in a limited sandbox, you may not be able to communicate with Zookeper at all. Also, you will have to start and end you backends programmatically, leading you to lots of work.
As for EC2, see this:
http://www.mail-archive.com/zookeeper-user#hadoop.apache.org/msg01083.html
I am interested to build a platform on Google App Engine where one Master App provides common functionality to several child Apps (Web portals).
Master App:
Offers Common Social Networking features
Common user database
Interacts using Web services API
Child Apps (Web Portals):
Connects to Master App for user database and social networking features
Uses Local business logic for local features
My Design plan is:
Master and Child Apps will be
individual Apps on GAE Appspot (for
easy management)
All will communicate via REST/JSON?
I will enable billing for all Apps so
nothing is against Google's usage
policy.
Will use SSL for login into Master App, later use a token
(cookie) for further interactions
Could anyone kindly spot risks and suggest any improvements?
Your "Master/Child" concept will be hard to maintain, introduce issues with cross-site-scripting and most importantly be completely against Google App Engine's terms of service for combining multiple apps into one.
4.4. You may not develop multiple Applications to simulate or act as a single Application or otherwise access the Service in a manner intended to avoid incurring fees.
You could use multiple app versions (defined on app.yaml) to deploy different codebases to the same appspot.com subdomain, and access them like http://app1.myapp.appspot.com, http://app2.myapp.appspot.com for "versions" app1 and app2. You could even have one version running on Java and another one on Python.
They'll all share the same datastore, although you'd be able to use namespaces to separate them if necessary.
For the distributed nature of the architecture you proposed, I'd suggest having a deep look at Facebook's graph API and Twitter's API. They both use oAuth for authentication/authorization and already have a proven implementation of what you're trying to accomplish.
Is it possible to query App Engine's Datastore from outside the cloud, i.e. a client application?
I could possibly write an app to be housed within AppStore and query the Datastore returning XML-formatted data; I want to know, however, if there are any Datastore endpoints which would allow me to do it directly.
Also, in case it is possible, am I able to do so via SSL?
Yes. The remote_api library supports exactly this use-case. If you're using Java, there's a Java remote_api handler available, and the client will be available at some point in the future.
You can use this over SSL in the same way as any other handler.
There's no reason you couldn't create your own app engine application that exposes the datastore as a web service (either http or https). In fact, here is a link to a python version.