App Engine allows you to have multiple modules within a single application. I'm trying to understand what the benefits of this are over having multiple App Engine projects.
In my situation, I have three components
A back-end component that does all the processing, stores all data, and is accessible with a REST API
A first front-end (e.g., request handlers) component under a first domain name that probably doesn't need its own datastore
A second front-end component under a second domain name that also probably doesn't need its own datastore.
Whether I use multiple modules or multiple apps, the communications between components are done using HTTP requests.
With modules, all the modules use the same datastore and memcache, but with different projects, they will each have their own memcache and datastore. I don't think this matters for me, because only the backend component needs a datastore.
I'm leaning towards using separate applications instead of separate modules because it seems easier to have complete separation.
Is there any reason I should prefer separate applications over modules or vice versa?
The question is somewhat opinion-based but there are many more reasons to use services (as they are now known) over separate projects.
You cite the main reason in your question: shared back-end services. Although you don't think that matters as they probably don't need Datastore, I would rather assume they may need them in the future than not (and then have to integrate via your other application's HTTP interface instead of direct Datastore RPC).
By using different services in the the same project, you benefit from simpler access to other Cloud Platform services (e.g. BigQuery) through things like service accounts.
You also get things like service discovery through the Modules Service. If you were to deploy as separate projects, App Engine doesn't know your projects from mine.
By using separate projects, you get pretty much the same separation as using services, but forego the benefits above.
Some people might want to use a separate project to benefit from an extra 28 free instance hours, but that wouldn't be a great long term design goal for my liking.
Related
I work on a React app consuming RESTful API. My API on top of basic CRUD operations exposes business functions falling under controller archetype (executable functions, with parameters and possibly return values). E.g.:
http://api.example.com/carts
http://api.example.com/users
http://api.example.com/documents
http://api.example.com/carts/{id}/checkout
http://api.example.com/users/{id}/block
http://api.example.com/documents/{id}/print
Within the team however we have disagreement regarding how API access layer should be called. We struggle to choose between *Repository and *Service.
On one hand these functionalities can be perceived as some collections abstractions and facades and it make sense to call them repositories. On the other hand however they expose some business, domain logic which definitely does not fall under repository pattern and is definitely a service.
Normally (in the back end) a repository could be either a service dependency or can be used standalone. Repository serves the purpose of accessing and modifying a collection, while a service is facade for some more complex business operations.
This separation however does not seem to make a lot of sense in the front end, where the only difference is the API endpoint website will call. Thus it's pointless to separate them, since realistically they are within the boundaries of the same concern.
What are your opinions about it? What kind of nomenclature do you use in your apps? Thanks for help and suggestions ;)
Sorry, if this is a naive question, but i've watched bunch of talks from google's staff and still don't understand why on earth i would use AE instead of CF?
If i understood it correctly, the whole concept of both of these services is to build "microservice architecture".
both CF and AE are stateless
both suppose to execute during limited period of time
both can interact with dbs and other gcp apis.
Though, AE must be wrapped into own server. Basically it utilizes a lot of complexities on top of the same capabilities as CF. So, when should i use it instead of CF?
Cloud Functions (CFs) and Google App Engine (GAE) are different tools for different jobs. Using the right tool for the job is usually a good idea.
Driving a nail using pliers might be possible, but it won't be as convenient as using a hammer. Similarly building a complex app using CFs might be possible, but building it using GAE would definitely be more convenient.
CFs have several disadvantages compared to GAE (in the context of building more complex applications, of course):
they're limited to Node.js, Python, Go, Java, .NET Core, and Ruby. GAE supports several other popular programming languages
they're really designed for lightweight, standalone pieces of functionality, attempting to build complex applications using such components quickly becomes "awkward". Yes, the inter-relationship context for every individual request must be restored on GAE just as well, only GAE benefits from more convenient means of doing that which aren't available on CFs. For example user session management, as discussed in other comments
GAE apps have an app context that survives across individual requests, CFs don't have that. Such context makes access to certain Google services more efficient/performant (or even plain possible) for GAE apps, but not for CFs. For example memcached.
the availability of the app context for GAE apps can support more efficient/performant client libraries for other services which can't operate on CFs. For example accessing the datastore using the ndb client library (only available for standard env GAE python apps) can be more efficient/performant than using the generic datastore client library.
GAE can be more cost effective as it's "wholesale" priced (based on instance-hours, regardless of how many requests a particular instance serves) compared to "retail" pricing of CFs (where each invocation is charged separately)
response times might be typically shorter for GAE apps than CFs since typically the app instance handling the request is already running, thus:
the GAE app context doesn't need to be loaded/restored, it's already available, CFs need to load/restore it
(most of the time) the handling code is already loaded; CFs' code still needs to be loaded. Not too sure about this one; I guess it depends on the underlying implementation.
App Engine is better suited to applications, which have numerous pieces of functionality behaving in various inter-related (or even unrelated) ways, while cloud functions are more specifically single-purpose functions that respond to some event and perform some specific action.
App Engine offers numerous choices of language, and more management options, while cloud functions are limited in those areas.
You could easily replicate Cloud Functions on App Engine, but replicating a large scale App Engine application using a bunch of discrete Could Functions would be complicated. For example, the backend of Spotify is App Engine based.
Another way to put this is that for a significantly large application, starting with a more complex system like App Engine can lead to a codebase which is less complex, or at least, easier to manage or understand.
Ultimately these both run on similar underlying infrastructure at Google, and it's up to you to decide which one works for the task at hand. Furthermore, There is nothing stopping you from mixing elements of both in a single project.
Google Cloud Functions are simple , single purpose functions which are fired while watching event(s).
These function will remove need to build your own application servers to handle light weight APIs.
Main use cases :
Data processing / ETL : Listen and respond to Cloud Storage events, e.g. File created , changed or removed )
Webhooks : Via a simple HTTP trigger, respond to events originating from 3rd party systems like GitHub)
Lightweight APIs : Compose applications from lightweight, loosely coupled bits of logic
Mobile backend: Listen and respond to events from Firebase Analytics, Realtime Database, Authentication, and Storage
IoT: Thousands of devices streaming events and which in-turn calls google cloud functions to transform and store data
App Engine is meant for building highly scalable applications on a fully managed serverless platform. It will help you to focus more on code. Infrastructure and security will be provided by AE
It will support many popular programming languages. You can bring any framework to app engine by supplying docker container.
Use cases:
Modern web application to quickly reach customers with zero config deployment and zero server management.
Scalable mobile backends : Seamless integration with Firebase provides an easy-to-use frontend mobile platform along with the scalable and reliable back end.
Refer to official documentation pages of Cloud functions and App Engine
As both Cloud Functions and App Engine are serverless services, this is what I feel.
For Microservices - We can go either with CF's or App Engine. I prefer CF's though.
For Monolithic Apps - App engine suits well.
Main differentiator as #Cameron points out, is that cloud functions reliably respond to events. E.g. if you want to execute a script on a change in a cloud storage bucket, there is a dedicated trigger for cloud functions. Replicating this logic would be much more cumbersome in GAE. Same for Firestore collection changes.
Additionally, GAE’s B-machines (backend machines for basic or manual scaling) have conveniently longer run times of up to 24 hours. Cloud functions currently only run for 9 minutes top. Further, GAE allows you to encapsulate cron jobs as yamls next to your application code. This makes developing a server less event driven service much more clean.
Of course, the other answers covered these aspects better than mine. But I wanted to point out the main advantages of Cloud Functions being the trigger options. If you want functions or services to communicate with each other, GAE is probably the better choice.
I have two parts of a Project that I think predate Google Cloud Console and now show up in Google Cloud Console separately:
An App Engine Project
A Google APIs and Google Cloud Storage Project
These two "Projects" are part of the same real-life software project.
Should I try to eventually migrate my API and Storage Project into the App Engine Cloud Project? Would there be any benefits?
There really isn't any easy way to do this, and the benefits will probably not outweigh the costs. Unless you're merging two app engine apps into one (that can result in significant cost benefits) it probably doesn't make a difference.
You should definitely try to migrate your API and Storage Project into the App Engine Cloud Project (by enabling the API in the associated Cloud Project, copying your resources and re-creating your credentials).
This will make it easier for you to use Google Cloud Datastore and other Cloud APIs in association with your App Engine application.
I think the question you should ask yourself is if these two components are distinct parts of your infrastructure or if they're effectively the same. It's kind of a subjective and abstract question, but ideally you want your software stack broken into logically cohesive portions.
There's also a more practical consideration related to the size of your organization. If you're working with a small organization with one or two teams, it's likely that you'll want to have a more "monolithic" infrastructure. Larger organizations will likely want an infrastructure based more on "microservices" where individual pieces of the pie are broken down into smaller pieces.
I suppose a good general rule of thumb you could use is that the number of projects you have should be on the order of the number of teams you have working on different components of your software stack. In other words, if you have a handful of teams working on a handful of components, you should have a handful of projects. If you have hundreds of teams working on hundreds of components, you should have hundreds of projects.
We are starting a new project that requires two main components:
Backend for task management, e.g retrieve a task from a queue and according to some specific logic validate it.
Run a real compiler on that specific task and create an executable that an end user should receive.
We love app engine, however the second part will require a concrete instance where an actual compiler will have to be installed, app engine is not capable here. We were thinking to mix both app engine and aws instances to accomplish the task (part 1 will be app engine and part 2 will be aws).
All of our senses say it's a bad idea:
unneeded traffic between the two providers, someone needs to pay for that unfortunately.
We'll have to deal with two systems, two deployments process, each system has its own quirks --> double the work.
But we love app engine.
Does anyone has any experience in combining the two systems? any recommendations ?
There's no reason why what you suggest won't work, especially if you separate your concerns well, by exposing a clean 'compiler' interface on AWS or a similar service. Yes, you will have to pay for traffic between the two services, but this is unlikely to be substantial. If you are serving up the end result to the user, you can link them directly to AWS, rather than fetching it with your app first.
AWS's EC2s are literally just vanilla linux boxes in the sky. I would also throw out the suggestion of just moving to it completely. Porting your system over may be easier than it sounds if you're unix savvy.
In addition to the datastore for your specific site, can you also share one datastore between all your websites? (Like connecting to a different MySQL database from your main MySQL database?)
Not really.
Two workarounds:
Use five "versions" of the same app instead of five different apps. They would share the same data store. The sites they power need not look alike at all (except sharing the domain).
Make the data store web-accessible by enabling the remote_api. It is up to you to configure the security for this, and performance is not likely to be great. Also, at the moment, the client-side remote_api is only available for Python (the server-side works on Java, too, though).
Short answer: no, your application has one and only one datastore, and it is completely segregated from every other application's datastore.
Longer answer: if you had an external datastore of some variety that was web-accessible, you could access it using urlfetch, but there is no way to access more than one AppEngine datastore using the datastore API.
RESTful services between the apps could be expensive and an alternative could be to use one multitenancy app for many client domains or namespaces to partition your data.