In addition to the datastore for your specific site, can you also share one datastore between all your websites? (Like connecting to a different MySQL database from your main MySQL database?)
Not really.
Two workarounds:
Use five "versions" of the same app instead of five different apps. They would share the same data store. The sites they power need not look alike at all (except sharing the domain).
Make the data store web-accessible by enabling the remote_api. It is up to you to configure the security for this, and performance is not likely to be great. Also, at the moment, the client-side remote_api is only available for Python (the server-side works on Java, too, though).
Short answer: no, your application has one and only one datastore, and it is completely segregated from every other application's datastore.
Longer answer: if you had an external datastore of some variety that was web-accessible, you could access it using urlfetch, but there is no way to access more than one AppEngine datastore using the datastore API.
RESTful services between the apps could be expensive and an alternative could be to use one multitenancy app for many client domains or namespaces to partition your data.
Related
GCP has put out several articles about how their various services work behind the scenes.
Is there any information out there illustrating how they keep projects and the data for those projects segregated?
Is my data stored on separate machines from other GCP customers? or is it the same machines with some kind of multi-tenancy implemented (like this article they have where they explain how i could implement multi-tenancy within my own datastore project https://cloud.google.com/datastore/docs/concepts/multitenancy)?
Datastore is a Non-SQL database, built on Megastore, which in turn builds on Bigtable. Datastore is essentially a layer on top of Bigtable that adds query semantics, transactions, and index management (a DBMS).
Perhaps this is interesting for you to know more about the internal of Google Cloud Datastore. Also, here you can find a further explanation on Megastore, from which most of Datastore is part of. The information on those slides can be found in this public paper.
Long story short: no, your data is not stored in a separate machine from other Google Cloud Platform users, as well as your data may reside in different physical machines.
App Engine allows you to have multiple modules within a single application. I'm trying to understand what the benefits of this are over having multiple App Engine projects.
In my situation, I have three components
A back-end component that does all the processing, stores all data, and is accessible with a REST API
A first front-end (e.g., request handlers) component under a first domain name that probably doesn't need its own datastore
A second front-end component under a second domain name that also probably doesn't need its own datastore.
Whether I use multiple modules or multiple apps, the communications between components are done using HTTP requests.
With modules, all the modules use the same datastore and memcache, but with different projects, they will each have their own memcache and datastore. I don't think this matters for me, because only the backend component needs a datastore.
I'm leaning towards using separate applications instead of separate modules because it seems easier to have complete separation.
Is there any reason I should prefer separate applications over modules or vice versa?
The question is somewhat opinion-based but there are many more reasons to use services (as they are now known) over separate projects.
You cite the main reason in your question: shared back-end services. Although you don't think that matters as they probably don't need Datastore, I would rather assume they may need them in the future than not (and then have to integrate via your other application's HTTP interface instead of direct Datastore RPC).
By using different services in the the same project, you benefit from simpler access to other Cloud Platform services (e.g. BigQuery) through things like service accounts.
You also get things like service discovery through the Modules Service. If you were to deploy as separate projects, App Engine doesn't know your projects from mine.
By using separate projects, you get pretty much the same separation as using services, but forego the benefits above.
Some people might want to use a separate project to benefit from an extra 28 free instance hours, but that wouldn't be a great long term design goal for my liking.
I am working on a system on GAE that includes app server, Datastore, a mobile app and a web client.
My each customer will need to have a customized (separate) app server, datastore, mobile app and a web client. I am not able to find any information on how I should design my system:
[Option A]:-Separate project for each customer, hence app server and datastore will be available exclusively for that customer. In the sense, do I need to clone my base project for each of my customers and then customize to have separate datastore and appservers ?
OR
[Option B]:-Same project with different versions of application for each customer. In this case, can I have individual (exclusive) data store and app server for each customer? In the sense, I want a separate datastore and app server for each customers. I am not sure if I can take advantage of using different datastore buckets for different customers for my requirement.
I could not find any reference link addressing my problem.
Any help will greatly be appreciated.
Thanks in advance!
You should consider App Engine's native support of multitenancy. If this is not good enough for your requirements, then you will have to create a separate project for each customer. The Datastore for a single project is shared across all versions, so you will not be able to segment it (unless you are ok using namespaces as described in the linked doc).
I would also recommend making sure that you actually need to have separate applications running for each customer. Unless you sign up for a premier account, you have a limited number of projects that you can create. Besides that, it will most likely make management of those apps much more difficult for you.
I'm trying to use AppEngine as sort of a RESTful web service. The service is supposed to do simple finds and puts from the Datastore so Objectify seems good for covering that part. It also does a few lookups to other services if data is not available in the Datastore'. I'm usingRedstone XMLRPC` for that part.
Now, I have a couple of questions about how to design the serving part in view of AppEngine' quotas (I guess one should think about efficiency in most case but AppEngine's billing make more people think about efficiency).
First lets consider I use simple Servlets. In this case, I see two options. Either I create a number of servlets each providing a different service with Json passed to each of them or I use a single (or a fewer number of) service and determine the action to perform based on a parameter passed with the Json. Will either design have any significance on the number of hours, etc. clocked by AppEngine?
What is the cost difference if I use a RESTful framework such as Restlet or RestEasy as opposed to the barebones approach ?
This question is something of a follow up to : Creating Java Web Service using Google AppEngine
It's not so important, because most costs are going to datastore, so frontend micro-optimisation doesn't matter.
You can save there may be few cents, by choosing 'simple servet', but... is it your goal? It's much more important to make good data structures, prepare all required data in background, make good caching strategy, etc.
I agree with #Igor.
However, there is an additional thing to consider: http sessions.
GAE supports http sessions. Since GAE is a distributed system, sessions are stored in Datastore (and cached in Memcache for efficient reading). Session is updated in every request (to support expiration), so on every request Datastore is accessed.
Sessions are not required for REST and should be turned off.
I am trying to figure out the best way to deploy a single Google App Engine application across multiple regions.
The same code is to be used, but the stored data is specific to each region. Motivating examples are hyperlocal review sites, like yelp.com or urbanspoon, where restaurants and other businesses to review are specific to a region (e.g. boston.app.com, seattle.app.com).
A couple options include:
Create multiple GAE applications,
and duplicate the code across them.
Create a single GAE application, and store all data for all regions
in the same Datastore, with a region
identifier field for each model
delimiting the relevant region.
Some of the trade-offs:
Option 2 seems like it will be increasingly inefficient (space: replicating a region identifier for each record of every model; time: filtering/indexing on the identifier for every query).
Option 1 requires an app ID for every region, while GAE only allows 10 apps per account. Moreover, deploying the code across every region, as well as Datastore migration, seems like it could be a pain to manage.
In the ideal world, I would have a single application instance. From that instance, I could route between subdomains (like here), as well as have a separate Datastore for each subdomain. But I believe GAE only allows a single datastore per application.
Does anyone have ideas on the best way to solve this problem? Or options that I am not considering?
Thanks for your time!
I would recommend your approach #2. Storage space is cheap (and region codes are short), and datastore performance does not degrade with size, unlike most databases. Using a single app also makes for easier management and upgrades, and avoids any issues with the TOS (which prohibit sharding your app to avoid billing charges).
If you use source code revision control, then it is not too bad to push identical code into multiple apps. You could set a policy whereby only full-fledged tags are allowed to be pushed up to GAE. Another option is to make your application version the same as the revision number.
With App Engine, I (and I believe most others) always migrate data from within my model code. You can't easily do bulk migrations in GAE and the usual solution is to migrate data as you come across it in code. In this way, you can keep your models pretty much identical across applications.
Having said that, I would probably still go with a unified application. It's more future-proof. What if users want to join their L.A. identity and their New York identity? Or what if an advertiser offers you a sweet deal for you to run some marketing reports on your own data?
Finally, a few bytes of data doesn't matter so much on App Engine. As your site grows, you will very quickly discover that you will always be bumping into ceilings. GAE limits are extremely small compared to a traditional web server and so you will have to work within those limits anyway. For example, you can only fetch 1,000 records at a time. So your architecture will already support a piecemeal paging solution. So don't worry too much about an extra field or two in your record.