It's been noted that Google App Engine is moving its datastore implementation from BigTable to MegaStore. What's the difference between the two?
As this article explains, "Megastore is a transactional indexed record manager built by Google on top of BigTable".
What Megastore adds on top of BigTable, again according to the URL I gave (of course I cannot discuss anything that Google hasn't yet made public!), is stuff that might not be easy to see from an App Engine app's viewpoint, depending on what App Engine may have already added on its own on top of BigTable. E.g., Megastore adds entity groups for transactional behavior... but App Engine has had those for a while. Do you really care how App Engine internally implements, or will implement in the future, identical APIs...?
Megastore supports schemas... but who's to know whether they'll be made available to App Engine apps (so that puts with the wrong combination of types will raise exceptions instead of silently succeeding), which so far have always been schemaless except for whatever you, yourself, or Google's open-source app-level code, implemented at application level.
Now more details of Megastore have emerged, including James Hamilton's summary which links to the CIDR 2011 paper.
the existing answers have described the differences between bigtable and megastore pretty well. i'll just add one thing: app engine isn't moving from bigtable to megastore. it's been on megastore since the beginning. (ok, well, at least since very very early in development, years before it launched publicly.)
for example, see this sept 2009 app engine blog post about megastore replication.
Related
I am starting out with Google Apps Script, but as learning graph is growing exponentially few terms are getting hard to grasp.
Can anyone simply explain me about relationship between GAS and Googles Cloud Platform. How they are related? When there is a need of GCP when working with GAS?
Whats difference between creating project on Developer Console and creating on Cloud console?
Please add comment if need any rectification in question, i'll be happy to do that.
Thank you.
Google Apps Script is part of the G-Suite products, generally not related to the Google Cloud Platform products in ways other than being both Google products.
From the developer/customer prospectve occasionally there may be some marginal interactions between the products, but in a very limited manner - special situations. There might be more interactions on Google's infra side, but they're transparent for developers/customers.
So normally there is no need for GCP when working with GAS (or viceversa) unless it's for one of those very particular interaction situations, specifically documented.
For example
the Cloud Resource Manager is aware of and can deal differently with G-Suite customers. From Acquiring an Organization resource:
G Suite:
The first time a user in your domain creates a project, the Organization resource is automatically created and linked to your
company’s G Suite account. The current project and all future
projects will automatically belong to the organization.
a while ago mapping a Google App Engine application to a custom domain was only possible through G-Suite (known as Google Apps at the time).
It's not uncommon for various documentation/guides, including those G-Suite related (either Google's or from third parties) to reference GCP products, but usually that's because of the convenience of using them, not because they are actually needed.
So you really need to be aware of which family the product/documentation you're using is part of. The GCP products and documentation are a lot better of consistently identifying themselves a being part of GCP.
Finally the Developer Console is just the older, but still widely used name of the Cloud Console, they're the same thing - used for GCP. Unless you're referring to the Admin Console which is used for G-Suite.
It's still unclear to me when I should or should not use Google App Engine to deploy a commercial web application.
It appears Google has "business" level support.
http://code.google.com/appengine/
Can someone bullet list when I should use Google App Engine and when I shouldn't use it for a web application
The question is surprisingly simple to answer after I've had a stab at google engine with my project for a few weeks. You should use it when:
you can't be arsed to set up a server
you want instant for-free nearly infinite scalability
your traffic is spikey and rather unpredictable
you don't feel like taking care of your own server monitoring tools
you need pricing that fits your actual usage and isn't time-slot based
you are able to chunk long tasks into 30 second pieces
you have the skill/will/desire to work with noSQL and deal with the consequences thereof
you are able to work without direct filesystem access
So actually, you can use it pretty much for anything, especially websites. The only thing it very quickly becomes too pricey for is running large background processes. If you're doing some hardcore number crunching 24/7 you're better off using your own server somewhere because no cloud service can really live up to that.
But think of it this way, where else are you going to get an architecture that can swallow 10+ requests per second load for ten dollars a month?
Basically it boils down to this: If you want to focus on developing your code, not your server architecture. GAE is for you. (unlike amazon which behaves more like a fancy VPS)
I can't really tell you whether you should use App Engine without knowing anything about what you need your web app to do, but I will tell you what App Engine can and can't (or won't) do.
App Engine is fantastically good at scaling. It is, in fact, designed to scale web apps to ridiculous lengths first and foremost, with ease of use and number of features being secondary goals.
That's not to say that App Engine doesn't have features, or isn't easy to use, just that if there ever becomes a choice between adding a feature and staying scalable, the App Engine team will choose scalability.
For example, App Engine doesn't have some of the features of a relational database, because those features don't scale to the size of an app that App Engine is designed to support. App Engine doesn't support requests taking more than 30 seconds, because App Engine is designed to serve a web app, not process long-running requests.
In general, when App Engine doesn't support something, it's not because it's impossible -- nothing is impossible -- but rather because it would detract from the scalability of App Engine.
There are workarounds that can be (and have been) implemented to get around this, particularly with things like the task queue, and App Engine is constantly getting new features and new frameworks built on top of it.
App Engine for Business adds SLAs and different pricing, but is otherwise pretty much the same App Engine.
I am thinking of making an (initially) small Web Application, which would eventually have a potential to grow. All things considered Google App Engine seems like a very attractive option. Say, user base and complexity grows and for one or other reason I needed to leave GAE behind. How difficult would it be to migrate away?
1) Does GAE provide a way to export the database? What format would it be? Would it be difficult to put it under MySQL (or similar)?
2) In which areas (ex. database access, others?) would I have to use GAE API? I.e. which parts of implementation would have to be abstracted away / interfaced?
Edit: 3) Alternatively, is it even worth to abstract away GAE API?
For question #1: I don't know if GAE specifically supports exports of a database but you can always roll your own, worst case scenario. If you are in a position where you need to, you'll probably have the resources to do it, too.
For question #2: You can and should always encapsulate those kinds of outside dependencies anyway. It doesn't matter whether or not they provide interfaces. Coupling to those interfaces should be kept to an absolute minimum.
For question #3: This question is not really super-clear so I cannot answer it.
I'm speaking strictly from a java webapp point of view...
Google App Engine for python has a backup/restore utility:
http://code.google.com/appengine/articles/gae_backup_and_restore.html
There is a huge interested in porting this to the java flavor.
You can use the higher level standard database apis (JDO/JPS) to allow you to move your app away from google's database services. I suggest purchasing the data nucleus tools in order to smooth the transition from big tables to something like mysql or oracle.
The packaged services GAE provides are enumerated at
http://code.google.com/appengine/docs/java/javadoc/
The stock JRE should handle porting of the urlfetch, mail, and memcache api packages.
You'll have to find a substitute technology for the users, blobstore, xmpp, and taskqueue packages.
Lately I've been reading a lot of information about App Engine; the Google service that looks very promising to me. However, it all seems too good to be true. Call me a negative person, but I would like to know any reasons NOT to use App Engine.
This is programmers related, since I'm asking as in the programmer point of view.
I just want the general "downsides" of App Engine, if any.
No relational database. (No normal ORM ...)
Limits on the number of files. (Can't just upload the 6,000 files in some standard packages...)
Specific VM required. (Depending on your language you may be a non-starter.)
Runtime limits. (Don't get bogged down.)
SSL only through https://your-app-id.appspot.com (Not directly to your domain.)
Root domains via URL redirection only
I am not criticizing GAE, I'm just listing limitations as the OP asked...
You are designing your app to a Google-specified framework. Presently you can ONLY run an app-engine application on Google's infrastructure. That means you are completely dependent on Google (at least for the moment). Whether that's a problem or not is dependent on the project in question, but it's something you had best be aware of. Google has been known to cancel things in the past (though that seems unlikely with App Engine).
I'm evaluating hosted production environments and currently have interest in Google App Engine.
Currently I'm enjoying the free quotas. I'm concerned if it is efficient to scale up using
Google App Engine. Portability is being analyzed as well.
Please advise if Google App Engine is good for scalability and portability.
Thank you in advance.
Portability is guaranteed by the fact that Google has open-sourced all the parts of App Engine that live "in front of" the RPC layer, thus facilitating the work (which would happen anyway of course!-) of third party like appcelerator and bdbdatastore that implement compatible environment running on different infrastructure -- you only need to stay on Google's systems if Google gives you better ROI for your apps, else can easily migrate them to alternative implementations (I'm sure many more third-parties will join the ranks of these two, offering a variety of such alternatives).
Scalability, when the apps are programmed appropriately, seems proven eg. by the Obama's Town Hall Meeting example -- the app, using an open-sourced Google codebase known as "Moderator", was handling 700 QPS for a total of many millions of visits in a few hours, and maintaining excellent latency and impeccable uptime.
A LOT has been written (and recorded on video) about the right techniques needed to obtain such seamless scalability with App Engine -- there's really no way to summarize all of the hits in this google search! Suffice it to say, it's not trivial, but in the end it's easier (for suitable kinds of apps, at least -- ones that are "front-end heavy" as opposed to ones focused on huge "batch" jobs) than with any other technology I know of.