In the Google Cloud ecosystem, what is the canonical storage option for basic user data including login credentials?
username
password
email
etc.
Let's optimistically say my app has 100K users, and I don't want to enable social login, but might want to in the future.
I'm looking at CloudSQL and Datastore at the moment, and am unsure about the tradeoffs.
The main reason why I have been choosing Cloud Datastore in favour of Cloud SQL is the horizontal scalability of Datastore which is fully managed by Google. Since it is backed by Bigtable, you don't have to worry about scaling your instances.
Another thing you have to keep in mind is the cost model: with Cloud Datastore you pay per read/write/delete and storage. And for Cloud SQL you pay per running instance and storage. When you are starting up a website with only a few users, Cloud Datastore is very cheap, because you don't have to pay for a 24/7 running instance of Cloud SQL. When the traffic grows, it depends on the usage pattern what technology will be more cost-effective.
With Cloud SQL, you are more flexible when performing ad-hoc queries: you don't always have to add an index beforehand. And even if you do, you don't have to re-write all entities to add the entity (like it is the case in Cloud Datastore). And of course you have the obvious advantages: ACID support and a wide range of frameworks to choose from to interact with MySQL or PostgreSQL.
Keep in mind that both technologies seem similar, but the characteristics are completely different. You will encounter that when designing your Datastore dataset for scale and reading the documentation about the limits.
Finally, it might also be worth to take a look at these two products:
Cloud Firestore: the successor of Cloud Datastore, currently in Beta
Firebase Authentication: an API that handles authentication and integrates nicely with other Firebase products
Related
I'm trying to setup a web app in Google App Engine standard environment using Flask and MySQL.
I know Google App Engine is capable of scaling it's instance dynamically based on the performance needed but I'm unsure how I can do this for my MySQL servers.
For the MySQL connection I try to connect to Google Cloud SQL. I know how this works for a single instance but I would like to know whether it's possible to connect multiple instances and thereby allow the possiblity to dynamically scale my storage while using the same data.
So far I've found information how to do this on Google Computing Engine instances and Google Cloud SQL via this link but couldn't find out how this would work on Google App Engine standard environment.
Each GAE service instance will connect to your Google Cloud SQL as a separate client, using the same credentials. There's nothing to worry about in this respect, except maybe the cost of scaling/sizing your Google Cloud SQL instances if you have many GAE instances or limiting the number of GAE instances to keep the performance decent. From Best practices (the whole page is probably a useful read):
App Engine has the ability to automatically create more instances when load increases. If the number of App Engine instances increases
beyond what the Cloud SQL instance is sized to handle, your Cloud SQL
instance will not be able to meet the demand. You can avoid this issue
by limiting the maximum number of App Engine instances. For more
information, see Scaling elements.
As #dan-cornilescu correctly mentioned, Google App Engine automatically scales horizontally and handles the connections to your Cloud SQL instance without you having to care about.
But taking a look at your last comment on his answer :
Thank you, I understand that Google App Engine can scale dynamically
and let the created instances connect to one Google Cloud SQL
Instance, but I'm looking for a solution for the limited space and
performance on this Cloud SQL instance. Is there a way I can connect
Google App Engine to multiple instances and so increase the database
space on Google Cloud SQL?
I will go ahead and give you some options in order to address the limited space and performance of your Cloud SQL instance.
Regarding the limited space, when creating your Cloud SQL instance you can check the option “Enable automatic storage increases” under the “Configure machine type and storage” dropdown.
Regarding having multiple instances, remember that Cloud SQL instances scale vertically but not horizontally, so the first option I would recommend you is to switch to Google Cloud Spanner if what you are seeking is horizontal scalability. At the moment is the only database within Google Cloud products that allows to do so and is the choice to go.
You can also “fake out” the horizontal scalability by creating multiple Cloud SQL instances, which is indeed a best practice. While doing so, you only have to take into consideration that the amount of instances allowed per project is 40 and you will have to request an increase if you wish to have more.
After reading available answers/comments on similar questions on this forum, it is now evident that GAE app is not straight-forward ready to be deployed on Compute engine. I fully understand that what all the managed services(mostly as APIs be it, datastore, document/index Search, memcache, cloud storage, task queues, cron jobs etc.), App Engine offered being a platform, won't be the same-fashioned accessible/integration-ready if available on Compute engine at all.
We have a 5 years old fully-grown App engine app now.
I am considering a scenario to support high-level of customization/control and adding third party softwares/middlewares to our server environment which is not possible with App engine. So if we have all solutions(Compute Engines, Container Engines etc.) other than App engine, to migrate our application to meet such requirements, what is the cost of such migration?
Need of server provisioning and configuration at Compute engines with different pricing model[Understood, should not be a problem :)]
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
In nut-shell, We wanted it fully managed in the beginning so PaaS seemed the right choice 5 years ago. Now we want App minus platform-managed plus customized/flexible to our choice. How complicated this transition is going to be?
Full or partial code rewrite to continue using the same APIs esp. Datastore, Cloud Storage, Task Queues, Cron jobs, Document Search, Memcache etc.[Need confirmation here and any reference/link to migration guide would be help!!]
Unfortunately, some of those service only be provided on GAE, such as Document Search. But most of service can be use directly for GCP, such as Datastore, Cloud Storage. GAE Flexible Environment is much like GCP environment, so you can read this article first Migration to GAE Flexible Environment
In following articles also have some answer：
How to migrate Google App Engine Project to Compute Engine completely
Google App Engine Blobstore to Google Cloud Storage Migration Tool
Does this lead to risk of losing any managed service/API offered from App Engine? Document Search, Memcache, Task Queues, Cron jobs seem the possible candidates. Please confirm.
Yes, Document Search only available on GAE.
As per my reading, Big Query, Cloud storage, Pub-Sub APIs integration should not be much affected with such migration(Client-libraries or Rest APIs should still help!). Please confirm.
Yes, but you may need to change SDK or library. It dependency on your language and how to call those service by Rest API directly or SDK.
I am developing an Android app using the Firebase.
It uses Firebase Auth, Real-time database, Storage.
And the App has a search feature.
For example, a customer can search another user using this feature.
If there is a user "Yoonho Aaron Kim", user want to search this user using some keyword like "yoonho", "aaron", "kim", etc...
But the Firebase Query doesn't support all of them.
It provides only "startAt", "endAt", "equalTo" methods.
Plus, I cannot use these 3 methods at the same time.
Because of this limitation, I am considering another module like "Cloud SQL", "App Engine".
Is there any good ideas please?
AppEngine is actually not a database engine, it is a "platform to build apps and scale automatically" and as such can connect to different types of databases: Cloud SQL (that you mention) a relational MySQL db (or PostgreSQL) or Cloud Datastore, a NoSQL database.
With Cloud SQL you could indeed perform some queries with a LIKE operator in a WHERE clause. With Datastore you will get the same limitations than Firebase database.
However, switching to AppEngine means that you will stop using Firebase and that you will go for another solution to develop and expose your APIs to your Android app, e.g. use a framework like Google Endpoints (https://cloud.google.com/endpoints/) .
I am building a mobile app with Google Cloud as back-end. It stores information about restaurants including their menu. I would need to search based on the menu items and geo location.
I am fairly confused about which Google Cloud solutions to use. Initially I have planned to use Google App Engine with Cloud Datastore for storing data, but later found out, it doesn't support geolocation search, and even if I use the App Engine search api, I won't get 'like' text search. About using cloud SQL, I am worried about its price. I am also thinking about using Google Compute Engine and installing required MongoDB or SQL in it and use some custom search libraries.
So my question is which of the below should be an feasible and economical solution for storing large amount of data and searching using text and geo-location
GAE with Cloud Datastore
GAE with Cloud SQL
GCE with MongoDB or SQL installed
Any other feasible solutions are also welcome
Google has a page to give high-level guidance on what storage option to choose on GCP, called Choosing a storage option.
Specifically for Cloud Datastore:
Description: A scalable, fully-managed NoSQL document database for your web and mobile applications.
Good for:
Semi-structured application data
Hierarchical data
Durable key-value
Common Workloads:
User profiles
Product catalogs
Game state
Specifically for Cloud SQL:
Description: A fully-managed MySQL database service that is built on the strength and reliability of Google’s infrastructure.
Good for:
Web-frameworks
Structured data
OLTP workloads
Common Workloads:
Websites, blogs, and content management systems (CMS)
Business Intelligence (BI) applications
ERP, CRM, and eCommerce applications
As you can see, your use case could border either option. Restaurants + Menus is very similar to product catalogs.
Cloud Datastore would require less ops work/thoughts on your side. For example, you don't need to think about what type of VM instance and memory size it should use. However, to do analytics on data rather than just serving the menus, you'd probably want to do daily dumps into BigQuery.
Cloud SQL will require a little more thought at the beginning, and at small sizes will be a little more expensive than Cloud Datastore. It does give you more flexibility on the analytics side in that it doesn't really need you to dump into BigQuery.
Running a different database on GCE is certainly an option if you want more control. The trade-off here will be in requiring much more active ops work on your behalf, such as installing, patching, tuning, etc.
So what makes google cloud storage preferred choice for saving files. As I know google cloud storage is saving the file as blob so it is immutable and cannot be edited.
if the main aspect of your website/application is a database (as is often the case), check out Google Cloud SQL. Again it allows you to host your MySQL database on Google’s infrastructure, increasing the speed, reliability and security.
Whereas Google Cloud Storage enables application developers to store their data on Google’s infrastructure with very high reliability, performance and availability, and can be used to distribute large data objects to users via direct download.