So what makes google cloud storage preferred choice for saving files. As I know google cloud storage is saving the file as blob so it is immutable and cannot be edited.
if the main aspect of your website/application is a database (as is often the case), check out Google Cloud SQL. Again it allows you to host your MySQL database on Google’s infrastructure, increasing the speed, reliability and security.
Whereas Google Cloud Storage enables application developers to store their data on Google’s infrastructure with very high reliability, performance and availability, and can be used to distribute large data objects to users via direct download.
Related
In the Google Cloud ecosystem, what is the canonical storage option for basic user data including login credentials?
username
password
email
etc.
Let's optimistically say my app has 100K users, and I don't want to enable social login, but might want to in the future.
I'm looking at CloudSQL and Datastore at the moment, and am unsure about the tradeoffs.
The main reason why I have been choosing Cloud Datastore in favour of Cloud SQL is the horizontal scalability of Datastore which is fully managed by Google. Since it is backed by Bigtable, you don't have to worry about scaling your instances.
Another thing you have to keep in mind is the cost model: with Cloud Datastore you pay per read/write/delete and storage. And for Cloud SQL you pay per running instance and storage. When you are starting up a website with only a few users, Cloud Datastore is very cheap, because you don't have to pay for a 24/7 running instance of Cloud SQL. When the traffic grows, it depends on the usage pattern what technology will be more cost-effective.
With Cloud SQL, you are more flexible when performing ad-hoc queries: you don't always have to add an index beforehand. And even if you do, you don't have to re-write all entities to add the entity (like it is the case in Cloud Datastore). And of course you have the obvious advantages: ACID support and a wide range of frameworks to choose from to interact with MySQL or PostgreSQL.
Keep in mind that both technologies seem similar, but the characteristics are completely different. You will encounter that when designing your Datastore dataset for scale and reading the documentation about the limits.
Finally, it might also be worth to take a look at these two products:
Cloud Firestore: the successor of Cloud Datastore, currently in Beta
Firebase Authentication: an API that handles authentication and integrates nicely with other Firebase products
In the docs, they claims that Cloud Firestore duplicates data in multiple servers located in different regions across the world.
My question is : By using Cloud Firestore, do i need to cache my data even though ?
Firestore is a document-based storage system. It's typically used as a database for your app, so for content that changes more regularly.
A CDN is a caching layer for delivering (mostly) static assets to regionally diverse users. I'd recommend looking at Cloud Storage for Firebase for that.
Cloud Firestore synchronously replicates data for durability and availability, rather than for serving speed. In this sense, it is very different from a CDN or a latency cache.
The Mobile and Web SDKs do store data locally though and provide a concept called latency compensation, so on that side it's got you covered out of the box.
I am building a mobile app with Google Cloud as back-end. It stores information about restaurants including their menu. I would need to search based on the menu items and geo location.
I am fairly confused about which Google Cloud solutions to use. Initially I have planned to use Google App Engine with Cloud Datastore for storing data, but later found out, it doesn't support geolocation search, and even if I use the App Engine search api, I won't get 'like' text search. About using cloud SQL, I am worried about its price. I am also thinking about using Google Compute Engine and installing required MongoDB or SQL in it and use some custom search libraries.
So my question is which of the below should be an feasible and economical solution for storing large amount of data and searching using text and geo-location
GAE with Cloud Datastore
GAE with Cloud SQL
GCE with MongoDB or SQL installed
Any other feasible solutions are also welcome
Google has a page to give high-level guidance on what storage option to choose on GCP, called Choosing a storage option.
Specifically for Cloud Datastore:
Description: A scalable, fully-managed NoSQL document database for your web and mobile applications.
Good for:
Semi-structured application data
Hierarchical data
Durable key-value
Common Workloads:
User profiles
Product catalogs
Game state
Specifically for Cloud SQL:
Description: A fully-managed MySQL database service that is built on the strength and reliability of Google’s infrastructure.
Good for:
Web-frameworks
Structured data
OLTP workloads
Common Workloads:
Websites, blogs, and content management systems (CMS)
Business Intelligence (BI) applications
ERP, CRM, and eCommerce applications
As you can see, your use case could border either option. Restaurants + Menus is very similar to product catalogs.
Cloud Datastore would require less ops work/thoughts on your side. For example, you don't need to think about what type of VM instance and memory size it should use. However, to do analytics on data rather than just serving the menus, you'd probably want to do daily dumps into BigQuery.
Cloud SQL will require a little more thought at the beginning, and at small sizes will be a little more expensive than Cloud Datastore. It does give you more flexibility on the analytics side in that it doesn't really need you to dump into BigQuery.
Running a different database on GCE is certainly an option if you want more control. The trade-off here will be in requiring much more active ops work on your behalf, such as installing, patching, tuning, etc.
I am working on a project which needs to store profiles of people on the cloud.
Information includes multiple photos and multiple text fields. I don't need Messaging.
There is lot if confusion in the documentation provided Google Cloud Services.
I am confused about what storage services should I opt for out of the 3:
1-Google Cloud Services,
2-Google Datastore,
3-Google Cloud SQL
So the the things i need to confirm are:
0.Is there a storage limit on using Google cloud SQL?
1.Does Google Cloud Storage and Google cloud Datastore provide unlimited storage?
2.Can an Android user write data on the cloud. I heard from some where that the applications only have access to read the data and the developer needs to put the data as blob on the cloud him self when using Google Cloud Services. Is this fact true for all the 3 storage services.
3.Is the data fully 'Sharable'+'Searchable'?
In other words:
If an Android user stores data in cloud in Google Datastore(text)+Google Cloud Storage(image), can this data be accessed by another android user without any headache of permissions or authentication(after I authenticate my app/app-engine)?
4.Is it the best option to store the images in Google Cloud Storage and their URLs in Google Datastore?
5.Does all the three storage services need app-engine to work?
6.Are any limitations on each of these services?
(0) Cloud SQL has currently a limit of 250Gb.
(1) With regards to Cloud Storage, there isn't a limit you could reach.
(2) and (3) They're not created for easiness of searchability. They should be accessed through applications, that are authorized, i.e. is not a substitute to Google Drive or Dropbox.
If you're the owner of the project, you can "browse" the contents of your Cloud Storage, but it's not meant for that.
Furthermore, objects in Cloud Storage can't be modified once created. A change needs to create a new copy of the object.
(4) It's a good idea, and something is used by many developers who have their applications in Google App Engine.
(5) No, they can exist without you using Google App Engine, but as I said earlier, you'd probably need an "application" to allow your Web/Android users to interact with the data, and there's where GAE comes handy.
(6) Yes, your budget.
If you provide a more detailed use case, I could tell you what you'd need to do to get it done with the whole array of Google Cloud products.
In my application on Google App Engine, I have files/documents uploaded to Blob Store. The size of Blob Store has grown to more than 100 GB.
I am in search of a mechanism by which I can backup my blob store data, may be to some other server location or Google Cloud Storage or some other safe place.
AFAIK there isn't a simple tool that you can use to backup your Blobstore data. There are at least a few approaches that you could take to write your own tool:
Use cron.xml and a servlet (or some similar scheduling) to copy batches of your entity blobs to Google Cloud Storage and then download from there (see below for tools).
Use remote-API or provide some other API (REST etc) where a remote tool you make can query your entities and then over a serving url (that you probably already have) download the Blobs to your local machine.
Those approaches are not very friendly and there are however tools such as GSUtil that would be helpful if you migrated your data to Google Cloud Storage (GCS). I've seen a few people migrate their data now that upload and serving with GCS is supported with the Blobstore API.
It's important to also note that the Files API for Blobstore is now deprecated (since 1.8.1?) and that there is a new preferred way through the App Engine GCS client.
Additionally if you check Posts about 1.8.1 Pre-release SDKs Available. you'll see that there is some indication from Google that blobs might be auto-migrated over to Cloud Storage (free of charge) before the end of the year (presumably close to when Blobstore will be deprecated). Depending on the urgency of your needs it may be feasible to just wait.
You can use Backup/Restore functionality of GAE Admin console.