I want to build a mobile app. Should i cache data on google cloud memcache or it's better to store data into the mobile phone local memory ?
which is faster ? and what type of data should be put in cache in the case of a mobile app ?
Thanks you.
Any data that you want to make available to your user in offline mode (device not connected to the Internet) should be cached on the device itself. You may also want to cache the data in online mode as well (e.g. an Email App may cache the emails for last 7 days) and updates the cache when the user refreshes or the server pushes new emails.
MemCache on Google Cloud is something you would use in your Application Server for caching data that does not change very often so you don't have to keep loading the same data from the database for every request, thus improving the performance.
It is always faster to access the data on the local device compared to accessing the data from Cloud.
Related
Being new to GCP, I have a question about which architecture to use in a particular case.
Suppose I have a Django website running on the App engine (flexible environment?). Users upload images to the website. I would like to first use Google Vision API to perform some label detection on the images and then feed the labels and images to a VM with GPU attached (all running on Google cloud), for additional computationally costly job on the images. After the job is completed by the VM, the resulting images are then available for the user to download or sent to the user email.
Because of the relatively large time spent on the VM+GPU side, and because the website will be accessed by users globally, I would like to reduce the overall latency time and pick the most efficient architecture for the job.
My first thought was to:
upload images to Google Cloud Storage;
use GC functions to perform some quick transformations and then call Google Vision API;
pull the resulting labels and transformed images to the VM and make computations on the VM side;
upload finalized images to Google Cloud Storage.
Now, that's a lot of bouncing back and forth between a storage bucket and APP engine plus VM on either side. I was wondering if there is a 1) quicker and 2) more efficient resources-wise way to achieve the same goal.
If your website is accessed globally, your App Engine choice is the wrong one: App Engine can be deployed in only one region, not globally.
For the frontend, I recommend to use Cloud Run instead (or VM, but I don't like VM) and to put a HTTPS load balancer in front of. Like that, the physical latency is reduced.
And, the files must be also store in the closest region, so in Cloud Storage in different region.
And finally, to duplicate the VM/GPU infrastructure in each region (it could be costly, but it's the best way to reduce latency.
Your process is the right one. I recommend you to expose an API on your VM to notify it when a file is ready. You can use the PubSub notification on Cloud Storage to sink the event in PubSub, and then create a push subscription to invoke your VM directly (instead of a cloud functions).
Like that, you remove a component and you perform all your processing on the VM side.
I was wondering, if data stored in IndexedDB can be used globally by other users? Or it's purpose is only for local usage? For example if I'll do web page with personal notes, can the notes be visible for other users on other computer (if I want them to be?). And how the IndexedDB works. If it's localy stored, then what if the user will uninstall the web browser or just even destroy the computer. Will the data be deleted?
Sorry, if it's silly question for You :D
IndexedDb is browser based storage mechanism. A few reading in the web will give enough information I will try to list of few crux of indexedDB
It is browser based Data storage mechanism , its scope limited to browser and machine. That means one you stored something to IndexedDB only from that browser and that machine and from same domain only you can access. Domain means if you have a website which useses indexedDb from no other website can be accessed.
Indexed db can use 6-10% of the hard disk of your machine
Indexed db data wont be loosed if you closed or rebooted your machine. even Ctrl+f5 wont loose the data. if user do a hard cache delete then the indexed db data will be loosed.
Check CanIUse for the browser compatibility
IndexedDB basically work of Key value pair data storage, we can do indexing on those itesm, in indexed db we can created multiple store to categorize the data, store concept wise we can relate to table in SQL
IndexedDB is a way for you to persistently store data inside a user's browser. Because it lets you create web applications with rich query abilities regardless of network availability, these applications can work both online and offline. IndexedDB is useful for applications that store a large amount of data
IndexedDB is an asynchronous API that can be used in most contexts, including Web Workers. It used to include a synchronous version also, for use in web workers, but this was removed from the spec due to lack of interest by the web community.
more about indexedDB please read here.
My application is currently on app engine server. My application writes the records(for logging and reporting) continuously.
Scenario: Views count in the website. When we open the website it hits the server to add the record with time and type of view. Showing these counts in the users dashboard.
Seems these requests are huge now. For now 40/sec. Google App Engine writes are going heavy and cost is increasing like anything.
Is there any way to reduce this or any other db to log the views?
Google App Engine's Datastore is NOT suitable for such a requirement where you have to continuously write to datastore and read less often.
You need to offload this task to a third party service (either you write one or use existing one)
Better option for user tracking and analytics is Google Analytics (Although you wont be directly able to show the hit counters on website using analytics).
If you want to show your user page hit count use a page hit counter: https://www.google.com/search?q=hit+counter
In this case you should avoid Datastore.
For this kind of analytics it's best to do the following:
Dump data to GAE log (yes, this sounds counter-intuitive, but it's actually advice from google engineers). GAE log is persistent and is guaranteed to not loose data you write to it.
Periodically parse the log for your data and then export it to BigQuery.
BigQuery has a quite powerful query language so it's capable of doing complex analytics reports.
Luckily this was already done before: see the Mache framework. Also see related video.
Note: there is now a new BigQuery feature called streaming inserts, which could potentially replace the cumbersome middle step (files on Cloud Storage) used in Mache.
New in GAE development and have some question regarding extracting data.
I have an app that collects data from end users and data is stored in high availability datastore, and there is a need to send subset of data the app collected to business partners on a regular basis.
Here are my questions,
1. How can I backup data in the datastore on a regular basis, say daily incremental backup and weekly full backup?
2. what are the best practices to generate daily data dump files that can be downloaded or send to my partners in a secured approach. I expect few hundred MB data files everyday and eventually will be in few GB range.
3. Can my business partners be authenticated though basic HTTP auth, or have to use OAuth?
Google is in effect backing up your data by storing it in multiple data centres.
You can however use the bulk loader if desired and back it up manually:
Uploading and Downloading Data
You can authenticate users however you choose, it's totally up to you. The "users" service is integrated into app engine directly however so if everybody has or could have google accounts that's even easier for you to use.
The users service
Due to the size of your files unless you want to piece them together from the datastore you'll have to use something else, as the datastore has a 1MB limit per model. It's perfectly possible to do that however.
But you should probably look at The Google Cloud Storage API instead as there are no file size limits.
I'm mostly a PHP developer. Now I have a client steering me in the direction of a standalone app that runs on Mac, Windows, and Linux. It deals with pilot weather data and he wants it to work offline during flights and then sync up fresh data in the airport wifi. Immediately I thought of Google Chrome Apps for this.
I need to know what the storage size limitation is for Google Chrome databases when used specifically in Google Chrome Apps. I've been having trouble finding this information.
Some extra, related questions are:
When someone clears their Chrome browser cache, does this mean their Databases, Local Storage, and Application Cache are wiped clean? Or, is it only some of those resources are cleaned? My fear is that someone clears their cache and there goes all my offline app storage in the Google Chrome App.
I hate to sound dumb, but is "WebSQL" different than Chrome Databases?
Why would I use Local Storage versus a Chrome Database? (See the difference when you do CTRL+SHIFT+I in Chrome, and then click Resources tab.)
Storage limit is 5MB by default. This limit can be disabled with unlimitedStorage permission (more details here).
I don't want to clear my cache to test it, but I am pretty sure storage is not cleaned. There is a related issue report which says that there is no way to delete storage explicitly right now.
WebDatabase, WebSQL, "a database api" all refer to the same thing - web database API that is currently based on SQLite.
Web database is pretty much a full scale database, localStorage is just a hashmap (an associative array) that stores key-value pairs.