I'm trying to deploy Errbot in App Engine. Errbot needs the data directory to be writable but an App Engine instance's local filesystem is not writable. Is there any way to work around this?
From Documentation:
You are correct, GAE local filesystem that your application is deployed to is not writeable. This behavior ensures the security and scalability of your application.
However, if your use case involves to store data in a persistent manner, you may consider using Cloud Storage to read and write files during runtime. App Engine creates a default bucket when you create an application.
This bucket provides the first 5GB of storage for free and includes a free quota for Cloud Storage I/O operations. You can create other Cloud Storage buckets, but only the default bucket includes the first 5GB of storage for free.
Related
Have a requirement of accessing large pandas data frame files to run some analytics in the worker of the app engine via google cloud tasks
Can somebody please suggest on what component in google cloud can be used for storing and accessing files quickly ?
Any reference to example would be of great help.
I think Google Cloud Storage is the best place to store and accessing the files quickly :
https://cloud.google.com/storage/docs/discover-object-storage-console
GCS can store large files and a big amount of files.
You can also use :
gsutil to move or copy files between buckets.
Storage transfer service
https://cloud.google.com/storage-transfer/docs/overview
Your application from app engine could use Cloud Storage.
I have a REST, .NET Core application running in Google App Engine Flexible.
It reads binary files from Cloud Storage (several MB in size, rarely hundreds of MB).
To make it running faster I'm caching these files in local file system (/tmp). But this approach doesn't work when the app is scaled and more instances are running simultaneously.
What are my options for fast file cache which is shared between app instances?
Cloud Filestore looks great but is not available for App Engine
Cloud Memorystore - I'm not sure it is suitable for me
You can use a redis Database to cache up to 30 mb of data for free from your App Engine and this is a solution easy to implement.
For this you only need to create a Redis Cloud database and modify the appsettings.json file of your app.
I have some configuration files in my application, to stop needing a deployment each time the configuration files change - I wanted to have a simple process which would sync these files from a store, such a google cloud storage into a folder on the machine(s) aka RAM.
I need the data in RAM for speed.
I do not want at this stage to add complexity by adding DBs and pub/sub etc.
Running a nodejs app on the flexible GAE environment.
In essence I am looking for a simple solution, at this stage.
You can use Cloud Storage FUSE which is a adapter for mounting Cloud Storage buckets to your local machine or GCE instances, as a file system.
You have the installing instructions here.
So what makes google cloud storage preferred choice for saving files. As I know google cloud storage is saving the file as blob so it is immutable and cannot be edited.
if the main aspect of your website/application is a database (as is often the case), check out Google Cloud SQL. Again it allows you to host your MySQL database on Google’s infrastructure, increasing the speed, reliability and security.
Whereas Google Cloud Storage enables application developers to store their data on Google’s infrastructure with very high reliability, performance and availability, and can be used to distribute large data objects to users via direct download.
According to this Wikipedia article on Google App Engine:
Developers have read-only access to
the filesystem on App Engine.
I have heard of App Engine apps that allow uploads and Google also sells additional storage - if so, wouldn't this statement appear to be incorrect?
No.
Those apps would likely store the uploaded data into the store directly rather than the filesystem.
The additional storage is not necessarily in the filesystem but in the storage itself.
From this Google App Engine page:
An App Engine application cannot [...] write to the filesystem. Applications must use the App Engine datastore for storing persistent data. Reading from the filesystem is allowed
Your app cannot write to the file system. Apps that allow uploads store the uploaded documents/data in the database.
Check out the Blob property, such as in this tutorial for storing images:
http://code.google.com/appengine/articles/images.html
There are also libraries for splitting a large file into pieces for storage as Blobs and put them back together again.