I have a process that gathers and stores data on my local dev_appserver, both on the datastore and blobstore. I do NOT want to move the process to production on Google App Engine, I just want to move the result of that process (structured data on on the datastore and some blobs).
What would be the best approach, and how could I get it done in a fairly automated way?
I've have a look at the appcfg.py options of dumping data out of the datastore, but from what I've seen it does not work with blobs.
My data structure is something like:
name --> ndb.TextProperty
content --> ndb.TextProperty
image --> ndb.StructuredProperty (structured property containing image BlobKey and image Serving URL)
I believe I need to upload the blobs to my Cloud Storage on one side, upload data to the Cloud Data Store on the other side, and then make sure the BlobKey relationship between the Data and Blobs is not lost.
The Cloud Storage API does not provide a clear explanation of how to do this from local. It seems like I need to create a POST request to the Cloud Storage API and the request should have the authorization (API Key) and the blob data. Is there a App Engine API that does this, or do I need to build the request myself?
Has anyone done this before? Any suggestions?
Thanks!
You could use the Cloud Datastore API to write to your production App Engine datastore using authorized HTTP requests.
The Python API is lower-level than NDB, but the entities section of the documentation has a table describing each property type.
Related
I need create a file report for user request. Each user select the filter for file report, and my application should generate a file in cloud storage and send a notification with the file link generated.
This is the application workflow:
the client selects a filter and request a report file
The application get this request and create a record in datastore with data about user selected filter
Stores the Datastore key URL Safe String from the new record in pubsub.
The Dataflow Pipeline read the key stored in PubSub.
Generate file in google cloud storage
Notifies the client with storage file url
It is possible to create a file for each pubsub entrance ?
How I do to create a file with custom name?
It is correct this architecture ?
Your use case sounds as if it would be more applicable to google cloud storage than cloud datastore. Google cloud storage is meant for opaque file-like blobs of data, and provides a method to receive pubsub notifications on file updates https://cloud.google.com/storage/docs/pubsub-notifications.
However, its a bit unclear why you're using the indirection of pubsub and datastore in this case. Could the server handling the client request instead directly make a call to the google cloud storage api?
I need carry out cloning application. Datastore can copy through
AppEngine console. How can copy Blobstore to another application?
Google does not provide a bulk option for download and upload the blobstore. Because of that you need to write your own code which do the procedure.
I can think about an upload endpoint in the destination application and an endpoint, in the source application, which cycle throgh the existing blob (you can know the existing blobs making a query on the datastore on the BlobInfo entity kind).
Remember that if you re-upload a blob in another application the BlobKey is not maintained, so you'll need to update your datastore entities too.
Because of the deprecation of the Blobstore, you can think about move to Cloud Storage in the source application first (changing the stored BlobKeys in a bucket/object Storage value) and then, after you have transferred the Datastore, you simply need to give access to the bucket to the destination application (only if you want keep the same file on two different applications)
How to backup (and restore) image files uploaded and stored as blobs in GAE Blobstore(Python)
I have gone through the GAE help doc on this topic. I could not find any way but I am sure there must be a very simple and intuitive way to do this since this is a fundamental need to develop any big commercial web app
Although a feature to download the backed up data would be better but I am even ok with Google Cloud Storage based approach if some definite guide is present for the same
I want to use the backup of my web app data in case of some accidental data deletion or corruption.. I plan to use the Datastore Admin to backup my NDB entities which could be easily restored using the same.. I was hoping for a similar solution(backup and also easy restore) for the image(picture) files stored as blobs in Blobstore..
I have gone through this GAE Blobstore help page and it does not say anything about its deprecation (Files API is deprecated and I am not using that)
I would advice against storing images in the AppEngine blobstore to store anything given that it's set for deprecation (and has been so for the last few months). So, in addition to back up I would strongly suggest migrating your app to store images directly in Google Cloud Storage asap.
The best way to back up images stored in Blobstore is to create a migration via TaskQueues. In this migration, grab each of the blobs and store them to a container which can be AWS S3 or Google Cloud Storage (via boto library). The reason you need to make is TaskQueue is because it will likely take a LONG time if you have lots of images stored in the blobstore.
Here's the SO question I asked and got a response about:
GAE Blobstore file-like API deprecation timeline (py 2.7 runtime)
I have around 1500 images which are dynamically generated in my local server. I want these to upload to Appengine's datastore. How can i do this? any help, any ideas?
BlobStore
You can use the blobstore if you have a billing account.
So I assume to use the bolbstore API to upload it in chunks with a client tool over http.
Before every upload, you can ask the blobstore to give you an unique upload URL to post MIME multipart posts to it.
There is a size limit for requests I think.
Simple DataStore
If you store your images in the datastore, you can use a python tool somehow to synchronize.
in Google App Engine,after using the makePersistent() to store the data in the datastore,i know how to get the data content by the key using getObjectById().
but now i wanna to get the data in the datastore by url. i think the url is created .
so the question is how the url can be created to get the data in the datastore
There is no built-in means to access the datastore through URLs. If you choose to, your application can implement URLs that return data from the datastore.
I invite you to take a look at titan-files. It's a powerful file-system abstraction on top of the DataStore and/or blob store. I'm using it for a commercial application and so far I've been very happy with it.