I need carry out cloning application. Datastore can copy through
AppEngine console. How can copy Blobstore to another application?
Google does not provide a bulk option for download and upload the blobstore. Because of that you need to write your own code which do the procedure.
I can think about an upload endpoint in the destination application and an endpoint, in the source application, which cycle throgh the existing blob (you can know the existing blobs making a query on the datastore on the BlobInfo entity kind).
Remember that if you re-upload a blob in another application the BlobKey is not maintained, so you'll need to update your datastore entities too.
Because of the deprecation of the Blobstore, you can think about move to Cloud Storage in the source application first (changing the stored BlobKeys in a bucket/object Storage value) and then, after you have transferred the Datastore, you simply need to give access to the bucket to the destination application (only if you want keep the same file on two different applications)
Related
How to backup (and restore) image files uploaded and stored as blobs in GAE Blobstore(Python)
I have gone through the GAE help doc on this topic. I could not find any way but I am sure there must be a very simple and intuitive way to do this since this is a fundamental need to develop any big commercial web app
Although a feature to download the backed up data would be better but I am even ok with Google Cloud Storage based approach if some definite guide is present for the same
I want to use the backup of my web app data in case of some accidental data deletion or corruption.. I plan to use the Datastore Admin to backup my NDB entities which could be easily restored using the same.. I was hoping for a similar solution(backup and also easy restore) for the image(picture) files stored as blobs in Blobstore..
I have gone through this GAE Blobstore help page and it does not say anything about its deprecation (Files API is deprecated and I am not using that)
I would advice against storing images in the AppEngine blobstore to store anything given that it's set for deprecation (and has been so for the last few months). So, in addition to back up I would strongly suggest migrating your app to store images directly in Google Cloud Storage asap.
The best way to back up images stored in Blobstore is to create a migration via TaskQueues. In this migration, grab each of the blobs and store them to a container which can be AWS S3 or Google Cloud Storage (via boto library). The reason you need to make is TaskQueue is because it will likely take a LONG time if you have lots of images stored in the blobstore.
Here's the SO question I asked and got a response about:
GAE Blobstore file-like API deprecation timeline (py 2.7 runtime)
I have a process that gathers and stores data on my local dev_appserver, both on the datastore and blobstore. I do NOT want to move the process to production on Google App Engine, I just want to move the result of that process (structured data on on the datastore and some blobs).
What would be the best approach, and how could I get it done in a fairly automated way?
I've have a look at the appcfg.py options of dumping data out of the datastore, but from what I've seen it does not work with blobs.
My data structure is something like:
name --> ndb.TextProperty
content --> ndb.TextProperty
image --> ndb.StructuredProperty (structured property containing image BlobKey and image Serving URL)
I believe I need to upload the blobs to my Cloud Storage on one side, upload data to the Cloud Data Store on the other side, and then make sure the BlobKey relationship between the Data and Blobs is not lost.
The Cloud Storage API does not provide a clear explanation of how to do this from local. It seems like I need to create a POST request to the Cloud Storage API and the request should have the authorization (API Key) and the blob data. Is there a App Engine API that does this, or do I need to build the request myself?
Has anyone done this before? Any suggestions?
Thanks!
You could use the Cloud Datastore API to write to your production App Engine datastore using authorized HTTP requests.
The Python API is lower-level than NDB, but the entities section of the documentation has a table describing each property type.
I've got a bunch of large files in the regular cloud storage that I'd like to programmatically move over to the blobstore for further processing using the mapreduce library. (Since there is a BlobstoreLineInputReader but not a Datastore version.) I've tried making a url for the gs file and having the blobstore try reading it in itself, also I've tried buffered reads, but for large files I still hit a memory error. (I avoid the deadline exceed error (more 60 seconds) for blobstore files by opening in append mode and finalizing only at the end.) It seems like there should be an efficient way to do this since both the datastore and blobstore are part of the same application context, but I haven't found it.
I'm confused because you mention cloud storage and datastore almost interchangeably here.
If your data is in Google Cloud storage then you can create BlobKeys for the files and use them with any current Blobstore API.
i.e.
blobkey = blobstore.create_gs_key('/gs/my_bucket/my_object').
If your files are in the datastore then you'll need to use the files API to move them to Cloud Storage/Blobstore and then process them from there.
I couldn't find articles match to my requirements.
Basically, what I want is that:
User uploads picture to the application from their local drive.
Application stores the picture uploaded to datastore.
Application retrieves images from datastore.
Any suggestions? Urgent.
That's exactly what is discussed in the documentation for the BlobStore API.
You can do this in much the same way as you would in any other framework or platform: Create an HTML form with a 'file' input and the mimetype set to 'multipart/form-data'. On the server side, extract the file data from the form field (using self.request.POST['fieldname'].value in webapp) and store the contents in a datastore model, in a db.BlobProperty field.
I'm building application on Google AppEngine with Java (GAE/J) and all my data will be stored in Google DataStore. Now, what if I want to save some binary file, let's say Images (JPG, PNG, etc), DOC, TXT, Video Files how do i deal with these? Or what if i want to stream video files (SWF) where and how should i store those files and when I redeploy my app i don't loose any data.
Depends on whether you're talking about static files or dynamic... If they're static created by you, you can upload them subject to a 10MB/3000 file max but Google doesn't offer a CDN or anything.
If they're dynamic, uploaded by your users or created by your application, the datastore supports BlobProperties: you can dump any kind of binary data you want in there as long as it's less than 1MB per entity. If they're larger you can consider another service like S3 or Mosso's cloud files. This can be a better solution for serving files directly to users because these guys can offer CDN service but it's not cheap. On the other hand your latency back to GAE will be much higher than storing the data in Google's Datastore and you'll have to pay for transit on both sides so it's something to take into account if you're going to be processing the files on App Engine.
Google App Engine Virtual File System (GaeVFS)