I've inherited a project which uses Google App Engine Blobstore to store files, and I need to download all the images so they can be migrated to a new system.
I was able to get all the Google Datastore data out with https://github.com/jodeleeuw/export-google-datastore and was hoping there was something similarly easy with Blobstore. Or at least an example of how to read all the files in the blobstore and download them.
The Datastore solution you mentioned relies on the Datastore API being available for apps not running on GAE. But AFAIK the Blobstore API is not available outside GAE, so a similar solution is likely impossible.
I see a couple of options:
enhance the GAE app code you inherited by adding the capability to get the data in the Blobstore (since it normally should be able to) and export it either directly to where you want the data moved or in an intermediate place, say GCS for example, from where you can ship the data to the final destination easier. Moving directly to the final location would be preferable IMHO if you need to keep the app working with the new location - you can build a nice, hitless migration story
use the developer console Blobstore browser/viewer which also allows downloading and deleting the blobs, either manually or using a GUI automation tool (like selenium, for example) for a programmatic approach.
Related
I have been using Google Cloud Storage to save photos that users upload from a mobile app (built with flutter and firebase), recently I had the need to resize/transform images and I wanted to explore if it's possible to do it directly from google cloud storage!
I found this project https://github.com/albertcht/python-gcs-image that you have to deploy on google app engine and if you call it with a bucket and an image it returns a URL to a Google CDN I think (something like this http://lh3.googleusercontent.com/*).
I looked at the code in the repository and the only thing it does is to return the result of google.appengine.api.images.get_serving_url and I don't understand why I cannot get this serving_url directly from my dart code?
What is the difference between Google Cloud Storage and lh3.googleusercontent.com?? Can I make the same image processing directly from cloud storage?
It seems odd that I have to run an app engine app that just returns a URL?
What am I missing?
The lib use this api. It's in Python 2.7 which have a end of life the 01/01/2020.
Moreover, the image api is available only with AppEngine 1st generation (python 2.7) and not available for the 2nd generation (python 3).
All of this for not recommended you to use this.
The best design today is to perform the resize/crop when the file is uploaded and to store the result in Cloud Storage. Example here and here
Then, you only have to serve, from Cloud Storage, the resized/cropped images.
Look # Firebase Extensions, there is already such an Extension provided there
What I found is:
The Java, Python, and go Standard environments for Google App Engine
include the GAE Images API (Java, Python, which can resize, rotate,
flip, and crop an image, as well as return an image serving URL which
allows for client side transformations, similar to Cloudinary and
Imgix.
This matches my previous understanding and experience as well. serving_url is really convenient for image manipulations.
Having said that, as you correctly pointed out, it's first and foremost an AppEngine feature and will require you to use AppEngine in one way on another.
If it's not what you want, you can create a service that will crop your images and deploy it serverless. It's a lot less burdensome that having AppEngine service running 24/7. What's more, AWS had several pre-baked templates to do just that - crop images that can be deployed in a couple of clicks.
If you are, like myself, interested in Google Cloud solution I can offer a similar function that I wrote. It can be deployed in Cloud Run as-is. See details in my other answer.
With it you can not only resize the images for mobile, but also map your own domain to the Cloud Run function and put it behind any CDN you like, which potentially can be faster that service from Google Storage. You can find plenty info on the Internet about why full-fledged CND is better than just bare Google Storage.
I am writing an Android application that allows users to upload and share photos. The server is based on Google App Engine. App Engine's datastore does not allow to save file, so currently I just have URLs saved. Looking for a way to store files I read about Google Cloud Storage. My question is - if I'm looking for a host for user uploaded files, is Google Cloud Storage what I'm looking for?
Yes. Google Cloud Storage is the way forward. There is also the BlobStore API on App Engine that allows you to store large amounts of information but the road map seems to be clear i.e. use Google Cloud Storage moving forward.
The reason for going with GCS will also be influenced that eventually you might want various tools or utilities that people have written that work directly with GCS. With Blobstore API, you will need to write those utilities yourself or rely on Admin console's support for taking backup,etc - which is not really much.
In summary, go with GCS.
Yes thats what you want. Says the same if you read the docs about google cloud storage.
MySQL has this Sample Sakila DB where we can start playing around with bunch of data already for our application, how about for Google App Engine/GAEJ is there something like this for the datastore?
I started recently to experiment with the Google App Engine and I was confronted with the same question. I was interested in a REST based app engine backbone which I could easily load/unload with data but couldn't find something to play around.
So I started to build up two projects on github which supports me in such kind of work.
clb-appEngineTemplate is a skeleton application for a Google App Engine Jave REST backend. It provides some sample code for a standardized REST API based persistency layer on Business Object level and can be easily extended (using Objectify and GSON).
clb-test which is a utility class which allows to load Test Data from Excel CSV file into your Google App Engine REST backend.
Both projects are maven based and allow me easily to define data objects which I can upload into the App Engine. Mainly I'm run them against the local test server, which serves me for initial testing.
I just released a first version and will incrementally extend over the next weeks.
AFAIK, there is no sample DB for GAE, probably because datastore write operations are expensive. There are demos bundled with GAE SDK. If you are using Eclipse you can import the samples to your workspace. Some of them involve datastore so you can run the application and add data yourself.
Another way is to use bulkloader to upload data at once using CSV files. But you can quickly run out of free quota for datastore writes.
I want to upload files into my Google App Engine project.
I've been reading a while on this issue, and there is a lot of answers arguing Blobstore is the best option.
But if I did understand this well, these are database objects. I would like to upload them as system files to a caching theme or to ease a possible future migration to another CDN.
Imagine I want to save several files for each user, a couple of images, text files, maybe some video, etc...
If some day I want to move these static files to another CDN, shouldn't it be out of the database?
Is that a good idea? Is there a solution?
I would recommend you to use GAE datastore to store the files references (for example, video, images, etc.) and upload the content on Amazon S3. You may even allow the clients to upload the content directly to Amazon S3 without to process the content through GAE using Amazon HTML POST Forms (Browser Uploads to S3 using HTML POST Forms). Amazon AWS also offers CDN (CloudFront) with tight integration with S3 service.
If you need to store only small pictures, you may also consider the Google images services as it offers "free" CDN and some cool out-of the box transformations.
You can't write to the filesystem on App Engine, so your options are to store data to the datastore, or the blobstore. Of the two, the blobstore is generally the better choice for storing file uploads.
Regarding porting to a CDN, no matter how you do this it's going to require changes. App Engine is not a CDN, no two CDNs are exactly alike, and it's unlikely the CDN would expose an interface that you can interact with using standard filesystem operations anyway.
I've got some data up on Google App Engine's live datastore, but would like to have a local copy as well for local development. What's the best way to do an occasional sync?
There's no trivial way to do this, but there are some possibilities:
Use the official bulk uploader (from the GAE-Python SDK) if your application is in Python (but not Django)
Use Gaebar if your application is in Django
Use the Datanucleus REST API if your app is in Java
You can also try some other open source projects meant to solve this problem. AppRocket was a promising one, but there's only one commiter to the project and there is no activity in the past months...
Also, star issues 59 and 776 to get this included officially by google.