This process allows users to upload (POST) data directly to a specified GCS bucket, without any GAE intermediation. For large files, this seems easier/faster/better for both users and devs. Is it possible to do something like this using the Google Cloud's Python SDK (or pure Python)?
You can allow users to upload directly from the web browser by using the Cloud Storage XML API. The example below uploads to a public bucket, naming the file with what the user inputs in the "key" text field:
<form action="http://storage-upload.googleapis.com/[BUCKET_NAME]/" method="post" enctype="multipart/form-data">
<input type="text" name="key" value="">
<input type="hidden" name="success_action_redirect" value="http://www.example.com/success_notification.html">
<input name="file" type="file">
<input type="submit" value="Upload">
</form>
There are multiple ways to secure this, the POST Object documentation provides an example of how to do it with a Policy Document generated with Python, but you could also use Signed URLs, which can be generated with the App Identity Python API.
Another interesting thing about the Cloud Storage XML API, is that it also allows you to implement Resumable uploads, which might be the best approach for uploading large files.
About the size limit, I could not find any information in the Official Docs about a limit for this API in specific, but I figure this could be easily determined by running some tests.
Related
I have a React web application in which I allow users to upload DICOM files to Google Healthcare API. The current implementation is that the files first gets uploaded to my back-end server which uploads them to Healthcare API. I am allowing users to upload a full DICOM study (100MB - 2+GB) which could have anywhere from 1-500+ DICOM files (each usually 50KB-50MB). Our current approach as worked thus far but as we are expanding, it seems insufficient use of my server.
My goal is to allow user to directly upload to Google Cloud Storage bucket from the React app. I want to perform some validation logic before I export it to Google Healthcare API. I have looked into signed urls but since the files being uploaded are medical images I wasn't sure if they would be secure enough. The users don't necessarily have a google account.
What is the best way I can allow user to directly upload a directory to GCS bucket without going through my server? Are there dangers involved with this approach if the user uploaded a virus? Also signed urls are valid for a set amount of time, can I deactivate a signed url as soon the uploads are complete?
Does anyone have an example of a GAE/Cloud Endpoints API method (in Java) that can take in an image from an Android app and upload it to Google Cloud Storage?
I cannot seem to find any samples on how to do this but it is possible from what I understand.
EDIT:
The tutorial here shows how to add a dependency to google app engine in eclipse and upload/download an image to Google Cloud Storage. Is it possible to do this with Cloud Endpoints somehow..? After all, they are both Google App Engine.
I want to offload as much of the upload/download code into my Cloud Endpoints API method(s), rather than coding everything inside of Android. This would allow me to reuse my Cloud Endpoints API on other clients.
More info I found: https://developers.google.com/api-client-library/java/apis/storage/v1#sample
Looks like this is the gradle dependency for the cloud endpoints backend?:
dependencies {
compile 'com.google.apis:google-api-services-storage:v1-rev66-1.21.0'
}
EDIT:
You should use this dependency inside cloud endpoints:
compile 'com.google.appengine.tools:appengine-gcs-client:0.5'
You can upload file to Google Storage using Json Api
You may or may not want to store file metadata to datastore thru Endpoints.
You may or may not want to authenticate your users thru Endpoints before give them possibility to store files to Storage.
What I want to say is that Storage / Endpoints / Datastore are three different things and you don't required to use them all together.
Useful link: https://github.com/pliablematter/simple-cloud-storage
You cannot directly upload (large) files to an Endpoints API method but instead need to receive them using the blobstore (or GCS) (https://cloud.google.com/appengine/docs/python/blobstore/ ). This requires the following:
Setup on your server a blobstore upload handler (which is just a regular webapp2 handler).
Expose an Endpoints method that calls blobstore.create_upload_url(), and then returns the upload URL to your App.
Within the App, upload the picture to that upload URL; the file will then be accessible within your upload handler, where you can move it to GCS, Datastore or somewhere else.
Solution : https://github.com/thorrism/GoogleCloudExample
Enable Google Cloud Storage : => https://console.developers.google.com/apis
Generate and download a key P12 : => https://console.developers.google.com/iam-admin/iam
Create a folder named "assets" and place there your key :
=> app/src/main/assets/"yourKey.P12"
For everyone can access your uploaded file reading do not forget to add the permissions on your Bucket
I'd like to create an object, give my users an upload url, and let them upload data. The resulting object must be public-readable. Is this possible with google cloud storage? If so, is it possible through google app engine, and where can I find documentation and/or examples for doing it?
To have a user upload directly to Google Cloud Storage, you can use the Signed URLs feature. This allows you to grant access to issue a PUT request to an object to a single user.
If you're using Python, there is a python example demonstrating signed URLs.
You can create an upload url using the blobstore service. See the create_upload_url function.
To make the object publicly accessible you may need to play with the acls of the bucket.
See also the Cloud Storage Overview.
Another option to upload directly to Google Cloud Storage is Resumable URLs.
If your object is big, such as a video, you can upload it in chunks this way. If the upload fails (e.g. client loses internet connection), you can resume from where you left off and not have to have the user start over again. Plus you save some money by not having to restart that upload.
However if your media is small, just use Signed URLs.
I couldn't find articles match to my requirements.
Basically, what I want is that:
User uploads picture to the application from their local drive.
Application stores the picture uploaded to datastore.
Application retrieves images from datastore.
Any suggestions? Urgent.
That's exactly what is discussed in the documentation for the BlobStore API.
You can do this in much the same way as you would in any other framework or platform: Create an HTML form with a 'file' input and the mimetype set to 'multipart/form-data'. On the server side, extract the file data from the form field (using self.request.POST['fieldname'].value in webapp) and store the contents in a datastore model, in a db.BlobProperty field.
I have around 1500 images which are dynamically generated in my local server. I want these to upload to Appengine's datastore. How can i do this? any help, any ideas?
BlobStore
You can use the blobstore if you have a billing account.
So I assume to use the bolbstore API to upload it in chunks with a client tool over http.
Before every upload, you can ask the blobstore to give you an unique upload URL to post MIME multipart posts to it.
There is a size limit for requests I think.
Simple DataStore
If you store your images in the datastore, you can use a python tool somehow to synchronize.