I have a GAE based application that pulls a file from cloud storage and then does some processing on that file. To run the application in the remote appengine environment, I first upload a file to cloud storage using the browser console, and then make requests to the application, which pulls the file I uploaded from cloud storage. I'd like to be able to do development locally, however there is not a sweet browser console for the local implementation of gcs, as discussed here: Local storage browser for Google Cloud Storage and dev_appserver.py.
I'm wondering if it's possible to use gsutil. It seems the local gcs implementation is accessible through a localhost endpoint, mentioned here: Google Cloud Storage on Appengine Dev Server.
Right now, what I want to do is just load a file into my local gcs instance. I could do this my writing a little utility, but it seems much better to use gsutil if I can get that to connect to my local instance.
Thank you,
Ben
Related
I use the Google App Engine Standard environment to develop my Python app using Development SDK 1.9.61.
I'm trying to learn to use Google Cloud Storage in my app by following these instructions. I verified that my default and staging buckets do exist via the cloud console, and manually uploaded a sample file to each bucket using my browser.
Next, I programmatically uploaded some files to a bucket (so I thought) via my local development app instance per Google's instructions.
However, when I checked my cloud storage buckets via my GCP Console in my browser, I could not find the files. After searching my local development SDK console, I eventually found the files located in the local "Blobstore Viewer".
I'm confused, based on Google's instructions I expected to find the files in my project's cloud storage bucket.
I searched the App Engine Python Release Notes for some potential SDK version changes to explain this behavior, but couldn't find anything relevant.
Is this the way it's supposed to work? Are Google's instructions in error?
If you upload files to a local development server, those exist in-memory on your machine. The GCP Console doesn't interact with your local development server, it interacts with the public (production) Google Cloud Storage API.
So in essence, the files on your local dev server are in a completely different namespace. If you want to interact with the production version of Google Cloud Storage and see the results in the GCP console, you'll need to use a non-dev-server deployment of your application.
I'm developing a google app engine application that uses cloud storage. I want to have a base set of files on the cloud storage that are shared by each user of the application. I know I can use gsutil to copy these files to the production server.
But I would like to test my application on my local development server, so I need these files in the dev cloud storage as well. I can't find any way to copy the files. Is there a way to use gsutil to copy files to the development server's cloud storage simuation?
We don't currently support the full GCS API in the local dev server.
Your best bet is to probably just write to a different bucket when running locally for now.
I am writing an application that will be running on GCE and am using the google-api-go-client from https://code.google.com/p/google-api-go-client/ to store data in GCS.
I have a GAE application running as well that the GCE application communicates with.
I can run my GCE application locally and communicate with the live GCS servers and store files and retrieve them.
Now I am trying to locally store files in GCS so that I can run a test environment.
I see that devappserver supports simulating GCS via blobstore. However I can't seem to figure out how to point my GCE application at the local devappserver for GCS storage.
I am running a devappserver through 'goapp serve' with my GAE application.
I have tried setting the basepath and that results in 500 errors from the devappserver.
service, err := storage.New(httpClient)
service.BasePath = "http://localhost:8080/_ah/gcs/"
results in
ERROR 2014-10-23 19:14:08,692 module.py:727] Request to '/_ah/gcs/b/default/o/6051711999279104' failed
Traceback (most recent call last):
File "/home/ben/go_appengine/google/appengine/tools/devappserver2/module.py", line 723, in _handle_request
return handler.handle(match, environ, wrapped_start_response)
File "/home/ben/go_appengine/google/appengine/tools/devappserver2/wsgi_handler.py", line 59, in handle
return self._wsgi_app(environ, start_response)
File "/home/ben/go_appengine/google/appengine/tools/devappserver2/gcs_server.py", line 56, in __call__
status_message = httplib.responses.get(e.args[1], '')
IndexError: tuple index out of range
INFO 2014-10-23 19:14:08,693 module.py:652] default: "GET /_ah/gcs/b/default/o/6051711999279104?alt=json HTTP/1.1" 500 -
I can point my browser to this URL
http://localhost:8080/_ah/gcs/bucket
and get an XML result:
<ListBucketResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
<IsTruncated>False</IsTruncated>
</ListBucketResult>
I have seen several posts about using GCS locally but I can't seem to get it working correctly. Is there something I am doing wrong here? Or does devappserver not support GCS or maybe its just the json api it doesn't support?
I want to make sure we're clear on the distinction between the "Google APIs Client Library for Go", which is a general-purpose Google API client for Go that can be used to access the Google Cloud Storage JSON API, and App Engine's "Google Cloud Storage Go Client", which is a client library specific to App Engine users who want to use Google Cloud Storage.
There are two Google Cloud Storage APIs: JSON and XML. The general purpose Google APIs Client Library is compatible with the JSON API. The App Engine Google Cloud Storage Go Client speaks XML API.
The App Engine client libary fully supports local operations. This is accomplished by providing a local implementation of the Google Cloud Storage XML API, which the client library uses. Unfortunately, only the XML API exists locally, and thus only the App Engine client library works locally. Pointing the other client to it will fail because the XML API doesn't understand the incoming JSON API requests.
Your two best options are to do one of the following:
Use the Google Cloud Storage Go Client Library for App Engine. Download instructions are here.
Instead of using a local implemention, use the real Google Cloud Storage with a test bucket.
I am looking to use Google Cloud Storage services outside of launching a google app engine application. My application is designed for running on a hosted server of our own, and I would like to still use the cloud storage service.
I've written a dev app that did this, and ran it with dev_appserver.py in the python GAE framework. It seems the dev_appserver.py is necessary as it fills in how to perform the RPC functions the cloud storage provides.
Is there a way to easily use google cloud storage service without the rest of it?
There is gsutil command line tool:
https://developers.google.com/storage/docs/gsutil
There is an xml/json restful service all at:
https://developers.google.com/storage/index
You'll need to enable the google cloud storage on the google console at:
https://code.google.com/apis/console/ and get the necessary api keys.
I would like to know if there is a way to upload a big file (>500MB) tp Google Cloud Storage passing through a Google App Engine application: I suppose that it is not possible because of the GAE servlet limitation (execution time <=60s).
Is it correct or do you know some new trick?
What do you mean by "passing through" an application?
You can use createUploadUrl and set a destination Google Storage bucket in the UploadOptions and the blob will be written to Google storage rather than blobstore, and your callback will be invoked when the upload is complete.
When you upload a file, it's between your client and the Cloud Storage. There should be no request to your front-end instance hanging until the upload is complete.
That being said, if for some reason your front-end should get involved as a "pass-through" link, there is no limit on a servlet that runs on the backend instance. You can link your upload widget to your backend instance, i.e. call myBackend1.myApp.appspot.com. You can read on how to configure the backend in GAE documentation.
Another option is to use Google Cloud Storage's direct upload mechanism:
https://developers.google.com/storage/docs/reference-methods#postobject
Did you already consider that as an option?