I use the Google App Engine Standard environment to develop my Python app using Development SDK 1.9.61.
I'm trying to learn to use Google Cloud Storage in my app by following these instructions. I verified that my default and staging buckets do exist via the cloud console, and manually uploaded a sample file to each bucket using my browser.
Next, I programmatically uploaded some files to a bucket (so I thought) via my local development app instance per Google's instructions.
However, when I checked my cloud storage buckets via my GCP Console in my browser, I could not find the files. After searching my local development SDK console, I eventually found the files located in the local "Blobstore Viewer".
I'm confused, based on Google's instructions I expected to find the files in my project's cloud storage bucket.
I searched the App Engine Python Release Notes for some potential SDK version changes to explain this behavior, but couldn't find anything relevant.
Is this the way it's supposed to work? Are Google's instructions in error?
If you upload files to a local development server, those exist in-memory on your machine. The GCP Console doesn't interact with your local development server, it interacts with the public (production) Google Cloud Storage API.
So in essence, the files on your local dev server are in a completely different namespace. If you want to interact with the production version of Google Cloud Storage and see the results in the GCP console, you'll need to use a non-dev-server deployment of your application.
Related
I installed the Cloud Storage Client Library in my local machine with composer.
My App Engine app uses StorageClient, like the sample code below, to write a file to a Cloud Storage Bucket.
require 'vendor/autoload.php';
use Google\Cloud\Storage\StorageClient;
$storage = new StorageClient();
$bucket = $storage->bucket('my_bucket');
// Upload a file to the bucket.
$bucket->upload(
fopen('/data/file.txt', 'r')
);
Are the libraries being copied to the respective App Engine instance when I deploy this app?
Was the authentication process/service account for copying the libraries down to my machine?
Is there a separate charge for using the libraries?
Are the libraries being copied to the respective App Engine instance when I deploy this app?
In a way they’re copied to the respective App Engine instance but not from your local machine.
This happens during the build stage of the deploy process and you can confirm it by viewing the build results. You can do that using the Google Cloud console or the Google Cloud SDK.
Here’s an example to view the build logs from the last deployment.
gcloud builds log $(gcloud builds list --format json | jq -r ".[0].id")
You can use it in Google Cloud Shell and also pipe it to grep like this:
gcloud builds log $(gcloud builds list --format json | jq -r ".[0].id") | grep composer
Is there a separate charge for using the libraries?
As explained in the documentation for App Engine Pricing, depending if you’re using the Environment Standard or Flexible, the billing rates differ.
With Standard Environment you choose an instance class and you’re billed per hour per instance accordingly while with Flexible Environment you’re billed for resources.
So in the former case you’d have to determine the amount of RAM memory you believe fits your needs while on the latter you can see more clearly the effect on your project’s billing of the amount of dependency libraries you have installed for your app.
Image1 available clearly shows that the bucket contains a folder namely cloudml-samples-master and a file namely setup.py
However on entering the command ls in Google Cloud Shell, it is giving no output. I am stuck at this problem and I have tried almost everything but I am unable to find the reason behind this.
Also when I am trying to access these files through my API, I get an error displaying that these files could not be found.
Both the files present in the directory were uploaded using WebUI.
To add my opinion, what I understood till now is that the files being uploaded using WebUI and those uploaded using command line are acting independently. Because I have seen that the files which I am uploading using command line are not showing in UI. Is there some error from my side or is it an issue related with Cloud Google Storage?
As you can see below from https://cloud.google.com/shell/docs/features, Google Cloud Shell is not connected directly with Google Cloud Storage. It gives you possibility to connect with that storage eg. via gsutil
When you start Cloud Shell, it provisions a g1-small Google Compute
Engine virtual machine running a Debian-based Linux operating system.
Cloud Shell instances are provisioned on a per-user, per-session
basis.
Also:
Cloud Shell provides the following:
A temporary Compute Engine virtual machine instance Command-line
access to the instance from a web browser Built-in code editor BETA
5 GB of persistent disk storage Pre-installed Google Cloud SDK and other tools Language support for Java, Go, Python, Node.js, PHP, Ruby
and .NET Web preview functionality Built-in authorization for access
to GCP Console projects and resources
I have a GAE based application that pulls a file from cloud storage and then does some processing on that file. To run the application in the remote appengine environment, I first upload a file to cloud storage using the browser console, and then make requests to the application, which pulls the file I uploaded from cloud storage. I'd like to be able to do development locally, however there is not a sweet browser console for the local implementation of gcs, as discussed here: Local storage browser for Google Cloud Storage and dev_appserver.py.
I'm wondering if it's possible to use gsutil. It seems the local gcs implementation is accessible through a localhost endpoint, mentioned here: Google Cloud Storage on Appengine Dev Server.
Right now, what I want to do is just load a file into my local gcs instance. I could do this my writing a little utility, but it seems much better to use gsutil if I can get that to connect to my local instance.
Thank you,
Ben
I'm developing a google app engine application that uses cloud storage. I want to have a base set of files on the cloud storage that are shared by each user of the application. I know I can use gsutil to copy these files to the production server.
But I would like to test my application on my local development server, so I need these files in the dev cloud storage as well. I can't find any way to copy the files. Is there a way to use gsutil to copy files to the development server's cloud storage simuation?
We don't currently support the full GCS API in the local dev server.
Your best bet is to probably just write to a different bucket when running locally for now.
I 'm building an AppEngine application that stores data in Google Cloud Storage. I use the Google Cloud Storage Client (GCS) library as suggested.
My app is working when deployed on AppEngine (reading/writing/listing objects) but I cannot make it work on the development server. The development server keeps returning error 404 and GCS raises NotFoundError. The dev-appserver is supposed to emulate the cloud storage functionality without any specific configurations etc. I see in the log files that the dev server is accepting requests at "/_ah/gcs" yet it seems that there is no handler for that url. I 've tried with version 1.8.5 and 1.8.6. Apart from my app, not even the demo app provided by Google works.
Is there something that I 'm missing here, e.g. a special configuration for the dev-appserver?
Sorry the following change was pushed out too early by mistake. It only works with 1.8.8 SDK. We are streamlining the release process of gcs client to align with SDK. Sorry
https://code.google.com/p/appengine-gcs-client/source/detail?r=125
Without this change, it works on 1.8.7 SDK.