I've got a fairly old AppEngine (python) application that has been running for several years without issue.
I use task queues for move images into Google Cloud Storage.
My code is using the GCS client (https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/setting-up-cloud-storage) to do this and basically looks like:
# open the file and write to it using the gcs client
f = gcs.open(filename, 'w')
f.write(blob)
f.close()
NB: The filename includes the bucket so it's effectively '/[bucketname]/abc123'
When I "trace' the issue using the tools provided by the cloud console I can see that a 403 error is being thrown via URLFetch when it calls "storage.googleapis.com".
I've been through the process of adding IAM permissions to the bucket for the particular app-engine (xyz#appspot.gserviceaccount.com) service account. Have changed this account to be a "Storage Admin" but it doesn't seem to have made any difference. I've also added this role via the permissions on bucket itself.
So I'm not sure why it's not working...
Does anybody have an advice as to what I might be doing wrong? This all seems a bit confusing really.
Thanks for any help in advance,
Matt
Related
I have an ASP.NET Core web application hosted on the GoogleCloud App Engine.
The application has a function that creates and displays a PDF.
Locally it works, however in the GCP it is giving permission denied according to the image:
enter image description here
Can someone please help me?
I will post my comment as an answer, as it is too long for the comment field.
“Permission Denied” errors typically mean the caller doesn't have the right Google Cloud IAM permission on certain resources.
When developing for GAE you have to check whether the libraries you are utilizing within your ASP.NET app are compatible with GAE and additionally authorize the requests performed to GAE.
Does the pdf document get stored in a Database? If yes, does your application have all permissions required to read and write files to the Database?
Could you please specify your configurations, as well as provide the .yaml file and the corresponding code of your application for further clarifications?
Our production app (python 2.7 standard environment) running on Google App Engine suddenly lost permissions to write (create objects) on Google Cloud Storage, without any change in the code on our side.
The code is able to create new buckets, but not new objects within them.
It seems that the default app engine service account is not granted the permission.
Needless to say, the service account has the Storage Object Creator role, as well as the Editor role on the project level.
Strangely, the exact same code running on the test environment project, continues to work perfectly.
We are using the api client library to obtain credentials, like so:
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build as discovery_build
credentials = AppAssertionCredentials(scope=scope)
http = credentials.authorize(httplib2.Http())
service = discovery_build('storage', 'v1', http=http)
And then using the service to make the api call.
All calls to create objects are suddenly failing with the message:
"Anonymous caller does not have storage.objects.create access to /"
Any ideas what could suddenly have gone wrong ??
This turned up to be an issue with Google Cloud Storage (GCS).
A payed support ticket was opened, after approximately 90 hours, a rollback was made by Google GCS engineers which solved the issue, however, the root cause of the issue was not found or reported.
Very troubling that a production app can be affected this way for such a long time and eventually there is no explanation.
I managed to get the "Quickstart for Python 3 in the App Engine Standard Environment" example up and running, and I thought I'd try and further my knowledge a little, perhaps by attempting to get a cron job running.
So I updated the python code, adding another endpoint, counter, like this:
#app.route('/counter')
def counter():
with open('counter.txt', 'a') as the_file:
the_file.write('Hello\n')
return 'Counter incremented'
I intend to have the cron job periodically hit /counter. When this endpoint is hit it will open the file and add a line to it. The /counter endpoint works on my local machine. After I deploy this updated code to the Google cloud, if I go to my blahblah.appspot.com/counter url it should update this 'counter.txt' file.
My question is: How do I see that file to know if it is being updated or not? How do I view that file in the cloud? Thanks.
It not possible to write files in the Google App Engine Standard Python3 environment except in the /tmp directory. As stated in the Python3 GAE official documentation:
The runtime includes a full filesystem. The filesystem is read-only except for the location /tmp, which is a virtual disk storing data in your App Engine instance's RAM.
I agree with #Josh J answer, you should use Google Cloud Storage instead.
You shouldn't write to the local file system in Google App Engine. It may or may not be visible to your app when it scales.
Google Cloud Storage is the preferred method of file storage.
https://cloud.google.com/appengine/docs/standard/python3/using-cloud-storage
For python 2.7 you can import a module from string instead of file, you can see in this answer:
https://stackoverflow.com/a/7548190/8244338
I studied and could successfully replicate the quickstart.py example on https://developers.google.com/drive/web/quickstart/quickstart-python to upload a file to my google drive using command line.
However, I wish to write an app that does the same, but through a cron job i.e. uploads a file everyday at 8am say, without the need to authenticate each time. Is there sample code/examples that I can look at to implement the oauth steps without the command line intervention?
Thanks!
You can use your App Engine app's built-in Service Account to authorize requests to the Google Drive API.
https://cloud.google.com/appengine/docs/python/appidentity/
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Your app will need to have an embedded Refresh Token, or some way of fetching it from a secure server. The Refresh Token acts a bit like a stored username/password, albeit with constrained access. Therefore you need to consider the security implications. For example, since it's uploading, it will only need drive.file scope, so your corpus of Drive files remain inaccessible.
If you're happy with the security implications, then the steps you need are described How do I authorise an app (web or installed) without user intervention? (canonical ?)
I am new to Google App engine and I have tried to run an demo application called guestbook to connect to Google cloud sql from the Google app engine with app-engine-sdk version 1-7.0. But each time I am getting an error saying "java.lang.IllegalStateException: System property rdbms.driver must be set at com.google.appengine.api.rdbms.dev.LocalRdbmsServiceLocalDriver.registerDriver(LocalRdbmsServiceLocalDriver.java:80)". I double check my code and every thing looks ok, and I still have no glue where the error coming from.
Below is a snippet of my connection code :
c = DriverManager.getConnection("jdbc:google:rdbms://my_instance/my_database");
and mysql-connector-java-5.1.21-bin is in the class path,
and I have enable Google cloud sql in the Google app engine,
and I have checked the use of Google Cloud instance in the app engine as well with the my instance of the database, database name, login , and password,
and I am using Eclipse Juno.
I think I have missed something important; so would you please help me if you know what I have missed.
Thank you very much in advance,
Minh
Once you are new to GAE I recommend to give a try on the Big Table database. Using it you will not have to setup any database localy. So just the eclipse plugin will be enough for your first test. The Guestbook uses this database so it will be easier to follow the tutorial.