I have an older (Python 2.7) Google App Engine app that I recently deployed using Google Cloud Platform (gcloud), since appcfg was retired. I'm having issues using the images.get_serving_url not working for uploaded image blobs. The Images API is returning 'TransformationError'. This code worked previously for years in production before the switch to google cloud platform.
One quick thing to note is that Google Cloud Platform created a default storage bucket for blobs. Previously, I think GAE was using the Blobstore, so even though this change is transparent to developers, probably a lot has happened under the hood. I think for some reason my app is having issues creating the image url for the file stored on the cloud storage bucket.
I made a very simple test case to replicate the issue. Are there new requirements, or should this work?
class testUpload(webapp.RequestHandler):
def get(self):
uploadUrl = blobstore.create_upload_url('/testUploadHandler')
print("Generated upload url: %s" % uploadUrl)
self.response.out.write("""
<html><body>
<form action="{0}" method="POST" enctype="multipart/form-data">
Upload File: <input type="file" name="file"><br>
<input type="submit" name="submit" value="Submit">
</form>
</body></html>""".format(uploadUrl))
class testUploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
logging.info("testUploadHandler called.")
upload = self.get_uploads()[0]
blob_key = upload.key()
image_url = images.get_serving_url(blob_key)
self.redirect(image_url)
The deployed code returns the following error on the web server (local dev server is fine):
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~puzzleflow-dev/20210325t013659.433969250928848570/puzzleflow.py", line 4171, in post
image_url = images.get_serving_url(blob_key)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 1866, in get_serving_url
return rpc.get_result()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 615, in get_result
return self.__get_result_hook(self)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/fdc6d631da52d25b/python27/python27_lib/versions/1/google/appengine/api/images/__init__.py", line 1970, in get_serving_url_hook
raise _ToImagesError(e, readable_blob_key)
TransformationError
Also, you can try this yourself by going to this link (it should be publicly available):
https://puzzleflow-dev.appspot.com/testUpload
The issue was that I had switched my access control to Uniform, since Google says this is the 'recommended' setting. I'm not well-versed enough on the differences between Fine-Grained and Uniform, but switching it back to Fine-Grained fixed the problem.
Related
I am trying to pull my reports from YouTube reporting API and upload them into GCS. Since I must use oauth2, I used Google's guide on creating a service account and giving it domain-wide authority. While I can get authenticated service to pull the reports at user level. I get "The caller does not have permission" when appengine runs the request. Any idea where to look next?
SCOPES = ['https://www.googleapis.com/auth/yt-analytics-monetary.readonly',
'https://www.googleapis.com/auth/youtubepartner-content-owner-readonly',
'https://www.googleapis.com/auth/devstorage.read_write']
SERVICE_ACCOUNT_FILE = 'file.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_credentials = credentials.with_subject('email#email.com')
storage_client = build('storage', 'v1', credentials=credentials)
youtube_reporting = build('youtubereporting', 'v1',
credentials=credentials)
Traceback:
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~test-project-67412/20180712t103610.411083715789214207/main.py", line 66, in get
jobs = list_jobs(youtube_reporting, ContentOwner, True)
File "/base/data/home/apps/s~test-project-67412/20180712t103610.411083715789214207/main.py", line 52, in list_jobs
includeSystemManaged=True).execute()
File "/base/data/home/apps/s~test-project-67412/20180712t103610.411083715789214207/lib/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~test-project-67412/20180712t103610.411083715789214207/lib/googleapiclient/http.py", line 840, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 403 when requesting https://youtubereporting.googleapis.com/v1/jobs?onBehalfOfContentOwner=ContentOwner&alt=json&includeSystemManaged=true returned "The caller does not have permission">
--UPDATE--
So I guess it had to do with Domain-wide authority for service account. I deleted the account, created a new one, gave it domain-wide authority, and it works just fine now.
Below is a fragment of my code, i tried passing an instance variable from the class DiscussPage to anothe class quesAnsweredPage
class DiscussPage(webapp2.RequestHandler):
def post(self):
self.Ques_tit = "Content"
classname = questAnsweredPage()
classname.setValue(self.Ques_tit,self.Ques_tit)
class questAnsweredPage(webapp2.RequestHandler):
Can_I = "yes"
def setValue(self,que,title):
self.response.write(que)
The Code above returns the error:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "B:\Eclipse World\Google App Engine\App Engine Dev\Jambite_Extended\JambitesCurrent\Jambitesupdated by bel\main.py", line 2686, in post
classname.setValue(self.Ques_tit,self.Ques_tit)
File "B:\Eclipse World\Google App Engine\App Engine Dev\Jambite_Extended\JambitesCurrent\Jambitesupdated by bel\main.py", line 2774, in setValue
self.response.write(que)
AttributeError: 'NoneType' object has no attribute 'write'
How do i resolve this error
You need to look at the signature of classes based on webapp2.RequestHandler ( specifically the init method)
From the docs http://webapp-improved.appspot.com/api/webapp2.html#webapp2.RequestHandler we get
class webapp2.RequestHandler(request=None, response=None)[source]
So when you instantiate an instance of a class inheriting from RequestHandler you need to initialise it correctly.
You code should read
classname = questAnsweredPage(self.request,self.response)
Now why you would use a variable name classname to hold an instance of a class is a whole separate style issue ;-)
The error is probably because self.response can only be called inside a get() or post() method.
I'm trying merge 2 files and to create one new in the BlobStore of the AppEngine. This files are 12.000 rows and have 12 MB. The problem occurs when i writte the new file, because in middle of write appear the next error:
ApplicationError: 10
Traceback (most recent call last):
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 716, in __call__
handler.post(*groups)
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 244, in post
self.principal(cont+1, numFicheritos,"csvTotal","uniendoCSV")
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 277, in principal
self.escribirtxt(f, cadenaSalida)
File "/base/data/home/apps/s~ono-hat-vv2/1.369681903434492066/com/__init__.py", line 308, in escribirtxt
f.write(linea)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 326, in write
self._make_rpc_call_with_retry('Append', request, response)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 427, in _make_rpc_call_with_retry
_make_call(method, request, response)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 252, in _make_call
_raise_app_error(e)
File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/files/file.py", line 195, in _raise_app_error
raise FileNotOpenedError(e)
FileNotOpenedError: ApplicationError: 10
I have read that the problem is in the time that I open the file, because this can't be greater that 30 s. How can i resolve this problem and merge this file?
Close the file and re-open every so often.
Have you looked at Google Cloud Storage? It may not have this limitation and also the app engine files api has been deprecated and we need to move in that direction anyway. Something to consider.
I've been having some trouble with the memcache viewer after updating the Python Dev Appserver to Google App Engine 1.7.6 (with Python 2.7).
It appears that my memcache isn't updated or isn't readable. I have tried to view memcache with the app engine memcache viewer but when I input the memcache key I get an error.
when I flush the cache everything proceeds as normal until memcache needs to be read again...
The hit ratio and memcache size increases as normal, so there is something in the cache. Also when I revert back to app engine 1.7.5 everything works just fine. Perhaps someone else has had this issue?
When I input the memcache key I get the following:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\admin\admin_request_handler.py", line 80, in dispatch
super(AdminRequestHandler, self).dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.1\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\admin\memcache_viewer.py", line 145, in get
values['value'], values['type'] = self._get_memcache_value_and_type(key)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\admin\memcache_viewer.py", line 74, in _get_memcache_value_and_type
except (pickle.UnpicklingError, AttributeError, EOFError, ImportError,
NameError: global name 'pickle' is not defined
I tried including an "import pickle" in my main.py but this was in vain.
I've included some samples of my code but hopefully this isn't necessary I hope its something more to do with the app engine update than my code...
some of my main.py file:
#import pickle
from google.appengine.api import memcache
from google.appengine.ext import db
and a sample function for how I handle memcache:
def mc_get(key):
a = memcache.get(key)
if a:
val = a
else:
val = None
return val
def mc_set(key, val):
memcache.set(key, val)
and if I want to query the users in my db I use:
def get_users(update=False):
mc_key = 'USERS'
entries = mc_get(mc_key)
if update or entries is None:
a = User.all()
logging.error('DB---Q - Users')
entries = list(a)
memcache.set(mc_key, entries)
return entries
UPDATE:
I added "import pickle" to the memcache_viewer.py file in Google\google_appengine\google\appengine\tools\devappserver2\admin\memcache_viewer.py
(is this a bug??)
and now when I type in a memcache key I get the following error under the memcache key input field:
Error fetching USERS: Failed to retrieve value from cache: No module named main
Any help would be greatly appreciated, thanks in advance.
I changed from the old datastore API to NDB(a bit of a chore changing the code). The automatic caching seems to have solved the problem, this may suggest that the problem was with my code, but still doesn't explain why everything worked fine when using app engine 1.7.5 and not with 1.7.6.
I'll remove this answer if anyone has an alternative, just thought I'd post my progress in case anyone else is having the same problem.
I am trying to use this script for federated login with google's app engine. This works on the deployed app engine but with localhost it gives this error:
Traceback (most recent call last):
File "/workspace/python/app_en/google_appengine/google/appengine/ext/webapp/__init__.py", line 515, in __call__
handler.get(*groups)
File "/workspace/python/app_en/bingousie/helloworld.py", line 23, in get
p['login_url'] = users.create_login_url(federated_identity=p_url)
File "/workspace/python/app_en/google_appengine/google/appengine/api/users.py", line 216, in create_login_url
apiproxy_stub_map.MakeSyncCall('user', 'CreateLoginURL', req, resp)
File "/workspace/python/app_en/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 86, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "/workspace/python/app_en/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 286, in MakeSyncCall
rpc.CheckSuccess()
File "/workspace/python/app_en/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
self.request, self.response)
File "/workspace/python/app_en/google_appengine/google/appengine/api/apiproxy_stub.py", line 73, in MakeSyncCall
if request.ByteSize() > self.__max_request_size:
File "/workspace/python/app_en/google_appengine/google/appengine/api/user_service_pb.py", line 178, in ByteSize
n += self.lengthString(len(self.destination_url_))
TypeError: object of type 'NoneType' has no len()
Is this because the openid servers do not allow logging in to local site? If it is then it why doesn't it at least work with myopenid.com (I use it for my django openid authentication on local server).
In the create_login_url call you should specify the dest_url parameter.
dest_url can be full URL or a path
relative to your application's domain.
I had the same problem here.