I have an application hosted on GAE where I am creating files within directories which will then get passed to the user. By creating files I mean they are text or image files stored in the working directory or in some sub-directory. However when I create the file to be modified Google app engine recognizes that there was a file changed and will not proceed any further. Is there a way to turn off file change detection completely in my application?
INFO 2016-08-18 05:47:39,456 module.py:788] default: "POST /GatherContent HTTP/1.1" 500 2188
INFO 2016-08-18 05:47:40,467 module.py:402] [default] Detected file changes:
The preferred way to store files in GAE is using the blobstorage or cloud storage. So my guess would be no, it is not possible to turn off.
Related
In Google App Engine (GAE), files that get stored to the local Cloud Storage show up in the admin console with a path. Example:
/gs/myapp.appspot.com.somefile.jpg
This one seems to get closer:
http://localhost:8080/_ah/img/encoded_gs_file:somefile.jpg
But that generates an error:
Error 404 ApplicationError: 6: Could not read blob.
This one works but it requires I know the key:
http://localhost:8080/_ah/img/encoded_gs_key:some_key
Is there a way to use the local url but use the filename instead of a key?
I think you should go through the details of this GitHub Code about how to read and write blobs. The code confirms that for image files, you always require the keys.
For Images, you require the Key http://localhost:8080/_ah/img/encoded_gs_file:[Keys]
While for other files: https://localhost:8080/_ah/gcs/default_bucket/file_name
I installed Google Cloud SDK and it dumped a .boto file directory in to the My Documents folder (e.g. C:\Users\John) which is a wildly inappropriate location. I do see many instances of the boto file in the Python files, a couple of dozens of instances / examples:
return os.path.join(self.LegacyCredentialsDir(account), '.boto')
os.path.expanduser(os.path.join('~', '.boto')),
Where do I go to change the path to something appropriate? An appropriate path would be something such as C:\Users\John\AppData\Roaming\gcloud\.boto in example.
At the top of the file:
This file contains credentials and other configuration information needed
by the boto library, used by gsutil. You can edit this file (e.g., to add
credentials) but be careful not to mis-edit any of the variable names (like
"gs_access_key_id") or remove important markers (like the "[Credentials]" and
"[Boto]" section delimiters).
[Credentials]
Google OAuth2 credentials are managed by the Cloud SDK and
do not need to be present in this file.
To add HMAC google credentials for "gs://" URIs, edit and uncomment the
following two lines:
The latest versions of Boto don't seem to be a great fit for App Engine. I ran into this issue about a year ago, and I don't remember all of the details, but I avoided Boto3 and stuck with Boto 2.47 and that worked well for me.
For my use case, I only needed help with SES. If you need many other AWS services then YMMV.
On my appspot website, I use a third party API to query a large amount of data. The user then downloads the data in CSV. I know how to generate a csv and download it. The problem is that because the file is huge, I get the DeadlineExceededError.
I have tried tried increasing the fetch deadline to 60 (urlfetch.set_default_fetch_deadline(60)). It doesn't seem reasonable to increase it any further.
What is the appropriate way to tackle this problem on Google App Engine? Is this something where I have to use Task Queue?
Thanks.
DeadlineExceededError means that your incoming request took longer than 60 secs, not your UrlFetch call.
Deploy the code to generate the CSV file into a different module that you setup with basic or manual scaling. The URL to download your CSV will become http://module.domain.com
Requests can run indefinitely on modules with basic or manual scaling.
Alternately, consider creating a file dynamically in Google Cloud Storage (GCS) with your CSV content. At that point, the file resides in GCS and you have the ability to generate a URL from which they can download the file directly. There are also other options for different auth methods.
You can see documentation on doing this at
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/
and
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/functions
Important note: do not use the Files API (which was a common way of dynamically create files in blobstore/gcs) as it has been depracated. Use the above referenced Google Cloud Storage Client API instead.
Of course, you can delete the generated files after they've been successfully downloaded and/or you could run a cron job to expire links/files after a certain time period.
Depending on your specific use case, this might be a more effective path.
I'm trying to trouble shoot some issues I'm having with an export task I have created. I'm attempting to export CSV data using Google Cloud Storage and I seem to be unable to export all my data. I'm assuming it has something to do with the (FAR TOO LOW) 30 second file limit when I attempt to restart the task.
I need to trouble shoot, but I can't seem to find where my local/development server writing the files out. I see numerous entries in the GsFileInfo table so I assume something is going on, but I can't seem to find the actual output file.
Can someone point me to the location of the Google Cloud Storage files in the local AppEngine development environment?
Thanks!
Looking at dev_appserver code, looks like you can specify a path or it will calculate a default based on the OS you are using.
blobstore_path = options.blobstore_path or os.path.join(storage_path,
'blobs')
Then it passed this path to blobstore_stub (GCS storage is backed by blobstore stub), which seems to shard files by their blobstore key.
def _FileForBlob(self, blob_key):
"""Calculate full filename to store blob contents in.
This method does not check to see if the file actually exists.
Args:
blob_key: Blob key of blob to calculate file for.
Returns:
Complete path for file used for storing blob.
"""
blob_key = self._BlobKey(blob_key)
return os.path.join(self._DirectoryForBlob(blob_key), str(blob_key)[1:])
For example, i'm using ubuntu and started with dev_appserver.py --storage_path=~/tmp, then i was able to find files under ~/tmp/blobs and datastore under ~/tmp/datastore.db. Alternatively, you can go to local admin_console, the blobstore viewer link will also display gcs files.
As tkaitchuck mentions above, you can use the included LocalRawGcsService to pull the data out of the local.db. This is the only way to get the file, as they are stored in the Local DB using the blobstore. Here's the original answer:
which are the files uri on GAE java emulating cloud storage with GCS client library?
I've been using App Engine with Python for a few months. Now that my application has a fair amount of code, I'm trying to solve a problem I've ignored so far:
Each time I turn off my computer, all my development datastore entities are removed.
I would like to keep this data until the next time I launch my development server. But I would also like to be able to turn off my computer without losing all of this data.
How should I proceed?
Thanks a lot
======== UPDATE ==========
When I set the datastore_path flag as explained by #moishe, my development server crashes as soon as it must write into the datastore.
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_file_stub.py", line 557, in __WritePickled
os.rename(tmp_filename, filename)
OSError: [Errno 13] Permission denied
Therefore, I gave this folder all UNIX permissions
chmod a+w /my_app_folder
But I have now another error which is
OSError: [Errno 21] Is a directory
Obviously the path should not be a directory. So I changed the path to:
/my_app_folder/data.datastore
And now it works! PFF...
Maybe the default data store path is in a /tmp directory that's being deleted on shutdown? You can manually set the path with the --datastore_path flag in dev_appserver.py. See the docs for details.
This clearing should not be the default behavior.
Check that this application in the Google AppEngine launcher doesn't have the --clear_datastore flag.
Select app in list and select Edit->Applications Settings...
Extra Command Line Flags should be empty.
I once set this to restart some tests and forgot to remove it.
Remove the existing application in the launcher and Create New Application. See if that helps.
Verify the OS isn't deleted the file. If you open the log for the app, then launch it, the output says where the sqlite file is being located (e.g. T:\temp\dev_appserver.rdbms)
flag when starting the dev server:
--storage_path=...
Path at which all local files (such as the Datastore, Blobstore files,
Google Cloud Storage Files, logs, etc) will be stored, unless
overridden by --datastore_path, --blobstore_path, --logs_path, etc.
found at https://developers.google.com/appengine/docs/python/tools/devserver?csw=1
I had the same problem, and installing the latest gae SDK solved it.
As in the case here: app engine datastore auto-clears every time project runs