I'm having issues downloading the deployed datastore down to my local directory.
The documentation seemed pretty straightforward
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Downloading_and_Uploading_All_Data
I'm a bit new to GAE... am I missing something obvious here?
I run the following command
appcfg.py download_data --url=http://myapp.appspot.com/_ah/remote_api --filename=ds_copy
I'm not even asked to authenticate,
and this is the output:
12:33 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20130507.123333
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 500: --- begin server output ---
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>If the problem persists, please report your problem and mention this error message and the query that caused it.</h2>
<h2></h2>
</body></html>
--- end server output ---
Any help is appreciated!
Have you configured the remote_api to be enabled in your app.yaml ?
https://developers.google.com/appengine/docs/python/tools/remoteapi
You simply need to enable the remote api
builtins:
- remote_api: on
Update your application then run the following commands:
appcfg.py download_data -A appName --url=http://appName.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data <directory>
this taken from https://stackoverflow.com/a/7944641/2954800
Related
In the local development environment, the upload_data command will not launch a browser for OAuth. Why is that?!?
The code works perfectly fine on App Engine but not the local development environment. Is there a trick to use the remote API for the dev environment.
Here's how I use the command...
appcfg.py upload_data --config_file=bulkloader.yaml --filename=./stops.txt --kind=StopLocationLoader --url=http://localhost:8082/_ah/remote_api
10:39 PM Uploading data records.
[INFO ] Logging to bulkloader-log-20161017.223916
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---
This is a bug: https://code.google.com/p/googleappengine/issues/detail?id=12445
It links to a workaround posted for another question:
gcloud auth login
gcloud auth print-access-token
appcfg.py upload_data --oauth2_access_token=<oauth2_access_token> --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<you#gmail.com>
I used the following command to download my GAE data:
appcfg.py download_data --log_file=bulkloader.log --kind=MyKind --url=http://myappid.appspot.com/rmt_api --filename=myfilename --db_filename=MyKind_db.sql3 --result_db_filename=MyKind_result_db.sql3 --config_file=bulkloader.yaml
It worked well a few days ago, but yesterday it returned me access_token is expired. I manually deleted gae oauth files at /Users/myuser/ (I am on OSX), but it didn't help - now I get the following output:
09:51 PM Downloading data records.
[INFO ] Logging to bulkloader.log
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 302: --- begin server output ---
--- end server output ---
I tried to add --verbose and --noisy parms, but nothing changed.
In result I can not download the data. Upload of the application appcfg.py --update works well.
It looks like GAE issue, but all such issues are rejected.
It appears that the thread became confused by users posting potentially-unrelated material. You should open a new ticket linking to that one, and we'll be happy to take a look. We monitor the relevant tags here on Stack Overflow and Serverfault, along with several Google Groups forums and code.google.com Public Issue Trackers (along with the various GitHub projects), so you'll be sure to get a quick response.
I would like to download my local fixture data into a csv, to keep it safe.
I have tried something similar 6 months ago, with the only difference that I downloaded the data from a deployed appengine project, rather than downloading it from my local datastore.
However how could I download it from my local datastore?
appcfg.py download_data --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv --email=x#w.eu --application="dev~myApp"
I get the following error message:
03:37 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20140626.153708
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20140626.153708.sql3
[INFO ] Opening database: bulkloader-results-20140626.153708.sql3
Password for x#w.eu:
[INFO ] Connecting to localhost:8080/_ah/remote_api/
[ERROR ] Unable to download kind stats for all-kinds download.
[ERROR ] Kind stats are generated periodically by the appserver
[ERROR ] Kind stats are not available on dev_appserver.
Any idea what I am doing wrong?
Go to the dev admin console (probably "localhost:8000")
Go to Datastore Stats
Click Generate Stats
This error occurs because the dev app engine does not automatically generate datastore statistics. Instead you must perform the steps above to manually generate these statistics.
I've got a Java appengine app with remote_api installed as per http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-engine/
<servlet>
<servlet-name>RemoteApi</servlet-name>
<servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>RemoteApi</servlet-name>
<url-pattern>/remote_api</url-pattern>
</servlet-mapping>
When I go to myapp.appspot.com/remote_api with a web browser, I see the message "This request did not contain a necessary header," which I understand is expected.
But when I run appcfg.py create_bulkloader_config --url=http://APPID.appspot.com/remote_api --application=APPID --filename=config.yml from my command line (with the proper APPID) I get
C:\ag\dev>appcfg.py create_bulkloader_config --url=https://correctid.appspot.c
om/remote_api --application=correctid --filename=config.yml
Creating bulkloader configuration.
[INFO ] Logging to bulkloader-log-20101114.081901
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20101114.081901.sql3
[INFO ] Opening database: bulkloader-results-20101114.081901.sql3
[INFO ] Connecting to correctid.appspot.com/remote_api
[INFO ] Authentication Failed
C:\ag\dev>
I've already tried the no_cookies option, which didn't help. I also tried using -e correctadminmail#gmail.com. Neither of these changed my output at all.
How can I specify my authentication parameters?
This might happen if your app is configured to use OpenID for logins - OpenID isn't compatible with remote_api.
This blog post describes the problem and a solution:
http://blog.notdot.net/2010/06/Using-remote-api-with-OpenID-authentication
The solution is in Python though - you might be able to do something equivalent in Java (or upload some Python code to a different version of your app, just for the remote api).
I want to make client version of GAE app that store exact data of online version.(myapp.appspot.com) If i can use sdk instead, is any library or tools to sync online and sdk version? I try using bulkloader but i can't load downloaded data to local SDK? Please help.
As explained in this article (link updated, thanks to Zied Hamdi)
You simply need to enable the remote api
builtins:
- remote_api: on
Update your application then run the following commands:
appcfg.py download_data -A s~YOUR_APP_NAME --url=http://YOUR_APP_NAME.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data .
Edit for After April 12th 2016 on Latest AppEngine SDK:
The above works for SDK version 1.9.0 and before. However with the depreciation of ClientLogin, the above will cause an error of
03:13 PM Uploading data records.
[INFO ] Logging to bulkloader-log-20160909.151355
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20160909.151355.sql3
2016-09-09 15:13:55,175 INFO client.py:578 Refreshing due to a 401 (attempt 1/2)
2016-09-09 15:13:55,176 INFO client.py:804 Refreshing access_token
2016-09-09 15:13:55,312 INFO client.py:578 Refreshing due to a 401 (attempt 2/2)
Recommended by Anssi here, we can use the API server directly without running into this error. For a typical dev_appserver startup you get the following output
INFO 2016-09-09 19:27:11,662 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2016-09-09 19:27:11,899 api_server.py:205] Starting API server at: http://localhost:52497
INFO 2016-09-09 19:27:11,905 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO 2016-09-09 19:27:11,918 admin_server.py:116] Starting admin server at: http://localhost:8000
instead of the above for upload use the API port, in this case
appcfg.py --url=http://localhost:52497/_ah/remote_api/ --filename=data.csv upload_data .
See the docs for details on how to download and upload your entire datastore. Simply bulk download from production, then bulk upload to your local datastore.
Bear in mind, however, that the local datastore is not designed to handle large volumes of data - you may run into performance or memory issues.