I want to make client version of GAE app that store exact data of online version.(myapp.appspot.com) If i can use sdk instead, is any library or tools to sync online and sdk version? I try using bulkloader but i can't load downloaded data to local SDK? Please help.
As explained in this article (link updated, thanks to Zied Hamdi)
You simply need to enable the remote api
builtins:
- remote_api: on
Update your application then run the following commands:
appcfg.py download_data -A s~YOUR_APP_NAME --url=http://YOUR_APP_NAME.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data .
Edit for After April 12th 2016 on Latest AppEngine SDK:
The above works for SDK version 1.9.0 and before. However with the depreciation of ClientLogin, the above will cause an error of
03:13 PM Uploading data records.
[INFO ] Logging to bulkloader-log-20160909.151355
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20160909.151355.sql3
2016-09-09 15:13:55,175 INFO client.py:578 Refreshing due to a 401 (attempt 1/2)
2016-09-09 15:13:55,176 INFO client.py:804 Refreshing access_token
2016-09-09 15:13:55,312 INFO client.py:578 Refreshing due to a 401 (attempt 2/2)
Recommended by Anssi here, we can use the API server directly without running into this error. For a typical dev_appserver startup you get the following output
INFO 2016-09-09 19:27:11,662 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2016-09-09 19:27:11,899 api_server.py:205] Starting API server at: http://localhost:52497
INFO 2016-09-09 19:27:11,905 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO 2016-09-09 19:27:11,918 admin_server.py:116] Starting admin server at: http://localhost:8000
instead of the above for upload use the API port, in this case
appcfg.py --url=http://localhost:52497/_ah/remote_api/ --filename=data.csv upload_data .
See the docs for details on how to download and upload your entire datastore. Simply bulk download from production, then bulk upload to your local datastore.
Bear in mind, however, that the local datastore is not designed to handle large volumes of data - you may run into performance or memory issues.
Related
I'm fairly new to working with live projects.
My project runs perfectly on localhost, I deploy the exact same copy to google app engine using the command gcloud beta app deploy.
My welcome page works perfectly:
As Well as my auth pages:
straight after the auth process i get the following response:
To verify that the account has been authenticated my route url is redirecting to the dashboard:
example.com/admin/users
my app.yaml file is as follows:
runtime: php
env: flex
runtime_config:
document_root: public
# Ensure we skip ".env", which is only for local development
skip_files:
- .env
env_variables:
# Put production environment variables here.
APP_LOG: errorlog
APP_KEY: App-key
STORAGE_DIR: /tmp
CACHE_DRIVER: file
SESSION_DRIVER: file
## Set these environment variables according to your CloudSQL configuration.
DB_HOST: localhost
DB_DATABASE: lara
DB_USERNAME: root
DB_PASSWORD: password
DB_SOCKET: /cloudsql/connection-name
MAIL_MAILER: smtp
MAIL_HOST: smtp.mailtrap.io
MAIL_PORT: 2525
MAIL_USERNAME:username
MAIL_PASSWORD: password
MAIL_FROM_ADDRESS: from#example.com
MAIL_FROM_NAME: {App-Name}
#we need this for the flex environment
beta_settings:
# for Cloud SQL, set this value to the Cloud SQL connection name,
cloud_sql_instances: connection-name
Here is my log?
This is the view it is looking for:
My routes:
It's funny how small things create the biggest issues, took me over 3 weeks to resolve this issue.
As I was conducting research I discovered that Google App engine is case sensitive and so here are steps I used to resolve this issue:
1st I checked my routes using php artisan route:list and my route is route: admin.users.index and my file structure was --path: views/Admin/Users/index.blade.php and so i change all my folders to lower-case to match the route.
Then I ran the following commands:
php artisan cache:clear
php artisan route:clear
php artisan view:clear
Lastly I added the following script under scripts on my composer.json file:
"post-install-cmd": [
"chmod -R 755 bootstrap\/cache",
"php artisan cache:clear"
]
Deployed using gcloud app deploy
Worked like a charm.
In the local development environment, the upload_data command will not launch a browser for OAuth. Why is that?!?
The code works perfectly fine on App Engine but not the local development environment. Is there a trick to use the remote API for the dev environment.
Here's how I use the command...
appcfg.py upload_data --config_file=bulkloader.yaml --filename=./stops.txt --kind=StopLocationLoader --url=http://localhost:8082/_ah/remote_api
10:39 PM Uploading data records.
[INFO ] Logging to bulkloader-log-20161017.223916
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---
This is a bug: https://code.google.com/p/googleappengine/issues/detail?id=12445
It links to a workaround posted for another question:
gcloud auth login
gcloud auth print-access-token
appcfg.py upload_data --oauth2_access_token=<oauth2_access_token> --config_file bulkloader.yaml --url=http://<yourproject>.appspot.com/remote_api --filename places.csv --kind=Place --email=<you#gmail.com>
I used the following command to download my GAE data:
appcfg.py download_data --log_file=bulkloader.log --kind=MyKind --url=http://myappid.appspot.com/rmt_api --filename=myfilename --db_filename=MyKind_db.sql3 --result_db_filename=MyKind_result_db.sql3 --config_file=bulkloader.yaml
It worked well a few days ago, but yesterday it returned me access_token is expired. I manually deleted gae oauth files at /Users/myuser/ (I am on OSX), but it didn't help - now I get the following output:
09:51 PM Downloading data records.
[INFO ] Logging to bulkloader.log
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 302: --- begin server output ---
--- end server output ---
I tried to add --verbose and --noisy parms, but nothing changed.
In result I can not download the data. Upload of the application appcfg.py --update works well.
It looks like GAE issue, but all such issues are rejected.
It appears that the thread became confused by users posting potentially-unrelated material. You should open a new ticket linking to that one, and we'll be happy to take a look. We monitor the relevant tags here on Stack Overflow and Serverfault, along with several Google Groups forums and code.google.com Public Issue Trackers (along with the various GitHub projects), so you'll be sure to get a quick response.
I would like to download my local fixture data into a csv, to keep it safe.
I have tried something similar 6 months ago, with the only difference that I downloaded the data from a deployed appengine project, rather than downloading it from my local datastore.
However how could I download it from my local datastore?
appcfg.py download_data --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv --email=x#w.eu --application="dev~myApp"
I get the following error message:
03:37 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20140626.153708
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
[INFO ] Opening database: bulkloader-progress-20140626.153708.sql3
[INFO ] Opening database: bulkloader-results-20140626.153708.sql3
Password for x#w.eu:
[INFO ] Connecting to localhost:8080/_ah/remote_api/
[ERROR ] Unable to download kind stats for all-kinds download.
[ERROR ] Kind stats are generated periodically by the appserver
[ERROR ] Kind stats are not available on dev_appserver.
Any idea what I am doing wrong?
Go to the dev admin console (probably "localhost:8000")
Go to Datastore Stats
Click Generate Stats
This error occurs because the dev app engine does not automatically generate datastore statistics. Instead you must perform the steps above to manually generate these statistics.
I'm having issues downloading the deployed datastore down to my local directory.
The documentation seemed pretty straightforward
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Downloading_and_Uploading_All_Data
I'm a bit new to GAE... am I missing something obvious here?
I run the following command
appcfg.py download_data --url=http://myapp.appspot.com/_ah/remote_api --filename=ds_copy
I'm not even asked to authenticate,
and this is the output:
12:33 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20130507.123333
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Error 500: --- begin server output ---
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>If the problem persists, please report your problem and mention this error message and the query that caused it.</h2>
<h2></h2>
</body></html>
--- end server output ---
Any help is appreciated!
Have you configured the remote_api to be enabled in your app.yaml ?
https://developers.google.com/appengine/docs/python/tools/remoteapi
You simply need to enable the remote api
builtins:
- remote_api: on
Update your application then run the following commands:
appcfg.py download_data -A appName --url=http://appName.appspot.com/_ah/remote_api/ --filename=data.csv
appcfg.py --url=http://localhost:8080/_ah/remote_api/ --filename=data.csv upload_data <directory>
this taken from https://stackoverflow.com/a/7944641/2954800