I'm trying to use the bulkloader to load my data to the App-Engine server. I run the following command using Python 2.5:-
appcfg.py upload_data --application=myappname --kind=mykind
--filename=data_archive.csv --url=http://myappname.appspot.com/remote_api
But its failing with this Authentication error:-
[INFO ] Connecting to myappname.appspot.com/remote_api
[ERROR ] Exception during authentication
URLError: <urlopen error [Errno 10061] No connection could
be made because the target machine actively refused it>
[INFO ] Authentication Failed
My idea is to do a bulk download from my development server and then use this dump to do a upload to the app-engine server. The bulk download worked fine. I used this format for this:-
appcfg.py download_data --application=myappname --kind=mykind
--url=http://localhost:8888/remote_api --filename=data_archive.csv
But the bulk upoad fails. A couple of things: the bulk download asked me for a userid and password, but the bulk upload does not. Also, I don't currently have a app.yaml file which I see mentioned a lot - do I need one to do this ?
Thanks in advance for any help.
M.
EDIT
For anyone else struggling with this, the problem was indeed being behind the proxy server, but there was another 'error' with what is above. The app-id needs the "s~" bit added to it.
appcfg.py upload_data --application=s~myappname --kind=mykind
--filename=data_archive.csv --url=http://myappname.appspot.com/remote_api
This isn't an authentication issue - that message is a red-herring - your machine is unable to contact the App Engine app at all. Do you have a proxy you need to transit through in order to make external connections?
You do not need --application=s~myappname when using bulkloader - Google have mentioned before:
Warning! Do not use the --application= flag to get the application ID
when using the bulk loader. Instead, use --url=.
For more detail take a look here:
https://developers.google.com/appengine/docs/python/tools/uploadingdata
app.yaml is how it finds your server. I am not sure how you can try and upload without one.
In addition to having an app.yaml that points to the production server, the production server also needs to have remote_api turned on (in it's app.yaml and in the version you are trying to reach):
builtins:
- remote_api: on
Related
I am getting the following error while deploying the google app engine
ERROR: gcloud crashed (SSLHandshakeError): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
I am using python 2.7 also tried turning off firewall settings but doesnot help. Any suggestions?
This is a common network issue seen when there is a networking proxy present on your network or antivirus and similar software that might prevent the connection.
As you mentioned the issue was solved when deactivating an antivirus software. If you still want to run the antivirus, you can configure it properly to allow the connection to GCP.
I tried this to avoid SSL certificate validation and successfully worked
gcloud config set auth/disable_ssl_validation True
I have a set of data that I would like to use for production but I can't seem to get appcfg.py to download the localhost datastore. I have localhost:8080/_ah/remote_api on in app.yaml then, in the SDK shell, i ran
appcfg.py download_data --filename=local.db --url=http://localhost:8080/_ah/remote_api -A [MYAPPID]
I get this error message:
You must be logged in as an administrator to access this.
I can't find a way to authenticate my google account as the admin. any advice?
This is a general issue of remote_api admin authentication to the local development server, you may want to star it: https://code.google.com/p/googleappengine/issues/detail?id=12445. Several workaround suggestions in there (YMMV according to other posts)
The most recent one (2 days old) looks promising:
Workaround for appcfg.py upload_data to local dev server,
"Refreshing due to a 401" issue. Tested using Java dev server 1.9.42:
Log into the local dev server Development Console in a browser.
Open your browser's dev tools and pull the value for the dev_appserver_login cookie.
Edit lib/oauth2client/oauth2client/client.py in the app.
Find "def new_request" function. Before line self.apply(headers) in this function, add:
headers['Cookie'] = 'dev_appserver_login=your-cookie-value-here';
Run the upload_data command.
There's also this recipe, successful at the time, but it might be a bit old now:
appcfg.py shows You must be logged in as an administrator
Google App Engine SDK for PHP (at local environment) returns error on any try to use Google Cloud Storage. Error message is:
Fatal error: Uncaught exception 'google\appengine\runtime\RPCFailedError' with message 'Remote implementation for app_identity_service.GetAccessToken failed' in /media/data/home/vladimir/setup/gae/google_appengine/php/sdk/google/appengine/runtime/RemoteApiProxy.php on line 92
It exactly repeats the problem described here:
App engine update breaks CloudStorage in dev php env
Test code from the question above shows the same result.
I tried App Engine SDK for PHP versions 1.9.19, 1.9.20, 1.9.21 without success.
On Win10 this issue can be solved by generating an application-default credentials file:
D:\Workspace\Sourcecode>gcloud auth application-default login
Credentials saved to file: [C:\Users\Otje\AppData\Roaming\gcloud\application_default_credentials.json]
And then setting the environment in commandline:
D:\Workspace\Sourcecode>SET GOOGLE_APPLICATION_DEFAULT=C:\Users\Otje\AppData\Roaming\gcloud\application_default_credentials.json
Seems to me GAE on local just outputs limited error information when it can't find the right credentials to succesfully connect to remote endpoint.
It seems that it was GAE's server side issue. They fixed it. I discovered that it started to work as expected today without any changes applied by me.
I needed to test some changes on my local dev server before pushing to production. Doing so required having the full dataset on my local machine.
A colleague directed me to:
https://developers.google.com/appengine/docs/python/tools/uploadingdata?csw=1
I downloaded the data using an administrator's username and password, but unfortunately, I was unable to upload the data to my localhost "dev" app engine server.
Ran this command from the commandline:
appcfg.py upload_data --filename=../data/data1.dat --url=http://localhost:9080/_ah/remote_api ./
Where:
9080 was my app port on my localhost copy of the app
I was running this command from my app directory
Had the downloaded data stored in relative directory
../data/data1.dat
Received this error:
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: app "dev~appname" cannot access app "appname"'s data
UPDATE: It seems that the answer was as simple as adding the following to my upload_data call:
--application="dev~appname"
Thanks #DavidBennett.
ORIGINAL ANSWER: (which also works)
After a ton of searching on SO and code.google.com, the solution I found that worked was a comment on this question:
devappserver2, remote_api, and --default_partition
I used my original command as described in the question:
appcfg.py upload_data --filename=../data/data1.dat --url=http://localhost:9080/_ah/remote_api ./
The username and password I entered when prompted were my apps username (in my case, my email) and the corresponding password. (If that doesn't work you might want to try blank or test#example.com based on other comments I've read, but have not tested that theory.)
I also restarted my app engine with the following flag: (don’t forget to remove the flag the next time you restart the server) (You might want to try without using this flag, since I can’t confirm that it affects anything - I’m including it here, since it was a setting that I used.)
--clear_datastore=yes
The commenter recommends to delete “dev~” in your local server code on line 84 in this file:
google/appengine/tools/devappserver2/application_configuration.py, line 84
Where:
that base directory 'google' is located inside of:
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
assuming your GoogleAppEngineLauncher.app directory is in your Applications directory on your Mac
IMPORTANT: Restart your local app engine server for the changes to take effect.
http://code.google.com/appengine/docs/python/tools/uploadingdata.html
Here it is explained how to download data from a gAE app,
First thing to do is setting up remote_api.
The bulk loader tool communicates with your application running on App Engine using remote_api, a request handler included with the App Engine runtime environment that allows remote applications with the proper credentials to access the datastore remotely. There are two ways to install remote_api: automatically using the builtins directive, or you manually using the url directive.
I enabled it using builtins directive:
i changed app.yaml accordingly
builtins:
- remote_api: on
Its given that this directive finds "include.yaml" file for the remote_api and maps the request handler to /_ah/remote_api. Only administrators of the application can access this URL.
but i never came across include.yaml
after that i tried downloading data using the the commands given there
appcfg.py download_data --application=<app-id> --url=http://<appname>.appspot.com/[remote_api_path] --filename=<data-filename>
i'm getting an error saying permission denied, i'm confused. i am also not able to use "create_bulkloader_config" command, getting the same error, I'm confused, Thanks
Are you using open ID / federated login for your app? The remote API does not work with open ID, but there is a workaround here:
http://blog.notdot.net/2010/06/Using-remote-api-with-OpenID-authentication
Replace
builtins:
- remote_api: on
With
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin
You Should run the commandline as an admin user. The permission denied error you are getting refers to the appcfg script not being able to access a local file.