http://code.google.com/appengine/docs/python/tools/uploadingdata.html
Here it is explained how to download data from a gAE app,
First thing to do is setting up remote_api.
The bulk loader tool communicates with your application running on App Engine using remote_api, a request handler included with the App Engine runtime environment that allows remote applications with the proper credentials to access the datastore remotely. There are two ways to install remote_api: automatically using the builtins directive, or you manually using the url directive.
I enabled it using builtins directive:
i changed app.yaml accordingly
builtins:
- remote_api: on
Its given that this directive finds "include.yaml" file for the remote_api and maps the request handler to /_ah/remote_api. Only administrators of the application can access this URL.
but i never came across include.yaml
after that i tried downloading data using the the commands given there
appcfg.py download_data --application=<app-id> --url=http://<appname>.appspot.com/[remote_api_path] --filename=<data-filename>
i'm getting an error saying permission denied, i'm confused. i am also not able to use "create_bulkloader_config" command, getting the same error, I'm confused, Thanks
Are you using open ID / federated login for your app? The remote API does not work with open ID, but there is a workaround here:
http://blog.notdot.net/2010/06/Using-remote-api-with-OpenID-authentication
Replace
builtins:
- remote_api: on
With
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin
You Should run the commandline as an admin user. The permission denied error you are getting refers to the appcfg script not being able to access a local file.
Related
I currently have a small Google App Engine project written in PHP. On a traditional web server, I would simply invoke this particular code via command line (e.g., php whatever.php). I'd like to set this up with cron.yaml to just run every hour or so without invoking a HTTP request if possible. How would you go about doing this?
You can do this by adding a handler url that points to your script as follows:
In your app.yaml it would look like this:
handlers:
- url: /mycron
script: cron.php
login: admin
The login parameter will only allow execution of the endpoint by Google system accounts or something like that so you enpoint won't be publicly accessible.
More information in the below link
ref: https://cloud.google.com/appengine/docs/standard/php/config/cron#securing_urls_for_cron
In order to schedule tasks using the cron.yaml file, otherwise known as cron jobs, you can use the following structure inside the file:
cron:
- description: "running my PHP code"
url: /your-app-url
target: your-service
schedule: every 60 minutes
You can edit the different cron fields to your convenience by following this syntax.
Place the cron.yaml file on the same directory as your app.yaml file (your application's root directory) and, before deploying, test it by going to http://localhost:8080/cron. If it works, you can deploy the application with the cron job by running this command:
gcloud app deploy cron.yaml
You can find additional information about cron jobs, such as how to retry failing cron jobs and securing or deleting them by following this link.
I have a set of data that I would like to use for production but I can't seem to get appcfg.py to download the localhost datastore. I have localhost:8080/_ah/remote_api on in app.yaml then, in the SDK shell, i ran
appcfg.py download_data --filename=local.db --url=http://localhost:8080/_ah/remote_api -A [MYAPPID]
I get this error message:
You must be logged in as an administrator to access this.
I can't find a way to authenticate my google account as the admin. any advice?
This is a general issue of remote_api admin authentication to the local development server, you may want to star it: https://code.google.com/p/googleappengine/issues/detail?id=12445. Several workaround suggestions in there (YMMV according to other posts)
The most recent one (2 days old) looks promising:
Workaround for appcfg.py upload_data to local dev server,
"Refreshing due to a 401" issue. Tested using Java dev server 1.9.42:
Log into the local dev server Development Console in a browser.
Open your browser's dev tools and pull the value for the dev_appserver_login cookie.
Edit lib/oauth2client/oauth2client/client.py in the app.
Find "def new_request" function. Before line self.apply(headers) in this function, add:
headers['Cookie'] = 'dev_appserver_login=your-cookie-value-here';
Run the upload_data command.
There's also this recipe, successful at the time, but it might be a bit old now:
appcfg.py shows You must be logged in as an administrator
This has worked for years. Today is not working.
Running appcfg.py download_data --flags..., and cannot get properly authorized.
Ran gcloud auth login
and got back:
Saved Application Default Credentials.
You are now logged in as [correct#gmail.com].
Your current project is [correct].
but when I run appcfg.py download_data ... I get this error:
raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The
Application Default Credentials are not available. They are available if
running in Google Compute Engine. Otherwise, the environment variable
GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file
defining the credentials. See
https://developers.google.com/accounts/docs/application-default
credentials for more information.
I updated the Google Cloud SDK, and now get a different error:
client.py:539 Attempting refresh to obtain initial access_token
client.py:797 Refreshing access_token
Error 302: --- begin server output ---
--- end server output ---
I have also set the credentials in Terminal:
set GOOGLE_APPLICATION_CREDENTIALS "https://www.myurl/credentials.json"
set CLOUDSDK_PYTHON_SITEPACKAGES 1
gcloud config set project "correct_project_name"
I have also tried setting the env_variable flag in the appcfg command:
--env_variable=GOOGLE_APPLICATION_CREDENTIALS:"https://www.myurl/credentials.json"
Seems to be a GAE bug. Removing the login:admin from the remote_api handler in app.yaml fixes it in production, but not local dev server. The 302 was telling me that the app was trying to redirect, perhaps to a Google login page. Was never seeing the prompt asking for email and password.
handlers:
- url: /remote_api
script: google.appengine.ext.remote_api.handler.application
#login: admin <-- removing admin fixes it in production
secure: always
On the dev server, removing the admin login yields this:
client.py:797 Refreshing access_token
Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---
Did Google change something in the --oauth2 workflow? I appear to be authorized, but the download crashes at the auth step.
Thanks.
Seems to be a bug in GoogleAppEngineLauncher-1.9.27, where it does not allow a login prompt. When reverting back to 1.9.26, it works. Interestingly, my gcloud components still have the 1.9.27 version:
>$ gcloud version
Google Cloud SDK 0.9.83
alpha 2015.10.08
app 2015.10.16
app-engine-python 1.9.27
beta 2015.10.08
core 2015.10.16
core-nix 2015.09.03
gcloud 2015.10.16
temporary solution: go to https://console.developers.google.com/storage/browser/appengine-sdks/featured/ to get version 1.9.26
Submitted bug report: https://code.google.com/p/google-cloud-sdk/issues/detail?id=340
I'm trying to use the bulkloader to load my data to the App-Engine server. I run the following command using Python 2.5:-
appcfg.py upload_data --application=myappname --kind=mykind
--filename=data_archive.csv --url=http://myappname.appspot.com/remote_api
But its failing with this Authentication error:-
[INFO ] Connecting to myappname.appspot.com/remote_api
[ERROR ] Exception during authentication
URLError: <urlopen error [Errno 10061] No connection could
be made because the target machine actively refused it>
[INFO ] Authentication Failed
My idea is to do a bulk download from my development server and then use this dump to do a upload to the app-engine server. The bulk download worked fine. I used this format for this:-
appcfg.py download_data --application=myappname --kind=mykind
--url=http://localhost:8888/remote_api --filename=data_archive.csv
But the bulk upoad fails. A couple of things: the bulk download asked me for a userid and password, but the bulk upload does not. Also, I don't currently have a app.yaml file which I see mentioned a lot - do I need one to do this ?
Thanks in advance for any help.
M.
EDIT
For anyone else struggling with this, the problem was indeed being behind the proxy server, but there was another 'error' with what is above. The app-id needs the "s~" bit added to it.
appcfg.py upload_data --application=s~myappname --kind=mykind
--filename=data_archive.csv --url=http://myappname.appspot.com/remote_api
This isn't an authentication issue - that message is a red-herring - your machine is unable to contact the App Engine app at all. Do you have a proxy you need to transit through in order to make external connections?
You do not need --application=s~myappname when using bulkloader - Google have mentioned before:
Warning! Do not use the --application= flag to get the application ID
when using the bulk loader. Instead, use --url=.
For more detail take a look here:
https://developers.google.com/appengine/docs/python/tools/uploadingdata
app.yaml is how it finds your server. I am not sure how you can try and upload without one.
In addition to having an app.yaml that points to the production server, the production server also needs to have remote_api turned on (in it's app.yaml and in the version you are trying to reach):
builtins:
- remote_api: on
I have an issue and I'd like to use the Interactive console you get on the local dev server (http://localhost:8080/_ah/admin) on the cloud server to make sure a query is working as I expect.
I read before you can enable it - but how is it done?
Nick's answer doesn't work anymore when using the "threadsafe" option. Instead, you have to use:
- url: /admin/.*
script: google.appengine.ext.admin.application
login: admin
Add the following to your app.yaml, before any .* handler:
- url: /admin/.*
script: $PYTHON_LIB/google/appengine/ext/admin
login: admin
Another option for your use-case is to enable remote_api, then use the remote_api_shell.py tool included with the SDK, allowing you to test things from a local Python shell.