I needed to test some changes on my local dev server before pushing to production. Doing so required having the full dataset on my local machine.
A colleague directed me to:
https://developers.google.com/appengine/docs/python/tools/uploadingdata?csw=1
I downloaded the data using an administrator's username and password, but unfortunately, I was unable to upload the data to my localhost "dev" app engine server.
Ran this command from the commandline:
appcfg.py upload_data --filename=../data/data1.dat --url=http://localhost:9080/_ah/remote_api ./
Where:
9080 was my app port on my localhost copy of the app
I was running this command from my app directory
Had the downloaded data stored in relative directory
../data/data1.dat
Received this error:
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: app "dev~appname" cannot access app "appname"'s data
UPDATE: It seems that the answer was as simple as adding the following to my upload_data call:
--application="dev~appname"
Thanks #DavidBennett.
ORIGINAL ANSWER: (which also works)
After a ton of searching on SO and code.google.com, the solution I found that worked was a comment on this question:
devappserver2, remote_api, and --default_partition
I used my original command as described in the question:
appcfg.py upload_data --filename=../data/data1.dat --url=http://localhost:9080/_ah/remote_api ./
The username and password I entered when prompted were my apps username (in my case, my email) and the corresponding password. (If that doesn't work you might want to try blank or test#example.com based on other comments I've read, but have not tested that theory.)
I also restarted my app engine with the following flag: (don’t forget to remove the flag the next time you restart the server) (You might want to try without using this flag, since I can’t confirm that it affects anything - I’m including it here, since it was a setting that I used.)
--clear_datastore=yes
The commenter recommends to delete “dev~” in your local server code on line 84 in this file:
google/appengine/tools/devappserver2/application_configuration.py, line 84
Where:
that base directory 'google' is located inside of:
/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
assuming your GoogleAppEngineLauncher.app directory is in your Applications directory on your Mac
IMPORTANT: Restart your local app engine server for the changes to take effect.
Related
I am using Google App Engine on Ubuntu within Linux Subsystem for Windows.
When I start dev_appserver.py I receive errors with the following line resulting in this, which I am understanding to be a corrupted sqlite data file.
File "/../google-cloud-sdk/platform/google_appengine/google/appengine/api/logservice/logservice_stub.py", line 181, in start_request
host, start_time, method, resource, http_version, module))
DatabaseError: database disk image is malformed
Based upon this post I am understanding there is a log.db referenced.
GoogleAppEngineLauncher: database disk image is malformed
However, when I run the script referenced, the resultant path does not contain a log.db leading me to believe this is a different issue.
Any help in identifying the appropriate database, for the purposes of removing, would be appreciated.
Per comment added --clear_datastore=1 and did not notice a change
dev_appserver.py --host 127.0.0.1 --port 8080 --admin_port 8082 --storage_path=temp/storage --skip_sdk_update_check true --clear_datastore=1 main/app.yaml main/sync.yaml
I have a set of data that I would like to use for production but I can't seem to get appcfg.py to download the localhost datastore. I have localhost:8080/_ah/remote_api on in app.yaml then, in the SDK shell, i ran
appcfg.py download_data --filename=local.db --url=http://localhost:8080/_ah/remote_api -A [MYAPPID]
I get this error message:
You must be logged in as an administrator to access this.
I can't find a way to authenticate my google account as the admin. any advice?
This is a general issue of remote_api admin authentication to the local development server, you may want to star it: https://code.google.com/p/googleappengine/issues/detail?id=12445. Several workaround suggestions in there (YMMV according to other posts)
The most recent one (2 days old) looks promising:
Workaround for appcfg.py upload_data to local dev server,
"Refreshing due to a 401" issue. Tested using Java dev server 1.9.42:
Log into the local dev server Development Console in a browser.
Open your browser's dev tools and pull the value for the dev_appserver_login cookie.
Edit lib/oauth2client/oauth2client/client.py in the app.
Find "def new_request" function. Before line self.apply(headers) in this function, add:
headers['Cookie'] = 'dev_appserver_login=your-cookie-value-here';
Run the upload_data command.
There's also this recipe, successful at the time, but it might be a bit old now:
appcfg.py shows You must be logged in as an administrator
I'm trying to run the Google App Engine Python 2.7 Hello World program and view it in a browser via Google App Engine Launcher. I followed the install and program instructions to the letter. I copied and pasted the code in the instructions to the helloworld.py file and app.yam1 and verified that they are correct and in the directory listed as the application directory. I hit run on the launcher and it runs with no errors, although I get no sign that is has completed (orange clock symbol next to app name). I get the following from the logs:
Running dev_appserver with the following flags: --skip_sdk_update_check=yes --port=8080 --admin_port=8000 Python command: /opt/local/bin/python2.7
When I try to open in the browser via the GAE Launcher, the 'browse' icon is grayed out and the browser won't open. I tried opening localhost:8080 in Firefox and Chrome as the tutorial suggests, but I get unable to connect errors from both.
How can I view Hello World in a browser? Is there some configuration I need to make on my machine?
I had the same problem. This seemed to fix it:
cd to google_appengine, run
python dev_appserver.py --port=8080 --host=127.0.0.1 /path/to/application
at this point there is a prompt to allow updates on running, I said Yes.
At this point the app was running as it should, also when I quit this and went in using the launcher again, that worked too.
I have to manually start python and make it point to my app folder, for instance in a command line window on Windows I am using python. I installed python in C:\Python27 and my sample app is in c:\GoogleApps\guestbook
C:\Python27>dev_appserver.py c:\GoogleApps\guestbook
and then I can start my app in the Google App Engine Launcher and hit localhost 8080
How about specifying --host argument? You can find it at the bottom of following doc.
https://developers.google.com/appengine/docs/python/tools/devserver
This might be a little late. But still someone might find it useful.
When ever you go and try changing the port number from 8080 to something else, it will not get updated. So the best option is:
Go to your user directory: eg: C:\Username
There will be a Google folder. Go inside
Open the file google_appengine_projects.ini
Change your port number from 8080 to whatever you like 8081
Save it and close the file.
Launch the GAE Launcher again and you will find the changes reflected and the app runs without issues.
7: Access the application using: http://localhost:NewPort/
This can be used to change ports both run port and admin port for your individual projects running locally.
Hope this helps!
The 8080 portion of your url is a port number. Firefox disables visiting url's of other ports by default. You have to enable them by doing the following: http://blog.christoffer.me/post/2012-02-20-how-to-remove-firefoxs-this-address-is-restricted/
Paraphrasing that website:
Open firefox and visit about:conf
In the Filter box, type in network.security.ports.banned.override
If you can't find such a preference, right click to open up the pop-up menu and pick New and then String
As preference name type network.security.ports.banned.override and 8080 as the value.
Done!
It's likely if this continues to not work that your browser is behaving properly (8080 is a fairly standard port). That means that its a problem with the server and we'd have to do some more debugging.
I have GWT app, which is deployed on GAE (Java). I'm trying to download data from App Engine datastore using appcfg.py . I did all the setup according to http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-engine/ .
GAE Python SDK version is 1.4.3
Python version is 2.5.4
appcfg.py is on my PATH. When I run appcfg.py on the command-line, I get the "help" message. But the problem is that no matter which command I use, it always returns the help message. I have not been able to run any command using appcfg.py.
It doesn't give any specific error message no matter what arguments I give. My app is using Google Accounts authentication, but I don't think it even gets to the point of authentication.
I'm able to use the Java appcfg (for other actions like rollback) without any problem. But the Python version simply refuses to work for all commands.
I've tried different formats like:
appcfg.py create_bulkloader_config --url=http://myappid.appspot.com/remote_api --application=myappid --filename=config.yml
appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url=http://myappid.appspot.com/remote_api
appcfg.py --filename=bulkloader.yaml --url=http://myappid.appspot.com/remote_api create_bulkloader_config
All give me the same help message:
Usage: appcfg.py [options]
Action must be one of:
create_bulkloader_config: Create a bulkloader.yaml from a running application.
cron_info: Display information about cron jobs.
download_app: Download a previously-uploaded app.
download_data: Download entities from datastore.
help: Print help for a specific action.
request_logs: Write request logs in Apache common log format.
rollback: Rollback an in-progress update.
set_default_version: Set the default (serving) version.
update: Create or update an app version.
update_cron: Update application cron definitions.
update_dos: Update application dos definitions.
update_indexes: Update application indexes.
update_queues: Update application task queue definitions.
upload_data: Upload data records to datastore.
vacuum_indexes: Delete unused indexes from application.
Use 'help <action>' for a detailed description.
Options:
-h, --help Show the help message and exit.
-q, --quiet Print errors only.
-v, --verbose Print info level logs.
--noisy Print all logs.
-s SERVER, --server=SERVER
...
...
...
Even when I try "appcfg.py help create_bulkloader_config" for a detailed description, it still shows me the same standard help.
I have also tried on the local development server using the url http://127.0.0.1:8888/remote_api but it still gives the same help message.
I'm totally clueless as to what the problem is. I'm new to GWT and GAE, and any help will be appreciated.
Thanks.
The following fix worked for me. It looks like appcfg.py doesn't like PYTHON27 and ALWAYS returns the help menu. I fixed it by executing it with PYTHON25 and hard coded all my file locations:
C:\Python25-archive\python "C:\Program Files (x86)\Google\google_appengine\appcfg.py" rollback C:\scripts\myapp
The right way is to change the environment variables on Windows 7:
Go to System Properties
Go to Advance System Settings
Click on Environment Variables
Append to Path variable the values C:\Python27\
Click Ok and restart your computer. (Yes, it is needed.)
Another way is to:
Open command Prompt
Locate your python.exe file. For example:
C:\Python27>_
Then, run a python command that looks like this.
python <appcfg_directory> download_app -A <your_app_id> -V <your_app_version> <output-dir>
Where <appcfg_directory> is equal to C:\Program Files\Google\google_appengine\appcfg.py. (Depending on your file location)
Don't forget to put quotes before and after <appcfg_directory>
This seems like it should be very easy but I don't see a link to it anywhere.
How do I download the source code of a google app engine project?
Windows
appengine-java-sdk\bin\appcfg.cmd -A <your_app_id> -V <your_app_version> download_app <output-dir>
Linux
./appengine-java-sdk/bin/appcfg.sh -A <your_app_id> -V <your_app_version> download_app <output-dir>
For completeness, using the Python implementation:
appcfg.py download_app -A $appID -V $appVersionNumber $downloadDirectory --oauth2
--oauth2 is of course optional, you can omit it and provide your email + app-specific password (or your password, and then go implement two-factor authentication right after), but it's easier, and frankly there's no reason not to.
Documentation.
App Engine actually recently added the ability for the developer who uploaded a given app version to download its source code.
As of October 2019 you can simply go to --> App Engine --> Services and in the tool dropdown select 'source' and the source code is there
Posting this since none of the listed methods above didn't take me to the code (by June 2021)
You could try accessing it through;
Google Cloud Platform > Debugger > choosing the version of the
Application from combo at top.
This will list the files of that version on the left pane. There is no way to download it automatically but you can copy-paste the code.
Hope you will find this helpful.
IMHO, the best option today (Aug 2018) is:
Under the main menu, under Products, go to Tools -> Cloud Build -> Build history.
There, click the ID of the build you want (for me - the last one).
Then, in the opened window (Build details), click the "source" link, the download of your compressed code begins.
As simple as that.
HTH.
Working with App engine standard using Go, the debugger isn't available yet.
How I managed to download the source code for an existing service was to use the gcloud tool.
First: Get the version id of your service using the app engine console or running: gcloud app versions list
Second: use the version and service name and run: gcloud app versions describe <versionID> --service=<service name>
the describe parameter will give you the storage locations for your source files that looks like this:
cmd/main.go:
sha1Sum: e3fe5848c2640eca7ac3591490e1debc2d3a9b09
sourceUrl: https://storage.googleapis.com/<project>/<file id>
Third: you can then use the storage console, using the file id, to download the files you are interested in.
this process based on java sdk
Its works for me...
Download Google cloud SDK
gcloud init
enter image description here
Follow through process of logging in using your credentials
Enter following command from SDK
C:\Program Files (x86)\Google\appengine-java-sdk-1.9.49\bin
enter image description here
Enter Following command to download source code
appcfg.sh -A [YOUR_APP_ID] -V [YOUR_APP_VERSION] download_app [OUTPUT_DIR]
Eg: appcfg.sh -A my-project-name-1234 -V 2 download_app C:\Users\india\Desktop\my project
Note: this progress based on java-appengine sdk so we use appcfg.sh instead of appcfg.py
check if your app is uploaded with same email id that is in your app engine. if you are not sure then in app engine > control > Clear deployment credentials and then click on any project, deploy to sign in again then use this
appcfg.py download_app -A {app id from google app engine} -V {1} "{c:\path}" --oauth2_credential_file=C:\Users\{your account name}/.appcfg_oauth2_tokens
change all {} to your needs
Things have changed since this question was asked so I'm adding an updated answer. Note that this only applies to GAE Standard Environment
Google has deprecated appcfg.py and so the previous responses appcfg.py download_app no longer works.
gcloud which is the SDK in use (it replaced appcfg) does not have the functionality to download your source code.
When you deploy your app via gcloud app deploy, it copies your source code to a bucket. The default bucket is staging.<project_name>.appspot.com. Your files will stay in this bucket for a maximum of 15 days before they are deleted. You can modify the rule so that the files are retained for longer or less time.
The file names in the bucket are encoded so you can't figure out what each file is unless you open it (i.e. download it). Google has a mapping of the encoded names to the original file names. To get this mapping, you run the gcloud app versions describe command and it will list the file names and their encoded names. To download the files, you have to manually click each url one by one. So essentially, you have to download each file manually and then use the mapping to rename them (or open the file, check the content and then rename them). Also note that downloading the files manually will not maintain the folder structure in which they were uploaded.
If you do not wish to go through all of the above hassles (imagine having to manually open each url for each file if you have a small to mid-sized project which has hundreds of files), our App - https://nocommandline.com - now supports downloading source code from the default bucket - staging.<project_name>.appspot.com (so far as your files are still there which means any deployment i.e update not older than 15 days from your current date unless you previously increased the deletion age on your staging bucket's lifecycle page).
In simple terms, you enter your project name, the version number and our App will take care of retrieving the original file name to encoded name mapping, automatically downloading the files and renaming them to the original names, while maintaining the folder structure. For more information, refer to https://nocommandline.com/help/#faq_download_source_code_from_gae.
Log in to the console.developers.google.com
Select the project you want to download the code from (Google App Engine Standard Envoronment).
Go to the App Engine Dashboard. Under Summary is Debug and Source. Click on Source.
Select each file one at a time and copy it (highlight the code, copy and paste into your local editor.)
Select the next file....
You need to use svn to checkout the files.
If you are on Windows, you can use tortoise svn for your GUI end.
Here are tutorials on how to do it, here is the related question.