not sure if Google App Engine changed something, but I was able to download logs until recently to parse the logs for generating custom analytics on usage etc. Now the log is getting exported but the issue is that the last log is from Nov 23 while I see logs as recent as 12/18 in the console. The commandline version I use is as below.
appcfg.cmd -n 90 --severity=1 request_logs <appDirPath> <full_log_file_path>
Has anything changed recently with logs export with GAE ? Does anyone know a workaround for this issue. If it matters, I am using appengine-java-sdk-1.9.64.
I finally figured out the issue. Posting here in case anyone else runs into similar silly issue. After upgrading to google-cloud-sdk and Java 8, I am getting the following log in console
Configuration Warning : <application>/<version> XML elements and --application/--version should not be specified when staging
The following parameters will be scrubbed from app.yaml
application : appId
version : chosenVersion
Since google app engine was ignoring this, it was creating a new version with dateTimeStamp with new deployment. But the command line in my post points to the appDirPath that has the appengine-web.xml listing a version and so it was downloading the logs for that version only. Since that version was overridden with a new dynamic version on Nov 24, the command line was giving only logs till Nov 23. It took me a while to figure out so posting for anyone else who run into this issue.
Related
After some time I needed to do changes on GAE Python (2.7) First Generation app, but when I develop and run locally, it's basically nonusable due to a very slow start.
These are printed logs:
INFO 2019-10-18 07:56:35,533 devappserver2.py:278] Skipping SDK update check.
INFO 2019-10-18 07:56:35,595 api_server.py:275] Starting API server at: http://localhost:36159
INFO 2019-10-18 07:56:35,599 dispatcher.py:270] Starting module "default" running at: http://127.0.0.1:8080
INFO 2019-10-18 07:56:35,600 admin_server.py:152] Starting admin server at: http://localhost:8000
INFO 2019-10-18 08:01:01,644 instance.py:294] Instance PID: 28496
What I notice that the last line (instance.py) is printed after ~5 minutes and only after that app responds to requests, not before. Interesting that the admin server (localhost:8000) is available right away. Of course, when I do some code change it automatically reloads and it repeats again.
Things I tried/found out:
it behaves like that on my all GAE projects Python First Gen.
tried to create a bare minimal version (webapp2 with one URL), clean virtualenv, still the same behavior
tried to reinstall Google Cloud SDK. delete the whole google-cloud-sdk folder and install again, no changes
tried to install older version of Cloud SDK
used clean VM and it works ok!!!, so it looks like there could be something wrong with my system (outside of SDK), but I'm not sure what.
It's interesting that the pause between the last two log lines is always about 5 minutes, not sure why exactly that time.
Python 2.7.14
OS: OpenSuse Leap 15.0
I'm running out of ideas so any advice would be appreciated.
I solved this accidentally.
I wanted to run Jupyter notebook, but I got the error:
error: [Errno 99] Cannot assign requested address
after debugging in /tornado/netutil.py, I noticed that it tries to work with IP 192.168.1.50 which I wasn't sure where did that come from, (probably I set it since I was playing with my home network some time ago), but when I deleted it from /etc/hosts, Jupyter, as well as GAE, works ok.
What a coincidence :)
Today we start recieving this message when trying to deploy.
...
Scanned 8750 files.
Scanned 9000 files.
Scanned 9250 files.
Scanned 9500 files.
Scanned 9750 files.
Initiating update.
Cloning 711 static files.
Cloning 9233 application files.
Cloned 2000 files.
Cloned 4000 files.
Cloned 6000 files.
Cloned 8000 files.
Deploying:
Uploading 0 files.
Initializing precompilation...
Deploying new version.
Rolling back the update.
com.google.appengine.tools.admin.HttpIoException: Error posting to URL: https://appengine.google.com/api/appversion/deploy?app_id=facturanube&version=1&
500 Internal Server Error
<h3>Server Error</h3><p>A server error has occurred.</p>
Debugging information may be found in C:\Users\usuario\AppData\Local\Temp\appengine-deploy6314649189927600314.log
Debugging information file contents:
Unable to update:
com.google.appengine.tools.admin.HttpIoException: Error posting to URL: https://appengine.google.com/api/appversion/deploy?app_id=si-nube&version=1&
500 Internal Server Error
<h3>Server Error</h3><p>A server error has occurred.</p>
at com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:303)
at com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:254)
at com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:233)
at com.google.appengine.tools.admin.LoggingClientDeploySender.send(LoggingClientDeploySender.java:47)
at com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:1100)
at com.google.appengine.tools.admin.AppVersionUpload.deploy(AppVersionUpload.java:949)
at com.google.appengine.tools.admin.AppVersionUpload.commit(AppVersionUpload.java:850)
at com.google.appengine.tools.admin.AppVersionUpload.uploadFilesTransaction(AppVersionUpload.java:176)
at com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:151)
at com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:474)
at com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:55)
at com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngineBridgeImpl.java:400)
at com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(DeployProjectJob.java:148)
at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:38)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Important considerations:
The appengine-web.xml file has no errors.
We tried deploying in different versions with no success.
The application can be deployed in another appengine application, but not in our production application.
We have no cuota-related problems. Our payments have no
problems.
Our current versions:
GWT SDK 2.4.0
App Engine 1.9.54
Java 7
Update
I was able to deploy today (2018/03/18) with no changes to code or configuration files. I guess Google did something to solve the issue.
500 A server error has occurred on deployment is usually related to malformed configs, including app.yaml and cron.yaml (and Java equivalents). Or if unsupported stanzas are included. Please check your GAE app configs and attempt to strip them down to the essentials just to narrow down which stanzas/flags are messing up the deployment.
I try to download the logs from my Google App Engines with the command:
appcfg.py request_logs [directory] mylogs.txt
However, I encountered the server error as below:
C:\Program Files (x86)\Google\google_appengine>appcfg.py request_logs E:\PhD\cod
e\python\webDevelop\similarAPI\similarAPI/ mylogs.txt
05:22 PM Application: similarapi
05:22 PM Host: appengine.google.com
05:22 PM Downloading request logs for app similarapi version 1.
Error 500: --- begin server output ---
Server Error (500)
A server error has occurred.
--- end server output ---
I have looked at the official document and related Stack Overflow questions, but no clues are found. Actually I can get the logs in Dec, 2016 but not now. And I tried it for one week without the success, so it should not be the error of Google's infrastructure.
Does anyone know how to solve that problem? Thanks in advance.
Just a workaround, not a direct answer.
The GAE SDK might be getting a bit less attention compared to the cloud SDK (which is the recommended one lately). I, for one, can't even get as far as you got since appcfg.py request_logs can't properly detect my multi-service app from its directory and was not happy with any syntax attempt I made to specify it via args.
But I have the cloud SDK installed as well and I was able to get the logs for my app using gcloud app logs read (after setting up the gcloud context for my app)
FWIW, this isn't yet enough to make me switch, I'm still primarily a GAE SDK user. YMMV.
Finally, it has be proved that it is a bug of Google's infrastructure from 22 Mar to 10 Apr, 2017. Three related bug reports can be seen in the Google Issue Tracker 36637246, 37192011, 36969442.
Now the command appcfg.py request_logs [directory] mylogs.txt works well.
Currently the GAE Managed VMs feature is broken for any project with a name longer than 27 characters.
The underlying issue is that Docker restricts image namespace to between 4-30 chars. This has been fixed (https://github.com/docker/docker/issues/10392) but is still awaiting a release at time of writing.
It seems when deploying a Managed VM to GAE that the namespace is automatically generated from your project name plus _m_ prefix. This leads to error when attempting to deploy the vm:
DEBUG: "POST /v1.10/images/gcr.io/_m_<my project name>/<my project name>.default.20150330t140211/push HTTP/1.1" 500 111
INFO: Exception 500 Server Error: Internal Server Error ("Invalid namespace name (_m_<my project name>). Cannot be fewer than 4 or more than 30 characters.") thrown in ProgressHandler. Retrying.
The obvious solution would be for GAE gcloud tools to respect the underlying limit via some auto-truncation or hashing scheme.
Does anyone know a way around this? Or I have to wait for Google to fix or Docker to release a new version and Google to update?
We're aware of the issue and we're working on a long-term fix. For now, you can switch to an old version of gcloud. You can do this by setting this variable to point to an old version (0.9.51):
gcloud config set --scope=installation component_manager/fixed_sdk_version 0.9.51
then run "gcloud components update"
Then run "gcloud config set app/hosted_registry false"
and you should be able to deploy. I'll update this answer when we've fixed the naming issue.
UPDATE:
The naming issue has been fixed as of this week's release (0.9.57).
I have GWT app, which is deployed on GAE (Java). I'm trying to download data from App Engine datastore using appcfg.py . I did all the setup according to http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-engine/ .
GAE Python SDK version is 1.4.3
Python version is 2.5.4
appcfg.py is on my PATH. When I run appcfg.py on the command-line, I get the "help" message. But the problem is that no matter which command I use, it always returns the help message. I have not been able to run any command using appcfg.py.
It doesn't give any specific error message no matter what arguments I give. My app is using Google Accounts authentication, but I don't think it even gets to the point of authentication.
I'm able to use the Java appcfg (for other actions like rollback) without any problem. But the Python version simply refuses to work for all commands.
I've tried different formats like:
appcfg.py create_bulkloader_config --url=http://myappid.appspot.com/remote_api --application=myappid --filename=config.yml
appcfg.py create_bulkloader_config --filename=bulkloader.yaml --url=http://myappid.appspot.com/remote_api
appcfg.py --filename=bulkloader.yaml --url=http://myappid.appspot.com/remote_api create_bulkloader_config
All give me the same help message:
Usage: appcfg.py [options]
Action must be one of:
create_bulkloader_config: Create a bulkloader.yaml from a running application.
cron_info: Display information about cron jobs.
download_app: Download a previously-uploaded app.
download_data: Download entities from datastore.
help: Print help for a specific action.
request_logs: Write request logs in Apache common log format.
rollback: Rollback an in-progress update.
set_default_version: Set the default (serving) version.
update: Create or update an app version.
update_cron: Update application cron definitions.
update_dos: Update application dos definitions.
update_indexes: Update application indexes.
update_queues: Update application task queue definitions.
upload_data: Upload data records to datastore.
vacuum_indexes: Delete unused indexes from application.
Use 'help <action>' for a detailed description.
Options:
-h, --help Show the help message and exit.
-q, --quiet Print errors only.
-v, --verbose Print info level logs.
--noisy Print all logs.
-s SERVER, --server=SERVER
...
...
...
Even when I try "appcfg.py help create_bulkloader_config" for a detailed description, it still shows me the same standard help.
I have also tried on the local development server using the url http://127.0.0.1:8888/remote_api but it still gives the same help message.
I'm totally clueless as to what the problem is. I'm new to GWT and GAE, and any help will be appreciated.
Thanks.
The following fix worked for me. It looks like appcfg.py doesn't like PYTHON27 and ALWAYS returns the help menu. I fixed it by executing it with PYTHON25 and hard coded all my file locations:
C:\Python25-archive\python "C:\Program Files (x86)\Google\google_appengine\appcfg.py" rollback C:\scripts\myapp
The right way is to change the environment variables on Windows 7:
Go to System Properties
Go to Advance System Settings
Click on Environment Variables
Append to Path variable the values C:\Python27\
Click Ok and restart your computer. (Yes, it is needed.)
Another way is to:
Open command Prompt
Locate your python.exe file. For example:
C:\Python27>_
Then, run a python command that looks like this.
python <appcfg_directory> download_app -A <your_app_id> -V <your_app_version> <output-dir>
Where <appcfg_directory> is equal to C:\Program Files\Google\google_appengine\appcfg.py. (Depending on your file location)
Don't forget to put quotes before and after <appcfg_directory>