I run my application in AppEngine (Java).
I want to use gcloud app logs command to view my application logs:
gcloud app logs tail
However, the output is just few data as below:
....
2019-10-04 12:46:43 default[api-v1-0-0] "GET / HTTP/1.1" 200
2019-10-04 12:46:43 default[api-v1-0-0] "GET / HTTP/1.1" 200
....
Actually, I expect gcloud app logs will show logs that my application printed to stdout and stderr. However, it does not.
My question is: How to get/view application stdout/stderr logs by using gcloud command?
From gcloud app logs tail:
DESCRIPTION
Streams logs for App Engine apps.
The command is designed as a blocking call primarily watching for the app logs being generated after the command is invoked (though at startup it may also display a few historical logs, generated before the command is issued).
To focus primarily on the historical logs you want to use the gcloud app logs read command:
DESCRIPTION
Display the latest log entries from stdout, stderr and crash log for
the current Google App Engine app in a human readable format.
Note that the presence of the stdout/stderr in the logs depends on the environment and runtime your app uses. For example in the 1st generation standard environment (at least for the python runtime) the messages printed directly to stdout/stderr do not show up in the logs, one has to specifically use the python logging calls.
I suspect the same might be true for Java, as I see stdout/stderr mentioned in the logging docs for Java11 (and flex environment), but not in the Java8 one.
I also could not use
> gcloud app logs tail
but i found out you can use:
gcloud logging logs list
This will return a list of log names like:
NAME
projects/intan1/logs/appengine.googleapis.com%2Frequest_log
projects/intan1/logs/cloudaudit.googleapis.com%2Factivity
Then you can call:
gcloud beta logging read logName=projects/intan1/logs/appengine.googleapis.com%2Frequest_log
This will return a log, with a lot of extra info, but maybe this can help you!
Related
In the browser cloud console for my Google App Engine app, I can choose to see the logs for /dev/log and stderr which gives me all of the log entries that I'm expecting to see.
However, when I use the command line gcloud app logs tail to stream the logs in my terminal, I can't get it to give me the /dev/log entries.
The docs say the default list of logs include: stderr,stdout,crash.log,nginx.request,request_log
So the /dev/log must be represented by some other identifier, but I can't find any docs on what it might be. I've tried a few guesses, but none work.
How can I can the terminal to stream the same logs I'm getting in my browser?
You can use command like gcloud logging read to interact with Stackdriver logging & get a non streamed version of those logs. Setup the Stackdriver GUI with the logs you wish to see, then convert to an advanced filter. You can then paste the advanced filter as is, in quotations after gcloud logging read. Examples in gcloud logging read documentation. I will get back to you in a comment on this post as to whether you can get the /dev/log logs with the gcloud app logs tail command. I will update on Saturday
I have a google app engine project in python 3.7 in which I would like to write some logs.
I am used to program in app engine python 2.7 and I was using the simple code:
logging.info('hi there!')
to write any log onto the google cloud log console.
That command above now doesn't work anymore and it says:
logging has no attribute 'info'
I searched and I found this possible new code
from flask import Flask
from google.cloud import logging
app = Flask(__name__)
#app.route('/l')
def hello():
logging_client = logging.Client()
log_name = LOG_NAME
logger = logging_client.logger(LOG_NAME)
text = 'Hello, world!'
logger.log_text(text, severity='CRITICAL')
return text
This code above doesn't give any error in the stack-driver report page BUT it displays nothing at all in the log page.
So how can I write a log for my app engine project in python3.7?
The second generation standard environment (which includes python 3.7) is IMHO closer to the flexible environment than to the first generation standard environment (which includes python 2.7).
Many of the APIs which had customized versions for the 1st generation (maintained by the GAE team) were not (or at least not yet) ported in the 2nd generation if the respective functionality was more or less covered by alternate, more generic approaches, already in use in the flexible environment (most of them based on services developed and maintained by teams other than the GAE one).
You'll notice the similarity between many of the service sections in these 2 migration guides (which led me to the above summary conclusion):
Understanding differences between Python 2 and Python 3 on the App Engine standard environment
Migrating Services from the Standard Environment to the Flexible Environment (i.e 1st generation standard to flexible)
Logging is one of the services listed in both guides. The 1st generation used a customized version of the standard python logging library (that was before Stackdriver became a standalone service). For the 2nd generation logging was simply delegated to using the now generally available Stackdriver logging service (which is what the snippet you shown comes from). From Logging (in the 1st guide):
Request logs are no longer automatically correlated but will still
appear in Stackdriver Logging. Use the Stackdriver Logging client
libraries to implement your desired logging behavior.
The code snippet you show corresponds, indeed, to Stackdriver Logging. But you seem to be Using the client library directly. I don't know if this is a problem (GAE is often a bit different), but maybe you could also try using the standard Python logging instead:
Connecting the library to Python logging:
To send all log entries to Stackdriver by attaching the Stackdriver
Logging handler to the Python root logger, use the setup_logging
helper method:
# Imports the Google Cloud client library
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
# Connects the logger to the root logging handler; by default this captures
# all logs at INFO level and higher
client.setup_logging()
Using the Python root logger:
Once the handler is attached, any logs at, by default, INFO level or
higher which are emitted in your application will be sent to
Stackdriver Logging:
# Imports Python standard library logging
import logging
# The data to log
text = 'Hello, world!'
# Emits the data using the standard logging module
logging.warn(text)
There are some GAE-specific notes in there as well (but I'm not sure if they also cover the 2nd generation standard env):
Google App Engine grants the Logs Writer role by default.
The Stackdriver Logging library for Python can be used without needing
to explicitly provide credentials.
Stackdriver Logging is automatically enabled for App Engine
applications. No additional setup is required.
Note: Logs written to stdout and stderr are automatically sent to Stackdriver Logging for you, without needing to use
Stackdriver Logging library for Python.
Maybe worth noting that the viewing the logs would likely be different as well outside the 1st generation standard env (where app logs would be neatly correlated to the request logs).
And there's also the Using Stackdriver Logging in App Engine apps guide. It doesn't specifically mention the 2nd generation standard env (so it might need an update) but has good hints for the flexible environment which might be useful. For example the Linking app logs and requests section might be of interest if the missing request logs correlation has anything to do with it.
Even though logging works differently in Python 2.7 and 3.7, the same method of logging provided in Reading and Writing Application Logs in Python 2.7 should work for Python 3.7 as well since logs written to stdout and stderr will still appear in the Stackdriver Logging.
Import logging
logging.getLogger().setLevel(logging.DEBUG)
logging.debug('This is a debug message')
logging.getLogger().setLevel(logging.INFO)
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
#logging.warn is deprecated
logging.warn('This is a warning' message)
=======================================
Import logging
app.logger.setLevel(logging.ERROR)
app.logger.error('This is an error message')
However, log entries are no longer automatically correlated with requests as in Python 2.7, which is why you see them in plain text. I have created a feature request to address this which you can follow here.
I like that I can use the Logs API (described here: https://cloud.google.com/appengine/docs/java/logs/) to programatically access and display app & request logs as I see fit--it's great.
Now that I'm using Managed VMs on AppEngine you can see on the Admin Console Logs Viewer that there are a ton of additional logs--including in my case a custom log which I found I could include in the viewer (decribed here: https://cloud.google.com/appengine/docs/managed-vms/custom-runtimes#logging).
My question is: Is there any way I can use the Logs API (or other pipelines already built?) to access these logs? My Managed VM module includes several components which could produce logs that I want to view:
App logs -- I can get these! No problem here.
Custom log files created by background processes I kick off in _ah/start (like "my_custom_1.log" in the screenshot)
STDERR & STDOUT from my background processes
Relevant Managed VM logs (e.g. for when an instance was restarted due to bad health... other system events like normal restarts?)
Basically I want "the total picture" at the instance level. Anyone tried to tame Managed VMs in this way with success? I'm not looking forward to rolling my own solution. And I wouldn't even know where to start on the problem of capturing STDERR and STDOUT. Any help appreciated.
There is a difference between App Engine logging and Google Cloud logging. Some of the Managed VM logs go to both, but much of it only goes to cloud logging.
Until recently there was not an API to read Cloud logs, only to write them. However, there is a new v2 beta API: https://cloud.google.com/logging/docs/api/introduction_v2
To do things at an instance level, entries in Cloud logging should have metadata set to denote which VM they came from. Both of these values seem to vary on logs from my VMs:
compute.googleapis.com/resource_name
compute.googleapis.com/resource_id
I am running a Java Microengine on GAE.
I have my own html pages for data input and output. When there is a data error and the engine cannot complete its execution (crashes) - the microengine spits out the "Response" as Server not available, please try later.
In order to debug, I run the dataset in the dev environment - as a Java application to identify the error in the console output.
Is there a way to capture the "error" (the console output equivalent when run as a Java application) - as an output string and send it as a content of the servlet response from the deployed Application in GAE..
thanks,
assuming you are using App Engine Managed VM, and a logging framework, you should also forward the log entry into a file, e.g. /var/log/app_engine/custom_logs/app.log
https://cloud.google.com/appengine/docs/managed-vms/custom-runtimes#logging
Subsequently, you'll be able to read the output from Google Cloud Logging.
Am doing development on google app engine and i want to print out values that are sent to the server from the browser. When i insert the print commands in my handlers nothing get printed on the console running the local server!
How do i get around this?
Gath
Rather than just printing out values, use the logging module. This is generally good practice anyway as it means you can potentially leave in some handy debug output even when you are running in production mode.
import logging
logging.error("This is an error message that will show in the console")
By default dev_appserver.py will not show any messages below logging-level INFO, but you can run dev_appserver.py with the --debug flag and then you will be able to see output from the lower logging.DEBUG level and scatter various logging calls in your code like:
logging.debug("some helpful debug info")
If you leave these in once your app is in production, then these will be visible in your app's admin area in the logs section.
As #Chris Farmiloe suggested you should use the logging module . Google appengine by default supports logging modlue. You can see various levels of logs for your applications in appengine console. Logging module comes in handy in the production enviroment. Where you can see all your logs.
For general info use
logging.info('This is just for information')
self.response.out.write("Hallo World")
Using Go Language (I don't know if this works with the other languages), you can do that through the appengine.Context.Debugf function
https://developers.google.com/appengine/docs/go/reference#Context
func yourfunction (r *http.Request){
c := appengine.NewContext(r)
c.Debugf("%v\n","Your message")
....
}
You may have to set the minimum log_level in the development server:
dev_appserver.py --log_level debug [your application]
When I want a quick and dirty way to print out some info, I do this:
assert False, "Helpful information"
after getting the needed info, I delete the line.