I like that I can use the Logs API (described here: https://cloud.google.com/appengine/docs/java/logs/) to programatically access and display app & request logs as I see fit--it's great.
Now that I'm using Managed VMs on AppEngine you can see on the Admin Console Logs Viewer that there are a ton of additional logs--including in my case a custom log which I found I could include in the viewer (decribed here: https://cloud.google.com/appengine/docs/managed-vms/custom-runtimes#logging).
My question is: Is there any way I can use the Logs API (or other pipelines already built?) to access these logs? My Managed VM module includes several components which could produce logs that I want to view:
App logs -- I can get these! No problem here.
Custom log files created by background processes I kick off in _ah/start (like "my_custom_1.log" in the screenshot)
STDERR & STDOUT from my background processes
Relevant Managed VM logs (e.g. for when an instance was restarted due to bad health... other system events like normal restarts?)
Basically I want "the total picture" at the instance level. Anyone tried to tame Managed VMs in this way with success? I'm not looking forward to rolling my own solution. And I wouldn't even know where to start on the problem of capturing STDERR and STDOUT. Any help appreciated.
There is a difference between App Engine logging and Google Cloud logging. Some of the Managed VM logs go to both, but much of it only goes to cloud logging.
Until recently there was not an API to read Cloud logs, only to write them. However, there is a new v2 beta API: https://cloud.google.com/logging/docs/api/introduction_v2
To do things at an instance level, entries in Cloud logging should have metadata set to denote which VM they came from. Both of these values seem to vary on logs from my VMs:
compute.googleapis.com/resource_name
compute.googleapis.com/resource_id
Related
I know that when you run a Google App Engine project locally you can view Log statements in your API methods:
log.warning("Some warning because you did something");
But can you view those logs for a deployed project? When I go to Logging in the google developers console I can see logs for various actions of my API but I cannot see my custom logs I put into my methods like the one above.
I am using Java.
There are multiple ways you can achieve this. Though I suggest using one of these two:
Use google's logging library which will write logs for you and they will be available in log viewer and you can export them to your cloud storage or use them as pub/sub events. https://cloud.google.com/logging/docs/api/libraries
If you just want your logs to be visible in the log viewer, another alternative is to use a logging library which writes the logs to syslog (http://www.networkmanagementsoftware.com/what-is-syslog/) and use google-fluentd on your instance. This will collect your custom logs as well as any other event logs to the log viewer.
I had to go into logging.properties in my project and change:
# Set the default logging level for all loggers to WARNING
.level = WARNING
to
# Set the default logging level for all loggers to WARNING
.level = INFO
Now it works. Not sure why though.
We were running on AppEngine but recently moved over to Managed VMs. For some reason AppStats is no longer available? We just get a 404 not found error when browsing to our appstats URL. Is appstats not supported on Managad VMs? If not, is there a way of isolating poorly performing endpoints within our application?
One way to isolate poorly performing endpoints is to use the advanced filter search in the GCP Logs Viewer. It is a little hard to find at first.
To get there, in your Google Cloud console, navigate to Logging for your project. At the right of the text box for "Filter by label or text search" you will see a small dropdown arrow. Click that and select "Convert to advanced filter". This will allow you to write your own sql-ish query where you can find requests that took longer than n to complete.
For example, add the following to the filter:
protoPayload.latency>"0.300s"
This will return a list of all requests that took longer than 300 milliseconds to process. If you have Cloud Trace enabled, you can click on the request response time to see the timeline for the individual service calls.
I'm trying to migrate an a web app from Google Appengine to a dedicated server and I've got stuck to the logging issue. Basically I would like to organise the logs per request/context(like on GAE) so that I can easily review the errors/trace on each request. The most advanced logging library I could find is the glog package but still I can't figure it out how to log per request/context.
Each request gives you a http.Request-object to work with.
If you're using sessions, then you'll have a sessions.Session-object to work with.
You will want to use those objects to help log per request/context, as they identify the request / session.
I am using CapeDwarf JBoss AS7 to host classic GAE+GWT application. On server side, I am logging directly to java.util.Logger framework.
The application works fine, the logs print nice to the console and files.
However, all my logs, together with some http request logs, are also stored in the datastore. Yes, every log line becomes one datastore entity. This should not surprise me, because it is stated in the docs (http://www.jboss.org/capedwarf/docs) and it is needed in order to display logs in gae-like admin console of CapeDwarf.
I was looking for a way to disable this useful but expensive feature. I cannot find any docs that would tell me what to do if I do not need/want my logs in the datastore (e.g. right now 98.3% of my datastore are __org.jboss.capedwarf.LogRequest__ and __org.jboss.capedwarf.LogLine__ entities!). Does anyone happen to know how to do it?
Already answered on our forum: https://community.jboss.org/message/810735#810735
I have an application running on JBoss AS 7 and creating log files in /standalone/log.
For security reasons I not allowed to browse JBoss directories.
Is there any build-in application to read these logs files from a browser ?
NB : I cannot use admin console either.
No, nothing built in. You can have the admins configure the logging service to put logs where you can get to them, or you can configure the logger to capture logs and post to a database or other.
Not yet, but there are some requests for it (one by me, BTW ;-) and it might appear in WildFlz 8. Hopefully. (Vote on them if you like.)
WFLY-1048 Allow hooking into logging subsystem through Management API
WFLY-1144 Provide the ability to view server logs through the web interface
WFLY-280 Provide an operation to retrieve the last 10 errors from the log
Until then, I suggest to ask the admins to allow access to that one particular log file.
If that doesn't pass through, you may declare dependency of your deployment on a logging service's modules (Dependencies: ... in MANIFEST.MF) and the log manager in JVM. Unless there's some additional obstacle like security manager or so.