We have several applications that use log4j for logging. I need to get a log4j parser working so we can combine multiple log files and run automated analysis on them. I'm not looking to reinvent the wheel, so can someone point me to a decent pre-existing parser? I do have the log4j conversion pattern if that helps.
If not, I'll have to roll our own.
I didn't realize that Log4J ships with an XML appender.
Solution was: specify an XML appender in the logging configuration file, include that output XML file as an entity into a well formed XML file, then parse the XML using your favorite technique.
The other methods had the following limitations:
Apache Chainsaw - not automated enough
jdbc - poor performance in a high performance distributed app
You can use OtrosLogViewer with batch processing. You have to:
Define you log format, you can use Log4j pattern layout parser or Log4j XmlLayout
Create java class that implements LogDataParsedListener. Method public void logDataParsed(LogData data, BatchProcessingContext context) will be called on every parsed log event.
Create jar
Run OtrosLogViewer with specifying your log processing jar, LogDataParsedListener implementation and log files.
What you are looking for is called SawMill, or something like it.
Log4j log files aren't really suitable for parsing, they're too complex and unstructured. There are third party tools that can do it, I believe (e.g. Sawmill).
If you need to perform automated, custom analysis of the logs, you should consider logging to a database, and analysing that. JDBC ships with the JdbcAppender which appends all messages to a database of your choice, but it has performance implications, and it's a bit flaky. There are other, similar, alternatives on the interweb, though (like this one).
You -can- use Log4j's Chainsaw V2 to process the various log files and collect them into one table, and either output those events as xml or use Chainsaw's built-in expression-based filtering, searching & colorizing support to slice & dice the logs.
Steps:
- Start Chainsaw V2
- Create a chainsaw configuration file by copying the example configuration file available from the Welcome tab - define one LogFilePatternReceiver 'plugin' entry for each log file that you want to process
- Start Chainsaw with that configuration
- Each log file will end up as a separate tab in the UI
- Pause the chainsaw-log tab and clear the events from that tab
- Create a new tab which aggregates the events from the various tabs by going to the 'view, crate custom expression logpanel' menu item and enter 'level >= DEBUG' in the box. It will create a new tab containing events from all of the tabs with level >= debug (which is why you cleared the chainsaw-log tab).
You can get an overview of the expression syntax used to filter, colorize and search from the tutorial (available from the Help menu).
If you don't want to use Chainsaw, you can do something similar - start a simple app that doesn't log but loads a log4j.xml config file with the 'plugin' entries you defined for the Chainsaw configuration, but also define a FileAppender with an xmllayout - all of the events received by the 'receivers' will be sent to the single appender.
Related
We're designing a system for a client where they are allowing authenticated users to upload images. We've created an API to upload the files but the client only wants the latest file and delete all previous ones so that there would only ever be one.
We've looked through the docs and can't come across a way for ADAM to handle this in both 2SXC and DNN's file system.
Internally when deleting images we see API calls like the following to the internal 2SXC API, but we're wondering if this is exposed somewhere within the public API?
https://somedomain.com/api/2sxc/app/auto/data/61393528-b401-411f-a001-f423ea46700a/b7d04e2c-c565-496c-8efb-aa133cf90d33/Photo/delete?subfolder=&isFolder=false&id=189&usePortalRoot=false&appId=3
We could probably use the same endpoint above, but we'd likely run into permission issues or changes to the APIs that could be problematic.
Thank you for any advice you can give! Perhaps #iJungleBoy can provide some thoughts on this.
As a solution from a completely different direction, if you are on the later release of 2sxc (v12.8+, v13+), and comfortable programming in C#, you might consider doing this as a "cleanup" from a Dnn Scheduled Task. This can be done with a relatively easy setup. We have a Gist in place that we use as a starter. You simply put the code in the /App_Code folder then setup a normal Dnn Scheduled Task. NOTE that you can scroll down to the first comment on the Gist to see a screenshot of a complete working setup.
Accuraty's AccuTasks template on GitHub Gists
There are two more key things to note:
You need to install Dnn's CodeDom 3.6 because the example uses the later versions C#'s string interpolation - OR remove the few $"ASL2021 - {this.GetType().Name}, Task Scheduled Email", bits or convert to string.Format() or something.
Since your task's code is NOT running in a (2sxc) module, if needed, you'll do stuff like this: 2sxc Docs - Use 2sxc Instance or App Data from External C# Code
So, if you are comfortable writing code that "finds and deletes stuff older than NN days" - this might be the way to go.
I have an embedded C project which uses subversion for source control. I want to use Tessy for unit testing and have these tests archived in subversion too. However, it generates many small files which will make analysing diffs for the actual source code changes a real pain. Trying to actually look at the source changes when there are hundreds of Tessy related files changed will make it impossible.
Does anyone know if there is a setting to have these stored in a less problematic format or any suggestions for a viable solution? What would be ideal is if it could store everything as, for example, an xml file - this would make browsing directory diffs easier and would allow the actual content to be human readable as well.
Any ideas?
I know this is an old question ...
Does anyone know if there is a setting to have these stored in a less problematic format or any suggestions for a viable solution?
The TESSY recommended way is to do utilize the database save feature found under in File menu (and in a variety of right-click menu's). This creates a binary .tmb file which contains everything related to your tests. By default the .tmb file is stored in the backup directory in your Tessy Project folder. The config folder, backup folder and the PDBX file would then all be stored in SVN. See the Tessy Users Manual (Backup, restore, version control chapter) for more specifics.
What would be ideal is if it could store everything as, for example, an xml file - this would make browsing directory diffs easier and would allow the actual content to be human readable as well.
That would be ideal, but unfortunately is not really an option. Having everything stored as a binary file makes it impossible to do a useful diff. The other problem with this method is that it disconnects a change to the test from the file that is checked into SVN - unless the tester specifically performs a database save.
Yes I realize xUnit testing frameworks don't have those limitations, but Tessy has some features (like MCDC and DO178B support) that the xUnit frameworks do not have baked in.
So how do you work in this environment. Key word - Discipline.
We set up internal procedures for who and how tests gets updated. When the proecedures are followed we are able to deal with the limitations presented above. It is not optimal, but with some internal discipline it can work.
I need to create application logs to capture users signing in/out and their requests, for that.
We're using Java EE, and thought that creating new log files (new txt file for each day) would be a good approach, but I see that people discourage doing that, the question is: why not do it that way, and what is the correct approach?
also - is there some way to get the application directory?
log4j is one of the popularly used logger for Java EE applications and the others are slf4j,logback
log4j has many features, one them being able to create daily log files.
and to answer your question,
creating daily log files does not cause any harm to your application.
Logging to text files and rolling them daily is quite a normal approach and discouraging it per se is not justified.
For some specific uses it may be improper, for example if you log sensitive data (passwords, card numbers, etc.). There may be also issues with some cluster configurations, but then you have to ask a more detailed question.
Log4J works fine, but once you have many different Applications logging to many different Log files in different locations, you encounter the problem of having to search many log files to find the trace of certain Transaction.
One colleague recommended GrayLog2 once, which makes the viewing of the Log Files a lot easier.
You might want to take a look at that as well, depending on how many Log files your planning to keep.
http://graylog2.org/about
I'm not guru in DBA, so I'll try explain what I want in terms I imagine it.
I have Oracle DB with network devices. each device has ports which has parent device/port
I want some tool which will automaticaly create visual map of this device relations.
Will create "Network Map" based on this relations.
It's would be better if this tool will have some output ready for web publishing, or web based tool from the begging. Also if it will automatically update "picture" as soon as I add new relation/object
From far it looks something like Gource http://youtu.be/E5xPMW5fg48
But not exactly what i need
Hope to get some suggestion.
Thanks in advance!
UPD: found another tool: Gephi
You could try graphviz. It was created specifically for visualising large graphs of network nodes.
It's not out of the box; you'll have to write some code that:
Reads data on the devices & their relationships
Creates the graphviz input file
generates the diagram by calling the graphviz binary.
There are many ways to do that. One of the easiest is to use python with the pydot library.
Note that graphviz generates static images (jpeg / tiff etc.) so you'd have to regenerate on demand.
There are more interactive toolkits available, e.g. protovis / infovis. Both are javascript based and render directly in the browser.
hth.
I'd like to know your approach/experiences when it's time to initially populate the Grails DB that will hold your app data. Assuming you have CSVs with data, is is "safer" to create a script (with whatever tool fits you) that:
1.-Generates the Bootstrap commands with the domain classes, run it in test or dev environment and then use the native db commands to export it to prod?
2.-Create the DB's insert script assuming GORM's version = 0 and incrementing manually the soon-to-be autogenerated IDs ?
My fear is that the second approach may lead to inconsistencies for hibernate will have the responsability for the IDs generation and there may be something else I'm missing.
Thanks in advance.
Take a look at this link. This allows you to run groovy scripts in the normal grails context giving you access to all grails features including GORM. I'm currently importing data from a legacy database and have found that writing a Groovy script using the Groovy SQL interface to pull out the data then putting that data in domain objects appears to be the easiest thing to do. Once you have the data imported you just use the commands specific to your database system to move that data to the production database.
Update:
Apparently the updated entry referenced from the blog entry I link to no longer exists. I was able to get this working using code at the following link which is also referenced in the comments.
http://pastie.org/180868
Finally it seems that the simplest solution is to consider that GORM as of the current release (1.2) uses a single sequence for all auto-generated ids. So considering this when creating whatever scripts you need (in the language of your preference) should suffice. I understand it's planned for 1.3 release that every table has its own sequence.