Log to a database using log4j - database

Since in log4j javadoc is
WARNING: This version of JDBCAppender is very likely to be completely replaced in the future. Moreoever, it does not log exceptions.
What should I do to log to a database?

If you are looking for a database appender which not only works, but also supports connection pooling, is maintained and properly documented, than consider logback's DBAppender.
Ironically enough, the warning in the javadocs about removing JDBCAppender in future versions of log4j was written by me.

You can use an alternative appender, but really Log4j 1.2 is going to be around and standard for a long time. They developed DBAppender as part of their receivers companions, which isn't officially released, but you can download the source code and get your own going as well.
Unless the issue of not logging exceptions bothers you, JDBCAppender is just fine. Any further upgrade to 2.0 is going to be more radical than just changing JDBCAppender (if 2.0 happens), so I wouldn't worry about using it, despite the warning. They clearly don't have a solid roadmap or timeline to introducing a new version, and 1.2.15 was released in 2007.

**log4j.properties file**
# Define the root logger with appender file
log4j.rootLogger = DEBUG, DB
# Define the DB appender
log4j.appender.DB=org.apache.log4j.jdbc.JDBCAppender
# Set JDBC URL
log4j.appender.DB.URL=jdbc:mysql://localhost/log
# Set Database Driver
log4j.appender.DB.driver=com.mysql.jdbc.Driver
# Set database user name and password
log4j.appender.DB.user=root
log4j.appender.DB.password=root
# Set the SQL statement to be executed.
log4j.appender.DB.sql=INSERT INTO actionlg(user_id, dated, logger, level, message) values('%X{userId}',' %d{yyyy-MM-dd-HH-mm}','%C','%p','%m')
# Define the layout for file appender
log4j.appender.DB.layout=org.apache.log4j.PatternLayout
**Java Class**
Log4jExamples.java
import java.sql.*;
import java.io.*;
import org.apache.log4j.Logger;
import org.apache.log4j.MDC;
public class Log4jExample {
/* Get actual class name to be printed on */
static Logger log = Logger.getLogger(Log4jExample.class.getName());
public static void main(String[] args)throws IOException,SQLException{
log.error("Error");
MDC.put("userId", "1234");
}
}
**libs required**
- mysql-connector-java-3.1.8-bin.jar
- log4j-1.2.17.jar

Related

EPiServer Error during import: System.ArgumentException: Cannot be the same as destination

I am trying to import content from our UAT server to our Production server.
The Production database was originally cloned from the UAT database, both code bases are the same.
My procedure when exporting/importing is really straight forward, first: http://screencast.com/t/UkT1W7t7g
Then I get the final file (7MB+) with all the content from UAT, then I go to Production and select the following: http://screencast.com/t/UkT1W7t7g
When clicking on import I get the following error: http://screencast.com/t/lgr7jeFX
This is what I get in the error log:
2016-11-28 10:31:04,995 [5] ERROR EPiServer.Core.Transfer.TransferLogger: 10.5.3 Export/import error: Exception: Cannot be the same as destination
Parameter name: contentLinkID
System.ArgumentException: Cannot be the same as destination
Parameter name: contentLinkID
at EPiServer.DataAccess.ContentSaveDB.Move(Int32 contentLinkID, Int32 destinationLinkID, Boolean archive)
at EPiServer.DataAbstraction.ContentStore.Move(Int32 contentLink, Int32 destinationLinkID, Boolean archive)
at EPiServer.DefaultContentProvider.Move(ContentReference contentReference, ContentReference destinationLink)
at EPiServer.Core.Transfer.ContentTransfer.MoveContent(IContent content, ContentReference parentLink, AccessLevel requiredDestinationAccess)
at EPiServer.Core.Transfer.ContentTransfer.Import(RawContent rawContent, AccessLevel requiredDestinationAccess, Guid& importedPageGuid)
at EPiServer.Core.Transfer.ContentTransfer.Import(ITransferContentData content, AccessLevel requiredDestinationAccess)
at EPiServer.Enterprise.DataImporter.ImportContents[T](XmlTextReader reader, ZipPackage package)
at EPiServer.Enterprise.DataImporter.ImportStream(ZipPackage package, XmlTextReader reader, String partName)
at EPiServer.Enterprise.DataImporter.ImportPartOfPackage(ZipPackage package, String partName)
at EPiServer.Enterprise.DataImporter.ImportRaw(ZipPackage package)
at EPiServer.Enterprise.DataImporter.Import()
System.ArgumentException: Cannot be the same as destination
Parameter name: contentLinkID
at EPiServer.DataAccess.ContentSaveDB.Move(Int32 contentLinkID, Int32 destinationLinkID, Boolean archive)
at EPiServer.DataAbstraction.ContentStore.Move(Int32 contentLink, Int32 destinationLinkID, Boolean archive)
at EPiServer.DefaultContentProvider.Move(ContentReference contentReference, ContentReference destinationLink)
at EPiServer.Core.Transfer.ContentTransfer.MoveContent(IContent content, ContentReference parentLink, AccessLevel requiredDestinationAccess)
at EPiServer.Core.Transfer.ContentTransfer.Import(RawContent rawContent, AccessLevel requiredDestinationAccess, Guid& importedPageGuid)
at EPiServer.Core.Transfer.ContentTransfer.Import(ITransferContentData content, AccessLevel requiredDestinationAccess)
at EPiServer.Enterprise.DataImporter.ImportContents[T](XmlTextReader reader, ZipPackage package)
at EPiServer.Enterprise.DataImporter.ImportStream(ZipPackage package, XmlTextReader reader, String partName)
at EPiServer.Enterprise.DataImporter.ImportPartOfPackage(ZipPackage package, String partName)
at EPiServer.Enterprise.DataImporter.ImportRaw(ZipPackage package)
at EPiServer.Enterprise.DataImporter.Import()
I do not get what the problem might be, any help is really appreciated.
P.D.: I already asked the same at EPiServer Dev Forum http://world.episerver.com/forum/developer-forum/-Episerver-75-CMS/Thread-Container/2016/11/error-during-import-system-argumentexception-cannot-be-the-same-as-destination
I think this is what happens?
Your exported package will be imported below the node you select when importing - not overwrite the selected destination node.
For example, you could export a start page beneath the root page in UAT, and then select Root as the destination in production. On import, the start page would end up below the Root page.
The error occurs when exporting the Root page in your UAT environment, and then trying to export that to the Root page in your production environment.
Essentially you're trying to import the source root page under the destination root page (hence the exception about content ID being the same as the destination ID).
Thanks for the explanation Ted. I did the following and it worked:
In UAT
Export starts: http://screencast.com/t/FgHRnOhuauR with "Export files that the pages link to" unchecked.
Export ends: http://screencast.com/t/aPe0Ntvb9aq
10583 content items were exported into the file.
Then in Production
Import starts: http://screencast.com/t/qqkBsL731P with "Update existing content items with matching ID" checked.
Import in-progress: http://screencast.com/t/reSfMnCV8
One thing I noticed is that at the end it says it only updated a portion of the content items, in fact I did it a couple of times as UAT is being actively edited and got results with 28, 16, 1, or 8 items imported (out of 10500+) I guess that is because those were the only with changes or new items.
I did checked a couple of blocks that I knew were updated and the were matching with the new version after the import.
As this did the work I am marking Ted´s answer as accepted.
I think is kinda confusing the selection of elements in the content tree since I wanted to take everything from root and move it to the root on the other server but it seems that is not how it works.
That error occurs when contentLinkID equals destinationLinkID
The reflected code confirms this
public void Move(int contentLinkID, int destinationLinkID, bool archive)
{
if (contentLinkID == destinationLinkID)
{
throw new ArgumentException("Cannot be the same as destination", "contentLinkID");
}
// ...
Check your episerver log files, they will tell you which contentLinkID that causes the error.
Best guess is that the
application pool account doesn't have sufficient rights on the database or on the file system
application pool cannot write or update the files in app_data
Typically this is where most people give up, instead migrating Episerver using database backup/restore instead. However, if you really want to go full nerd on this I suggest customizing the import, check the article Patrick van Kleef about content migration http://www.patrickvankleef.com/2015/08/18/episerver-content-migration/.
This is an issue Episerver really should fix asap!

Play Slick config

I'm trying to get started with play and slick.
Strategy; take hello-slick-3.1 project from the activator tutorials.
If works fine with the H2 in memory database. I want to connect to a sql server. After a battle I have some configuration which appear to connect using jdts.
In application .conf
driver=net.sourceforge.jtds.jdbc.Driver
url="jdbc:jtds:sqlserver://%%%%:1433;databaseName=%%%%;user=%%%;password=%%%%%"
This is using the jtds driver instead of
com.typesafe.slick.driver.ms.SQLServerDriver
Which appears to have been made deliberately difficult to use. I have not found a sucessful config with it. JTDS manages to create the 'suppliers' table based on it's schema, but all subsequent requests fall over with a nebulous 'data truncation' message;
object HelloSlick extends App {
val db = Database.forConfig("sqlServerLocal")
try {
// The query interface for the Suppliers table
val suppliers: TableQuery[Suppliers] = TableQuery[Suppliers]
val setupAction: DBIO[Unit] = DBIO.seq(
// Create the schema by combining the DDLs for the Suppliers and Coffees
// tables using the query interfaces
//(suppliers.schema).create,
// Insert some suppliers
suppliers += (101, "Acme, Inc.", "99 Market Street", "Groundsville", "CA", "95199"),
suppliers += ( 49, "Superior Coffee", "1 Party Place", "Mendocino", "CA", "95460"),
suppliers += (150, "The High Ground", "100 Coffee Lane", "Meadows", "CA", "93966")
)
Telling me that
background log: info: 10:58:48.465 [sqlServerLocal-1] DEBUG slick.jdbc.JdbcBackend.statement - Preparing statement: insert into "SUPPLIERS" ("SUP_ID","SUP_NAME","STREET","CITY","STATE","ZIP") values (?,?,?,?,?,?)
background log: error: Exception in thread "main" java.sql.DataTruncation: Data truncation
Does anyone have any ideas? Is connecting to SQL server with slick - 3.1 even a sensible thing to attempt?
EDIT::
#szeiger makes the excellent point that I'm importing the wrong thing into the model classes. I had the old H2 driver import from the Hello Slick example, which should be replaced with this;
import com.typesafe.slick.driver.ms.SQLServerDriver.api._
In order to work as a SQL server language.
After making this change, the DB configuration which 'did something' no longer works, advertising this
background log: info: Running HelloSlick
background log: error: Exception in thread "main" java.lang.NoClassDefFoundError: slick/profile/BasicProfile$SimpleQL
I've tried altering the instantiation of the actual db variable to match the 'recommended' strategy suggested by szeiger.
val db = Database.forURL("jdbc:sqlserver://%%%:1433;user=%%%%;password=%%%%", driver="com.typesafe.slick.driver.ms.SQLServerDriver",
executor = AsyncExecutor("test1", numThreads=10, queueSize=1000))
Unfortunately, this fails like so;
background log: error: Exception in thread "main" java.lang.NoClassDefFoundError: slick/profile/BasicProfile$Implicits
with the jtds driver, it has the same error message with the slick MSSQL driver, and despite my best efforts, I cannot instantiate the MS JDBC driver. It's in a folder called 'lib' in the application , although I suspected that might be naive and the wrong place.
Managing this dependency through play-slick would be excellent. I had thought that I would be doing that via the inclusion of this line in SBT;
"com.typesafe.play" %% "play-slick" % "1.1.0",
but appears to have no effect in isolation, and I'm unsure how to configure this to reach the 'official' slick MS SQL driver.
EDIT 2:: Finally, this appears to build what I want.
name := """hello-slick-3.1"""
lazy val root = (project in file(".")).enablePlugins(PlayScala)
resolvers += "Typesafe Releases" at "http://repo.typesafe.com/typesafe/maven-releases/"
resolvers += "Scalaz Bintray Repo" at "https://dl.bintray.com/scalaz/releases"
resolvers += Resolver.url("Typesafe Ivy releases", url("https://repo.typesafe.com/typesafe/ivy-releases"))(Resolver.ivyStylePatterns)
scalaVersion := "2.11.6"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.0",
"com.typesafe.slick" %% "slick-extensions" % "3.1.0",
"com.typesafe.play" %% "play-slick" % "1.1.0",
"org.scalatest" %% "scalatest" % "2.2.4" % "test"
)
I had lots of problems because I forgot to include the correct resolvers.
Don't forget to include the correct resolvers :-).
You're confusing Slick drivers and JDBC drivers. Which Slick driver are you using? The imports are missing from your snippet but there should be something like import com.typesafe.slick.driver.ms.SQLServerDriver.api._. Using a wrong driver here could explain String columns being created as VARCHAR(1).
The recommended way of configuring database connections in Slick 3.1 is via DatabaseConfig, which allows you to configure the Slick driver together with the actual connection parameters. When you're writing a Play app, use the play-slick plugin instead to handle database connections. It is also based on the DatabaseConfig syntax.
The alternative to net.sourceforge.jtds.jdbc.Driver would be Microsoft's own JDBC driver, com.microsoft.sqlserver.jdbc.SQLServerDriver which is available as a separate download from Microsoft. You have to manually add sqljdbc4.jar to your build to make this work.
BTW, Slick 3.2 will change terminology and use the word "driver" exclusively for JDBC drivers. Slick drivers will be called "profiles" in 3.2.
After bonus pain, it looks like slick creates columns in SQL Server as type 'varchar(1)'.
Which can't hold very much data.
Not using Slick to create the tables, and commenting out these lines then means that the intro works as advertised.

log4j2 how to disable "date:" lookup - log4j throws exception

edit seems not to be possible at the moment filed an issue.
i am using log4j2 in my apache camel application. In camel file names can be configured this way "?fileName=${date:now:yyyyMMdd-HHmmss}ID.${id}.gz"
if i set log level to debug camel tries to log what it is doing but log4j seems to try to lookup/interpret the string with "date:" and throws an exception:
2014-11-24 11:29:19,218 ERROR Invalid date format: "now:yyyyMMdd-HHmmss", using default java.lang.IllegalArgumentExcepti
on: Illegal pattern character 'n'
at java.text.SimpleDateFormat.compile(Unknown Source)
at java.text.SimpleDateFormat.initialize(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at org.apache.logging.log4j.core.lookup.DateLookup.formatDate(DateLookup.java:60)
at org.apache.logging.log4j.core.lookup.DateLookup.lookup(DateLookup.java:53)
at org.apache.logging.log4j.core.lookup.Interpolator.lookup(Interpolator.java:144)
at org.apache.logging.log4j.core.lookup.StrSubstitutor.resolveVariable(StrSubstitutor.java:1008)
at org.apache.logging.log4j.core.lookup.StrSubstitutor.substitute(StrSubstitutor.java:926)
at org.apache.logging.log4j.core.lookup.StrSubstitutor.substitute(StrSubstitutor.java:816)
at org.apache.logging.log4j.core.lookup.StrSubstitutor.replace(StrSubstitutor.java:385)
at org.apache.logging.log4j.core.pattern.MessagePatternConverter.format(MessagePatternConverter.java:71)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:36)
at org.apache.logging.log4j.core.layout.PatternLayout.toSerializable(PatternLayout.java:189)
at org.apache.logging.log4j.core.layout.PatternLayout.toSerializable(PatternLayout.java:53)
at org.apache.logging.log4j.core.layout.AbstractStringLayout.toByteArray(AbstractStringLayout.java:52)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:
104)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:97)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:428)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:407)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:365)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:112)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1347)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1312)
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:132)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:518)
at org.apache.camel.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:570)
at org.apache.camel.util.IntrospectionSupport.setProperties(IntrospectionSupport.java:454)
at org.apache.camel.util.EndpointHelper.setProperties(EndpointHelper.java:249)
at org.apache.camel.impl.DefaultComponent.setProperties(DefaultComponent.java:272)
at org.apache.camel.component.file.GenericFileComponent.createEndpoint(GenericFileComponent.java:67)
at org.apache.camel.component.file.GenericFileComponent.createEndpoint(GenericFileComponent.java:37)
at org.apache.camel.impl.DefaultComponent.createEndpoint(DefaultComponent.java:123)
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:514)
at org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:547)
Is there a way to turn off this "date:" lookup? Why does it try to interpret stuff coming from log at all? I think it should not be touched in any way?!
Edit, very easy to reproduce in test:
public class LogTest {
private Logger log = LoggerFactory.getLogger(LogTest.class);
#Test
public void test() {
log.info("${date:now:buhu}");
}
}
It is crucial to us ${date:} - only "data:now" is working.
So this problem is completely independent from camel, but camel uses ${date:...} pattern for several things. Here is a simple route that reproduces the problem - the exception will be thrown on camel set up phase - no test code needed - logging level must be "debug"!:
public class LogTest extends CamelTestSupport{
private Logger log = LoggerFactory.getLogger(LogTest.class);
#Test
public void test() {
//log.info("${date:now:yyyyMMdd-HHmmss}");
}
#Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("direct:a").to("file:./?fileName=${date:now:yyyyMMdd-HHmmss}ID.${id}.gz");
}
};
}
}
This issue was fixed in 2.7 version of Log4j2.
The solution is to upgrade to that version (or higher) and add in the pattern attribute the option "{nolookups}" to %msg .
%msg{nolookups}
For example
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %class{1} %L %M %t - %msg{nolookups}%n%xEx%n" />
The problem can be avoided, if the simple-Expression is written as $simple{..} instead of ${..}. Then log4j2 won't use his Date-Lookup.
So, if you change your Route from:
from("direct:a").to("file:./?fileName=${date:now:yyyyMMdd-HHmmss}ID.${id}.gz");
to:
from("direct:a").to("file:./?fileName=$simple{date:now:yyyyMMdd-HHmmss}ID.${id}.gz");
it should work, even if you debug Camel.
To disable the date lookup locally, you can add a "$" in front of the expression:
log.info("$${date:now:buhu}");
This will print ${date:now:buhu} instead of throwing an exception printing the stack trace.
As for how to avoid this using Camel, I'm not sure. The cleanest fix would probably be a log4j2 update to disable their DateLookup feature. A temporary fix is to disable DEBUG level logs from the org.apache.camel package:
<loggers>
<logger name="org.apache.camel" level="INFO" />
<root level="debug">
<appender-ref ref="Console" />
</root>
</loggers>
It's not ideal, but we can increase the log level if we ever need to debug Camel context creation since the log statements are not necessary for general everyday development.
The correct solution is now to upgrade the log4j-core library to 2.15.0 or above. At time of writing, the latest and current recommended version is 2.16.0.
The variable substitutions happening in logged messages here are symptoms of the same feature exploited in CVE-2021-44228, aka Log4Shell.
The feature is disabled by default in 2.15.0 and removed in 2.16.0.
It's not news to anyone by now, but it's really important to take steps to disable this feature, as a security measure, even if not using Apache Camel or encountering the issue as described.
As an aside, I found this question when searching for early warning signs of the Log4Shell vulnerability. I've quoted it in my write-up.

How can I get AppEngine to log info level only for my app?

So I've tried configuring AppEngine logging according to this guide, ensuring I've configured the logging.properties file to be used in web.xml. I've configured logging.properties the following way:
.level = WARNING
nilsnett.chinese.backend.level = INFO
The package name of my logging wrapper is nilsnett.chinese.backend. The problem is that even with this configuration, info-level log output from my app is filtered. Evidence:
I've also tried the following config, which yielded the same result (including the logger class name at the end of the package name):
.level = WARNING
nilsnett.chinese.backend.JavaUtilLogger.level = INFO
To demonstrate that the logging.properties-file is actually read, and that I actually do write info-level logging data to app-engine in this service call, let me show you what happens when I set.level=INFO:
So my desired result is to have INFO and higher-level log outputs from my packages, while other packages, like org.datanucleus, only shows output if WARNING or more severe. In the example above, I want only the two lines marked with the purple star. Am I doing anything wrong?
change your config to:
.level = WARNING
# Set the default logging level for the datanucleus loggers
DataNucleus.JDO.level=WARNING
DataNucleus.Persistence.level=WARNING
DataNucleus.Cache.level=WARNING
DataNucleus.MetaData.level=WARNING
DataNucleus.General.level=WARNING
DataNucleus.Utility.level=WARNING
DataNucleus.Transaction.level=WARNING
DataNucleus.Datastore.level=WARNING
DataNucleus.ClassLoading.level=WARNING
DataNucleus.Plugin.level=WARNING
DataNucleus.ValueGeneration.level=WARNING
DataNucleus.Enhancer.level=WARNING
DataNucleus.SchemaTool.level=WARNING
# FinalizableReferenceQueue tries to spin up a thread and fails. This
# is inconsequential, so don't scare the user.
com.google.common.base.FinalizableReferenceQueue.level=WARNING
com.google.appengine.repackaged.com.google.common.base.FinalizableReferenceQueue.level=WARNING
this is are coming from logging config template, so to set datanucleus to warning you have todo like in this template.
https://developers.google.com/appengine/docs/java/#Logging
and then just add your own logging config:
nilsnett.chinese.backend.level = INFO
this should solve it

Tomcat 7 - Retrieve the version of a webapp (versioned WAR)

I've been unable to find any easy way of figuring out the version string for a WAR file deployed with Tomcat 7 versioned naming (ie app##version.war). You can read about it here and what it enables here.
It'd be nice if there was a somewhat more supported approach other than the usual swiss army knife of reflection powered ribcage cracking:
final ServletContextEvent event ...
final ServletContext applicationContextFacade = event.getServletContext();
final Field applicationContextField = applicationContextFacade.getClass().getDeclaredField("context");
applicationContextField.setAccessible(true);
final Object applicationContext = applicationContextField.get(applicationContextFacade);
final Field standardContextField = applicationContext.getClass().getDeclaredField("context");
standardContextField.setAccessible(true);
final Object standardContext = standardContextField.get(applicationContext);
final Method webappVersion = standardContext.getClass().getMethod("getWebappVersion");
System.err.println("WAR version: " + webappVersion.invoke(standardContext));
I think the simplest solution is using the same version (SVN revision + padding as an example) in .war, web.xml and META-INF/MANIFEST.MF properties files, so you could retrieve the version of these files later in your APP or any standard tool that read version from a JAR/WAR
See MANIFEST.MF version-number
Another solution described here uses the path name on the server of the deployed WAR. You'd extract the version number from the string between the "##" and the "/"
runningVersion = StringUtils.substringBefore(
StringUtils.substringAfter(
servletConfig.getServletContext().getRealPath("/"),
"##"),
"/");
Starting from Tomcat versions 9.0.32, 8.5.52 and 7.0.101, the webapp version is exposed as a ServletContext attribute with the name org.apache.catalina.webappVersion.
Link to the closed enhancement request: https://bz.apache.org/bugzilla/show_bug.cgi?id=64189
The easiest way would be for Tomcat to make the version available via a ServletContext attribute (org.apache.catalina.core.StandardContext.webappVersion) or similar. The patch to do that would be trivial. I'd suggest opening an enhancement request in Tomcat's Bugzilla. If you include a patch then it should get applied fairly quickly.

Resources