Classloader issues with Kafka Connector and Remote Executor (Apache Flink) - apache-flink

When trying to execute a Flink job on a remote cluster, i keep getting the same ClassLoader exception.
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:207)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:223)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
10/07/2016 06:34:57 Job execution switched to status FAILING.
I am packaging my jar using the maven-shade plugin and this class is definetely packaged with the jar, as can be seen when I look inside my jar.
$:jar -tf MyJar.jar |grep "org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer"
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.class
Here's my pom.xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.10</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka-0.9_2.10</artifactId>
<version>1.1.2</version>
</dependency>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>FloorData</finalName>
<shadeTestJar>false</shadeTestJar>
<shadedArtifactAttached>false</shadedArtifactAttached>
<createDependencyReducedPom>false</createDependencyReducedPom>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>floordata.cli.launcher.FlinkLauncher</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
I am configuring my remote environment as follows in my code. I am not sure if this is correct, but do I need to specify the location of the kafka connector as an argument to createRemoteEnvironment??
InetSocketAddress address = parseHostPortAddress((String) configuration.get(FLINK_JOB_MANAGER_ADDRESS));
StreamExecutionEnvironment.createRemoteEnvironment(address.getHostName(), address.getPort());

Related

statefun is giving org.apache.flink.client.program.ProgramInvocationException classloader.parent-first-patterns.additional;

I am running stateful-fun 2.0 basic hello job with the following command
./bin/flink run -c org.apache.flink.statefun.flink.core.StatefulFunctionsJob ./stateful-sun-hello-java-1.0-SNAPSHOT-jar-with-dependencies.jar
and my POM.xml is
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>stateful-sun-hello-java</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-sdk</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-flink-distribution</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-kafka-io</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
<build>
<defaultGoal>clean generate-sources compile install</defaultGoal>
<plugins>
<!-- compile proto file into java files. -->
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.6.0.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<includeMavenTypes>direct</includeMavenTypes>
<inputDirectories>
<include>src/main/protobuf</include>
</inputDirectories>
<outputTargets>
<outputTarget>
<type>java</type>
<outputDirectory>src/main/java</outputDirectory>
</outputTarget>
<outputTarget>
<type>grpc-java</type>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.15.0</pluginArtifact>
<outputDirectory>src/main/java</outputDirectory>
</outputTarget>
</outputTargets>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<!-- get all project dependencies -->
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!-- MainClass in mainfest make a executable jar -->
<archive>
<manifest>
<mainClass>org.apache.flink.statefun.flink.core.StatefulFunctionsJob</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- bind to the packaging phase -->
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
It is giving the following exception
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Invalid configuration: classloader.parent-first-patterns.additional; Must contain all of org.apache.flink.statefun, org.apache.kafka, com.google.protobuf
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966
Please suggest how to fix this.
You have to add below configuration parameters to your flink-conf.yaml file.
classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf
jobmanager.scheduler: legacy
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/packaging.html#flink-jar --> Official document says:
The following configurations are strictly required for running StateFun application.
classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf
We need to add this jar
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>statefun-flink-distribution</artifactId>
<version>2.1.0</version>
</dependency>
Package a statefun job to submit to an existing Flink cluster deployment style is no longer mentioned in statefun 3.1 documents. Is this deployment style still possible or supported? If so, where should the module.yaml file be packaged?

DataNucleus enchancer cannot instantiate org.datanucleus.api.jdo.JDOAdapter

My code complies but when I try to run the DataNucleus enhancer, I am unable to get the post compilation step to complete. I presume I am missing a jar file but which one?? I have included the error and the pom.xml
I copy the instructions from the google pages:
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>3.2.0-m1</version>
<configuration>
<api>JDO</api>
<props>${basedir}/datanucleus.properties</props>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.1.3</version>
</dependency>
</dependencies>
</plugin>
And I get this error.
And I get this error.
[ERROR] --------------------
[ERROR] Standard error from the DataNucleus tool + org.datanucleus.enhancer.DataNucleusEnhancer :
[ERROR] --------------------
[ERROR] Exception in thread "main" Error : An error occurred trying to instantiate an instance of the API adapter "org.datanucleus.api.jdo.JDOAdapter" (perhaps you dont have the requisite datanucleus-api-XXX jar in the CLASSPATH, or the
jar for the persistence spec you are using?) : {1}
org.datanucleus.exceptions.NucleusUserException: Error : An error occurred trying to instantiate an instance of the A
adapter "org.datanucleus.api.jdo.JDOAdapter" (perhaps you dont have the requisite datanucleus-api-XXX jar in the CLAS
TH, or the api jar for the persistence spec you are using?) : {1}
at org.datanucleus.api.ApiAdapterFactory.getApiAdapter(ApiAdapterFactory.java:104)
at org.datanucleus.AbstractNucleusContext.(AbstractNucleusContext.java:115)
at org.datanucleus.enhancer.EnhancementNucleusContextImpl.(EnhancementNucleusContextImpl.java:48)
at org.datanucleus.enhancer.EnhancementNucleusContextImpl.(EnhancementNucleusContextImpl.java:37)
at org.datanucleus.enhancer.DataNucleusEnhancer.(DataNucleusEnhancer.java:161)
at org.datanucleus.enhancer.CommandLineHelper.createDataNucleusEnhancer(CommandLineHelper.java:148)
at org.datanucleus.enhancer.DataNucleusEnhancer.main(DataNucleusEnhancer.java:1108)
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<groupId>com.thechrisoneil.mygroupstogo</groupId>
<artifactId>mygroupstogo</artifactId>
<properties>
<appengine.app.version>1</appengine.app.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<prerequisites>
<maven>3.1.0</maven>
</prerequisites>
<dependencies>
<!-- Compile/runtime dependencies, as defined by Google default maven project -->
<!-- https://cloud.google.com/appengine/docs/java/tools/maven -->
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-api-1.0-sdk</artifactId>
<version>1.9.18</version>
</dependency>
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-endpoints</artifactId>
<version>1.9.18</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.1.3</version>
</dependency>
<!-- Dependencies added for datastorage persistents -->
<!-- Datanucleaus (http://www.datanucleus.org/products/datanucleus/jdo/maven.html) -->
<dependency>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.1.3</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.1.3</version>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-testing</artifactId>
<version>1.9.18</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-api-stubs</artifactId>
<version>1.9.18</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
</dependencies>
<build>
<!-- for hot reload of the web application -->
<outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>display-dependency-updates</goal>
<goal>display-plugin-updates</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<version>3.1</version>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<webXml>${project.build.directory}/generated-sources/appengine-endpoints/WEB-INF/web.xml</webXml>
<webResources>
<resource>
<!-- this is relative to the pom.xml directory -->
<directory>${project.build.directory}/generated-sources/appengine-endpoints</directory>
<!-- the list has a default value of ** -->
<includes>
<include>WEB-INF/*.discovery</include>
<include>WEB-INF/*.api</include>
</includes>
</resource>
<!--Development of groupstogo front end is imported to deployment server -->
<resource>
<directory>C:/software/angularjs/my-gtg/app</directory>
<filtering>true</filtering>
<includes>
<include>**/*.js</include>
<include>**/*.html</include>
<include>**/*.png</include>
<include>**/*.css</include>
</includes>
<targetPath>app</targetPath>
</resource>
</webResources>
</configuration>
</plugin>
<plugin>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-maven-plugin</artifactId>
<version>1.9.18</version>
<configuration>
<enableJarClasses>false</enableJarClasses>
<!-- Comment in the below snippet to bind to all IPs instead of just
localhost -->
<!-- address>0.0.0.0</address> <port>8080</port -->
<!-- Comment in the below snippet to enable local debugging with a remove
debugger like those included with Eclipse or IntelliJ -->
<!-- jvmFlags> <jvmFlag>-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n</jvmFlag>
</jvmFlags -->
</configuration>
<executions>
<execution>
<goals>
<goal>endpoints_get_discovery_doc</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>3.2.0-m1</version>
<configuration>
<api>JDO</api>
<props>${basedir}/datanucleus.properties</props>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.1.3</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
I had the same problem and was able to solve it by comparing https://cloud.google.com/appengine/docs/java/datastore/jdo/overview-dn2 and http://www.datanucleus.org/products/accessplatform_3_2/jdo/maven.html
There might be a simple typo in Google's pom.xml snippet. As you can read on the DataNucleus page the maven plug-in will automatically use the latest available datanucleus core. To prevent this use:
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-maven-plugin</artifactId>
<version>3.2.0-release</version>
<configuration>
<api>JDO</api>
<props>${basedir}/datanucleus.properties</props>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.1.3</version>
</dependency>
</dependencies>
</plugin>
And the enhancer will work! So the difference is that Google used <artifactId>datanucleus-api-jdo</artifactId>
which wasn't necessary for me and of course it did not override the choice of the datanucleus-core version that should be used. Please also note that the plugin was renamed from maven-datanucleus-plugin to datanucleus-maven-plugin starting with 3.2.0-m2. So I also changed this to use the official 3.2.0-release.
Furthermore Google describes to use copies of the JARs found in appengine-java-sdk-1.9.21/lib/opt/user/datanucleus/v2 which are:
asm-4.0.jar
datanucleus-api-jdo-3.1.3.jar
datanucleus-api-jpa-3.1.3.jar
datanucleus-appengine-2.1.2.jar
datanucleus-core-3.1.3.jar
geronimo-jpa_2.0_spec-1.0.jar
jdo-api-3.0.1.jar jta-1.1.jar
But since I'm not using ant but maven I simply had to add this dependencies to pom.xml to be able to use JDO with DataNucleus with the versions explicitly supported by Google:
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.1.3</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>2.1.2</version>
</dependency>
BTW: I spotted another bug in the pom.xml as provided by the current appengine-skeleton-archetype. The maven goal "appengine update" failed because the appengine-maven-plugin tried to upload my application with version set to 1.9.21. This is obviously the version of the used GAE SDK and not the version of my app. And it fails because it violates the allowed format for version ids as allowed by GAE. The fix was to correctly set the version in the plugin configuration by adding the line <version>${app.version}</version> like this:
<plugin>
<groupId>com.google.appengine</groupId>
<artifactId>appengine-maven-plugin</artifactId>
<version>${appengine.version}</version>
<configuration>
<enableJarClasses>false</enableJarClasses>
<!-- Comment in the below snippet to bind to all IPs instead of just localhost -->
<!-- address>0.0.0.0</address>
<port>8080</port -->
<!-- Comment in the below snippet to enable local debugging with a remote debugger
like those included with Eclipse or IntelliJ -->
<!-- jvmFlags>
<jvmFlag>-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n</jvmFlag>
</jvmFlags -->
<version>${app.version}</version>
</configuration>
</plugin>
Have fun!
I have my project building properly, being able to generate metaclasses and run the bytecode enhancer. But it's SBT, not Maven.
In case you are interested, please have a look at
http://github.com/frgomes/poc-scala-datanucleus

App Engine, JDO and Maven. Error on startup

I'm trying to create a simple test using JDO with App Engine and a Maven configuration.
My compile and data enhancement steps succeed. But at runtime (both mvn:test and appengine:devserver) I get:
1) Error in custom provider, javax.jdo.JDOFatalInternalException:
Class "com.google.appengine.datanucleus.DatastoreManager" was not found in the CLASSPATH.
Please check your specification and your CLASSPATH.
However, my classpath (target/demo/WEB-INF/lib) does contain: datanucleus-appengine-2.1.1.jar
And my dependencies are the same as those specified in the Google datanucleus project's POM:
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>[3.1.1, 3.2)</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>[3.1.1, 3.2)</version>
</dependency>
<dependency>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>2.1.1</version>
</dependency>
Appreciate any suggestions.
RB
I have everything working now. I thought I'd share a couple of the gotchas (since it took me several days to plough through all of this):
1). All of the versions really matter (esp. matching the App Engine ORM 2.1.1 to DataNucleus 3.1.1 -- including the plugin).
http://www.datanucleus.org/products/accessplatform_3_2/datastores/appengine.html
Here's what I ended up with:
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.1.1</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.1.2</version>
</dependency>
<dependency>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-appengine</artifactId>
<version>2.1.2</version>
</dependency>
...
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<log4jConfiguration>${basedir}/log4j.properties</log4jConfiguration>
<verbose>false</verbose>
<fork>false</fork>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
</plugin>
2). Check the tail of the datanucleus.log to confirm that your classes were enhanced (via mvn datanucleus:enhance). I eventually realized that my test classes (in src/test) were being ignored.
I have added false in pom.xml and it works for me
<plugins>
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>3.1.2</version>
<configuration>
**<fork>false</fork>**
<log4jConfiguration>${basedir}/log4j.properties</log4jConfiguration>
<verbose>true</verbose>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>

Maven GAE Plugin using JDO

I'm using Maven GAE plugin with JDO, I started with jappstart example (https://github.com/tleese22/google-app-engine-jappstart/blob/master/pom.xml), but for some reason classes are not enhanced when I run gae:run command, even I have configured enhancement.
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>1.1.4</version>
<configuration>
<mappingIncludes>**/jdo/*.class</mappingIncludes>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
<api>JDO</api>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>${datanucleus.version}</version>
<exclusions>
<exclusion>
<groupId>javax.transaction</groupId>
<artifactId>transaction-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
<version>${datanucleus.version}</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-enhancer</artifactId>
<version>1.1.4</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
<version>2.3-ec</version>
</dependency>
</dependencies>
</plugin>
Exception:
javax.jdo.JDOUserException: Persistent class "Class com.my.Entity does not seem to have been enhanced. You may want to rerun the enhancer and check for errors in the output." has no table in the database, but the operation requires it. Please check the specification of the MetaData for this class.
I had something wrong with my mappings includes. Now it works if entities are in package that has last name entity (example com.my.app.entity):
<mappingIncludes>**/entity/*.class</mappingIncludes>

Conflicting versions of datanucleus enhancer in a maven google app engine project

I'm having a problem setting up datanucleus enhancer to use with a google app engine project. If I use the datanucleus eclipse plugin everything goes well, but in my maven project I get a strange conflicting version error.
My POM has these datanucleus references:
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>1.1.0</version>
</dependency>
...
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>1.1.0</version>
<configuration>
<mappingIncludes>**/*.class</mappingIncludes>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
<api>JDO</api>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
</plugin>
When I try to build the project I get the following error:
Exception in thread "main" Plugin (Bundle) "org.datanucleus" is already registered.
Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/Users/drome/.m2/repository/org/datanucleus/datanucleus-core/1.1.0/**datanucleus-core-1.1.0.jar**" is already registered, and you are trying to register an identical plugin located at URL "file:/Users/drome/.m2/repository/org/datanucleus/datanucleus-core/1.1.3/**datanucleus-core-1.1.3.jar**."
org.datanucleus.exceptions.NucleusException: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/Users/drome/.m2/repository/org/datanucleus/datanucleus-core/1.1.0/datanucleus-core-1.1.0.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/Users/drome/.m2/repository/org/datanucleus/datanucleus-core/1.1.3/datanucleus-core-1.1.3.jar."
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:437)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:343)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:227
)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.jav
a:159)
at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
at org.datanucleus.OMFContext.(OMFContext.java:164)
at org.datanucleus.enhancer.DataNucleusEnhancer.(DataNucleusEnhancer.java:171)
at org.datanucleus.enhancer.DataNucleusEnhancer.(DataNucleusEnhancer.java:149)
at org.datanucleus.enhancer.DataNucleusEnhancer.main(DataNucleusEnhancer.java:1157)
I don't understand why datanucleus required maven to download datanucleus-core-1.1.3.jar since this is not referenced in the pom.xml
I also do not understand why datanucleus-core-1.1.3.jar is being registered...
Any ideas?
Thanks in advance...
The DN M2 plugin pulls in the latest versions of the available DN jars that it needs to do its job (there is no other sensible way to do it other than use the latest). You want to restrict "core" to a different version, either by specifying the plugin dependency of core, or by specifying that in your application to
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>1.1.0</version>
<scope>runtime</scope>
</dependency>
Unfortunately the answer is "hidden" in the comments:
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>1.1.0</version>
<scope>runtime</scope>
</dependency>
That worked for me!
I ran into the same issue while testing a maven gae plugin archetype.
I fixed it by adding exclusions in my gae runtime transitive dependencies
<!-- Google App Engine meta-package -->
<dependency>
<groupId>net.kindleit</groupId>
<artifactId>gae-runtime</artifactId>
<version>${gae.version}</version>
<type>pom</type>
<exclusions>
<exclusion>
<groupId>com.google.appengine.orm</groupId>
<artifactId>datanucleus-core</artifactId>
</exclusion>
</exclusions>
</dependency>
and then adding the nucleus core as a runtime dependency
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>${datanucleus-core.version}</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>javax.transaction</groupId>
<artifactId>transaction-api</artifactId>
</exclusion>
</exclusions>
</dependency>
as keeping the gae plugin section simple:
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>${maven-datanucleus-plugin.version}</version>
<configuration>
<!--
Make sure this path contains your persistent classes!
-->
<mappingIncludes>**/model/*.class</mappingIncludes>
<verbose>true</verbose>
<enhancerName>ASM</enhancerName>
<api>JDO</api>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
</plugin>
After reading "How to override a plugin's dependency in Maven", I found another way to fix this. Here is my POM:
<plugin>
<groupId>org.datanucleus</groupId>
<artifactId>maven-datanucleus-plugin</artifactId>
<version>3.1.0-m3</version>
<configuration>
<verbose>true</verbose>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.0.4</version>
</dependency>
</dependencies>
</plugin>
clearing your old version of datanucleus from your local maven repository also solving the problem.
Maven-datanucleus-plugin has stopped pulling in the latest versions of the available datanucleus-core since version 3.1.1.
Check the differences between the POM files for Maven-datanucleus-plugin 3.1.1 (http://repo1.maven.org/maven2/org/datanucleus/maven-datanucleus-plugin/3.1.1/maven-datanucleus-plugin-3.1.1.pom) and 3.1.0-release (http://mvnrepository.com/artifact/org.datanucleus/maven-datanucleus-plugin/3.1.0-release).
For maven-datanucleus-plugin 3.1.1 the version range of datanucleus-core dependency is (3.0.99, 3.1.99), and for maven-datanucleus-plugin 3.1.0-release it is (3.0.99, ). No wonder for the older versions of maven-datanucleus-plugin, it automatically pulls in the latest versions of datanucleus-core.

Resources