Pentaho Windows Task Scheduler - batch-file

I am trying to schedule a job using a .bat file. When i run it it gets stuck on start of job execution. Does anyone know how I can fix this issue so I can run my jobs?
Does this have anything to do with Java? I do not have the variable PENTAHO_JAVA_HOME. I only have JAVA_HOME. Not sure if it has to do with my kitchen.bat file. I only using Spoon.bat.
I think its a problem with my Kitchen.bat (.kjb) because if i run the batch file with Pan.bat (.ktr) my file runs fine.
C:\Users\bxt0\Desktop>c:
C:\Users\bxt\Desktop>cd /d "C:\data-integration"
C:\data-integration>call Kitchen.bat
/file:C:\Users\bxt\Desktop\CCMStatsJob.
kjb "-param:TABLE_NAME=region" -logfile=C:\Users\bxt058y\Documents\Pentaho
Jobs\
ccmjob.txt
DEBUG: Using JAVA_HOME
DEBUG: _PENTAHO_JAVA_HOME=C:\Program Files\Java\jre1.8.0_74
DEBUG: _PENTAHO_JAVA=C:\Program Files\Java\jre1.8.0_74\bin\java.exe
C:\data-integration>"C:\Program Files\Java\jre1.8.0_74\bin\java.exe" "-
Xms1024m
" "-Xmx2048m" "-XX:MaxPermSize=256m" "-
Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2" "
-Djava.library.path=libswt\win64" "-DKETTLE_HOME=" "-DKETTLE_REPOSITORY=" "-
DKET
TLE_USER=" "-DKETTLE_PASSWORD=" "-DKETTLE_PLUGIN_PACKAGES=" "-
DKETTLE_LOG_SIZE_L
IMIT=" "-DKETTLE_JNDI_ROOT=" -jar launcher\pentaho-application-launcher-
7.1.0.0-
12.jar -lib ..\libswt\win64 -main org.pentaho.di.kitchen.Kitchen -initialDir
"C
:\data-integration"\ /file:C:\Users\bxt058y\Desktop\CCMStatsJob.kjb "-
param:TABL
E_NAME=region" -logfile C:\Users\bxt058y\Documents\Pentaho Jobs\ccmjob.txt
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
sup
port was removed in 8.0
log4j:ERROR Could not create an Appender. Reported error follows.
java.lang.ClassCastException: org.apache.log4j.ConsoleAppender cannot be
cast to
org.apache.log4j.Appender
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.ja
va:248)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurat
or.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfi
gurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOM
Configurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseCategory(DOMConfigurator.ja
va:436)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1004)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java
:872)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java
:755)
at org.apache.log4j.xml.DOMConfigurator.configure(DOMConfigurator.java:8
96)
at org.pentaho.di.core.logging.log4j.Log4jLogging.applyLog4jConfiguratio
n(Log4jLogging.java:81)
at org.pentaho.di.core.logging.log4j.Log4jLogging.createLogger(Log4jLogg
ing.java:89)
at org.pentaho.di.core.logging.log4j.Log4jLogging.init(Log4jLogging.java
:68)
at org.pentaho.di.core.KettleClientEnvironment.initLogginPlugins(KettleC
lientEnvironment.java:155)
at org.pentaho.di.core.KettleClientEnvironment.init(KettleClientEnvironm
ent.java:118)
at org.pentaho.di.core.KettleClientEnvironment.init(KettleClientEnvironm
ent.java:79)
at org.pentaho.di.kitchen.Kitchen$1.call(Kitchen.java:91)
at org.pentaho.di.kitchen.Kitchen$1.call(Kitchen.java:84)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
log4j:ERROR Could not create an Appender. Reported error follows.
java.lang.ClassCastException: org.apache.log4j.ConsoleAppender cannot be
cast to
org.apache.log4j.Appender
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.ja
va:248)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurat
or.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfi
gurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOM
Configurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:4
92)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1006)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java
:872)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java
:755)
at org.apache.log4j.xml.DOMConfigurator.configure(DOMConfigurator.java:8
96)
at org.pentaho.di.core.logging.log4j.Log4jLogging.applyLog4jConfiguratio
n(Log4jLogging.java:81)
at org.pentaho.di.core.logging.log4j.Log4jLogging.createLogger(Log4jLogg
ing.java:89)
at org.pentaho.di.core.logging.log4j.Log4jLogging.init(Log4jLogging.java
:68)
at org.pentaho.di.core.KettleClientEnvironment.initLogginPlugins(KettleC
lientEnvironment.java:155)
at org.pentaho.di.core.KettleClientEnvironment.init(KettleClientEnvironm
ent.java:118)
at org.pentaho.di.core.KettleClientEnvironment.init(KettleClientEnvironm
ent.java:79)
at org.pentaho.di.kitchen.Kitchen$1.call(Kitchen.java:91)
at org.pentaho.di.kitchen.Kitchen$1.call(Kitchen.java:84)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2018/06/28 09:35:32 - Kitchen - Start of run.
2018/06/28 09:35:34 - CCMStatsJob - Start of job execution
2018/06/28 09:35:34 - CCMStatsJob - CCMStatsJob

I think its a problem with my Kitchen.bat (.kjb) because if i run the batch file with Pan.bat (.ktr) my file runs fine.

Windows does not like white spaces in file names (linux neither). Correct the -logfile.
The parameters must not be between quotes. Remove.
In Window the arguments are introduced by "/" not by "-" (as in linux). Change.
The command call executes the process in the background, so you will see nothing, no log not even a signal that the process has stopped. To check it did run correctly you need to type the logfile.
kitchen.bat /file:C:\Users\bxt058y\Desktop\CCMStatsJob.kjb /param:TABLE_NAME=region /logfile:"C:\Users\bxt058y\Documents\Pentaho Jobs\ccmjob.txt"

Related

Getting Too many Open Files Error in logs

Recently we got "(Too many open files)" error in production and this leads to very high CPU spike(98%) on Linux machine and finally brought machine down. We have to reboot the machine to bring it back.
Flow of code is like this :-
Consume message from one of IBM-MQ queue -> start processing it -> make some update entries into SQL-Server DB and commit the transaction.
We are using Hikari for connection pool management and value is set as 30
hikariConfig.setMaximumPoolSize(30);
While checking the logs, i found following stack trace :-
2021-03-24T20:00:22.404 WARNING org.apache.catalina.loader.WebappClassLoaderBase Failed to open JAR [null]
java.io.FileNotFoundException: /opt/apache-tomcat-7.0.88/webapps/ROOT/WEB-INF/lib/HikariCP-2.5.1.jar (Too many open files)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:219)
at java.util.zip.ZipFile.<init>(ZipFile.java:149)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:221)
at org.apache.catalina.loader.WebappClassLoaderBase.openJARs(WebappClassLoaderBase.java:3102)
at org.apache.catalina.loader.WebappClassLoaderBase.findResourceInternal(WebappClassLoaderBase.java:3414)
at org.apache.catalina.loader.WebappClassLoaderBase.findResource(WebappClassLoaderBase.java:1494)
at org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream(WebappClassLoaderBase.java:1722)
at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2677)
at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2662)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2661)
at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1501)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1465)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1361)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:773)
at com.ibm.mq.jmqi.JmqiException.buildMessage(JmqiException.java:428)
at com.ibm.mq.jmqi.JmqiException.getMessage(JmqiException.java:543)
at com.ibm.mq.jmqi.JmqiException.getMessage(JmqiException.java:515)
at java.lang.Throwable.getLocalizedMessage(Throwable.java:391)
at com.ibm.mq.jmqi.internal.JmqiTools.getExSumm(JmqiTools.java:947)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2282)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1294)
at com.ibm.msg.client.wmq.internal.WMQSession.connect(WMQSession.java:387)
at com.ibm.msg.client.wmq.internal.WMQSession.<init>(WMQSession.java:331)
at com.ibm.msg.client.wmq.internal.WMQConnection.createSession(WMQConnection.java:909)
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.createSession(JmsConnectionImpl.java:896)
at com.ibm.mq.jms.MQConnection.createSession(MQConnection.java:337)
at org.springframework.jms.connection.SingleConnectionFactory.createSession(SingleConnectionFactory.java:437)
at org.springframework.jms.connection.CachingConnectionFactory.getSession(CachingConnectionFactory.java:236)
at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:604)
at com.sun.proxy.$Proxy195.createSession(Unknown Source)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:192)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:475)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:526)
at com.test.config.messaging.MainframeMessageProducer.lambda$sendMessageToQueueWithHeaders$0(MainframeMessageProducer.java:120)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
at com.test.config.messaging.MainframeMessageProducer.sendMessageToQueueWithHeaders(MainframeMessageProducer.java:118)
at com.test.config.messaging.MainframeMessageProducer.sendMessageToMainframeQueue(MainframeMessageProducer.java:101)
at com.test.config.messaging.MainframeMessageProducer$$FastClassBySpringCGLIB$$f2d092d2.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.test.common.aop.RecordTimingAspect.recordTimingAspect(RecordTimingAspect.java:28)
at sun.reflect.GeneratedMethodAccessor505.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.test.config.messaging.MainframeMessageProducer$$EnhancerBySpringCGLIB$$dc5ae691.sendMessageToMainframeQueue(<generated>)
at com.test.posting.GDISPurchasePostServiceImpl.postMessage(GDISPurchasePostServiceImpl.java:315)
at com.test.posting.GDISPostingService.processMessage(GDISPostingService.java:56)
at com.test.posting.GDISPostingService$$FastClassBySpringCGLIB$$3c0c40af.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.test.common.aop.CorrelationIdAspect.correlationIdAdvice(CorrelationIdAspect.java:40)
at sun.reflect.GeneratedMethodAccessor554.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.test.posting.GDISPostingService$$EnhancerBySpringCGLIB$$dd870248.processMessage(<generated>)
at com.test.posting.GDISPostingMessageListener.onMessage(GDISPostingMessageListener.java:167)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:746)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:684)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
I am confused here what leads to max open files error here :-
is it due to Hikari CP as this is first line we got as an error (java.io.FileNotFoundException: /opt/apache-tomcat-7.0.88/webapps/ROOT/WEB-INF/lib/HikariCP-2.5.1.jar (Too many open files) ) ?
Is it due to IBM-MQ connection not available ?
You could use the lsof command to LiSt Open Files.
It produces lots of output, so you'll want to capture and post process it ( eg grep or sort)
You can list open files by pid, userid, disk etc.
You might try lsof > before wait... lsof > after then use diff to see the differences

Kylin Build Cube failed somtimes at" #19 Step Name: Hive Cleanup" java.lang.RuntimeException: Failed to read kylin_hive_conf.xml

The error occurs sometimes ,and after reboot kylin(kylin.sh stop and then kylin.sh start), it will find the conf dir location and pass this step.
I am using Kylin version "2.6.2", and KYLIN_CONF="/opt/kylin/conf" is already set correctly.
The errors hints are different , as i have countered the following:
1.
java.lang.RuntimeException: Failed to read kylin_hive_conf.xml at '/opt/apache-kylin-2.6.2-bin-hadoop3/bin/meta/kylin_hive_conf.xml'
at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:88)
at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:63)
at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:49)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2.
java.lang.RuntimeException: Failed to read kylin_hive_conf.xml at '/opt/apache-kylin-2.6.2-bin-hadoop3/bin/meta/kylin_hive_conf.xml'
3.
java.lang.RuntimeException: Failed to read kylin_hive_conf.xml at '/opt/apache-kylin-2.6.2-bin-hadoop3/conf/meta/kylin_hive_conf.xml'
who can kindly help me find the root cause and fix this problem ?
Thanks in advance.
I hope you have already solve the issue. I had encounter same problem and investigated about it.
Prefer to https://github.com/apache/kylin/blob/kylin-2.6.2/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java#L481
When we use MapReduce, KYLIN_CONF will be set to different folder.
System.setProperty(KylinConfig.KYLIN_CONF, metaDir.getAbsolutePath());
I think to workaround with it, we have to create simple link for all XML xml configurations.
Try to check your Kylin log
cat YOUR_PATH/apache-kylin-2.6.3-bin-hbase1x/logs/kylin.log | grep "The absolute path"
You possibly see the result
2019-10-14 23:47:04,438 INFO [LocalJobRunner Map Task Executor #0] common.AbstractHadoopJob:482 : The absolute path for meta dir is /SOME_FOLDER/meta

camel ftp in recursive manner results in ftp error code: 226

I have a very simple ftp route that should recursively download files from an URL. It would be very important to get it running still today.
There are no proxies, no extra authentication, no firewall. However, it only downloads the first file and then the socket will be closed. I have experimented with the different timeouts but they did not solve the issue. Either error code 226 is retuned if I don't use any timeout or extra config option or if I use the commented out things, error code 221 is returned. The 226 does not seem to be an error as it just counts as an indication that the server completed the transfer. Stack traces I copied below. I would appreciate the responses and thanks also in advance.
The route I am using in pollEnrich as I have to start it depending on a timer.
The code:
[...]
from("direct:ftp").routeId("ftp")
.log("### FTP is in progress ")
.pollEnrich().simple(ConfigData.getConfigData().getFtpUrl()
+ "?binary=true&"
+ "recursive=true"
/* + "soTimeout=300000&"
+ "stepwise=true&"
+ "ignoreFileNotFoundOrPermissionError=true&"
+ "ftpClient.dataTimeout=30000&"
+ "disconnect=true&"
+ "consumer.delay=1000" */
)
.to("file:modelFiles")
.end();
[...]
Update 1
I removed pollEnrich() and without it it works fine. However, I cannot start it from another route e.g. from a timer. So this is a quick hack to get FTP running. I also copy the code here as I did it with an idempotent consumer, so that only those files will be downloaded that are not yet on the disk. It can maybe be useful also to other people. The idempotent consumer example you can find here (Apache Camel ftp consumer loads the same files again and again) for more information.
Any further comments?
from(ConfigData.getConfigData().getFtpUrl()
+ "?binary=true&"
+ "recursive=true&"
+ "passiveMode=true&"
+ "ftpClient.bufferSize=10000000&"
+ "localWorkDirectory=" + ConfigData.getConfigData().getLocalTmpDirectory())
.idempotentConsumer(header("CamelFileName"), FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat")))
.to("file:modelFiles")
.log("Downloaded file ${file:name} complete.")
.end();
Stack Trace
2016-03-19 15:27:53,921 [WARN|org.apache.camel.component.file.remote.FtpConsumer|MarkerIgnoringBase] Error processing file RemoteFile[2009-03-25/BioModels_Database-r13-sbml_files.tar.gz] due to File operation failed: null socket closed. Code: 221. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - File operation failed: null socket closed. Code: 221]
org.apache.camel.component.file.GenericFileOperationFailedException: File **operation failed: null socket closed. Code: 221**
[...]
Caused by: java.net.SocketException: socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:483)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:608)
at org.apache.commons.net.ftp.FTP.cwd(FTP.java:828)
at org.apache.commons.net.ftp.FTPClient.changeWorkingDirectory(FTPClient.java:1128)
at org.apache.camel.component.file.remote.FtpOperations.doChangeDirectory(FtpOperations.java:769)
2016-03-19 12:42:16,068 [WARN|org.apache.camel.component.file.remote.FtpConsumer|MarkerIgnoringBase] Error processing file RemoteFile[2008-12-03/BioModels_Database-r12-sbml_files.tar.gz] due to File operation failed: null Socket Closed. Code: 226. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - **File operation failed: null Socket Closed. Code: 226]**
[...]
Caused by: java.net.SocketException: Socket Closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at java.io.BufferedReader.fill(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
at org.apache.commons.net.ftp.FTP.getReply(FTP.java:692)
at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1813)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1885)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1845)
at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:367)
Solution:
As added in Upadete 1, I do not use pollEnrich(). The route cannot be started from a timer now but it works. So I will close the question. I really like the idea of the idempotent consumer (unrelated to the original question).
Don't forget that a pollEnrich delete the oldExchange and keep the newExchange.
So you don't want to do a from("ftp").pollEnrich("timer") but a from("timer").pollEnrich("ftp").
See the Solution at the end of the question.

StateMachineBase.java not updated when designer.jar saves theme file

StateMachineBase.java is not updating. I've tried moving the file so that the designer should re-create it, I've tried running it from the command line (no error messages are displayed).
Is there anything I can do to troubleshoot designer.jar and the StateMachineBase.java file?
the designer.jar file has a last modified date of Sept 1, 2015. I normally work in IntelliJ Idea but given that I'm running the designer from the command line, I assume this helps eliminate the IDE from the equation.
What steps can I take to troubleshoot this? what files or settings could affect the behavior of the Designer as it writes the StateMachineBase.java file?
Here is the contents of my codenameone_settings.properties file:
#
#Fri Mar 04 17:16:27 EST 2016
codename1.ios.appid=OBFUSCATED.ca.zettabot.trainingtool
codename1.ios.release.provision=
codename1.arg.rim.obfuscation=true
codename1.j2me.nativeTheme=nbproject/nativej2me.res
codename1.arg.ios.project_type=ios
codename1.arg.ios.interface_orientation=UIInterfaceOrientationPortrait\:UIInterfaceOrientationPortraitUpsideDown\:UIInterfaceOrientationLandscapeLeft\:UIInterfaceOrientationLandscapeRight
codename1.displayName=TrainingTool
guiResource=theme.res
codename1.android.keystoreAlias=
codename1.ios.release.certificate=
codename1.android.keystorePassword=
codename1.arg.ios.dsym=false
codename1.arg.android.release=true
package=generated
codename1.arg.ios.statusbar_hidden=false
codename1.languageLevel=5
codename1.android.keystore=
mainForm=SplashScreen
userClassAbs=C\:\\Users\\vankej\\Desktop\\Personal\\android\\codename1\\TrainingTool\\src\\userclasses\\StateMachine.java
codename1.vendor=Codename One
codename1.arg.win.ver=8
codename1.arg.zooz.sandbox=false
codename1.ios.debug.certificatePassword=
codename1.mainName=MyApplication
codename1.ios.release.certificatePassword=
codename1.arg.ios.prerendered_icon=false
codename1.arg.vserv.scaleMode=false
codename1.ios.debug.certificate=
libVersion=99
codename1.arg.vserv.category=29
codename1.arg.ios.application_exits=false
codename1.secondaryTitle=TrainingTool
codename1.description=
codename1.ios.debug.provision=
baseClass=src/generated/StateMachineBase.java
codename1.arg.j2me.nativeThemeConst=0
codename1.rim.certificatePassword=
codename1.version=1.0
codename1.icon=icon.png
codename1.rim.signtoolCsk=
codename1.arg.android.debug=false
codename1.rim.signtoolDb=
codename1.arg.vserv.transition=300000
userClass=src/userclasses/StateMachine.java
codename1.arg.ios.includePush=false
codename1.packageName=ca.zettabot.trainingtool
Here is the command line output of the designer command when I open and try to save the theme file:
C:\Users\vankej\Desktop\Personal\android\codename1\TrainingTool>java -jar designer.jar src\theme.res > output.txt
2016-03-08 21:59:34.602:INFO:oejs.Server:jetty-8.0.y.z-SNAPSHOT
2016-03-08 21:59:34.638:INFO:oejs.AbstractConnector:Started SelectChannelConnector#0.0.0.0:9000 STARTING
Mar 08, 2016 9:59:34 PM org.jdesktop.application.LocalStorage getId
WARNING: unspecified resource Application.id using ResourceEditorApp
Mar 08, 2016 9:59:34 PM org.jdesktop.application.LocalStorage getId
WARNING: unspecified resource Application.vendorId using UnknownApplicationVendor
javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; systemId: file:/C:/Users/vankej/Desktop/Personal/android/codename1/TrainingTool/res/theme.xml; lineNumber: 1; columnNumber: 1; Premature end of file.]
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(Unknown Source)
at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(Unknown Source)
at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(Unknown Source)
at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(Unknown Source)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(Unknown Source)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(Unknown Source)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(Unknown Source)
at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(Unknown Source)
at com.codename1.ui.util.EditableResources.openFileWithXMLSupport(EditableResources.java:409)
at com.codename1.designer.ResourceEditorView$LoadResourceFileAction.exectute(ResourceEditorView.java:4112)
at com.codename1.ui.resource.util.BlockingAction.run(BlockingAction.java:88)
at java.lang.Thread.run(Unknown Source)
Caused by: org.xml.sax.SAXParseException; systemId: file:/C:/Users/vankej/Desktop/Personal/android/codename1/TrainingTool/res/theme.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
... 10 more
java.lang.NullPointerException
at com.codename1.designer.UserInterfaceEditor.persistToXML(UserInterfaceEditor.java:3161)
at com.codename1.designer.UserInterfaceEditor.persistToXML(UserInterfaceEditor.java:3083)
at com.codename1.designer.UserInterfaceEditor.appendComponentXMLBody(UserInterfaceEditor.java:2821)
at com.codename1.designer.UserInterfaceEditor.persistToXML(UserInterfaceEditor.java:3510)
at com.codename1.designer.UserInterfaceEditor.persistToXML(UserInterfaceEditor.java:3083)
at com.codename1.ui.util.EditableResources.writeUIXml(EditableResources.java:1287)
at com.codename1.ui.util.EditableResources.saveXMLFile(EditableResources.java:1236)
at com.codename1.ui.util.EditableResources.saveXML(EditableResources.java:1311)
at com.codename1.designer.ResourceEditorView$SaveResourceFileAction.exectute(ResourceEditorView.java:4295)
at com.codename1.ui.resource.util.BlockingAction.run(BlockingAction.java:88)
at java.lang.Thread.run(Unknown Source)
Check the output of the designer when running it from the command line.
Make sure your codenameone_settings.properties is a valid file and contains the following.
This must point at a valid res file which must be the one containing your GUI's:
guiResource=theme.res
This must point at the generated package for the StatemachineBase.java:
package=generated
This must point at the StateMachine.java source file, not as crucial but important:
userClass=src/userclasses/StateMachine.java
This must point at a valid main form within your GUI:
mainForm=Main
You can try and regenerate the state machine directly from the command line using the command:
java -jar path_to_designer_1.jar -regen path_to_resfile.res
I'd really like to know if that works or fails.

Getting sonar-runner to work with C code

I am trying to use sonar. The server is up and running (using the default embedded derby database), serving a web page with no projects. Then I was planning to use sonar-runner to examine the source code, but I cannot get it to work even with the simplest hello world program. Any hints on what is missing/I am doing wrong?
(hlovdal) localhost:/work/sonar>wget -q http://repository.codehaus.org/org/codehaus/sonar-plugins/sonar-runner/1.1/sonar-runner-1.1.zip
(hlovdal) localhost:/work/sonar>unzip sonar-runner-1.1.zip
Archive: sonar-runner-1.1.zip
creating: sonar-runner-1.1/
...
inflating: sonar-runner-1.1/lib/sonar-runner.jar
(hlovdal) localhost:/work/sonar>export SONAR_RUNNER_HOME=/work/sonar/sonar-runner-1.1
(hlovdal) localhost:/work/sonar>$SONAR_RUNNER_HOME/bin/sonar-runner -h
usage: sonar-runner [options]
Options:
-h,--help Display help information
-X,--debug Produce execution debug output
-D,--define <arg> Define property
(hlovdal) localhost:/work/sonar>
But trying to run sonar-runner fails:
(hlovdal) localhost:/work/sonar>cd helloworld/
(hlovdal) localhost:/work/sonar/helloworld>cat main.c
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
printf("hello world\n");
return EXIT_SUCCESS;
}
(hlovdal) localhost:/work/sonar/helloworld>grep -v -E '^#|^$' sonar-runner.properties
sonar.projectKey=helloworld
sonar.projectName=Hello world
sonar.projectVersion=1.0
sonar.language=c
sources=/work/sonar/helloworld
sonar.sourceEncoding=UTF-8
(hlovdal) localhost:/work/sonar/helloworld>$SONAR_RUNNER_HOME/bin/sonar-runner
Runner settings: /work/sonar/sonar-runner-1.1/conf/sonar-runner.properties
Runner version: 1.1
Server: http://localhost:9000
Work directory: /work/sonar/helloworld/.sonar
[INFO] Database dialect class org.sonar.jpa.dialect.Derby
[INFO] Initializing Hibernate
Exception in thread "main" org.sonar.batch.bootstrapper.BootstrapException: org.picocontainer.PicoLifecycleException: PicoLifecycleException: method 'public void org.sonar.batch.index.DefaultIndex.start()', instance 'org.sonar.batch.index.DefaultIndex#3d57211f, java.lang.RuntimeException: wrapper
at org.sonar.runner.Runner.delegateExecution(Runner.java:155)
at org.sonar.runner.Runner.execute(Runner.java:58)
at org.sonar.runner.Main.main(Main.java:52)
Caused by: org.picocontainer.PicoLifecycleException: PicoLifecycleException: method 'public void org.sonar.batch.index.DefaultIndex.start()', instance 'org.sonar.batch.index.DefaultIndex#3d57211f, java.lang.RuntimeException: wrapper
at org.picocontainer.monitors.NullComponentMonitor.lifecycleInvocationFailed(NullComponentMonitor.java:77)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.monitorAndThrowReflectionLifecycleException(ReflectionLifecycleStrategy.java:132)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:115)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89)
at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
at org.picocontainer.behaviors.Stored.start(Stored.java:110)
at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:996)
at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:989)
at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:746)
at org.sonar.batch.bootstrap.Module.start(Module.java:88)
at org.sonar.batch.bootstrap.BootstrapModule.doStart(BootstrapModule.java:96)
at org.sonar.batch.bootstrap.Module.start(Module.java:89)
at org.sonar.batch.Batch.execute(Batch.java:74)
at org.sonar.runner.Launcher.executeBatch(Launcher.java:60)
at org.sonar.runner.Launcher.execute(Launcher.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.sonar.runner.Runner.delegateExecution(Runner.java:152)
... 2 more
Caused by: java.lang.RuntimeException: wrapper
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.monitorAndThrowReflectionLifecycleException(ReflectionLifecycleStrategy.java:130)
... 22 more
Caused by: java.lang.NullPointerException
at org.sonar.api.resources.Resource.hashCode(Resource.java:242)
at java.util.HashMap.put(HashMap.java:389)
at org.sonar.batch.index.DefaultIndex.doStart(DefaultIndex.java:98)
at org.sonar.batch.index.DefaultIndex.start(DefaultIndex.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110)
... 21 more
(hlovdal) localhost:/work/sonar/helloworld>
The projectKey does not look correct, it should be something like xxx:yyy

Resources