I am using Camel version 2.17.1 over a clustered environment with 2 nodes to process files. I use an idempotent readlock in my file consumer endpoint using a JDBCMessageIdRepository as the idempotent repository to stop multiple servers trying to process the same file.
Occasionally I see warnings in the SystemOut logs of one node saying a file could not be renamed while the other node processes the same file successfully.
Warning example:
[12/10/16 10:21:03:312 BST] 0000008f SystemOut O 12 Oct 2016 10:21:03 [r-inboundCamelContext_Worker-1] FileConsumer WARN Endpoint[file://drop?delay=30000&idempotentKey=${file:name}-${file:modified}-${file:size}&idempotentRepository=#filesInIdempotentRepository&move=done/${date:now:yyyyMMdd}/${date:now:yyyyMMddHHmmssSSS}-${file:name}&moveFailed=error/${date:now:yyyyMMdd}/${date:now:yyyyMMddHHmmssSSS}-${file:name}&preMove=processing&readLock=idempotent&readLockRemoveOnCommit=true&runLoggingLevel=DEBUG&scheduler=quartz2&scheduler.cron=*+*+*+*+*+?] cannot begin processing file: GenericFile[drop\file-20161012_101320.xml] due to: Cannot rename file: GenericFile[drop\file-20161012_101320.xml] to: GenericFile[drop\processing\file-20161012_101320.xml]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename file: GenericFile[drop\file-20161012_101320.xml] to: GenericFile[drop\processing\file-20161012_101320.xml]]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot rename file: GenericFile[drop\file-20161012_101320.xml] to: GenericFile[drop\processing\file-20161012_101320.xml]
at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.begin(GenericFileRenameProcessStrategy.java:43)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:367)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:226)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:190)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
at org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:61)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
stack
Is there a way to stop both nodes trying to process the same file?
Related
After booting SWUpdate yocto-generated image for the first time, executing swupdate results in error message:
Error parsing configuration file: 'globals' section missing, exiting.
I tried to strictly follow SWUpdate's documentation, but it gets short when it comes to yocto integration. I'm using meta-swupdate, meta-swupdate-boards, and meta-openembedded layers together with poky example repository all at Kirkstone tag, building via bitbake update-image and having modyfied local.conf as:
MACHINE ??= "raspberrypi4-64"
ENABLE_UART = "1"
RPI_USE_U_BOOT = "1"
IMAGE_FSTYPES = "wic ext4.gz"
PREFERRED_PROVIDER_u-boot-fw-utils = "libubootenv"
IMAGE_INSTALL:append = " swupdate"
Is there anything else I need to modify to generate the configuration file and be able to run SWUpdate binary properly?
Side question: In the documentation, it's recommended to append swupdate-www to achieve a better web server. However, if I append it, there is no swupdate-www binary inside the `/usr/bin' directory.
As with other recipes folders the recipes-support/swupdate/swupdate/raspberrypi4-64 folder was missing inside the meta-swupdate-boards layer. Therefore, an empty config file was always generated. After adding this folder and all related files, strongly inspired by raspberrypi3 folder, the error was gone and swupdate -h provided the expected output.
There was also one new error during build process thrown by yocto. It was related to missing systemd requirement and was solved by adding:
DISTRO_FEATURES_append = " systemd"
to local.conf
Can anyone tell me why this is happening:
$ dev_appserver.py nmg_server
INFO 2017-07-08 17:15:37,369 application_configuration.py:461] No version specified. Generated version id: 20170708t171537
WARNING 2017-07-08 17:15:37,369 application_configuration.py:166] The Managed VMs runtime is deprecated, please consider migrating your application to use the Flexible runtime. See https://cloud.google.com/appengine/docs/flexible/python/migrating for more details.
INFO 2017-07-08 17:15:37,472 devappserver2.py:116] Skipping SDK update check.
INFO 2017-07-08 17:15:37,513 api_server.py:312] Starting API server at: http://localhost:54096
INFO 2017-07-08 17:15:37,517 api_server.py:938] Applying all pending transactions and saving the datastore
INFO 2017-07-08 17:15:37,517 api_server.py:941] Saving search indexes
Traceback (most recent call last):
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 103, in <module>
_run_file(__file__, globals())
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 97, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 381, in <module>
main()
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 369, in main
dev_server.start(options)
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 196, in start
options.api_host, apiserver.port, wsgi_request_info_, options.grpc_apis)
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/dispatcher.py", line 223, in start
_module.start()
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1647, in start
self._add_instance()
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1799, in _add_instance
expect_ready_request=True)
File "/Users/dgaedcke/gcloud_tools/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/go_runtime.py", line 189, in new_instance
self._go_application.maybe_build()):
TypeError: maybe_build() takes exactly 2 arguments (1 given)
I'm trying to run my server for testing on localhost and it keeps exiting with this error
This appears to be a bug in Cloud SDK which is affecting dev_appserver.py when using with App Engine Managed VMs. It does not seem to be affecting App Engine Standard or App Engine Flex environments.
Until Google releases a new Cloud SDK with the fix, you can modify the CLOUD_SDK_INSTALL_DIR//platform/google_appengine/google/appengine/tools/devappserver2/go_managedvm.py file locally as shown below (added both the patchable unified diff as well as before/after just for convenience).
Also consider moving to App Engine Flex since Managed VMs are deprecated and will not be supported after October 27, 2017.
Warning: The Managed VMs beta environment (applications deployed with
vm:true) is deprecated and will be decommissioned. This page is for
users who are already using the flexible environment with vm:true in
their app.yaml and want to upgrade to the latest release. If you are
updating your application from the standard environment, see the
Migrating Services from the Standard Environment to the Flexible
Environment instead.
Diff in patchable format
--- /Users/tuxdude/google-cloud-sdk-orig/platform/google_appengine/google/appengine/tools/devappserver2go_managedvm.py 2017-07-08 11:11:11.000000000 -0700
+++ /Users/tuxdude/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/go_managedvm.py 2017-07-08 11:11:11.000000000 -0700
## -152,15 +152,9 ##
logging.debug('Build succeeded:\n%s\n%s', stdout, stderr)
self._go_executable = exe_name
- def maybe_build(self, maybe_modified_since_last_build):
+ def maybe_build(self):
"""Builds an executable for the application if necessary.
- Args:
- maybe_modified_since_last_build: True if any files in the application root
- or the GOPATH have changed since the last call to maybe_build, False
- otherwise. This argument is used to decide whether a build is Required
- or not.
-
Returns:
True if compilation was successfully performed (will raise
an exception if compilation was attempted but failed).
## -173,9 +167,6 ##
self._work_dir = tempfile.mkdtemp('appengine-go-bin')
atexit.register(_rmtree, self._work_dir)
- if self._go_executable and not maybe_modified_since_last_build:
- return False
-
if self._go_executable:
logging.debug('Rebuilding Go application due to source modification')
else:
Before:
def maybe_build(self, maybe_modified_since_last_build):
"""Builds an executable for the application if necessary.
Args:
maybe_modified_since_last_build: True if any files in the application root
or the GOPATH have changed since the last call to maybe_build, False
otherwise. This argument is used to decide whether a build is Required
or not.
Returns:
True if compilation was successfully performed (will raise
an exception if compilation was attempted but failed).
False if compilation was not attempted.
Raises:
BuildError: if building the executable fails for any reason.
"""
if not self._work_dir:
self._work_dir = tempfile.mkdtemp('appengine-go-bin')
atexit.register(_rmtree, self._work_dir)
if self._go_executable and not maybe_modified_since_last_build:
return False
if self._go_executable:
logging.debug('Rebuilding Go application due to source modification')
else:
logging.debug('Building Go application')
self._build()
return True
After:
def maybe_build(self):
"""Builds an executable for the application if necessary.
Returns:
True if compilation was successfully performed (will raise
an exception if compilation was attempted but failed).
False if compilation was not attempted.
Raises:
BuildError: if building the executable fails for any reason.
"""
if not self._work_dir:
self._work_dir = tempfile.mkdtemp('appengine-go-bin')
atexit.register(_rmtree, self._work_dir)
if self._go_executable:
logging.debug('Rebuilding Go application due to source modification')
else:
logging.debug('Building Go application')
self._build()
return True
I need to implement a handler that reacts on ZipException to move away corrupted gz files, otherwise the route will endlessly retry to unmarshal the gz.
The problem is that at the moment the exception is thrown there is a lock on this file (on linux canWrite() returns false) and there is the Camel lock file.
Is there an elegant Camel way to say/configure the onException that the lock is released (get write access and remove lockfile - if there is one)?
At the moment my code looks like that:
onException(ZipException.class)
.handled(true)
.process(corruptedFileProcessor)
.stop();
Thanks in advance.
The following route reads gzipped files from srcDir, writes unzipped files to destDir (without the .gz extension) and when a ZipException occurs, sends the file to errorDir.
from("file://srcDir/?delete=true")
.onException(ZipException.class)
.handled(true).useOriginalMessage()
.to("file://errorDir?autoCreate=true")
.end()
.unmarshal().gzip()
.to("file://destDir?autoCreate=true&fileName=${file:name.noext}");
We are working on a POC to use Spring integration and Rabbit MQ. We have two modules producer module and consumer module both are runs in different JVMs. The Producer module listen on a Folder (input folder) as soon as new files arrives, creates a message then push to (incoming.q.in) queue and also move to process folder.
In the producer module we have below code. When I dropped about 100 files in incoming folder about 90 files processed and moved to process folder but 10 files didn't move to process folder.
For failed cases these are the messages in log file
....
[07/30/13 07:34:23:023 EDT] [taskExecutor-3] DEBUG org.springframework.integration.file.FileReadingMessageSource Added to queue: [test.xml]
[07/30/13 07:34:23:023 EDT] [taskExecutor-3] DEBUG org.springframework.integration.endpoint.SourcePollingChannelAdapter Received no Message during the poll, returning 'false'
....
for Successful case
....
[07/30/13 07:34:32:032 EDT] [taskExecutor-1] DEBUG org.springframework.integration.file.FileReadingMessageSource Added to queue: [test_0.xml]
[07/30/13 07:34:32:032 EDT] [taskExecutor-1] INFO org.springframework.integration.file.FileReadingMessageSource Created message: [[Payload=/apps/incoming/test_0.xml][Headers={timestamp=1375184072466, id=d8d4cea4-a25d-4869-b287-e76cfb76f554}]]
....
Here is the code
<file:inbound-channel-adapter id="inboundAdapter" channel="inboundChannel" directory="file:${incoming_folder}" prevent-duplicates="true" filename-pattern="*.*" auto-startup="true" >
<int:poller id="fileInboudPoller" fixed-rate="3" receive-timeout="3" time-unit="SECONDS" max-messages-per-poll="1" task-executor="taskExecutor"/>
<file:nio-locker />
</file:inbound-channel-adapter>
It generally means the locker couldn't lock the file (presumably because the file is in use elsewhere).
BTW, a common error with applications like these is copying files "in place" such that the consumer might see an incomplete file.
A common technique to avoid these issues is to copy the file with a temporary name and rename it only when it is completely written.
Here's the problem I've been grappling with for a while...I'm using Camel (v2.10.2) to set up many file routes to move data across file systems, servers, and in/out of the organisation (B2B). There are data and signal files in their respective dirs with some of the routes being short lived, while others run as services on different VMs/servers. These processes (routes) are run under different unix 'functional' ids, but there is an attempt to make them belong to the same unix group(s) if possible...
Of course on unix there is always the potential for file/dir permission problems...and that is the issue I'm facing/trying to solve.
I use the DefaultErrorHandler and log success or failure for an exchange via a custom RoutePolicy within the onExchangeDone(...) checking the Exchange.isFailed(). The signal file is either moved to the destination on success or moved to .error dir on fail, with an alert written to a system-wide alert log (checked by Tivoli)
The file route is configured to propagate errors occurring while picking up files, etc via the consumer.bridgeErrorHandler=true
Basically, if I have any unix permission related errors, then I want to stop (and maybe remove) the effected route, indicating clearly that this has happened and why - a permission issue is not easily solvable programmatically, so stop and alert is the only option.
So I'll illustrate a test case that causes an issue...
App_A creates some data files in ./data/. Then App_A creates the same number of signal files in ./signal/, but due to some 'data' related bug it also creates a signal file ./signal/acc_xyz.csv that doesn't have a corresponding data file.
Route starts to process ./signal/acc_xyz.csv and the 'validation process' finds that ./data/acc_xyz.csv doesn't exist and throws an exception to indicate this, hence stopping the exchange being processed further.
The File component is configured with moveFailed=.error to move the signal file to ./signal/.error/, but this dir is locked (don't worry why this is) to the functional user id executing the Java process and internal Camel processing throws a GenericFileOperationFailedException indicating the cause to be an underlying 'Permission denied' issue.
Oh dear, the same signal file is then processed again, and again, and...
I have tried to get this 'secondary error' propagated to my code, but have failed, hence I can't stop the route.
How can I get this and other internal Camel errors propagated to my code/exception handler/whatever and not just seeing it be logged and swallowed?
thanks in advance
ok more detail from log4j...note the sequence of times
Camel DefaultErrorHandler:
2013-04-25 15:06:26,001 [Camel (camel-1) thread #0 - file:///FTROOT/fileTransfer/outbound/signal] ERROR (MarkerIgnoringBase.java:161) - Failed delivery for (MessageId: ID-rwld601-rw-discoverfinancial-com-60264-1366902384246-0-1 on ExchangeId: ID-rwld601-rw-discoverfinancial-com-60264-1366902384246-0-2). Exhausted after delivery attempt: 1 caught: java.lang.IllegalStateException: missingFile: route [App_A.outboundReceipt] has missing file at /FTROOT/fileTransfer/outbound/data/stuff.log
java.lang.IllegalStateException: missingFile: route [App_A.outboundReceipt] has missing file at /FTROOT/fileTransfer/outbound/data/stuff.log
at com.myco.mft.process.BaseFileRouteBuilder.checkFile(BaseFileRouteBuilder.java:934)
My alert logger via the RoutePolicy.onExchangeDone(...) - at this pont the exchange has completed with a failure:
2013-04-25 15:06:26,011|Camel (camel-1) thread #0 - file:///FTROOT/fileTransfer/outbound/signal|exchange|App_A.outboundReceipt|signalFile=/FTROOT/fileTransfer/outbound/signal/stuff.log|there has been a routing failure|missingFile: route [App_A.outboundReceipt] has missing file at /FTROOT/fileTransfer/outbound/data/stuff.log
Camel endpoint post-processing - this is the stuff that Camel doesn't propagate to me:
2013-04-25 15:06:26,027 [Camel (camel-1) thread #0 - file:///FTROOT/fileTransfer/outbound/signal] WARN (GenericFileOnCompletion.java:149) - Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileDeleteProcessStrategy#104e28b for file: GenericFile[/FTROOT/fileTransfer/outbound/signal/stuff.log]
2013-04-25 15:06:28,038 [Camel (camel-1) thread #0 - file:///FTROOT/fileTransfer/outbound/signal] WARN (MarkerIgnoringBase.java:136) - Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Error renaming file from /FTROOT/fileTransfer/outbound/signal/stuff.log to /FTROOT/fileTransfer/outbound/signal/.error/stuff.log]
org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming file from /FTROOT/fileTransfer/outbound/signal/stuff.log to /FTROOT/fileTransfer/outbound/signal/.error/stuff.log
at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:72)
...
Caused by: java.io.FileNotFoundException: /FTROOT/fileTransfer/outbound/signal/stuff.log (Permission denied)
at java.io.FileInputStream.open(Native Method)