My purpose is to not to read same files again and again from a shared directory by different applications.
Is there a sharable idemponent repository for multiple applications which use camel file component?
THNX
Yes read about the idempotentRepository on the Camel file doc page: http://camel.apache.org/file2
You can use a shared repository such as a JDBC based, so you can use a database table,
or write your own repository implementation.
You can also read more here: http://camel.apache.org/idempotent-consumer.html
Related
I need to configure file names for files created by StreamingFileSink.
I use ParquetAvroWriters.forGenericRecord to create parquet files.
Discovered that i cant use .withOutputFileConfig() when i use .forBulkFormat() (which is available when use .forRowFormat().
It started working with Flink 1.11
I have zero experience with ETL.
Whenever a file(a .csv) is moved into a specific folder, it should be uploaded to SalesForce I don't know how to get this automated flow.
I hope I was clear enough.
I gotta use the opensource version, any helpful links or resources will be appreciated.
Thank you in advance
You could definitely use Talend Open Studio for ESB : this studio contains 'Routes' functionalities : you'll be able to use a cFile component, which will check your folder for new files, and raise an event that will propagate throughout the route to a designed endpoint (for example a salesForce API). Talend ESB maps Apache Camel components , which are well documented.
Check about Routes with Talend for ESB, it should do the trick.
We have tFileExists component, you can use that and configure to check the file.
Also you have tFileWait component, where you can defile the frame of the arrival of he files and the number of iterations it has to check the file.
But i would suggest id you have any scheduling tool, use file watcher concept and then use talend job to upload the file to a specific location.
Using talend itself to check the file arrival is not a feasible way as the jobs has be in running state continuously which consumes more java resource
I Am having two servers which poll to same ftp location. when a file is placed in that location both the servers are picking the files. But I need only one server to pick the file and process and delete.I am using Camel 2.16.2 Version. Is there any way to solve this issue?
Camel FTP uses most of camel-File2 API internally. All, camel file the options are inherited.
There are many strategies to avoid parallel processing of same file.
Use preMove, moveFailed, readLock, readLockRemoveOnCommit Camel-File options.
For instance you could use below parameters.
ftp://{{input.folder}}?readLock=rename&preMove=.inprogress&moveFailed=.error
Is it possible to use apache camel is a cluster when using the file compoment?
For example the following rout:
from("file://C:/temp/camel/in?moveFailed=/temp/camel/error&delay=10000&delete=true")
.to("file://C:/temp/camel/out")
Will camel know it file has already been taken by one member of the cluster?
What if both camel instances take the file at the same time?
Thanks
Yes read the docs about the read lock option, where you have different options to setup this.
Is it possible to access files beyond WEB-INF folder in Google App Engine's development server ? Maybe I can somehow force GAE dev server to do not block external connections/file access ?
I'm trying to create Java version of gae-sqlite. But I can't access external sql server (mysql or h2db) due to access denied exceptions.
Call stack:
Caused by: java.security.AccessControlException: access denied (java.io.FilePermission C:\work\test_projects\gae_test\out\artifacts\gae_test_war_exploded\database.lock.db write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:252)
at java.lang.SecurityManager.checkWrite(SecurityManager.java:962)
at java.io.File.createNewFile(File.java:882)
at org.h2.store.fs.FilePathDisk.createFile(FilePathDisk.java:121)
at org.h2.store.fs.FileUtils.createFile(FileUtils.java:59)
at org.h2.store.FileLock.lockFile(FileLock.java:327)
at org.h2.store.FileLock.lock(FileLock.java:128)
at org.h2.engine.Database.open(Database.java:542)
at org.h2.engine.Database.openDatabase(Database.java:222)
According to GAE for Java Questions:
Why can't I read from this file?
It is possible to read from a file which is uploaded as part of your
application provided that it is in the following locations:
war/WEB-INF
in a location matching the <resource-files> pattern in appengine-web.xml (which by default includes everything)
If the file location is not the issue, the problem may be that the
method you are using to read from the file is not whitelisted. Your
application can use any IO classes that are useful for reading from
the file system, such as File, FileInputStream, FileReader, or
RandomAccessFile. For a full list of whitelisted classes, please see
the JRE Class White List. If you need to get file access to your own
resources (such as properties files), you could put these files inside
of jars and use Class or ClassLoader to load them.
So to answer your question, the only available place besides the WEB-INF directory to read files from is whatever you specified in the <resource-files> entry in appengine-web.xml.
Now one thing I noticed is that you're trying to read a DB file which should be okay if you're only considering it read-only. You cannot write to any "local" files because (from the same link):
Why can't I write to this file?
Writing to local files is not supported in App Engine due to the
distributed nature of your application. Instead, data which must be
persisted should be stored in the distributed datastore. For more
information see the documentation on the runtime sandbox.
As of GAE SDK 1.6.1 it's not possible. If you guys later find the way to do it please let me know.