I have simple Camel route (Java DSL) configured to:
Read a CSV file from a location.
Unmarshal is using 'BindyCsvDataFormat'.
Split it and push each record(row) to Kafka.
The route works fine in a happy path scenario.
But while reading the CSV if any of the records in the CSV are invalid the entire route fails and none of the records are processed. What I need is to skip the invalid record(row) and continue processing the other valid records and just log an exception for the invalid records.
Related
I have about 10K messages in a CSV file. Each message has an associated timestamp for it. When the time is reached, I want the message delivered to an MQ. The timestamps are not uniformly spread. Is this possible with Apache Camel?
As far as I know Apache Camel by default has no consumer endpoint components that you could configure to trigger with specific messages at specified times.
There is however a timer component that you can setup to trigger for example once a second. Then in the route you could use a processor to check if a list contains any messages that should be send at the given time and send them to the MQ.
You could also trigger route from java code using a ProducerTemplate that you can create using the CamelContext.
The list could be populated using your csv file and ordered by timestamp so you could use it like a stack and only check the first few entries instead of going through all 10K every second.
The real problem here would be persistence i.e to figure out which of the messages listed on the csv have already been sent if the application closes before all 10K messages have been sent.
Im working on a Wordpress project that will take entries from a 3rd party API and add them as individual post on the site with the data saved as metadata. Up until now it was going well but with the number of entries on the API increasing im starting to run in to issue regarding the server timing out while making the API request.
What I have done now is to wright the response from the API that is all the interies in json format to a file then creating the post on the site from the file. But still running in to time out issues.
That brings me to my question.
I want to break the data (from the JSON file) up in to smaller manageable request to the server. only processing lets say 20 entries at a time out if the 1000+. But I need to find out how do I select the entry number in the json file, for example after processing the first 20 entries I then want the function to go back to the json file but this time start at the 21st entry in the file, if at all possible. Or is there a better method of programmatically creating post from a json file with a large amount of entries.
// Just some more info.
Im using wp_remote_post() with blocking set to false to run the function that creates the post in batches in the background.
I have some steps in my logic apps eg Parse JSON. If they fail, I can see the reason for the failure when I open the step in logic apps eg string instead of integer.
How can I log these error messages in my Azure storage account.
The dynamic content dialogue box doesn't specify error messages.
I have created a storage account, created files, populated them with a string and put them into the storage account. I just need to get hold of the error message.
I will be processing JSON from HTTP requests. If the JSON is invalid ie does not conform to the expected schema, I need the error logged, so people can query it with the provider of the data.
If you just want to log the runs error message, it is not necessary to be so troublesome. You could just use outputs to implement it.
You set the create blob action run after parse json action fails, the blob content could be outputs('Parse_JSON')['errors'], if just want to get the error message it should be outputs('Parse_JSON')['errors'][0]['message'].
I have a route that reads from an FTP server, then processes the message. The route has DeadLetterChannel error handler that routes the message to some bean when an exception is thrown while processing the message.
Now when an exception is handled by the error handler, Camel presumes everything passed fine and still deletes the FTP file.
If I remove the error handler, Camel doesn't delete the file when there is an exception.
Now my question is, how can i have a DeadLetterChannel error handler and at the same time stop Camel from deleting FTP file when processing fails?
You can set the option noop=true on the ftp endpoint. Then the file will be left alone.
Though you would then have to consider how you can skip picking up the files in the future? And for that you can use the idempotent repository to keep track of which files you have processed before. Or an alternative is to move the file when you are done etc.
As the ftp component extends the file component see details at: http://camel.apache.org/file2
You have several options to do that:
You do not use the delete=true option at all and handle the delete of the file in the "success" scenario by yourself. This would be relatively transparent.
In case you enter the DLC you can manipulate the endpoint from which you are consuming. Therefore just define your own processor for the DLC in onPrepareFailure. See example: deadLetterChannel("jms:dlc").onPrepareFailure(new ErrorProcessor())
After that you can use the getContext() method to get the camel context and one of the getEndpoint() methods to get your consumer endpoint.
When you have the endpoint you can see which 'process factory' class is used with the getProcessStrategyand there you can update the delete flag to avoid deleting your file.
For this endpoint it is also possible to define your own 'processStrategy' class with the method setProcessStrategy. Please take a look for yourself which process strategy class is used in your case. You then can override the according delete method like deleteLocalWorkFile and just do nothing.
I have the below requirement
A large text file around 10Mb to 25Mb (With 50,000 to 100,000 lines of data) is uploaded into the web application. I have to validate the file line by line and write the output to another location and then display a message to the user.
The App Server is WebLogic and its is accessed through Web Server through Apache Bridge. Apache Bridge times out pretty quickly during the upload + processing activity. Is there any way to solve this issue without changing the timeout of the Apache Bridge
What is best possible solution ? Below are my current thoughts.
Soln 1 Upload the file and return back to the page. Then trigger a Ajax to run the validation in a separate thread and check its status through further Ajax requests.
Soln 2. Use sc_partial_content(206) http Code to keep the connection alive.