I try to get a blob out of a database using camel (2.15.1). The data size is about 20 mb. This is my route:
<route streamCache="true">
<from uri="file://c:/tmp/trigger" />
<to uri="jdbc:datasourceCosima" />
<log message="done 1."/>
<to uri="direct:a" />
</route>
<route streamCache="true">
<from uri="direct:a" />
<to uri="file:blobdata" />
<log message="done 2."/>
</route>
To start it i place a file with the sql command in c:/tmp/trigger. The content is something like select bls_data from blob_storage where io_id='2fe0...'
This is the result:
[2016-02-18 12:44:15.539] INFO: org.apache.camel.util.CamelLogger log - done 1.
[2016-02-18 12:44:16.553] SEVERE: org.apache.camel.util.CamelLogger log - Failed delivery for (MessageId: ID-GHD13044-49732-1455793836517-2-2382 on ExchangeId: ID-GHD13044-49732-1455793836517-2-2381). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[org.apache.camel.component.file.GenericFileMessage#7a0613]
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route9 ] [route9 ] [file://c:/tmp/trigger ] [ 1635]
[route9 ] [to13 ] [jdbc:datasourceCosima ] [ 822]
[route9 ] [log12 ] [log ] [ 1]
[route9 ] [to14 ] [direct:a ] [ 813]
[route10 ] [to15 ] [file:blobdata ] [ 812]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id ID-GHD13044-49732-1455793836517-2-2381
ExchangePattern InOnly
Headers {breadcrumbId=ID-GHD13044-49732-1455793836517-2-2380, CamelFileAbsolute=true, CamelFileAbsolutePath=c:\tmp\trigger\select.txt, CamelFileContentType=null, CamelFileLastModified=1455793414658, CamelFileLength=179, CamelFileName=select.txt, CamelFileNameConsumed=select.txt, CamelFileNameOnly=select.txt, CamelFileParent=c:\tmp\trigger, CamelFilePath=c:\tmp\trigger\select.txt, CamelFileRelativePath=select.txt, CamelJdbcColumnNames=[bls_data], CamelJdbcRowCount=1, CamelRedelivered=false, CamelRedeliveryCounter=0}
BodyType java.util.ArrayList
Body [{bls_data=255044462D312E360D25E2E3CFD30D0A3...
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[org.apache.camel.component.file.GenericFileMessage#7a0613]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1635)
at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:308)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:433)
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
I thought that streamCache would do the job and cache the data on disc or whatever.
So the question is: How to get blob data from a database into a file without size restriction?
Related
I want to validate a JSON file with a JSON schema file, which contains a reference to another JSON schema file.
I added a JSON Schema Validator for my Apache Camel route. Because I use JSON schema version draft-07 and the default version is draft-04, I exposed a JsonSchemaLoader, see also https://stackoverflow.com/a/63417253/5277820.
The first schema test-schema-1 is loaded from class path, but the second test-schema-2 is loaded from internet. The used ID http://mycompany/test-schema2.json is only an ID not a real resource, therefore I get an exception.
Is there any way to load the second JSON schema also from class path?
Spring Boot application
#SpringBootApplication
public class TestApplication {
public static void main(String[] args) {
SpringApplication.run(TestApplication.class, args);
}
#Bean
public JsonSchemaLoader mySchemaLoader() {
return (camelContext, schemaStream) ->
JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7).getSchema(schemaStream);
}
#Bean
public EndpointRouteBuilder routeBuilder() {
return new EndpointRouteBuilder() {
#Override
public void configure() throws Exception {
from(file("d:/tmp/camel/"))
.to(jsonValidator("/test/test-schema1.json").advanced().schemaLoader(mySchemaLoader())).stop();
}
};
}
}
JSON schema
test-schema-1:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://mycompany/test-schema1.json",
"type": "object",
"properties": {
"obj": {
"$ref": "test-schema2.json"
}
},
"required": [
"target"
]
}
test-schema-2:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "http://mycompany/test-schema2.json",
"type": "object",
"properties": {
"value": {
"type": "string"
}
}
}
Logs
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [from[file://d:/tmp/camel/] ] [ 5]
...
[route1 ] [to1 ] [json-validator:///test/test-schema1.json?hash=93f27f49 ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
com.networknt.schema.JsonSchemaException: java.net.UnknownHostException: mycompany
at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:348) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.RefValidator.getRefSchema(RefValidator.java:76) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.RefValidator.<init>(RefValidator.java:41) ~[json-schema-validator-1.0.43.jar:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at com.networknt.schema.ValidatorTypeCode.newValidator(ValidatorTypeCode.java:131) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonMetaSchema.newValidator(JsonMetaSchema.java:342) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.ValidationContext.newValidator(ValidationContext.java:53) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchema.read(JsonSchema.java:198) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchema.initialize(JsonSchema.java:76) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.PropertiesValidator.<init>(PropertiesValidator.java:36) ~[json-schema-validator-1.0.43.jar:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
at com.networknt.schema.ValidatorTypeCode.newValidator(ValidatorTypeCode.java:131) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonMetaSchema.newValidator(JsonMetaSchema.java:342) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.ValidationContext.newValidator(ValidationContext.java:53) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchema.read(JsonSchema.java:198) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchema.initialize(JsonSchema.java:76) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchemaFactory.newJsonSchema(JsonSchemaFactory.java:254) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:296) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:304) ~[json-schema-validator-1.0.43.jar:na]
at test.TestApplication.lambda$0(TestApplication.java:22) ~[classes/:na]
at org.apache.camel.component.jsonvalidator.JsonValidatorEndpoint.getOrCreateSchema(JsonValidatorEndpoint.java:162) ~[camel-json-validator-3.5.0.jar:3.5.0]
at org.apache.camel.component.jsonvalidator.JsonValidatorEndpoint.onExchange(JsonValidatorEndpoint.java:94) ~[camel-json-validator-3.5.0.jar:3.5.0]
at org.apache.camel.support.ProcessorEndpoint$1.process(ProcessorEndpoint.java:61) ~[camel-support-3.5.0.jar:3.5.0]
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66) ~[camel-support-3.5.0.jar:3.5.0]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:169) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:404) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:287) ~[camel-base-3.5.0.jar:3.5.0]
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:483) ~[camel-file-3.5.0.jar:3.5.0]
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:237) ~[camel-file-3.5.0.jar:3.5.0]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:198) ~[camel-file-3.5.0.jar:3.5.0]
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:190) ~[camel-support-3.5.0.jar:3.5.0]
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:107) ~[camel-support-3.5.0.jar:3.5.0]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.net.UnknownHostException: mycompany
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:608) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:557) ~[na:na]
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[na:na]
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) ~[na:na]
at com.networknt.schema.uri.URLFetcher.openConnectionCheckRedirects(URLFetcher.java:57) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.uri.URLFetcher.fetch(URLFetcher.java:43) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.uri.URISchemeFetcher.fetch(URISchemeFetcher.java:50) ~[json-schema-validator-1.0.43.jar:na]
at com.networknt.schema.JsonSchemaFactory.getSchema(JsonSchemaFactory.java:325) ~[json-schema-validator-1.0.43.jar:na]
... 46 common frames omitted
There is a configuration property for URI mappings, see Configuration:
uriMappings
Map of public, typically internet-accessible schema URLs to alternate locations; this allows for offline validation of schemas that refer to public URLs. This is merged with any mappings the sonSchemaFactory may have been built.
The type for this variable is Map<String, String>.
Modified code:
#Bean
public JsonSchemaLoader mySchemaLoader() {
final SchemaValidatorsConfig config = new SchemaValidatorsConfig();
config.setUriMappings(
Map.of("http://mycompany/test-schema2.json", "resource:/test/test-schema2.json"));
return (camelContext, schemaStream) ->
JsonSchemaFactory.getInstance(SpecVersion.VersionFlag.V7).getSchema(schemaStream, config);
}
I am trying to use PropertyBusinessObject.getPropertyIndex.loadJsonList(storage) to load a list of propertybusinessobjects. It keeps failing for this json which I have verified as valid json.
[{
"_id": "5e8ef0485053da7500020c79",
"name": "CriminalAct",
"description": "A criminal act",
"label": "Criminal Act",
"_created": "2020-04-09T09:52:08.689Z",
"comment_services": [
"5c49c41840e295040000582a",
"5c49c2b140e2950400005811",
"5b632dfabba4456c000002b1"
],
"service_attributes": [
"5ba1192eda1139170000482c",
"5e8f02c45053da7500020d04",
"5e8f03355053da7500020d08",
"5e8f03d45053da7500020d0e",
"5e8f06585053da7500020d23",
"5e8f08c35053da7500020d35",
"5e8f0c035053da7500020d58",
"5c347ec511f06f26000027a4",
"5c3471f711f06f26000025fe",
"5c347d2811f06f2600002768"
],
"category": [
"5ca7939d0eaf051400001c5f",
"5d8e2f84bc58cd2800000854",
"5e8ee9655053da7500020c41"
],
"parent": [],
"providers": ["5a57a54c9d89946a00002216"],
"logo": ["5e8ef2a15053da7500020c8e"]
}]
I get this error
java.lang.ArrayIndexOutOfBoundsException: -1
[EDT] 0:0:2,193 - Exception during JSON parsing at row: 1 column: 9 buffer:
[EDT] 0:0:2,197 - Exception: java.lang.NullPointerException - null
[EDT] 0:0:2,198 - Exception: java.lang.NullPointerException - null
at java.util.ArrayList.elementData(ArrayList.java:418)
at java.util.ArrayList.remove(ArrayList.java:495)
at com.codename1.io.JSONParser.endArray(JSONParser.java:625)
at com.codename1.io.JSONParser.parse(JSONParser.java:387)
at com.codename1.io.JSONParser.parseJSON(JSONParser.java:484)
at com.codename1.properties.PropertyIndex.loadJSONList(PropertyIndex.java:667)
I am quiet puzzled as to what could be wrong.
I am trying to test route app.cash.source-endpoint-quartz from my junit which internally redirect the flow to another route app.accrual.source-endpoint-direct.
In my case the scenario is like app.cash.source-endpoint-quartz sends few messages to MQ, after that the processing continues and based on certain condition it will redirect to route app.accrual.source-endpoint-direct and which eventually sends few more messages to same MQ.
How do I test this?
Cash Route
from("{{app.cash.source-endpoint-quartz}}")
.routeId("cash-route")
.log(LoggingLevel.INFO, logger,"***** CASH ROUTE STARTED *****")
.doTry()
...
....
.to("direct:cashTransactionRoute") //Sub Route
.process(c -> {
TransactionMaster transactionMaster = (TransactionMaster) c.getIn().getHeader(Constants.HEADER_TRANSACTION_MASTER_CASH);
transactionMasterService.updateMsgStatus(transactionMaster, Status.SUCCESS);
})
.bean(transactionManager, "markSuccess")
...
...
Cash Sub Route
from("direct:cashTransactionRoute")
.routeId("cash-transaction-route")
...
.split(simple("${body}"))
.parallelProcessing()
...
.end()// End of split() and parallelProcessing()
.end()
.process(e -> {
...
})
.choice()
.when(simple("${body.size} != 0"))
.process(e -> {
e.getIn().getBody();
})
.to("{{app.accrual.source-endpoint-direct}}") //Redirect to accrual route
.end() //End of choice
.end();
Accrual Route
from("{{app.accrual.source-endpoint-direct}}") //Accrual Route
.routeId("accrual-route")
.log(LoggingLevel.INFO, logger,"***** ACCRUAL ROUTE STARTED *****")
...
...
application-test.yaml
app:
cash:
source-endpoint-quartz: direct-vm:cash
txn-type: CASH
accrual:
source-endpoint-direct: direct-vm:accrual
source-endpoint-quartz-1: direct-vm:accrual-quartz-1
source-endpoint-quartz-2: direct-vm:accrual-quartz-2
Below is my JUnit which I tried but getting error.
#RunWith(CamelSpringBootRunner.class)
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT, properties = {"camel.springboot.java-routes-include-pattern=**/Cash*, **/Accrual*"})
#EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class, DataSourceTransactionManagerAutoConfiguration.class, SecurityAutoConfiguration.class})
#DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
#ActiveProfiles(profiles = {"test"})
public class CashRouteTest {
#EndpointInject(value = "{{app.cash.source-endpoint-quartz}}")
private ProducerTemplate producerTemplate;
#EndpointInject(value = "{{app.accrual.source-endpoint-direct}}")
private ProducerTemplate producerTemplateAccrual;
#EndpointInject(value = "{{app.mqservice}}")
private MockEndpoint mock;
#Test
public void cashRouteTest_PaymentWithAccrual() throws Exception {
Mockito.when(...).thenReturn(.....);
Mockito.when(...).thenReturn(.....);
...
producerTemplateAccrual.start();
producerTemplate.start();
producerTemplate.sendBody(null);
//producerTemplateAccrual.sendBody(null);
mock.expectedMessageCount(4);
mock.expectedBodiesReceived();
Assert.assertEquals(4, mock.getExchanges().size());
String xml = String.valueOf(mock.getExchanges().get(0).getIn().getBody());
MessageEnvelope messageEnvelope = (MessageEnvelope) XmlUtil.toObject(xml);
String actualPayload = XmlUtil.toXml(messageEnvelope.getPayload());
String expectedPayload = "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>....";
Collection<TransactionMaster> txnMasters = (Collection<TransactionMaster>) txnMasterRepo.findAll();
Collection<Transaction> txns = (Collection<Transaction>) txnRepo.findAll();
logger.info("actualPayload : {} ", actualPayload);
Assert.assertEquals(expectedPayload, actualPayload);
Assert.assertEquals(2, txnMasters.size());
Assert.assertEquals(4, txns.size());
Assert.assertEquals(Status.SUCCESS, Status.forValue(txnMasters.iterator().next().getRefStatusId()));
Assert.assertEquals(Status.SUCCESS, Status.forValue(txns.iterator().next().getRefStatusId()));
mock.assertIsSatisfied(5000);
}
}
When I run this JUnit I get below error.
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-route started and consuming from: direct-vm://cash
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-enrich-route started and consuming from: direct://cashEnrichRoute
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-transaction-route started and consuming from: direct://cashTransactionRoute
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Total 3 routes, of which 3 are started
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[cash-route ] [cash-route ] [direct-vm://cash ] [ 32018]
[cash-route ] [log1 ] [log ] [ 9]
[cash-route ] [doTry1 ] [doTry ] [ 0]
[cash-route ] [bean1 ] [bean[com.app.service.DbTransactionManager] ] [ 500]
[cash-route ] [bean2 ] [bean[com.app.service.CashTransactionSearch] ] [ 15]
[cash-route ] [choice1 ] [when[simple{Simple: ${body.size} == 0}]choice[] ] [ 204]
[cash-route ] [process1 ] [Processor#0x20bc4c09 ] [ 15]
[cash-route ] [process2 ] [Processor#0x1ca6323c ] [ 16]
[cash-route ] [to1 ] [direct:cashEnrichRoute ] [ 130]
[cash-enrich-route ] [split1 ] [split[Simple: ${body}] ] [ 130]
[cash-route ] [process3 ] [Processor#0x5aac9d67 ] [ 235]
[cash-route ] [process4 ] [Processor#0x753cc26d ] [ 75]
[cash-route ] [to2 ] [direct:cashTransactionRoute ] [ 0]
[cash-transaction-r] [split2 ] [split[Simple: ${body}] ] [ 385]
[cash-transaction-r] [process16 ] [Processor#0x20b3bbe7 ] [ 0]
[cash-transaction-r] [choice3 ] [when[simple{Simple: ${body.size} != 0}]choice[] ] [ 0]
[cash-transaction-r] [process17 ] [Processor#0x5190ae57 ] [ 0]
[cash-transaction-r] [to5 ] [{{app.accrual.source-endpoint-direct}} ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.component.directvm.DirectVmConsumerNotAvailableException: No consumers available on endpoint: direct-vm://accrual. Exchange[ID-SPLS1800411-10N-1564022207351-0-3]
I see that when it loads the context it does not start the accrual route. Even though I have specified producerTemplateAccrual.start(); in my junit.
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-route started and consuming from: direct-vm://cash
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-enrich-route started and consuming from: direct://cashEnrichRoute
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Route: cash-transaction-route started and consuming from: direct://cashTransactionRoute
2019-07-25 10:36:47,179 [main] INFO o.a.camel.spring.SpringCamelContext - Total 3 routes, of which 3 are started
#SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT, properties = {"camel.springboot.java-routes-include-pattern=**/Cash*, **/Accrual*"})
Cash*, **
Because of the space after , the AccrualRoute was getting ignored.
After removing the space both the routes are getting loaded.
We are migrating from solr 6.6 to solr 7.5. We have a solr Join which facets using facet. thread set to 32. We are seeing the repeated solrCore close exception and also further querying fails.Below are Exception stack trace. and solr Query
2019-07-09 17:37:30.053 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore [promotion_rules-v1_shard1_replica_n1] **CLOSING SolrCore org.apache.solr.core.SolrCore**#44de0f89
2019-07-09 17:37:30.054 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.promotion_rules-v1.shard1.replica_n1, tag=44de0f89
2019-07-09 17:37:30.056 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter#36a60356: rootName = null, domain = solr.core.promotion_rules-v1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.promotion_rules-v1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry#79be463d
2019-07-09 17:37:30.148 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.promotion_rules-v1.shard1.leader, tag=44de0f89
2019-07-09 17:37:30.159 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.u.p.DocExpirationUpdateProcessorFactory Triggering Graceful close of DocExpiration Executor
2019-07-09 17:38:03.062 **ERROR (facetExecutor-67-thread-26-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-4] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org**
2019-07-09 17:38:03.067 ERROR (facetExecutor-67-thread-28-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-5] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org**
2019-07-09 17:38:06.176 ERROR (facetExecutor-67-thread-25-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-7] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:38:07.602 ERROR (facetExecutor-67-thread-24-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-6] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:38:11.868 ERROR (facetExecutor-67-thread-23-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-8] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:38:32.522 ERROR (facetExecutor-67-thread-4-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-9] on** org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:08.862 ERROR (facetExecutor-67-thread-3-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-10] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:08.988 ERROR (facetExecutor-67-thread-6-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-12] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:09.048 ERROR (facetExecutor-67-thread-5-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-13] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:10.474 ERROR (facetExecutor-67-thread-2-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-11] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:10.572 ERROR (facetExecutor-67-thread-7-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-14] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
2019-07-09 17:39:10.646 ERROR (facetExecutor-67-thread-9-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-15] on org.apache.solr.core.SolrCore#44de0f89. Please report this exception to solr-user#lucene.apache.org
SolrQuery
q=*:*&defType=edismax&fl=comboid,skuid&start=0&rows=15&facet.limit=100&variant=A&debugQuery=true&debug.explain.structured=true&fq=!status_facet:inactive&fq=!mode:AF_ONLY&fq=**{!join+from=skuid+to=skuid+fromIndex=promotion_rules}id:promo-opera-test-1000_promo-opera-test-1000**&enableManagedResourceRange=true&facet=true&facet.mincount=1&**facet.threads=32**&facet.field=dynamiccategory_facet&facet.recommend.maxfacets=30&facet.recommend.debug=true&sort=ratingandreviewsort+desc&qt=/promo&timeAllowed=5000
I created a route that look like this:
from("cxfrs:bean:server?bindingStyle=SimpleConsumer")
.setExchangePattern(ExchangePattern.InOnly)
.streamCaching()
.onCompletion()
.modeBeforeConsumer() // <-- Problem
.to(CommonRoutes.ENDPOINT_DIRECT_LOGGING)
.process(createResponse)
.end()
.split().body().stopOnException()
.process(domainWrapProcessor)
.to(OutAmq.OUT_PLUG)
;
The Logging route is this:
from(ENDPOINT_DIRECT_LOGGING)
.routeId(routeId_direct_logging)
.doTry()
.process(createAuditMessageProcessor)
.marshal().json(JsonLibrary.Jackson)
.to(ENDPOINT_VM_LOG_DELIVERY) //mocked out in Test
.doCatch(Throwable.class)
.process(e -> {
log.warn("Failed to send audit message", e);
})
.end()
.end();
Because i only want to send an empty body with status back to caller i added .process(createResponse) together with .modeBeforeConsumer(). For logging it is not relevant if this happens before or after consumer.
The Problem is by adding .modeBeforeConsumer() camel produces a StackoverflowError.
onCompletion is triggered some hundred times but the exchange is always fed back after .to(CommonRoutes.ENDPOINT_DIRECT_LOGGING) - createResponse is never reached. But the mock:ENDPOINT_VM_LOG_DELIVERY registers some hundred exchanges.
I found 2 workarounds (but both destroy the desired semantic):
"mock and skip" ENDPOINT_DIRECT_LOGGING (log route is not called)
delete/comment .modeBeforeConsumer()
So what is going on here? What could cause this?
Here is the camel error log (repeats many times):
[immobilenserviceIn] [to3 ] [direct:logging ] [ 769]
[routeId_directLogg] [doTry1 ] [doTry ] [ 12]
[routeId_directLogg] [process1 ] [Processor#0x11f0a5a1 ] [ 0]
[routeId_directLogg] [marshal1 ] [marshal[org.apache.camel.model.dataformat.JsonDataFormat#20435c40] ] [ 12]
[routeId_directLogg] [to1 ] [vm:log-delivery ] [ 0]
[immobilenserviceIn] [process3 ] [Processor#0x142269f2 ] [ 0]
[immobilenserviceIn] [to3 ] [direct:logging ] [ 757]
[routeId_directLogg] [doTry1 ] [doTry ] [ 12]
And this is a part of the stack trace (until first repetition):
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-D624342-61550-1491998476554-0-2]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1706)
at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:348)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:160)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
at org.apache.camel.processor.OnCompletionProcessor.doProcess(OnCompletionProcessor.java:151)
at org.apache.camel.processor.OnCompletionProcessor$OnCompletionSynchronizationBeforeConsumer.onAfterRoute(OnCompletionProcessor.java:350)
at org.apache.camel.util.UnitOfWorkHelper.afterRouteSynchronizations(UnitOfWorkHelper.java:154)
at org.apache.camel.impl.DefaultUnitOfWork.afterRoute(DefaultUnitOfWork.java:273)
at org.apache.camel.processor.CamelInternalProcessor$RouteLifecycleAdvice.after(CamelInternalProcessor.java:316)
at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:245)
at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:256)
at org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86)
at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:101)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)