Camel MQTT Component - Limited number of dynamic endpoints? - apache-camel

Question

We are using Camel MQTT-Component to subscribe to several topics and perform routing logic.
We are using dynamic endpoints in some cases to set the publish topic name at runtime which works up to 15 concurrently connected endpoints.
As soon as the 16-th dynamic endpoint is created (toD()-call) there is a timeout while connecting to the mqtt-Broker.
org.apache.camel.FailedToCreateProducerException: Failed to create Producer for endpoint: <my connection here>&connectAttemptsMax=1&reconnectAttemptsMax=1&publishTopicName=<topic>. Reason: java.util.concurrent.TimeoutException
at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:579) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:406) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:119) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197) [camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.component.mqtt.MQTTConsumer.processExchange(MQTTConsumer.java:51) [camel-mqtt-2.18.3.jar:2.18.3]
at org.apache.camel.component.mqtt.MQTTEndpoint$2.onPublish(MQTTEndpoint.java:257) [camel-mqtt-2.18.3.jar:2.18.3]
at org.fusesource.mqtt.client.CallbackConnection$8.onPublish(CallbackConnection.java:521) [mqtt-client-1.14.jar:1.14]
at org.fusesource.mqtt.client.CallbackConnection.toReceiver(CallbackConnection.java:911) [mqtt-client-1.14.jar:1.14]
at org.fusesource.mqtt.client.CallbackConnection.processFrame(CallbackConnection.java:808) [mqtt-client-1.14.jar:1.14]
at org.fusesource.mqtt.client.CallbackConnection.access$1700(CallbackConnection.java:73) [mqtt-client-1.14.jar:1.14]
at org.fusesource.mqtt.client.CallbackConnection$6.onTransportCommand(CallbackConnection.java:425) [mqtt-client-1.14.jar:1.14]
at org.fusesource.hawtdispatch.transport.TcpTransport.drainInbound(TcpTransport.java:713) [hawtdispatch-transport-1.22.jar:1.22]
at org.fusesource.hawtdispatch.transport.TcpTransport$6.run(TcpTransport.java:592) [hawtdispatch-transport-1.22.jar:1.22]
at org.fusesource.hawtdispatch.internal.NioDispatchSource$3.run(NioDispatchSource.java:209) [hawtdispatch-1.22.jar:1.22]
at org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run(SerialDispatchQueue.java:100) [hawtdispatch-1.22.jar:1.22]
at org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:77) [hawtdispatch-1.22.jar:1.22]
Caused by: java.util.concurrent.TimeoutException: null
at org.fusesource.mqtt.client.Promise.await(Promise.java:83) ~[mqtt-client-1.14.jar:1.14]
at org.apache.camel.component.mqtt.MQTTEndpoint.connect(MQTTEndpoint.java:342) ~[camel-mqtt-2.18.3.jar:2.18.3]
at org.apache.camel.component.mqtt.MQTTProducer.doStart(MQTTProducer.java:38) ~[camel-mqtt-2.18.3.jar:2.18.3]
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:75) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.impl.DefaultCamelContext.deferStartService(DefaultCamelContext.java:1316) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.impl.DefaultCamelContext.doAddService(DefaultCamelContext.java:1244) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.impl.DefaultCamelContext.addService(DefaultCamelContext.java:1214) ~[camel-core-2.18.3.jar:2.18.3]
at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:577) ~[camel-core-2.18.3.jar:2.18.3]
... 20 common frames omitted
Is there a limit for concurrent connected Endpoints for Mqtt-Component if these endpoints are created at runtime?
There seems to be no limit for static endpoints. We are able to start up more than 15 routes with endpoints subscribing to topics and consuming messages.
Update: As a fallback we tried to use camel-paho component and it works with the same broker settings and dynamic topic.
Any help with this problem?

Answer 1

While this may be a limitation of camel-mqtt component, you could use one single endpoint and set the publish topic dynamically in a Header.
MQTT component docs suggest the use of CamelMQTTPublishTopic header (from Camel 2.14 onwards), so your route might be like
<route>
<from uri="direct:start" />
<to uri="bean:publishTopicProcessor" />
<to uri="mqtt:singleEndpoint" />
</route>
Inside the processor you can implement all the logic you need
public void process(Exchange exchange) {
String dynamicTopic = "your/dynamic/topic";
// set the topic you want in the string
exchange.getIn().setHeader("CamelMQTTPublishTopic", dynamicTopic);
}
Now the single endpoint will publish your message on different topics.
I did the exact same thing in this example using Camel 2.18