messages not being consumed by apache camel using servicemix as a broker - apache-camel

So I'm trying to use apache servicemix/camel to aggregate some messages coming through a JMS queue. The logic I have is dead-simple I just want it to use the last message received and only send it through 3 seconds after the last one.
I have servicemix setup as a message broker and I'm able to use it in such a capacity however it doesn't seem to trigger the route. I've never done this before so odds are I'm horribly off-base but here is what I have so far (put this in deploy/fedora-messaging/camel-context.xml)
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd">
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="activemq:topic:fedora.apim.update.merge"/>
<aggregate completionTimeout="3000">
<correlationExpression>
<simple>header.pid</simple>
</correlationExpression>
<to uri="log:events"/>
<to uri="activemq:topic:fedora.apim.update"/>
</aggregate>
</route>
</camelContext>
<bean id="activemq"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://localhost:61616"/>
</bean>
</beans>
The osgi application appears to start and run without error, and messages are received and queued in the topic:fedora.apim.update.merge however they just sit in the queue and never get consumed or pushed out to the destination queue.

I tested your route as a standalone application (outside of a ServiceMix container) with following broker configuration and everything worked as expected:
<broker xmlns="http://activemq.apache.org/schema/core" useJmx="true"
persistent="false">
<transportConnectors>
<transportConnector uri="tcp://localhost:61616" />
</transportConnectors>
</broker>
So, I guess, we can assume that your Camel route definition is not the problem but the ActiveMQ configuration and/or ServiceMix set-up.
EDIT:
Note, that messages to JMS topic destinations are lost if no active subscriber is connected to the destination. If you need some kind of persistence use durable subscribers or JMS queue destinations.

Related

Limit netty4 tcp endpoint to a single instance

I have a mock endpoint at 8001 that will echo anything provided to it.
I have an http endpoint that will submit the end of the URL to the mock endpoint, and provide a response from the endpoint's response.
That works fine.
The challenge is, I want the http route to use only 1 tcp connection to the 8001 endpoint.
I created a worker group as explained elsewhere, and set the worker count to 1. Looking through the source code, I'm thinking this approach should work.
However, when I do this bash command:
for a in {1..5}; do curl "http://localhost:8080/upstream/REQUESTNUM$a" > $a.txt & done;
I see multiple connections to 8001. I would have expected the http endpoint requests would have to share a single pool worker, but that doesn't seem to be the case.
Perhaps I am missing something, or perhaps there is another way to accomplish my goal of using only 1 back-end tcp connection for all the http requests.
How do I accomplish it?
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd">
<camelContext
xmlns="http://camel.apache.org/schema/spring">
<route id="mockUpstream">
<from
uri="netty4:tcp://localhost:8001?sync=true&textline=true&keepAlive=true&disconnect=false&reuseChannel=true" />
<log message="Incoming to upstream: ${body}" />
<transform>
<simple>${body}</simple>
</transform>
</route>
<route id="httpServer">
<from
uri="netty4-http:http://0.0.0.0:8080/upstream?matchOnUriPrefix=true" />
<!-- optional just use CamelHttpQuery from header, for full query -->
<log
message="Incoming http command: ${in.headers[CamelHttpPath]}" />
<transform>
<simple>${in.headers[CamelHttpPath]}</simple>
</transform>
<to
uri="netty4:tcp://localhost:8001?workerGroup=#sharedPool&sync=true&textline=true&keepAlive=true&disconnect=false&reuseChannel=true" />
<transform>
<simple>${body}</simple>
</transform>
</route>
</camelContext>
<bean id="poolBuilder"
class="org.apache.camel.component.netty4.NettyWorkerPoolBuilder">
<property name="workerCount" value="1" />
</bean>
<bean id="sharedPool" class="io.netty.channel.EventLoopGroup"
factory-bean="poolBuilder" factory-method="build"
destroy-method="shutdown">
</bean>
</beans>
Looking at the logs, with TRACE level logging, I saw the NettyProducer's pool was indeed set to use 1 max active connection, but the NettyProducer was allowed 100 idle connections. I changed the following line and it is now behaving as expected:
<to
uri="netty4:tcp://localhost:8001?workerGroup=#sharedPool&sync=true&textline=true&keepAlive=true&disconnect=false&reuseChannel=true&producerPoolMaxActive=1&producerPoolMaxIdle=1" />
I assumed the "producer" settings were only good for the producer side (netty in mock host route) of the connection, but it looks like they can be used by the consumer end (netty in http route), too.
edit: I confused the terms producer and consumer and got that backwards above. Seeing that the "to" element is producing a request for something to consume, the producer* parameters make sense for (netty in http route). The (netty in mock host route) is the consumer of requests.

Implementing Durable Subscription on ServiceMix

I currently am using ServiceMix to route messages using QPID libraries.
My working config is below :
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0
http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
<bean id="amqp" class="org.apache.camel.component.amqp.AMQPComponent">
<property name="connectionFactory">
<bean class="org.apache.qpid.jms.JmsConnectionFactory">
<property name="remoteURI" value="failover:amqps://remote-broker:9551?transport.trustStoreLocation=/vault/QA_UM_A.jks />
</bean>
</property>
</bean>
<bean id="kubeActivemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="tcp://10.100.10.13:61616" />
</bean>
<route>
<from uri="amqp:queue:///queue-1" />
<to uri="kubeActivemq:local-queue?jmsMessageType=Text" />
</route>
</camelContext>
</blueprint>
The above config works.
Now I need to implement durable subscription so that if servicemix goes down, it is able to recover all messages when it comes back up (the ones sent when servicemix is down.)
Based on the documentation , I have implemented a route :
<route>
<from uri="amqp:queue:///queue-1?clientId=1&durableSubscriptionName=bar1" />
<to uri="kubeActivemq:queue:///local-queue" />
</route>
But this implementation gives the ERROR :
2018-05-28 13:28:58,421 | ERROR | mix-7.0.0/deploy | BlueprintCamelContext | 40 - org.apache.camel.camel-blueprint - 2.16.4 | Error occurred during starting Camel: CamelContext(camel-1) due A durable subscription requires a topic (pub-sub domain)
java.lang.IllegalArgumentException: A durable subscription requires a topic (pub-sub domain)
at org.springframework.jms.listener.AbstractMessageListenerContainer.validateConfiguration(AbstractMessageListenerContainer.java:435)
at org.springframework.jms.listener.AbstractJmsListeningContainer.afterPropertiesSet(AbstractJmsListeningContainer.java:157)
at org.apache.camel.component.jms.JmsConsumer.prepareAndStartListenerContainer(JmsConsumer.java:163)
at org.apache.camel.component.jms.JmsConsumer.doStart(JmsConsumer.java:155)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:3234)
at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:3528)
at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:3464)
at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:3394)
at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:3162)
at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:3018)
.
.
.
Is there something Ive missed here ? Arent the topic names supposed to be same as queue names ?
Your route is instructing Camel to connect to a queue, and you need to change it to use a 'topic' for Durable Subscriptions. (Queue subscriptions are durable by default)
amqp:queue:///queue-1...
And Durable Subscriptions require a topic
amqp:topic:///topic-1...

Activemq Not Starting with HTTP uri

I am using activemq with camel for consuming messages from a queue and send them to a http server. I am using following camel configuration :-
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="activemq:queue:Consumer.A.VirtualTopic.Orders"/>
<to uri="http://localhost:8080/" />
<!-- <to uri="file:///Users/vinod/activemq.txt"/> -->
<!-- <to uri="activemq:queue:sssss"/> -->
</route>
</camelContext>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://localhost?create=false"/>
</bean>
</property>
</bean>
In first block creates a route which consumer messages from activemq:queue:Consumer.A.VirtualTopic.Orders queue and send them to server at http://localhost:8080/. The other two commented destinations are working fine, but when I start activemq with above configuration for sending messages over http, the server stops without throwing any error message. Activemq log for this is https://gist.github.com/kumar003vinod/1e5944cb246edb74c47fef7a0b433387
Please provide some insight.
Make sure to include camel-http and camel-http-common JARs in the ActiveMQ lib/camel directory. You may also need to include the transitive dependencies from camel-http in that directory so ActiveMQ has all the needed JARs in its classpath.
That would be commons-httpclient and commons-codec JARs but I think they are already included in lib/optional.

set endpoint of an OSGI Blueprint file in ServiceMix

I know I can define Camel routes in stand-alone xml file, using the Blueprint syntax. If I move one of this file in the "deploy" folder of ServiceMix, it automatically becomes an OSGI bundle. My question is, can I set an endpoint to this new bundle, accessible from outside?
I would like to do something like this:
blue_route1.xml
<blueprint>
<camelContext>
<route>
<from uri="http:my_servicemix:8181/blue_route1_endpoint" />
<to uri="jetty:http://server1" />
</route>
</camelContext>
</blueprint>
blue_route1 becomes an OSGI bundle once deployed, but where should I define "blue_route1_endpoint" ? Is it doable?
[UPDATE]
summering, I want that an external WS is able to send messages to blue_route1_endpoint, where the blue_route1 bundle will redirect messages according to Camel routes, without the need to create a new WS "Blue_route1" to deploy in ServiceMix
______________________
| ____________ |
external-->(blue_route1_endpoint)==|==-->|blue_route1|--|-->(http://server1)
WS | |___________| |
|____________________|
ServiceMix
Found it! I didn't get that it's so easy. To make ServiceMix listen on a port, I just have to specify the endpoint using the Camel-jetty component.
So, to answer my question, I solved in this way:
install the camel-jetty component in ServiceMix
features:install camel-jetty
Write the camel-route with Blueprint, in blue_route1.xml file
<?xml version="1.0" encoding="UTF-8"?>
<blueprint
xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0
http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route>
<from uri:="jetty:http:my_servicemix:8181/blue_route1_endpoint">
<to uri="http://localhost:8080/user/services/user?bridgeEndpoint=true&throwExceptionOnFailure=false"/>
</route>
<route>
<from uri="jetty:http://0.0.0.0:6969/sp_role?matchOnUriPrefix=true"/>
<setHeader headerName="Content-Type">
<groovy>"text/xml; charset=utf-8"</groovy>
</setHeader>
<to uri="http://server1"/>
</route>
</camelContext>
</blueprint>
I used a random port 8181 to listen to... but I could choose every number, ServiceMix will automatically start a jetty component, listening and consuming on that port/endpoint.
For SOAP messages you need CXF Component and wsdl file of your webservice. You can configure your endpoint outside of the camelContext like this:
<cxf:cxfEndpoint id="yourId" address="/your/address/to/endpoint"
serviceClass="your.java.ServiceClass"
wsdlURL="path/to/your/wsdl/file.wsdl" />
And in your route use tag like this:
<from uri="cxf:bean:yourId"/>
You need to add namespace and schemaLocation to your blueprint to use cxf namespace, use this:
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint/cxf http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

IllegalStateException on camel cxf route in FUSE (servicemix) ESB

I try to set up a camel (Ver. 2.4.0) route in a FUSE (Ver. 4.3.0) ESB/OSGi container.
It should be a simple cxf-proxy to route a WebService call from a 'proxy' address to a real service.
I read several documentation:
CXF Proxy Example
camel cxf component
and set up the following spring configuration:
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xmlns:osgi="http://www.springframework.org/schema/osgi"
xmlns:cxf="http://camel.apache.org/schema/cxf"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
http://www.springframework.org/schema/osgi
http://www.springframework.org/schema/osgi/spring-osgi.xsd
http://camel.apache.org/schema/osgi
http://camel.apache.org/schema/osgi/camel-osgi.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://camel.apache.org/schema/cxf
http://camel.apache.org/schema/cxf/camel-cxf.xsd">
<import
resource="classpath:META-INF/cxf/cxf.xml" />
<import
resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
<import
resource="classpath:META-INF/cxf/cxf-extension-http.xml" />
<import
resource="classpath:META-INF/cxf/osgi/cxf-extension-osgi.xml" />
<!-- the proxy service -->
<cxf:cxfEndpoint
id="myServiceProxy"
address="http://localhost:9003/cxf/myService"
serviceClass="foo.bar.iface.MyServiceInterface" />
<!-- my real existing cxf soap service -->
<cxf:cxfEndpoint
id="myService"
address="http://foo.bar/services/myService"
wsdlURL="http://foo.bar/services/myService?wsdl"
serviceClass="foo.bar.iface.MyServiceInterface"
serviceName="s:MyService"
endpointName="s:MyServiceEndpoint"
xmlns:s="http://foo.bar/iface/" />
<!-- route -->
<camel:camelContext>
<camel:route>
<camel:from
uri="cxf:bean:myServiceProxy" />
<camel:to
uri="cxf:bean:myService" />
</camel:route>
</camel:camelContext>
</beans>
Trying to start the bundle in FUSE leads to this exception
karaf#root> Exception in thread "SpringOsgiExtenderThread-22" org.apache.camel.RuntimeCamelException: java.lang.IllegalStateException: Endpoint address should be a relative URI wrt to the servlet address (use '/xxx' for example)
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1126)
at org.apache.camel.spring.SpringCamelContext.onApplicationEvent(SpringCamelContext.java:103)
at org.apache.camel.spring.CamelContextFactoryBean.onApplicationEvent(CamelContextFactoryBean.java:231)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
I do not know what's wrong. I doubt that my Endpoint address is wrong and I do not know what my servlet address is (there is no cxf:cxfEndoint servelt address attribute).
Any help to lead me in the right direction to solve this problem would be appreciated.
Thanks
Klaus
I finally found out what's wrong.
Instead
<!-- the proxy service -->
<cxf:cxfEndpoint
id="myServiceProxy"
address="http://localhost:9003/cxf/myService"
serviceClass="foo.bar.iface.MyServiceInterface" />
it has to be
<!-- the proxy service -->
<cxf:cxfEndpoint
id="myServiceProxy"
address="/myService"
serviceClass="foo.bar.iface.MyServiceInterface" />
As the first approach worked well running the camel project outside FUSE (in this case a the http server will be started to provide the service) it has to be a relative address inside FUSE.
Inside FUSE the embedded HTTP server which runs on (localhost:8181) will be used and the service url will be extended to http://localhost:8181/cxf/myService.

Resources