I have a generic message router that has its routes created at run time by various route builders based on some configuration.
The configuration is stored as XML and when loaded in memory it gets converted into a RouteConfig domain object exposing via its getters how the route should be build. Such a RouteConfig will have a getFromUri(), getDestinations(), getDeadLetterUri(), isTransacted(), etc methods defined.
An example such a route builder would look like below:
public class NonTransactedRouteBuilder extends AbstractRouteBuilder {
#Override
protected void buildRoute(String endPoint, RouteConfig routeConfig) {
RouteBean routeBean = getRouteBean(routeConfig.getBean());
final String[] destinations = routeConfig.getDestinations();
from(endPoint).routeId(createRouteId())
.autoStartup(false)
.threads(routeConfig.getThreads())
.filter(body().isNotNull())
.process((Processor) routeBean)
.filter(body().isNotNull())
.choice()
.when(header("dead.letter").isNotNull())
.to(getDeadLetterUri())
.otherwise()
.loadBalance().random()
.to(destinations)
.endChoice();
}
}
The above is just an example to give you an idea about what kind of routes we build. The only one relevant thing is that the route is not transacted. Now to unit test it works as expected we extend the TestNG flavor of CamelTestSupport and pass a mocked RouteConfig instance that will result in a route being configured to move messages between a direct:test and two mock:test1 and mock:test2 end points. The route bean has a bit of logic in it and depending of the message content help us with all testing scenarios:
#Test
public void shouldDiscardNullMessages() throws Exception {
...
}
#Test
public void shouldDiscardScamMessages() throws Exception {
...
}
#Test
public void shouldRouteMessageToDeadLetterQueue() throws Exception {
...
}
#Test(timeOut = 1000)
public void shouldRouteMessagesInALoadBalancedWay()
...
}
Everything works fine and we are very happy we can make code changes and test them immediately. However most of our routes builders will build transacted routes. From our integration and end to end tests we know the transacted functionality works fine but would be so much better to be able to test this at your code changing point with an unit test.
So my question is:
Having the same route as above with just an autoStartup(false).transacted() change on it would it be a way to getting the message back in the sender direct:test end point so we can cover this part of the functionality as well. We would be happy with any work around suggestion just to prove this aspect works.
Thank you in advance for your inputs.
UPDATE 1:
One of the things I tried was to configure my test camel context with a TransactionErrorHandler that had a mock jta transaction manager injected. Something like bleow:
#Test
public void shouldBeAbleToRollback() throws Exception {
TransactionErrorHandlerBuilder errorHandlerBuilder = new TransactionErrorHandlerBuilder();
errorHandlerBuilder.setTransactionManager(jtaTransactionManagerMock);
context().setErrorHandlerBuilder(errorHandlerBuilder);
template.sendBody(FROM_1, "rollback message");
...
}
Then I hoped that I will be able to capture a jtaTransactionManagerMock.rollback() but this did not happen. Wat would be the reason not to work.
UPDATE 2:
Unable to achieve the above I stepped back and started integrating ActiveMQ as a transactional resource in my unit tests and everything worked fine. In reality our routes include also file and database end points but for the purpose of unit testing a generic queue builder using just a JMS resource is enough. I did not realized how easy would be when you deal all day long with Webshere MQ manager where you need to have MQ manager created and configured and you have to have the queues already there and all the heavy infrastructure that you have to build for this. ActiveMQ just did a very good job.
However I will still be interested in whether there would be an way to mock this transnational behavior.
Related
I am migration to Camel 3.20.1 from 2.20.1 and am facing some difficulties with my current workflow.
I have three RouteBuilders with routes which are called in hierarchical order like this.
RouteBuilderA
#Override
public void configure() throws Exception {
errorHandler(deadLetterChannel("direct:dead")
.maximumRedeliveries(2)
.redeliveryDelay(1000));
from("direct:A")
.to("direct:B");
}
RouteBuilderB
#Override
public void configure() throws Exception {
from("direct:B")
.to("direct:C");
}
RouteBuilderC
#Override
public void configure() throws Exception {
errorHandler(noErrorHandler())
from("direct:C")
.throwException(new RuntimeException("Something went wrong"))
.to("direct:end");
}
Current flow:
As you can see in the code, route in RouteBuilder3 will trigger exception which will be passed to RouteBuilder2. From there it will not be handled, as no error handler is defined for that route, and it will fail with stacktrace...
Desired flow:
What I need is to pass that exception to RouteBuilder1 and handle it there where (my global deadLetterChannel) error handler is defined.
One could say, just add errorHandler(noErrorHandler()) into RouteBuilderB, right. That is clear and it works, BUT I don't want that because...
Because, in that case redelivery will kick in and route message through routes in RouteBuilderB and RouteBuilderC. But, as I got exception in RouteBuilderC, I only want to route message through route where it failed, that is RouteBuilderC.
Background:
In old Camel version I somehow managed to make it work by using includeRoutes(injectedRouteBuilder), which is not available anymore. RouteBuilderC bean is injected into RouteBuilderB, which is injected in RouteBuilderA.
Question:
Does anybody knows how to solve that? I cannot find a way to make this work. I tried a lot of things, including getCamelContext().addRoutes(injectedRouteBuilder);.
*p.s. I am aware of recommendation to use inheritance for RouteBuilders, but in my case this is not an option. I need to pass it to RouteBuilder1 as only that one knows to which error queue, message needs to be send. *
I have a simple Apache Flink job that looks very much like this:
public final class Application {
public static void main(final String... args) throws Exception {
final var env = StreamExecutionEnvironment.getExecutionEnvironment();
final var executionConfig = env.getConfig();
final var params = ParameterTool.fromArgs(args);
executionConfig.setGlobalJobParameters(params);
executionConfig.setParallelism(params.getInt("application.parallelism"));
final var source = KafkaSource.<CustomKafkaMessage>builder()
.setBootstrapServers(params.get("application.kafka.bootstrap-servers"))
.setGroupId(config.get("application.kafka.consumer.group-id"))
// .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
.setStartingOffsets(OffsetsInitializer.earliest())
.setTopics(config.getString("application.kafka.listener.topics"))
.setValueOnlyDeserializer(new MessageDeserializationSchema())
.build();
env.fromSource(source, WatermarkStrategy.noWatermarks(), "custom.kafka-source")
.uid("custom.kafka-source")
.rebalance()
.flatMap(new CustomFlatMapFunction())
.uid("custom.flatmap-function")
.filter(new CustomFilterFunction())
.uid("custom.filter-function")
.addSink(new CustomDiscardSink()) // Will be a Kafka sink in the future
.uid("custom.discard-sink");
env.execute(config.get("application.job-name"));
}
}
Problem is that I would like to provide an integration test for the entire application — sort of like an end-to-end (set of) test(s) for the entire job. I'm using Testcontainers, but I'm not really sure how to move forward with this. For instance, this is how the test looks like (for now):
#Testcontainers
final class ApplicationTest {
private static final DockerImageName DOCKER_IMAGE = DockerImageName.parse("confluentinc/cp-kafka:7.0.1");
#Container
private static final KafkaContainer KAFKA_CONTAINER = new KafkaContainer(DOCKER_IMAGE);
#ClassRule // How come this work in JUnit Jupiter? :/
public static MiniClusterResource cluster;
#BeforeAll
static void init() {
KAFKA_CONTAINER.start();
// ...probably need to wait and create the topic(s) as well
final var config = new MiniClusterResourceConfiguration.Builder().setNumberSlotsPerTaskManager(2)
.setNumberTaskManagers(1)
.build();
cluster = new MiniClusterResource(config);
}
#Test
void main() throws Exception {
// new Application(); // ...what's next?
}
}
I'm not sure how to implement what's required to trigger the job as-is from that point on. Basically, I would like to execute what was defined before, without (almost) any modifications — I've seen plenty of examples that practically build the entire job again, so that's not an option.
Can somebody provide any pointers here?
MessageDeserializationSchema is unbounded, so isEndOfStream returns false. Not sure if that's an impediment.
In order to make the pipeline more testable, I suggest you create a method on your Application class that takes a source and a sink as parameters, and creates and executes the pipeline, using those connectors.
In your tests you can call that method with special sources and sinks that you use for testing. In particular, you will want to use a KafkaSource that uses .setBounded(...) in the tests so that it cleanly handles just the range of data intended for the test(s).
The solutions and tests for the Apache Flink training exercises are organized along these lines; for example, see RideCleansingSolution.java and RideCleansingIntegrationTest.java. These examples don't use kafka or test containers, but hopefully they'll still be helpful.
I would suggest you instrument your application as an opaque-box test by interacting with it through its public API. This can be done either as an out-process test (e.g. by running your application in a container as well, using Testcontainers) are as an in-process test (by creating your Application and calling its main() method).
Now in your comments you explained, that you want to check for the side-effects of interacting with your application (Kafka messages being published). To check this, connect to the KafkaContainer with your own KafkaConsumer from within the test and use a library such as Awaitiliy to wait until the messages have been received.
I understand that this is because of the way proxies are created for handling caching, transaction related functionality in Spring. And the way to fix it is use AspectJ but I donot want to take that route cause it has its own problems. Can I detect self-invocation using any static analyis tools?
#Cacheable(value = "defaultCache", key = "#id")
public Person findPerson(int id) {
return getSession().getPerson(id);
}
public List<Person> findPersons(int[] ids) {
List<Person> list = new ArrayList<Person>();
for (int id : ids) {
list.add(findPerson(id));
}
return list;
}
If it would be sufficient for you to detect internal calls, you could use native AspectJ instead of Spring AOP for that and then throw runtime exceptions or log warnings every time this happens. That is not static analysis, but better than nothing. On the other hand, if you use native AspectJ, you are not limited to Spring proxies anyway and the aspects would work for self-invocation too.
Anyway, here is what an aspect would look like, including an MCVE showing how it works. I did it outside of Spring, which is why I am using a surrogate #Component annotation for demo purposes.
Update: Sorry for targeting #Component classes instead of #Cacheable classes/methods, but basically the same general approach I am showing here would work in your specific case, too, if you simply adjust the pointcut a bit.
Component annotation:
package de.scrum_master.app;
import static java.lang.annotation.ElementType.TYPE;
import static java.lang.annotation.RetentionPolicy.RUNTIME;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;
#Retention(RUNTIME)
#Target(TYPE)
public #interface Component {}
Sample classes (components and non-components):
This component is to be called by other components should not lead to exceptions/warnings:
package de.scrum_master.app;
#Component
public class AnotherComponent {
public void doSomething() {
System.out.println("Doing something in another component");
}
}
This class is not a #Component, so the aspect should ignore self-invocation inside it:
package de.scrum_master.app;
public class NotAComponent {
public void doSomething() {
System.out.println("Doing something in non-component");
new AnotherComponent().doSomething();
internallyCalled("foo");
}
public int internallyCalled(String text ) {
return 11;
}
}
This class is a #Component. The aspect should flag internallyCalled("foo"), but not new AnotherComponent().doSomething().
package de.scrum_master.app;
#Component
public class AComponent {
public void doSomething() {
System.out.println("Doing something in component");
new AnotherComponent().doSomething();
internallyCalled("foo");
}
public int internallyCalled(String text ) {
return 11;
}
}
Driver application:
Please note that I am creating component instances throughout this sample code with new instead of requesting beans from the application context, like I would do in Spring. But you can ignore that, it is just an example.
package de.scrum_master.app;
public class Application {
public static void main(String[] args) {
new NotAComponent().doSomething();
new AComponent().doSomething();
}
}
Console log when running without aspect:
Doing something in non-component
Doing something in another component
Doing something in component
Doing something in another component
Now with the aspect, instead of the last message we would expect an exception or a logged warning. Here is how to do that:
Aspect:
Sorry for using native AspectJ syntax here. Of course, you could also use annotation-based syntax.
package de.scrum_master.aspect;
import de.scrum_master.app.*;
public aspect SelfInvocationInterceptor {
Object around(Object caller, Object callee) :
#within(Component) &&
call(* (#Component *).*(..)) &&
this(caller) &&
target(callee)
{
if (caller == callee)
throw new RuntimeException(
"Self-invocation in component detected from " + thisEnclosingJoinPointStaticPart.getSignature() +
" to "+ thisJoinPointStaticPart.getSignature()
);
return proceed(caller, callee);
}
}
Console log when running with aspect:
Doing something in non-component
Doing something in another component
Doing something in component
Doing something in another component
Exception in thread "main" java.lang.RuntimeException: Self-invocation in component detected from void de.scrum_master.app.AComponent.doSomething() to int de.scrum_master.app.AComponent.internallyCalled(String)
at de.scrum_master.app.AComponent.internallyCalled_aroundBody3$advice(AComponent.java:8)
at de.scrum_master.app.AComponent.doSomething(AComponent.java:8)
at de.scrum_master.app.Application.main(Application.java:6)
I think, you can use this solution and maybe rather log warnings instead of throwing exceptions in order to softly guide your co-workers to inspect and improve their AOP-dependent Spring components. Sometimes maybe they do not wish self-invocation to trigger an aspect anyway, it depends on the situation. You could run the Spring application in full AspectJ mode and then, after evaluating the logs, switch back to Spring AOP. But maybe it would be simpler to just use native AspectJ to begin with and avoid the self-invocation problem altogether.
Update: In AspectJ you can also make the compiler throw warnings or errors if certain conditions are met. In this case you could only statically determine calls from components to other components, but without differentiating between self-invocation and calls on other methods from other components. So this does not help you here.
Please also notice that this solution is limited to classes annotated by #Component. If your Spring bean is instantiated in other ways, e.g. via XML configuration or #Bean factory method, this simple aspect does not work. But it could easily be extended by checking if the intercepted class is a proxy instance and only then decide to flag self-invocations. Then unfortunately, you would have to weave the aspect code into all of your application classes because the check can only happen during runtime.
I could explain many more things, such as using self-injection and call internal methods on the injected proxy instance instead of via this.internallyCalled(..). Then the self-invocation problem would be solved too and this approach also works in Spring AOP.
Can I detect self-invocation using any static analysis tools?
In theory you can, but be aware of Rice's theorem. Any such tool would sometimes give false alarms.
You could develop such a tool using abstract interpretation techniques. You may need more than a year of work.
You could subcontract the development of such tools to e.g. the Frama-C team. Then email me to basile.starynkevitch#cea.fr
Requisite disclaimer about being new to Camel--and, frankly, new to developing generally. I'd like to have a string generated as the output of some function be the source of my camel route which then gets written to some file. It's the first part that seems challenging: I have a string, how do I turn it into a message? I can't write it into a file nor can I use JMS. I feel like it should be easy and obvious, but I'm having a hard time finding a simple guide to help.
Some pseudo-code using the Java DSL:
def DesiredString() {return "MyString";}
// A camel route to be implemented elsewhere; I want something like:
class MyRoute() extends RouteBuilder {
source(DesiredString())
.to("file://C:/out/?fileName=MyFileFromString.txt");
}
I vaguely understand using the bean component, but I'm not sure that solves the problem: I can execute my method that generates the string, but how do I turn that into a message? The "vague" is doing a lot of work there: I could be missing something there.
Thanks!
Not sure if I understand your problem. There is a bit of confusion about what the String should be become: the route source or the message body.
However, I guess that you want to write the String returned by your method into a File through a Camel route.
If this is correct, I have to clarify first the route source. A Camel Route normally starts with
from(component:address)
So if you want to receive requests from remote via HTTP it could be
from("http4:localhost:8080")
This creates an HTTP server that listens on port 8080 for messages.
In your case I don't know if the method that returns the String is in the same application as the Camel route. If it is, you can use the Direct component for "method-like" calls in the same process.
from(direct:input)
.to("file:...");
input is a name you can freely choose. You can then route messages to this route from another Camel route or with a ProducerTemplate
ProducerTemplate template = camelContext.createProducerTemplate();
template.sendBody("direct:input", "This is my string");
The sendBody method takes the endpoint where to send the message and the message body. But there are much more variants of sendBody with different signatures depending on what you want to send it (headers etc).
If you want to dive into Camel get a copy of Camel in Action 2nd edition. It contains everything you need to know about Camel.
Example:Sending String(as a body content)to store in file using camel Java DSL:
CamelContext context = new DefaultCamelContext();
context.addRoutes(new RouteBuilder() {
public void configure() {
from("timer:StringSentToFile?period=2000")
.setBody(simple(DesiredString()))
.to("file:file://C:/out/?fileName=MyFileFromString.txt&noop=true")
.log("completed route");
}
});
ProducerTemplate template = context.createProducerTemplate();
context.start();
I'm trying to verify my Camel routes I need to prevent the endpoints from starting, the XMPP in particular as it contains concrete host information in their URI. Unfortunately I don't seem to figure out how.
My test class is as follows:
#RunWith(CamelSpringJUnit4ClassRunner.class)
#ContextConfiguration(
classes = {
ApplicationConfig.class
},
loader = CamelSpringDelegatingTestContextLoader.class)
#UseAdviceWith
#MockEndpointsAndSkip
public class XMPPRouteBuilderTest {
#Autowired
ApplicationContext applicationContext;
#Autowired
CamelContext camelContext;
#Test
public void testConfigure() throws Exception {
camelContext.start();
Collection<Endpoint> endpoints = camelContext.getEndpoints();
}
}
Whenever I call start() the actual endpoints are started which causes the XMPP routes to fail with host not found exceptions; I was expecting the mocks to replace the real ones.
Can anyone suggest what am I doing wrong?
Best,
Edoardo
#MockEndpointsAndSkip is only for producers (eg not consumers) so all the route from endpoints is not mocked.
You can use the replaceFromWith with the advice-with builder. See the section Replace from with another endpoint in the official Camel documentation for an example:
http://camel.apache.org/advicewith.html