I recently joined a project in which lots of autosys jobs are replaced with camel routes. most of the processing flows follow the same pattern:
Read a file from a certain folder
Apply some deserialization of some sort into a java structure
Invoke a stored procedure or another based on some values included in the java structuure
Send the processing results to some third party consumer
The easiest way to implement the stored procedure invoke was to use the sql-stored component. Let's just say I have this route:
from("file://d:/temp/in/?include=myFile*.csv")
.process("myHeaderSetter")
.choice()
.when(header("myheader")).to("sql-stored:proc_1()?dataSource=#dataSource")
.otherwise().to("sql-stored:proc_2()?dataSource=#dataSource")
.end()
.to("reportGenerator")
.to("file://d:temp/out/?fileName=report.txt");
Each processor is very well unit tested but we want to make sure that the route logic goes as we expect. The stored procedures can take a while to execute and running the real route is not an option part of the unit testing so I need a way to test that the stored procedures are invoked without actually getting them to run.
So what would be a good approach to unit test a route like above.
Thank you in advance for your inputs
Take a look at advice-with (http://camel.apache.org/advicewith.html) when testing as you can use it to replace parts of your routes with other bits such as routing to mocks instead of the sql
Related
I am working on an application that uses Apache Camel to flow a single request message (input) through some initial Camel components/logic and then to a multicast at which point the route branches out into multiple branches. The purpose of each branch is to retrieve data from a specific web service (or other back-end data source, e.g. database) and then after the web service invocation / data retrieval operation completes, each of the branches dumps its output data in the same way via a custom bean endpoint. I expect to eventually have approximately 40 different branches in the Camel route, each of which might flow through a different set of Camel components in order to prepare its request, submit the request, process the response, etc... I anticipate that a fair number of the branches will be quite similar (e.g. all SOAP invocations quite similar, all REST invocations quite similar, etc.) and so have concocted an approach whereby a config file stores the list of back-end data sources to be invoked/retrieved-from along with the ability to define (indirectly) the route that should be taken to reach each of those sources. The config file looks something like this:
[a]
route=X + Y
Y.url=http://someservice
[b]
route=Z
Z.someproperty=123
And then I have code that reads through that config file and treats each of the "sections" (e.g. "[a]", "[b]", etc.) as a branch (i.e. a destination out of the multicast) and relies on classes that are dynamically instantiated (e.g. XRouteSegment, YRouteSegment, ZRouteSegment) in order to each in-turn populate/define the route for its specific branch. As some examples, I have built RouteSegment helper classes for wiring up components such as Velocity, CXF, CXF-RS, for data marshalling/unmarshalling, etc... based on properties set in config file.
As far as initialization of the Camel context goes, it starts out in a fairly typical way with a single RouteBuilder which builds out the first part of the route up to the multicast. But then I go into a for loop and loop through all of the sources found in the config file (e.g. "a", "b", etc.) and create seda nodes for each of those which the multicast flows to. And then I call into each of the RouteSegment instances associated with a given source (e.g. X + Y) and allow those to add to the RouteDefinition as they need (e.g. from their seda start point going forward). And then back in my "main" RouteBuilder I tack on some final routing/components that is to be the same for all of the branches (i.e. the logic that forces each of the branches to store its data via the same custom bean).
The code works just fine, but I am questioning whether this approach is overkill and/or whether there is some easier/cleaner way of doing this that I am overlooking. Would I be better off just having individual Java classes (i.e. RouteBuilders) for each of the branches (in addition to the "trunk" and "tails" of the route)? What I was trying to avoid was having too much duplicated logic/code across all of those classes ... e.g. 20 classes all pulling data from SOAP web services in pretty much exactly the same way. So I am using a RouteSegment instance like "X" referenced above as re-usable shorthand for what would otherwise be a sequence of different Camel Java DSL calls (e.g. from/to/process/log/etc ... with parameters to control the specifics of the individual statements). Are there any other strategies/approaches I should consider in order to dynamically build out a Camel route (+ sizable number of branches) at runtime (e.g. within a for loop, or via some sort of reflection/discovery process (app runs using Spring Boot))?
Thanks in advance for any ideas you might be able to provide/suggest that I might not have thought of / tried yet!
I just want to throw in some subjects I am missing in your description.
If I understand your description correct, all your branches components that are called by multicast are not real components but kind of building blocks to build Camel routes at runtime. That sounds like they are not testable and not startable standalone (but perhaps you just not explained that aspect).
If you would build individual small components (every one with its own RouteBuilder) you would have something similar to a microservice architecture: small units to develop and deploy individually.
Since you use Spring Boot, you have autodiscovery of Routes, so they are kind of "pluggable". The components are also testable using Camel routetests etc.
The components would be much more "static" and small standalone projects. This also ensures a fast development roundtrip when you work on the components.
But as you write, this can lead to lots of redundant code. So I guess you have to decide what is more important for you.
I am currently working on writing automated tests using Selenium Webdriver. We use MTM to run our test suites. I need some ideas as to what would be a good way to write these tests.
Currently before running these tests, we perform a basic setup that sets the username and password that would be required to login to the site, set the browser that the test should use, and few other things.
Currently the data that is required for each of the test is setup manually and is already present in the database . The test simply performs a keyword search, finds the necessary data it needs and then performs the assertions. What we would like to achieve is find such data that is already present in the database and use it instead of creating it manually. That way I can run these tests across different environments(dev,qa,production).
The site I am testing is an e-commerce website. I mostly write tests for specific features that my team develops, and thus many of these tests require some specific data. e.g setting up a store that has products with certain shipping rates, with particular offers etc. I would like to find a way to automate or almost remove this manual process of setting up the data. That way I have the flexibility to run these tests across environments. Could you please direct me to some articles/suggestions that can help me achieve this ?
If I am understanding your question correctly, you want to automate the test data setup.
You can achieve this in following ways:
If possible, write a sql script which inserts the desired data in db. Now you can execute this while running your tests. If you are using TestNG framework, then there is already an annotation available like #BeforeTest. You can execute that sql script in this annotation, it will be executed once before your test and data is ready.
Prepare data in a spreadsheet. Create an algorithm, fill the data dynamically in spreadsheet and from there either read directly and fetch it to your test using #BeforeTest or if required, data in spreadsheet can be inserted in db also.
Quartz.net and me don't seem to think the same way. Please help.
I'll have Quartz running as a Windows Service.
I'll have an Ado Jobstore setup on my SQL server.
I'll have the connection string setup that allows Quartz to access the jobstore.
I'll have a trigger job data map (stored in the jobstore?).
I see that I can set a Job-name, and can have Job Data Map key/value pairs that I can store for example a stored proc name and maybe a param. So far so good. I also see that I can write code that will implement iJob and in it grab the key/values from the context. My code could then call the stored proc with standard ADO code. I could do a similar thing with a webservice name and param, where my custom code would call the service.
Here are my questions:
1) Do I really have to create a separate piece of code to execute the stored proc or web service? I would think something as sophisticated as Quartz would be able to "natively" handle calls to stored procs, web services, maybe execute ftp commands, etc. Am I looking for a no-code solution when I shouldn't be?
2) Assuming I do have to write my own "do the work" code, where do I put that code? Do I compile into a DLL and place it somewhere? How do I tell Quartz where to look for my DLLs? And how do I associate the Job-Name in the config with my class in my DLL? Do I just use the Job-Name setting as the actual name of my class?
Thanks!
1) Yes, you still have to write separate job classes.
2) All your class has to do is implement the IJob interface and the scheduler will pick it up. Read the documentation.
Quartz is a scheduler, it's all it does and it does it well. It does not "natively" handle calls to stored procs, web services, or ftp commands. You have to write the code to do that in your class that implements IJob and is instantiated by Quartz on the schedule you specify.
The best thing to do is to create a separate class library (DLL) that you will reference from your app which creates an instance of Quartz scheduler and provide it with a fully qualified name of the class it needs to instantiate (e.g. MyLibrary.MyNameSpace.MyClass) and it will instantiate that class on the schedule and execute your code found in the overriden Execute() method of your class...
It's that simple...
I am quite new to Java and JUnit.
We have loads of procedures in the database which we want to test, so we were thinking of the possibility of using JUnit to test them if it is feasible.
The database procs/functions which need to be tested along with the parameters, output expected will all be passed through a file.
I am not sure whether this is something where JUnit would help as we are looking at getting the information from the file, run the tests for each proc and then provide an output which specifies how many had failed along with the error messages.
Based on my understanding when I read few posts and internet search, it seems that I need a separate method with #Test to test every database procedure which would mean that we need to create Java methods in advance. But the test would be to help database developers who don't have much knowledge in Java, to just specify the proc to be bested in a file and JUnit framework will take care of the rest.
Is this feasible with JUnit, any advice please?
CodeIgniter comes with a Unit Testing class built in, and I would very much like to use it. However, almost all functions I would want to test interact with the database by adding records, deleting records, etc. How would I, for example, write tests for the 'create user' function without actually creating users every time I run the test?
Upon some further research, it seems I need to be using Mock objects for external services like the database, etc. I haven't been able to find much in the way of docs on how to do that besides this one forum thread:
http://codeigniter.com/forums/viewthread/106737
Is there any actual documentation?
If your database driver allows transactions, use them. Do whatever needs to be tested, then rollback (on success or failure).
I've found that it's hard to run unit tests with controller actions. If you find a good way of doing that, let us know!