The application has several camel contexts, each doing its own thing and as such do not need to communicate with each other. They are in the same module because they share some classes.
Are there any issues one needs to watch out for in the case of multiple contexts in a single osgi module ?
What is the recommendation and best-practice in this case ?
It is fairly subjective. IMHO: The two big things to consider are process control and upgrade impacts. Remember-- during a bundle upgrade all the contexts will stop and then restart.
You still have the ability to do fine grain process control (start, stop, pause, resume) at the Camel Context and Route level without having to rely on bundle start | stop.
If you wanted fine grain upgrade ability, you could put the Java classes in their own bundle, export the packages. Then put Camel Contexts in their own bundles and import the Java classes from the shared bundle. You then have the ability to do individual upgrades of the Camel Contexts w/o having to upgrade all the Contexts at once (and force them all to stop).
One single recommendation: have stateless beans/processors/aggregators.
All the state related information about the processing of your body must live in the Exchange headers/properties.
static final constants are good.
Configuration read only properties are fine too.
Related
For integration purposes, we're using Apache Camel, Karaf with OSGi, so we are creating OSGi bundles. However, what Best Practices exist when it comes to structuring the bundles?
The integrations are fairly straightforward, with an incoming document type (via some protocol like HTTPS, SFTP, JMS), transformation to another document type, and again transportation via some protocol. The basic setup is always the same and follows the VETO Pattern: validate, enrich, transform, operate. Each unique combination of the mentioned protocol/docType defines an integration.
We decouple the connectivity (which includes validation) from the other steps via JMS. When we look at the ETO steps we separate those into their own Java classes and their corresponding XSLT. However, what's the added value of the OSGi framework and how should we divide the integrations between the OSGi bundles?
Take into account performing changes, maintenance and deployments? Consider 2 dozen integration points (unique endpoints) with 50 different integrations running in between, in other words 50 unique transformations between two different docTypes. We can put all code & XSLT's of all 50 integrations in 1 bundle (the other bundle handling connectivity), or 50 bundles with 1 integration each. What are best practices, if any, when it comes to deployment strategy? What to take into account?
You can check out examples from Apache Karaf github repository to see how bundles for OSGi applications are structured there. Christian Schneider has also done talk about OSGi best practices and has some examples in his repository as well.
Generally you want to keep your bundles small with least amount of dependencies as possible. Due to this I would recommend having only one integration per bundle. This makes installing integrations and their dependencies easier and offers some flexibility if you ever decide to split integrations between multiple Karaf instances.
For connectivity stuff your best bet is usually to use/create/publish OSGi services. For example with pax-jdbc-config you can use config files to create new DataSource type services which you can then use to connect to different databases from your integration bundles.
Publishing own custom services is pretty easy with declarative services and could easily be used to share connections to internal systems, blob storages, data access objects, etc while maintaining loose coupling by keeping actual implementations hidden with interfaces. For services the recommended approach is to have separate API and implementation bundle so bundles that use the service can just add dependency to the API bundle.
For deployment you can create your own custom Karaf distribution with bundles pre-installed, deploy bundles using Karaf features or use the hot deploy folder. For the two latter ones you might want to configure Karaf to use external folder for bundle configurations and hot deploy as process of updating Karaf is basically replacing it with new installation.
[Edit]
If we have 50+ transformations and put each in its own bundle, then I would have more than 50 bundles to manage. This is just a single instance. Other customers would have their own instances, again running 50+, 100+ bundles
Would say that the key thing here is to try to simplify and identify repetition in bundles. Often these can be converted to something more generic and re-usable.
With OSGi you can use Declarative services and OSGiDefaultCamelContext to instantiate Camel integration instances per configuration file which can be useful if you have multiple integrations that work pretty much the same but only have minor variations. Don't know if camel has support for this with blueprints.
With many bundles efficient use of Karaf features or OSGi features (R8) can be vital for handling the added complexity. Features make it considerably easier to install and update OSGi applications which consist from multiple bundles, configuration files and other features.
Still there's really no rule on how big is too big for single OSGi bundle. Grouping closely related things in to single bundle can make a lot of sense and help avoid splintering things too much.
I am from a microsoft background where I always used to keep server and client applications in separate projects.
Now I am writing a client-server application with express as back-end and react js as front-end. Since i am totally a newbie to these two tools, I would like to know..
what is the general practice?:
keeping the express(server) code base and react(client) code base as separate projects? or keeping the server and client code bases together in the same project? I could not think of any pros & cons of either of these approaches.
Your valuable recommendations are welcome!.
PS: please do not mark this question as opinionated.. i believe have a valid reason to ask for recommendations.
I would prefer keeping the server and client as separate projects because that way we can easily manage their dependencies, dev dependencies and unit tests files.
Also if in case we need to move to a different framework for front end at later point we can do that without disturbing the server.
In my opinion, it's probably best to have separate projects here. But you made me think a little about the "why" for something that seems obvious at first glance, but maybe is not.
My expectation is that a project should be mostly organized one-to-one on building a single type of target, whether that be a website, a mobile app, a backend service. Projects are usually an expression of all the dependencies needed to build or otherwise output one functioning, standalone software component. Build and testing tools in the software development ecosystem are organized around this convention, as are industry expectations.
Even if you could make the argument that there are advantages to monolithic projects that generate multiple software components, you are going against people's expectations and that creates the need for more learning and communication. So all things being equal, it's better to go with a more popular choice.
Other common disadvantages of monolithic projects:
greater tendency for design to become tightly coupled and brittle
longer build times (if using one "build everything" script)
takes longer to figure out what the heck all this code in the project is!
It's also quite possible to make macro-projects that work with multiple sub-projects, and in a way have the benefits of both approaches. This is basically just some kind of build script that grabs the output of sub-project builds and does something useful with them in a combination, e.g. deploy to a server environment, run automated tests.
Finally, all devs should be equipped with tools that let them hop between discreet projects easily. If there are pains to doing this, it's best to solve them without resorting to a monolothic project structure.
Some examples of practices that help with developing React/Node-based software that relies on multiple projects:
The IDE easily supports editing multiple projects. And not in some cumbersome "one project loaded at a time" way.
Projects are deployed to a repository that can be easily used by npm or yarn to load in software components as dependencies.
Use "npm link" to work with editable local versions of sub-projects all at once. More generally, don't require a full publish and deploy action to have access to sub-projects you are developing along with your main React-based project.
Use automated build systems like Jenkins to handle macro tasks like building projects together, deploying, or running automated tests.
Use versioning scrupulously in package.json. Let each software component have it's own version# and follow the semver convention which indicates when changes may break compatibility.
If you have a single team (developer) working on front and back end software, then set the dependency versions in package.json to always get the latest versions of sub-projects (packages).
If you have separate teams working on front and backend software, you may want to relax the dependency version to be major version#s only with semver range in package.json. (Basically, you want some protection from breaking changes.)
I'm exploring JHipster, the Spring Boot + AngularJS application generator based on Yeoman.
So far, it's been quite interesting and fun. I've been able to get a vanilla webapp up and running without too much trouble.
Now I want to take a step further, moving the entities, repositories, and services away from the original webapp project.
The webapp project generated by JHipster declares spring-boot-parent as its parent project:
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
<version>1.3.1.RELEASE</version>
<relativePath />
</parent>
So I changed it to my own parent project, which looks something like:
<parent>
<groupId>br.net.neuromancer.bigpolis</groupId>
<artifactId>bigpolis-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
Immediately, I was swamped with Maven build exceptions. :(
I then followed the steps described on Spring Boot - parent pom when you already have a parent pom
I could get to the point where the project builds ok, but errors still showed up while running mvn spring:boot. I was just getting too much trouble for little gain, so I decided to rollback the changes.
Instead I declared spring-boot-parent as the parent of my own parent.
Though this strategy seems to work both for building and running, now I have two major questions.
First, what are the implications of declaring spring-boot-parent as the main parent for all my sub-projects ? Not that I dislike the idea so much, since most (if not all) of the sub-projects do depend on Spring Boot anyway.
But are there any side effects to this strategy that I should be aware?
The second question regards JHipster more closely.
What would be the best practices to make certain components available to non-JHipster projects? Entities, repositories, and services seldom belong exclusively to the webapp, and should be shared with other non-frontend components.
Could I move them to a separate project, perhaps losing the refactoring functionality provided by Yeoman?
Or should I declare the JHipster project as a regular dependency, just letting all the Angular stuff gets packed along the WAR? Somehow that doesn't seem too efficient.
Any pointers, thoughts or comments on what I've just described will be really appreciated. Thanks for reading this far... :)
Follow up a few days later...
I'd like to apologize for asking unrelated questions in the same post. Making JHipster modules available elsewhere is a topic which deserves a post of its own.
That said, here is some follow-up about the Maven parenting issue.
I finally decided against making spring-boot-starter-parent the parent of my own parent:
I prefer to avoid the introduction of too many unnecessary dependencies to other modules, which might rightfully not even care about spring-boot-starter at all.
So I rolled back the earlier changes, then ran the Maven dependency:resolve target to get a list of transitive dependencies for the original setup.
I then added this section to my JHipster pom.xml:
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
I had to explicitly declare missing version tags, until the builds within and outside Eclipse were happy.
Only then I ran again mvn dependency:resolution.
The comparison with the previous transitive dependencies list gave me good hints on the causes for errors I was getting on my first attempt.
I patiently fixed each mismatch, declaring variables and dependencies on my POM, until I had exactly the same versions for all dependencies on both setups.
After this manual labor, I was able to get the spring-boot:run target alive again. :)
Contrary to what was stated elsewhere, I did not have to re-configure plugins. But I guess it was only my good luck. I can see why the advice stands.
The downside is that the new POM ended up much larger (and more complex) than before.
https://github.com/javayuga/BigPolis/blob/master/bigpolis-parent/tserd14Browser/pom.xml
I'm pretty sure I can still do more cleanup, but results are enough to let me move on.
I'd like to thank everyone who responded, giving me some food for thought. :)
As for your first question, are all of your creations going to be Spring-Boot projects? That defines whether the parent needs to be Spring-Boot. The parent is there to save you the trouble of specifying dependencies.
Your second question is about unknown side-effects. If you specify the parent, you should look at it and understand it.
For your third question, it sounds like you need to learn how to create your own generators. I'm doing about the same thing as you. I first created a JHipster app to learn on. I've created a bunch now. I have one I delete and re-create, hack, overwrite, etc. called "goof". Every time I want to test something, I mkdir goof, cd goof, and yo jhipster. I have already written a Mule component (REST client) that talks to the API that's described in the Swagger API rendering using the admin login. I'm going to make a stripped down JHipster that's loses all the Angular stuff and only hosts the API. But, it won't be JHipster anymore, it will be my thing. I'm going to learn how to create a Yeoman generator and it will be my creation not JHipster's. Use JHipster because it's there, do your thing (using Spring-Boot parent if needed) but make your own generator for each thing you make. It sound's like you want your own generator to me. JHipster is complex and it's open source. Hack it up and learn from it. But go learn more about Yeoman generators.
Question1: Well it really depends. If you keep as your main parent, spring-boot-parent that means that all your sub-module will inherit from it. So is this OK? Is there any case that a submodule is not spring related at all? You need to think about that cases. If they are all spring related, most probably you will be alright, and I am saying most probably since this particular parent even in spring related projects- might fit 100% where you might need to do something very specific (corner cases). The cleanest solution would be to import this parent as 'BOM' as the previous topic suggests. The reason that does not work is that you need to configure the plugins on your own.
I have a complex appengine service that was written in PHP, now I want to migrate it to Python part by part.
Let's say that my service has 2 parts: /signIn/.... and /data/.... I just want to migrate /signIn/ part first, then /data/ later.
However, since my service is big, so I want to build new /signIn/ part in Python, then use Traffic Splitting to make some A/B Testing on this part.
My problem is that Traffic Splitting can be applied on versions only, so my old and new versions have to be in same module, and same module means that they have to written in same language (I was wrong here, see updated part). But I am migrating from PHP to Python.
What is the best solution for me?
Thanks,
Solution
With Dan Cornilescu's helping, this is what I do:
Split the app into 2 modules: default and old-version.
Dispatch /signIn/ into default module, the rest to old-version module.
Make another version of /signIn/ (default module) in Python
Configure Traffic Splitting to slowly increase requests percent into Python part. This will allow us to test and make sure there is no serious bug happen.
Note:
The /signIn/ part must be default module, since GAE's traffic splitting works at default module only.
I confirmed that we can make 2 versions in different language for a module.
One possible approach is to split your PHP app in modules in a 1st step. It's not a completely wasted effort, most of that will be needed anyways to just allow your app to work in multiple modules, not related to the language change. I suspect this is actually why you can't use A/B testing - mismatch between the modules. Unavoidable.
Once the split in modules is done then you can go on with your 2nd step - switching the language for selected module(s), with A/B testing as you intended.
A more brave approach is to mix the 2 and write the /signin/ module directly in python. On the PHP side you'd just remove the /signin/ portion (part of the earlier mentioned 1st step). Should work pretty well as long as you're careful to only use app language independent means for inter-module communication/operation: request paths, cookies, datastore/memcache keys, etc. A good module split would almost certainly ensure that.
You have testing options other than A/B, like this one: https://stackoverflow.com/a/33760403/4495081.
You can also have the new code/module able to serve the same requests as the old one, side-by-side/simultaneously and using a dispatch.yaml file to finely control which module actually serves which requests. This may allow a very focused migration, potentially offering higher testing confidence.
I'm also not entirely sure you can't actually have 2 versions of the same module in different languages - the versions should be pretty standalone instances, each serving their own requests in their own way using, at the lowest layer, the language-independent GAE infra services. AFAIK nothing stops a complete app re-write and deployment, with the same version or not - I've done that when learning GAE. But I didn't switch languages, it's true. I'd give it a try, but I don't have time to learn a new language right now :)
TL;DR Is there a way to deploy App Engine modules in parallel?
I've built a go application using Google's App Engine SDK for Go. This application defines multiple modules. These modules are self-contained, and do not require any sort of dependency across other modules.
When I attempt to deploy the modules to the Google Cloud, I can't help but notice that the modules are uploaded sequentially. This would be fine if deployment was relatively quick, but each module requires it's own redundant compilation of the Go binary. Hence, on top of the regular upload time, I have to wait for my app to compile [module count] x [compilation time] every time I want to deploy.
The obvious (quick) solution is to deploy in parallel, so I created a simple bash script to deploy each module independently. The problem I immediately encountered with this "solution" was a HTTP 500 response from the App Engine API. The whole umbrella application, spanning across all the modules, seems to "lock" whenever any individual module is updated. This scenario creates a race condition, under which only the first module to trigger a deploy succeeds and the others fail.
I fear that this is a holdover from the legacy languages in App Engine. Since every module uses the same Go binary, it doesn't really necessitate multiple compilations of the same code. Repeated compilation is redundant, and there is no way to circumvent the lock.
One hypothetical solution, which I have only a vague understanding of, is to compile in parallel and deploy in series. I imagine that this approach would involve taking apart the configuration tool and reworking it to execute in the aforementioned manner- though I can't say for sure (yet).
Any help here would be much obliged. Thanks!
You can deploy to another "version" of your App Engine app, then when all modules are deployed, do a very fast version switch?
Versions also allow for traffic splitting if you need/want that kind of thing.