I am trying to find a true open source alternative to our SAP PI ESB that is going to be discontinued in our company.
We are using it for integrating several systems. We have around 100 interface SOAP-SOAP (both synchronous and asynchronous), file (FTP) => SOAP and SOAP => file.
I have read many blogs and watched quite some videos about open source ESBs. But I think I still miss the whole picture. From what what i understood I can you Apache Camel as a core integration engine with appropriate adapters for delivering and transforming the messages, then JBoss Fuse (or perhaps OpenShift) to orchestrate the individual Camel "applications".
I am still missing last piece of the puzzele. I was not able to find any ready to use solution for message monitoring. If I search for monitoring i always find monitoring of the system (procesor load, memory, number of processes, number of messages...).
But I am looking for a message monitoring where I can filter the messages by date/time range, interface name, sender/receiver, status (completed, error, in queue...) and look into the message payload (XML or file) before and after the transformation. Fulltext search by message content and email notifications on errors would be also great.
Can anyone give me hint what to look for?
Related
Hy there,
I'm currently writing a camel client, that integrates into our companies file transfer tool.
As our next feature, we want to be able to scan files, that are send via our software with some antivirus software.
For different reasons we decided to do this with our camel client.
After resarching a bit I could not find any article or project, that talks about camel and scanning files for viruses.
Does anyone have any experience with this topic or is there something planed for camel to have an component for this in the future?
I think as the first step having a clamav client as a component would be awesome.
Any thoughts?
Greets
Chris
After reading many Solr books and article all over on the net, now I have an idea of the power of this server.
But... how to integrate it in a real application? For example: a web site written in PHP, etc.
Right now, I understand that Solr produces XML, JSON etc results... so to integrate this in a web application, the "simple" work is to convert this information for render in a page or there are other technique to avoid this?
I'm my case, I have to develop a search engine to scan many documents and find result.
My idea was:
Use Solr to build an index and search documents
Use a web application to show the result
Looking on the net I haven't find anything that explains how to integrate Solr in a real application, all the reading are about "How to use Solr... with Solr..." Anything about a real integration.
Does someone have some useful resource how to integrate Solr in a real application, with some clean examples?
Edit: It looks like Apache maintains their own list of recommended
client APIs, and their recommended tool for PHP is Google's
library (though they refer to it as SolPHP). Given this, I imagine that this is the best place
to start.
A Solr library for the programming language you're using could save you some of the trouble in implementing the integration. For instance, if your site is written in PHP, you could try Google's Solr library for PHP.
I have done most of my Solr work in Java, so I have used SolrJ quite a bit. This is a well supported tool because it comes from Apache in parallel with the Solr product itself.
If you are doing work in any other languages, you are likely to find libraries available for them. The amount of time they save you may vary according to the quality of the library itself.
When I was using Solr in my project, only my application server (that is Tomcat) was communicating with Solr server. I wrote a class, which executes GET requests to Solr server based on input provided by end user. When Solr returns XML/JSON back to an application server you may parse it and process as every other bussiness data (render an *.html). So, summing up, Web Browser never communicates directly with Solr, all goes through an application server:
WebBrowser -> GET to application server -> GET to Solr server
show *.html <- parse XML/JSON, render *.html <- return XML/JSON
I am studying Apache Camel.
Could some one please explain the difference between a processor, component and endpoint with regard to Apache Camel.
A component allows you to talk with other systems. It allows you to send or receive messages and encapsulates the protocol to deal with another system. For e.g. jms-component allows to talk with JMS brokers.
An endpoint is nothing but the channel by which you send or receive a message through component e.g. "jms:queue:order" this defines a jms endpoint which is a queue from where your (jms) component will either consume or publish a message.
While a processor is piece of code which goes in between routes. There you write code to manipulate (transform/enrich/extract etc.) the message or have some integration logic.
For more details refer to camel's documentation
All whom are new to Apache Camel I suggest to read this article which explains really well what Camel is, and has an example to go along.
http://java.dzone.com/articles/open-source-integration-apache
Another great piece is chatper 1 of the Camel in Action book, which can be freely downloaded from here: http://www.manning.com/ibsen/Camel_ch01_update.pdf
Disclaimer: I am co-author of that book.
And there is this old by good tutorial that still applies today: http://camel.apache.org/tutorial-example-reportincident.html
And you can find more tutorials / examples on the Camel website
http://camel.apache.org/tutorials.html
http://camel.apache.org/examples.html
And there is also some links to 3rd party blogs/articles/videos about Camel, where you can find some great information:
http://camel.apache.org/articles.html
We have been using the Apache ActiveMQ and Camel products for a while now but want to look at a good base ESB. I've been reading the Redhat site about Fuse but have been unable to find a good summary of the significant differences between Fuse and Apache for coders.
From a designer's/developer's point of view what are the significant differences between Fuse and the Apache Camel and ActiveMQ that we have been using? I get the lovely overview stuff, FuseIDE and the ESB management tools. But I really just want to know of the differences at the code level, i.e. does it introduce more useful Camel endpoints? are there additional libraries of genuinely useful things that will make my life as a designer/coder easier? are there any pitfalls to look out for?
I just need a few pointers to help me in my search, not a tome. Or better still a quick link to a document that goes over all this (ever hopeful :o) !) I have a short time to form a view to go forward on or the opportunity will pass me by.
Thank you.
SK
At the code level there is "no" difference. The process is that we develop on the Apache projects, and sync the code changes to Red Hat / Fuse git repos. There we cherry pick the changes we want to go into our branches, to keep the product stable. As well backport fixes to older branches if our customers need that / etc (eg you can influence that)
The products from Red Hat is also supported on a much longer timespan than the community support from Apache. There is a guranteed lifetime which you can find here: https://access.redhat.com/support/policy/updates/jboss_notes/
There is only a few additional Camel components from Fuse / JBoss Fuse products, which is part of the open source project Fuse Fabric (http://fuse.fusesource.org/fabric/) which is part of the JBoss Fuse products. Fuse Fabric is in the process of being donated to Apache ServiceMix, so it can benefit that community as well, allowing ServiceMix to bundle Fabric out of the box as well. Fabric has a few Camel components that allows sending messages to a any Camel endpoint that load balances automatic in a clustered environment / cloud environment. And there is another Camel component for selecting a master, and only run the route on the master node, and if the master dies, then another node takes over.
I also think that this move is a testimony of the open source
willingness the Fuse team has and continues to have. We do as much as possible
in the opening. For example the new project - hawtio (http://hawt.io/)
is also fully open source, ASL license, github project, anyone can contribute/fork etc.
And the JBoss Fuse product allows to patch itself in production. So if you need a hotfix asap, we can provide a fix as a .zip file which can be patched using a built-in patch tool in the product. This isn't possible from Apache.
A few links for further material (from our old site and the jboss community site)
http://fusesource.com/enterprise-support/support-offerings/
http://fusesource.com/community/apache-committers-and-fuse/
http://www.jboss.org/products/fuse
http://www.davsclaus.com/2013/04/apache-camel-web-dashboard-with-hawtio.html
Disclosure: I work for Fusesource / Red Hat.
On a code level, the difference is very small, if any at all.
What you get from the commersial RedHat package is support, a package that has been tested and operational benefits (that you mention).
It's all about what happends after the code is made - when you put your things to production and the coder is not still around to handle incidents.
Apache ActiveMQ and Camel are open source projects. Redhat fuse bundles them and possibly many other components into one package and so it can be used as one ESB package. I see the biggest difference as the support that you can get. You can get support for something that your organization has not produced. And the tools that comes with the package does help during development and maintenance in my view.
I am creating a Silverlight 2 user interface to a remote instrument. There are two concurrent users at different sites interacting with the instrument (operator at the instrument and remote scientist) and any number of observer users not interacting with it, just watching. However, whenever one of the two active users changes something these changes must be immediately reflected in the UIs of all users, e.g. panning or zooming an image or annotating or selecting part of an image, adding items to a collection displayed in a listbox. Within the client I use observable collections which easily reflect changes made by that user, but it is harder seeing changes made by another user. I can poll for changes from each client but something like push notifications would be better. I have extensively Googled for examples but not found anything which is quite what I need. There are all sorts of security issues with Silverlight interacting with WCF services which mean many potential examples just don't work. I have essentially run out of time on this project and need help fast. Does anyone have any suggestion of a suitable simple example which illustrates how to do this? I am an experienced developer but have had to teach myself Silverlight and WCF services and there is noone in my area who knows anything about these. Even tho' I have done a fair amount of ASP.NET work I am not a web/Javascript guru. Thanks.
Push notification is supported in Silverlight 2 using the new WCF PollingDuplexHttpBinding support. There are two assemblies installed with the Silverlight SDK (one for Silverlight app one for WCF server).
I have a few blog posts and a full sample application that demonstrate how to 'push' Stock updates from a Console Application server that self-hosts a WCF service to connected clients. It also shows how each client can add notes against a Stock and have those notes synchronized (pushed from server) to all other connected clients.
The latest version of the sample (Part 4) shows how to synchronize pushed updates between both Silverlight and WPF clients using two server endpoints as follows:
using System;
using System.ServiceModel;
using System.ServiceModel.Description;
namespace StockServer
{
public class StockServiceHost : ServiceHost
{
public StockServiceHost(object singletonInstance, params Uri[] baseAddresses)
: base(singletonInstance, baseAddresses)
{
}
public StockServiceHost(Type serviceType, params Uri[] baseAddresses)
: base(serviceType, baseAddresses)
{
}
protected override void InitializeRuntime()
{
this.AddServiceEndpoint(
typeof(IPolicyProvider),
new WebHttpBinding(),
new Uri("http://localhost:10201/")).Behaviors.Add(new WebHttpBehavior());
this.AddServiceEndpoint(
typeof(IStockService),
new PollingDuplexHttpBinding(),
new Uri("http://localhost:10201/SilverlightStockService"));
this.AddServiceEndpoint(
typeof(IStockService),
new WSDualHttpBinding(WSDualHttpSecurityMode.None),
new Uri("http://localhost:10201/WpfStockService"));
base.InitializeRuntime();
}
}
}
WPF clients connect to the WSDualHttpBinding endpoint and Silverlight clients connect to the PollingDuplexHttpBinding endpoint of the same WCF service. The app also shows how to handle the Silverlight client access policy requirements.
Clients (Silverlight or WPF) can add notes against a Stock in their UI and these notes propagate back to the server to be pushed to all other clients. This demonstrates communication in either direction and hopefully performs all of the necessary communication required for your app.
You can see a screenshot of the demo application running here.
Not that am pushing Flex in fan boy fashion, but matter of factly this is the kind of architecture we build into all our Flex-based applications routinely. Here is what we do on Flex - no doubt it could be suitably translated to Silverlight:
We take three ingredients and integrate them together to accomplish this capability:
Comet pattern (an HTTP compatible way to do server push notifications - look on Wikipedia for more info)
JMS messaging topics (publish/subscriber queues)
The Adobe BlazeDS servlet
The latter item implements the Comet pattern, supports AMF object marshaling (Adobe's binary serialization format for ActionScript3 objects), and bridges to a JMS queue or topic. When bridging to a topic, then multiple Flex clients running in a browser can be proxied in as subscribers to a JMS topic. So if any client publishes a message (or the server-side code publishes into the topic), all client subscribers will have the message pushed to them via BlazeDS and the Comet Pattern implementation.
Effectively you need to locate or write a component that accomplishes what BlazeDS does. You might also need to implement some client code that interacts with the Comet pattern of this server-side component.
Does WCF support the Comet Pattern and bi-directional messaging? Especially where complies to HTTP and port 80 or port 443 for SSL. Looks like you've already looked into that and not found anything for bi-directional messaging. So you may need to roll your sleeves up and do some coding.
Some things to note about doing server push to a web app:
BlazeDS supports two primary modes of implementing the Comet pattern (there's actually a 3rd polling option but am ignoring it):
long-polling
HTTP streaming
The long-polling one you should find to be more universally supportable to most web browsers. So you might streamline to just supporting that initially. Or you could spend the time to make your client code try HTTP streaming first and switch to long-polling if necessary.
As to a message broker that can provide publish/suscribe capatibility, you might consider using ActiveMQ JMS. It is open source and free with active community support (you can buy support too). Plus you can use NMS to integrate as a .NET client.
Having a message broker sitting in the middle-tier is actually important because it will be a place for messages to be placed safely. If your clients are doing long-polling, you wouldn't want them to miss any new message during an interval when they're not actually connected.
Another thing to consider in high traffic volume scenarios (hundreds or thousands of clients, such as a web site on the Internet), you need to have an approach to the Comet Pattern that is scalable.
In the Flex/Java world, the BlazeDS servlet (which is open source) has been modified to work with asynchronous model. In Java a socket listener can be built to use NIO channels and Java Concurrency Executor thread pools. The Tomcat web server has a NIO listener and support for asynchronous Servlet 3.0 events. BlazeDS in particular has been modified, though, to work with the Jetty web server. The bottom line is that the scalability of this asynchronous approach means a single physical web server can be enhanced to support up to around 20,000 concurrent Comet-style client connections.
It's been a while since I've done serious .NET programming but used to the io capabilities were much like Java 1.1 except with an asynchronous result handler capability. This, though, is not the same thing as creating asynchronous socket listeners via Java NIO channels. A NIO channel implementation can support hundreds to thousands of socket connections with a relatively small thread pool. But C# and .NET has gone through two or three significant revs - perhaps there have been new io capabilities added that are comparable to NIO channels.
I just wanted to clarify that the PollingDuplexHttpBinding doesn't implement 'true' push notifications, as reveals its name (polling). From the msdn documentation:
When configured with this binding, the Silverlight client periodically polls the service on the network layer, and checks for any new messages that the service wants to send on the callback channel. The service queues all messages sent on the client callback channel and delivers them to the client when the client polls the service.
However it is more efficient than the traditional way of polling a web service, since after each poll, the server will keep the channel open for a certain time (say 1 minute), and if a message arrives in that time it will directly 'push' the message to the client. The client has to repeatedly renew its connection, it so to say polls the service.
If you want to implement real push notifications with silverlight I believe you need to work with sockets, and I recommend reading some of Dan Wahlin's blog posts on the subject.
Alternatively,
if you want a native silverlight API with no proxies, bridges or webservers involved you could use Nirvana from my-Channels as your messaging middleware. Check out Nirvana from my-Channels and their showcase site. (sorry i am a new user and cant submit links):
Alex
EDIT: it's actually working fine. I got badly bitten by the "hidden variable" in a closure :(
I used the PollingDuplex for SL2 and I think that it's not ready for production yet.
My main issue is the fact that it doesn't discriminate on the clients on the same machine. If I run 2 clients then one of them won't be able to poll the server anymore and will die of timeout. There is a SessionId that is different for the 2 clients but it's just ignored on the client side.
Likewise, if I kill a client and then create a new one afterwards then the new client will get the push updates from the previous client for a while.
Did anyone encounter the same issues or are they fixed in SL3?
Actually I ran some more demo codes and realised that for some reason you have to specify the InstanceContextMode and InstanceMode so that the service is session based and not a singleton (as far as I can tell). There are clear performance issues in the simple demo code that I pulled.
It is quite unfortunate that this behaviour wasn't documented.
My organization found the Silverlight 2.0/WCF push implementation to be a little "not ready for prime time", at least for what we were planning to use it for.
We ended up going with XMPP/Jabber, because it is a more well formed beast, and you can implement it fairly easily in Silverlight, by just getting some resources off the internet.
I do believe that Silverlight 3.0 will implement a newer/more well formed push implementation, from what I can tell from publicly available information.
The PollingDuplexHttpBinding is probably the most elegant way to do it.
One possilby less involved alternative is to use a TCP socket from your Silverlight client. Whenever one of the Silverlight clients needs to have an update pushed you can send it a TCP message which contains the name of the WCF service it needs to call or some other light weight piece of information.
I use this approach for an application and it works well.
One much simpler and more powerful solution at the site http://www.udaparts.com/document/Tutorial/slpush.htm