I am thinking about replacing the business layer with BizTalk orchestrations exposed as WCF services as a standard architecture for many of our apps. Essentially ASP.NET and WinForms apps will call these services to retrieve and update data in many of our LOB databases among other things. Some of the services will also be exposed to partners.
As for the data access, I can certainly use the SQL Adapter, but I think it's not the cleanest way to do it, and the fact that it's tightly coupled to SQL Server also makes it a bad idea for me. I would like to use Entity Framework based custom DAL's or perhaps generated from tools like SubSonic, etc.
Is this a good idea? From my Google searches, I can't find many people doing this kind of thing or any comments on how it might have worked out for them.
What's your take on this? Any ideas on where to cache data, concurrency issues, etc.?
You will probably have an easier time using straight WCF and something like NHibernate or EF. These services consuming and returning DTO's and not raw entities. If you do have heavy business logic or mapping that needs done, BizTalk can sit in front or with ESB even expose mapping services to your data service on the side.
Also check out the new WebApi stuff. http://wcf.codeplex.com/
And BizTalk is not great at doing low latency ... you will get some overhead on all the service calls.
If you have to do a lot of service aggregation there MIGHT be a case for doing it this way ... but beware the latency and overhead you get from an integration platform that is aimed at giving you all kind of services around messaging integrity etc.
Related
We are currently looking into replacing one of our apps with possibly an ESB or some similar tool and was looking for some insights into how best to approach this.
We currently have a stand alone service that consumes/interact with different external services and data sources, some delivered through SOAP Web Services and others we just use a DB connection. This service is exposed through SOAP and we have other apps that consume this service but are very tightly coupled to it, now we also have other apps that need to consume some of the external services and would like to replace this all together with an ESB or some sort of SOA platform.
What would be the best way to replace this 'external' services integration layer with an ESB? We were thinking of having a 'global' contract/API in which all of the services we consume are exposed as one single contract where all the possible operations and data structures that we use are exposed under one single namespace, would this be the best way of approaching this? and if so are there any tools that could help us automate this process or do we basically have to handcraft this contract/API?. This would also mean that for any changes to the underlying services/API's we will have to update this new API as well.
If not then the other option I see is to basically use the 'ESB' as a 'proxy' layer in which all of our sources are exposed as they are, so we would end up with several different 'contracts' / API endpoints, but I don't really see the value in this.
Also given the above what would be the best tool for the job? is a full blown ESB an overkill or are we much better rolling our own using something like Apache Camel or Spring Integration?.
A few more details:
We are currently integrating over 5 different external services with more to come in the future.
Only a couple of apps consuming our current app at the moment but several other apps/systems in the future will need to consume some of these external services.
We are currently using a single method of communication (SOAP) between these services but some apps might use pub/sub messaging in the future, although SOAP will still be the main protocol used.
I am new to ESB integration so I apologize in advance if I'm misunderstanding a lot of these technologies and the problems they are meant to solve.
Any help/tips/pointers will be greatly appreciated.
Thanks.
You need to put in some design thoughts of what you want to achieve over time.
There are multiple benefits and potential pitfalls with an ESB introduction.
Here are some typical benefits/use cases
When your applications are hard to change or have very different release cycles - then it's convenient to have an ESB in the middle that can adopt the changes quickly. This is very much the case when your organization buys a lot of COTS products and cloud services that might come with an update the next day that breaks the current API.
When you need to adapt data from one master data system to several other systems and they might not support the same interfaces, i.e. CRM system might want data imported via web services as soon as it's available, ERP want data through db/staging tables and production system wants data every weekend in a flat file delivered via FTP. To keep the master data system clean and easy to maintain, just implement one single integration service in the master data system, and adapt this interface to the various other applications within the ESB plattform instead.
Aggregation or splitting of data from various sources to protect your sensitive systems might be a use case. Say that you have an old system that can take a small updates of information at a time and it's not worth to upgrade this system - then an integration solution that can do aggreggation or splitting or throttling can be a good solution.
Other benefits and use cases include the ability to track and wire tap every message passing between systems - which can even be used together with business intelligence tools to gather KPI:s.
A conceptual ESB can also introduce a canonical message format that is used for all services that needs to communicate. If a lot of applications share the same data with several other applications (not only point to point) - then the benefits of a canonical message format can outweight the cost (which is/can be high). An ESB server might be useful to deal with canonical data as it is usually very good at mapping from one format to another.
However, introducing an ESB without a plan what benefits you are trying to achieve is not really a good thing, since it introduces overhead - you need another server to keep alive, you need perhaps another team to understand all data flows. You need particular knowledge with your integration product. Finally, you need to be able to have some governance around it so that your ESB initiative does not drift away from the goals/benefits you have foreseen.
You should choose some technology that you are comfortable with - or think you can be comfortable with. Apache Camel is indeed very powerful and my favorite integration engine - but it's not an ESB as it does not come with a runtime that you can use to deploy/manage/monitor your integration services with. You can use it together with most Java EE application servers or even better - Apache ServiceMix (= Karaf+Camel+ActiveMQ+CXF) which is built for this task.
The same goes with spring integration - you need to run it somewhere, app servers or what not.
There is a large set of different products, both open source and commercial that does these things.
I am doing something I consider to be pretty normal (although I personally haven't had to do it before), and I have assumed there'd be a 'no-brainer' way forward, but Im yet to find it - which is really frustrating.
I will be creating a WPF application, which is a data-oriented business application. My data will come from a remote IIS server (that I control) that has a standard SQL server 2008 database, so Web services/WCF seem to be the way forward. The remote service needs to be secure (reasonably) via a user (of the WPF client) username/password login.
I dont want to use 3rd party ORM products, but I expect the data layer (between the service and the database) to be able to cope with very simple ORM type functionality (I really dont want to hand-craft a data retrieval and persistence layer). Im not worried about concurrency very much as this will be a fairly simple app.
My options seem to be one of the following:
ADO.NET Entity Framework over WCF
Linq2Sql over WCF
WCF Data Services
On further investigation, none of the above seem to be the 'no brainer' Im after
1) ADO.NET entity Framework - Ive had a play with this and getting all sorts of issues serializing objects over WCF. Even when I try to generate POCO entities and use them, Im having to decorate service contracts with custom attributes just to get it to not error all the time, and I seem to have to hand-crank anything more than a flat object graph. It seems to me that EF simply isn't designed to be exposed via a service.
2) Linq2Sql - This doesn't seem much better than EF. I seem to have to hand-crank too much stuff. Ive tried the designer and SQLMetal but nothing seems to 'just work' - it all needs fiddling with.
3) WCF Data Services - this seems like a good option on the face of it, but essentially it seems like I'm just exposing my SQL database tables 'in the raw' over the service layer. Im not an expert in this technology by any means but it seems like a potentially dangerous approach, and on top of that it doesnt seem to support any kind of access security as standard (you have to hack it to require authentication it seems).
As I said, this scenario feels like it should have a no-brainer solution, but Im still scratching my head. Ive done lots of things with .NET technologies, but to be honest this area represents a bit of a hole in my understanding, so I apologize if any of my comments or assumptions are naive.
Of course, it may well be that the 'hacky' long-way-round on EF or Linq2SQL may be all I can do, in which case I can roll up my sleeves, and accept the fact that I haven't missed a more elegant solution.
Any help/advice will be much appreciated.
This is a tad subjective, but i'll offer my opinion.
First of all, forget L2SQL - it's basically obsolete and doesn't have the full POCO support of EF4 (it can be done, but needs XML tinkering, or SQLMetal generation), which means serializaing your entities will be a left-to-right entity cloning nightmare.
I would go with ADO.NET Entity Framework over WCF, Entity Framework 4.0 specifically. You will have a wealth of flexibility in your model (including the ability to apply OO principles such as inheritance).
Use Self-Tracking-Entities. Yes, you have to decorate service contracts - this is by design, and there are many reasons for this.
You could always use DTO's, as opposed to serializing the actual EF entities.
OData is really good as well in it's flexibility and simplicity. But if your only consuming your model via a single client application, a specialized service layer (WCF) is a better approach IMO.
3) WCF Data Services - this seems like
a good option on the face of it, but
essentially it seems like I'm just
exposing my SQL database tables 'in
the raw' over the service layer.
That might be a first impression - but it's fundamentally wrong. What you're exposing over the web is a model - and you have full control over what gets into that model, and how consumers of your WCF Data Services might be able to see and/or even update entities in that model.
That's where Entity Framework comes in and shines (and where Linq-to-SQL miserably fails): you can grab your database (or at least parts of it) into an Entity Data Model, and then modify it. You can tweak your entity names to be totally different from your table names, you can add computed attributes, you can remove certain attributes and much more.
If you're talking about a fairly simple app, that's definitely the way I'd go:
grab your database and turn it into an Entity Data Model using EF
expose that EDM over WCF Data Services and define what can be seen read-only, and what might even be updated over the wire
VS2010: What is the best way to access a remote database over the internet from a WinForms application? By that I don't mean talking to the database (SQL Server) directly, but rather through a service. Best if I could use http as a protocol to avoid firewall issues.
Would recommend that you look at WCF
If you have control over both ends of the connection, you then can change protocol etc through configuration, but yes, http:80 is usually the easiest way to adhere to firewall policies.
If you wish to keep this simple and your needs are limited you could always use a typed dataset that you pass over WCF (or .Net Remoting). However you will quickly hit problems if your app gets more complex or you have lots of data.
SqlServer will exposed stored procs over soap, this may work well if your applications uses a few stored procs that are not too chatty.
Exposing your data with WCF Data Services is an option if you want simple CRAD operations.
Otherwise you will have to define all the methods you need on WCF interfaces.
If you are not tied to WinForm, look at WCF RIA Services with Silverlight as this will do a lot of the hard work for you with most CRAD type applications. But you will need to learn Silverlight.
The problem is there is no simple and
easy solution to “presenting the
internet is not in the way” and moving
to a 3 tier application will always
make your code more complex. A 3 tier
architecture can be great for more
complex and demanding applications,
but is often over kill.
Also consider writing a 2 tier WinForms application and using Terminal Services to expose it to your clients over the internet. This is often the cheapest solution and tends to give the least pain with system admin. However a lot of programmers think it is “cheating” for some reason.
I really like the idea of "WCF Data Services" but how does it work in a real life scenario? WCF Data Services provide just a nice way for the client to CRUD the data. However it's very limited in what you can pass and get back. So one ends up having all the business logic written on a client side. It's probably ok for small applications who just need a database back-end. You don't want that in serious enterprise applications, your client side will grow too large and if your business logic is some kind of know-how it can be easily disassembled.
Don't be misled that SOAP is for enterprise and REST is for sucky little side web apps. Many people have wasted a lot of time on SOAP frameworks, me included and the trouble that these frameworks cause for inter-enterprise communication would count into the Billions of dollars.
REST provides an opportunity to only care about the data being passed too and from services and the semantics used to operate against services, the rest (excuse the pun) is handled by transport level mechanisms. Do you want encrypted data channels? Well HTTPs is there for that. Do you need authentication? there are plenty of frameworks on HTTP that support this already rather than use complex WS-* protocols. Do you want Reliable Messaging? you can engineer it quite simply using message queue software - I have only ever seen one SOAP framework handle this well and it wasn't very interoperable at that point.
Whilst I am not discounting SOAP as enterprise-grade, all I am saying it don't discount REST based services as an excellent way for your enterprise modules to communicate.
I personally have integrated multi-million dollar systems using REST and SOAP and currently prefer REST based services for their ease of development and 3rd party integration, understanding, ease of documentation and their ability to rapidly deploy services across businesses.
I can understand your confusion given the naming... WCF Data Services are REST based which are notoriously poor for enterprise environments. Howvever, you can have normal SOAP based WCF services which work fine for the enterprise.
There are some rumors floating around that the team at my company will soon be using web services for all future application development. The architecture is supposed to be something like this:
Application --> Web Service --> Database
The stated reasoning behind it is security. This sounds like a huge waste of time for little if any benefit. My question is, in what ways does a web service make your data more secure than a database? I would think that if an attacker wanted to get all your data and had already gotten onto the app server, it would be fairly trivial to figure out how the application is getting it's data.
Please keep in mind that these web services would be purely for data, and would have little if any business/validation logic, and would also be outside the application developers control (at least that's the way it's worked with all previous applications that have used web services).
If it's true that there will be no business logic or validation on the web services, then there is only a limited security benefit to adding the additional layer of abstraction. I say limited because the interface between your application and the database is still more limited than if they were directly talking to each other.
If you add validation and business logic to the equation, there is a significant security benefit, as anyone who has access to the application account can only do the database what the application is able to do. Additionally, this is a better design because it reduces coupling between your application and implementation details of how the data is stored in the database. If you wanted to change the database schema, you only need to update the web services, and not entire applications.
One important thing about Web Services is interoperability so that different applications from different platforms later can utilize the services and data. Your company will benefit a lot by doing so. And you are right about the security, it is definitely one of the good reasons to use web service rather than expose a public endpoint of the database, it is dangerous!
Web Services enable the accessibility of your data, For example, your data can be accessed within browser by javascript. There is no way to access the database on the server directly within Javascript.
All in all, go for it, that is the right approach.
the security argument is questionable; authenticating to a web service is no different than authenticating to the database
there are legitimate reasons for moving db operations to web services and SOA in general, but security isn't one of them
If you use a webservice hopefully you will also be using some kind of queue when sending the data to the database. If you are using a webservice and queue combo then the security come into place with less chance of lost data. If you do not have a webservice and queue combo if you send data to the database and it never gets there you have no were for it to go it just disappears.
You are correct though if someone wants to break into your system a webservice isnt going to help if anything it might make it worse if you make the webservice public and they find the name of your webservice because then they can just query your DB using the webservice and any security features on your servers will just think it is you applications getting the information.