Synchronizing data between server and client - sql-server

When it comes to mobile applications a lot of the time the client is going to be offline.
Right now when the user submits data on the client it is saved locally and then when they press the sync button everything is pushed and pulled between the client and the server.
What would be the best practices when it comes to synchronization of data?
Server side I have a very large MS SQL Server and client side I have sqlite at my hands.

You might find this (This is Microsoft Application Architecture Guide v2.0, Oct 2009) useful. I am facing the same problem here at work, and I did find these guidelines useful early in the process of thinking.
To share my experience, I think the major decision comes from the problem you are trying to solve. If You're facing a single, fixed, mature business domain (i.e, rarely changing your db schema and you face just one or two clients) you can build your own data synchronization framework. This is far better than using commercial or non-commercial frameworks in terms of ease-of-use and adjustment + performance. + Store-and-forward is a very good feature that can overcome your connectivity issues.
However, If you are facing a changing (not necessarily expanding) business domain, or maybe different clients with different requirements you might want to use a mature framework (like Microsoft Synch Framework) to do some parts of the work for you and you can focus on the adjustments needed to your requirements.
I hope this isn't too general.

Related

Best approach to database and UI development?

This question covers a number of areas, so may ultimately have to broken up into constituent parts. However, I don't want to do this, at least until I have some general idea of what direction I should be taking!
I want to create an online booking system under the .Net Framework. The GUI would have two separate vital areas. 1. would be the typical form GUI for setting up a user account. 2. would be a Calendar (much in the same vein as Google Calendar - if not actually Google Calendar) which would show users available times that users can book, and hopefully allow them to actually book by simply clicking on an available time as displayed on the calendar.
The available times that would be displayed on the calendar would depend on two things: the type of user the user is (would be specified in the signup form) and also (obviously) whether or not other users have already taken available slots.
One big question I have is whether it is possible to use Microsoft Access (any version) for this task. My preference is due to familiarity, and the fact that most other technologies being used in this development are Microsoft based. I know it is possible to migrate Access DBs with SQL Server - but would this include the vital front end (i.e. the form)?
Finally, The strategy for approaching the calendar aspect of the front-end is doing my head in a little! While it is certainly possible to integrate SQL Server and Google (docs/calendar) I feel that attempting to shoehorn Google Calendar into such a complex relational database would be ultimately infeasible - but I am struggling to think of an alternative without having to do my own development (which, in this case, I am seriously attempting to avoid!)
Any guidance on the above would be greatly appreciated!
My $.02: as #Luxspes said, Sql Server has no "forms" or "reports", but the SQL Server engine is better suited to web-based applications than Access's "Jet" engine. You could, however, develop your application in Access and then "upsize" your database, which tells Access "hey, those forms and reports are great, but now I want you to put the back-end database into SQL Server Express (free) and keep the front-end in Access." You can then publish your forms and reports to the web using Access's own tools, getting them "for free" while getting the better database engine on the back.
Be aware that doing this is not generally acceptable in the commercial world. An Access application looks like an Access application and most of the world will recognize it, and most don't like it: it will seem like you're cutting corners unless you can make the forms look really, really good. For in-house use it's fine because it has a fast turn-around time so the ROI is great. If you're making a commercial site, I'd take a look at the Microsoft MVC 3 framework: it will take a while to get your noodle around it, but once you're up to speed you can develop sites pretty quickly and robustly. If "robustly" is a word.
Is it possible to use Microsoft Access (any version) for this task, the task being using Access as the database for a online booking system under the .Net Framework: Yes. (But I would prefer to use MSSQL Express Edition)
I know it is possible to migrate Access DBs with SQL Server - but would this include the vital front end (i.e. the form)?: No, SQL Server has not concept of "forms". If you already have them in Access you would have to re-create them in, for example, ASP.NET. If you are looking for the easiest fastest way to do it, I would recommend you LightSwitch ( http://www.microsoft.com/visualstudio/en-us/lightswitch )
Finally, The strategy for approaching the calendar aspect of the front-end is doing my head in a little!: If you do no want to code it, you can find it in Google, just write ".NET Schedule control"

Building a REAL database application using Datasnap

I have built an extensive 2-tier application in D2010, using ADO and devexpress. I want to upgrade this to using Datasnap mainly to provide HTTPS communication instead of just TCP/IP to the vulnerable SQL server. I have followed all the Datasnap tutorials I could find. I have Cary Jensen's Delphi In Depth: ClientDatasets. All good and well, but the examples are pretty useless because in a REAL database application, grids are populated from joining multiple tables together and almost never from a single table. This obviates the "autoresolve" capability of clientdatasets right off the bat. Even the proposed beforeupdateevent handlers won't work in a datasnap application because the DB is only accessible to the datasnap server. So it seems to me I have to create a method on the datasnap server for EACH insert/update I am going to need, then expose those methods to the client and call them from the client as required to request the datasnap server to perform the required update/inserts. This seems like a lot of work!
Is there an easier way to implement https comms to a SQL Server?
Oh in case you're wondering, the application is already pseudo 3-tier in that grids are wired to TdxMemData, and never directly to TADOQueries. I handle all insert/updates myself in the same way that I would have needed to if I had used TClientdatasets.
If you think your database is vulnerable think twice about using D2010 Datasnap. It is very, very vulnerable. Don't be fooled by HTTPS, there are still lot of pieces missing to fully protect the channel. For example once you use Datasnap, SQL server Windows integrated authentication (kerberos based...), is gone.
For a full explanation see: Why Datasnap 2010 is a toy library. It's of course my personal opinion, but is is based on my experience using Midas/Datasnap since Delphi 3, and my current work about IT security.
Anyway you're wrong about insert/updates/deletes. You have to use providers' events to control them on the datasnao server side. It's a bit more complex than handling them in a two-tier application, but you don't need ad-hoc methods for each operation.
[2016 Update: DataSnap in 2016 is even more woefully behind in terms of security and features now than it was when this question was written. I do not recommend its use in any new designs at all, ever.]
DataSnap is a solution to the problem of building multi-tier (Three or more) applications. Directly connecting to SQL over the internet from a thick client that contains all the business logic in the client has many well-understood problems, including the fact that business logic changes then require that you update ALL your clients at once. A middle tier improvement (business logic change) that is inside your data-snap (or other) middle tier logic, is not distributed to each client. The clients are thinner, and contain less of the business logic. Secondly, a well designed data-snap "API" that you build yourself only exposes you to the risks that you create yourself, rather than exposing you to the entire set of MS SQL vulnerabilities.
Frankly, losing Kerberos authentication from your thick client, is not a reason to abandon the idea of a middle tier. I don't understand ldsandon's point at all here. Is he advocating a two-tier application architecture that connects to internet or LAN clients, and that contains all the business logic, as "more secure" than a multi-tier application?
The implicit question suggested by your title is unanswerable, and undefined. What does "real" mean? Many industries deploy two-tier thick clients inside their own corporate LANs. Many have found it beneficial to use a middle tier inside their own LAN, and many have found that external applications that run over the internet should definitely NOT be surfacing the SQL connectivity to thick clients, and so they provide some kind of "web method" (SOAP, REST+JSON, etc) architecture. It has been carefully pointed out that Data-Snap is not a Purely "RESTful" architecture, but it does use JSON, and is in many ways REST-ful in design, although not fully.
If you don't understand the problem that DataSnap was created to solve, it is easy to think DataSnap is worthless, or (alternatively, and equally wrong) some kind of silver bullet. It exists for a particular purpose, one that many people find useful for their development needs. If you intend to take on the work of making a middle tier, DataSnap makes it easier than to do it 100% as a "roll your own middle tier", but it is more work than not having a middle tier.

Access database sharing strategies

What are the strategies you employ to let multiple people work on an access database?
Is it possible to host it online and have its features still functional without having to develop a custom frontend?
MS Access as a software has a few nice features that don't require any programming to configure:
Drop down lists - choose one
Multi Checkbox lists - choose multiple
Is it possible to get all of these features available even when hosted online? I'm basically thinking of an alternate way to quickly get people to work with data using GUI features like the above without going the webapp<>MySQL way.
You have some good comments here. Keep in mind that things have changed quite a bit for access 2010.
Access 2010 allows you to build web applications. The development process is very much the same as it’s been for years, but you can’t use VBA in forms for these web applications (you use a new macro language). This new feature set allows you to publish applications you build to a website. Here is an video of an application of mine running in access 2010, and at the halfway point in the video I switch to running the access application 100% in a web browser:
http://www.youtube.com/watch?v=AU4mH0jPntI
The above is for access 2010…due out this year. The above will require you to be running SharePoint services, or use an hosting service that supports "access web" services.
For previous versions of access, for all intents and purposes, it’s not a web based system at all. Now when you say multiple users, you have to clarify what kind of users and where they plan to be. If your users are on a local office network, then MS access can be used as a multi user system right out of a box with no additional coding and programming required. It is recommended however that you split your application into a front end part that’s deployed on each user’s computer. This Concept as outlined in the following article of mine.
http://www.members.shaw.ca/AlbertKallal/Articles/split/index.htm
Now, perhaps the users are going to be on notebooks and in different locations all over the country? In this type of case you are attempting to connect over a wide area network, or have users connect to the application over the Internet. This is a different problem. In this type of scenario, a good solution is to use something like SQL server for the backend, and you continue to deploy the Access front ends to each user’s computer. This application tends to be about the most cost affordable also. And using sql server + ms-access means you get to continue developing in Access for the most part like you always done. Another way to accomplish wide area use without resorting to sql server is to use something called terminal services. I outline these possibilities in the following article:
http://www.members.shaw.ca/AlbertKallal//Wan/Wans.html
As mentioned, a few others here posted links to some of the new SharePoint features that you can consider using, but they not out untill later this year.
Multi-user Access apps are pretty easy to do for small workgroup user populations in the 15-25 ranger or smaller. Above that, a developer should consider upsizing to a server back end, with the trade-off being greater administrative overhead for the server vs. having to program the app more carefully if you retain the Jet/ACE back end.
As to online access, you this isn't possible over HTTP, but if you have a Windows Terminal Server available, you can host your app there and give users access to that. This is actually an extremely easy and efficient and inexpensive way to support remote users of an app, though the larger the user population, the more problematic it becomes. But by the time an Access app has a user population that would strain a Windows Terminal Server setup, you're no longer going to be using a Jet/ACE back end.
And with a server back end, you could give access to a SQL Server on a VPN over the Internet, and if you write your Access app really efficiently, even over a standard broadband connection, your users could still work productively.
Then there's the future of Access: in Access 2010, a great deal of work has been done to integrate with a host of new features in Sharepoint 2010. If you create your A2010 app using the new type of Access web forms and reports, your app can be uploaded to a Sharepoint server running the new Access Services, and it can then be used running in a web browser (not limited to IE and not dependent on any plugins or web controls, as was the case in the past with the completely worthless Access Data Access Pages). The data store can either be a SQL Server, or you could keep it Jet/ACE for users not accessing it via the web browser, and have the data stored in Sharepoint for the online users. Also, you can have an app integrated with Sharepoint running locally in Access that uses Sharepoint when connected to the Internet, and still be able to work offline when disconnected. When connected again, you synch your local changes with the Sharepoint server, resolve any differences and continue working.
The features are really quite remarkable, and according to what I've heard and seen, if the Access app is built entirely of web forms and reports, it will look and function identically when run in Access and when run in the web browser via Sharepoint. And if you need to have client-side features that you don't expose to the users running the app in the browser, you can still use traditional Access objects!
The Access development team's blog has a number of posts on what's coming in A2010, and there's a good video posted there demonstrating how A2010 integrates with Sharepoint 2010's new Access Services.
This constitutes a quantum leap in Access's web capabilities, which were previously almost non-existent, and I'm quite excited about this. I was formerly quite wary of the changes being made to Access that seemed entirely to make it a servant of Sharepoint, but now I can see that the benefit to Access users and Access developers will be huge.
One way i've heard of, is to import the access database into a SQL Server database.
(Almost any version will do.).
Then link to the SQL Server database with Access and let users use it as they did before.
Look at this link: http://office.microsoft.com/en-us/access/HA010345991033.aspx
If you want an online solution i'd recommend going with a normal web application architecture. (Talking to a proper database.).
I have never needed to support it myself, but from what I heard so far, performance dramatically breaks down as soon as you need to support multiple users writing simultaneously. I think this is because Access uses simple file locking to implement isolation, and this just is not the right technique for a concurrent database system.
Hosted on-line? Do you mean on the network? Technically it will work on a network but there is a reason MS-Access in not in Visual Studio - it is not considered a development platform - it is a desktop application. When MS-Access first hit the scene many people built applications using it. The multiuser functionality just is not there. Upto four or five users is ok. But I would not go for more.

Replication vs Sync Framework vs Service Broker

I've asked about each of these technologies separately, and really haven't found a suitable answer.
We have a server in our central office running SQL Server 2005 Enterprise that has several large (large in the sense that DSL is the limiting factor) databases that we need local copies of at each of our locations. We currently have a few dozen locations, and are needing to bring even more online. The total number of locations we'll need to sync these databases to will be in the several hundreds in the next 2 years.
We are trying to overcome issues with the WAN connection at each location. These are DSL lines and the wiring at the locations isn't always the best. We currently have issues with some of the locations going down as often as every hour. While we are working to resolve these issues with rewiring and assistance from the local telcos, it mainly highlights the problem at hand: we need a two-way sync that can handle being occasionally-connected.
We tried transactional replication for a while, and while it worked some of the time, it was too high maintenance for us, and it seemed to randomly error out often with no possible explanations, forcing us to reinitialize subscriptions (which could take upwards of 4 hours assuming the location would stay connected long enough to get the entire snapshot in one go). We've looked at rolling our own solution from scratch, but I don't feel this would be the best idea given the scale and reliability we are needing.
So far we've also looked at Sync Framework, and as suggested by someone else, Service Broker. Sync Framework seems a better fit, but I was told that Service Broker scales better and is more reliable? I can't find any empirical data on the overhead involved with Sync Framework or Service Broker, so it's proving impossible to compare the two in this regard.
What we really need is a two-way sync between the central office server and a remote client that can run autonomously and can report to an admin in the event of a failure that requires our intervention.
There are so many possible solutions to this problem, all involving completely different technologies, that I need a fresh eye on this.
What do you think would be the optimal solution for our situation, and why?
EDIT: Obviously, upgrading to SQL Server 2008 would solve this problem easily. However, we would like to try to less expensive options first.
I don't have any hard data to offer on this, but we used the sync framework on a project a while ago. My experience with it is really bad. It's slow (even when synchronizing relatively small tables across a LAN), scales terribly and requires a lot of work to manually handle error conditions (it'll happy produce larger packets than WCF can handle by default -- and is only able to split updates into batches when syncing one way, not the other.) And it only works with a few select databases (the client must use MS SQL Compact Edition, as I recall), unless you're willing to write your own SyncAdapter.
Overall, a lot of work just to get a fragile and inefficient solution to your problem. I wouldn't recommend it.
You can Use sync framwork with SQL express 2008 R1/R2 on one end and multitenant db SQl server enterprise on central end. Below is the sample application for n-tier sync over secure WCF channel.You could write windows service to sync data from backend:
http://www.rajneeshnoonia.com/blog/2012/03/n-tier-sync-framework/
It sould be capable enough to handle large number of clients (thousands).
I think we'll look into the SQL Server 2008 upgrade route. It seems the native change tracking support will be the easiest way to accomplish this.

What are the cons of a web based application

I am going to write a database application for the camp I work for. I am thinking about writing it in C# with a Windows GUI interface but using a browser as the application is seeming more and more appelaing for various reasons. What I am wondering is why someone would not choose to write an application as a web application. Ex. The back button can cause you some trouble. Are there other things that ayone can think of?
There are plenty of cons:
Speed and responsiveness tend to be significantly worse
Complicated UI widgets (such as tree controls) are harder to do
Rendering graphics of any kind is pretty tricky, 3D graphics is even harder
You have to mess around with logins
A centralised server means clients always need network access
Security restrictions may cause you trouble
Browser incompatibilities can cause a lot of extra work
UI conventions are less well-defined on the web - users may find it harder to use
Client-side storage is limited
The question is.. do enough of those apply to your project to make web the wrong choice?
One thing that was not mentioned here is the level of complexity and knowledge required to generate a good web application. The problem being unless you are doing something very simple, there is no "Single" knowledge or technology that goes into these applications.
For example if you were to write an application for some client server platform.. you may develop in Java or C++. For a complex web application you may have to have expertise in Java, Java Script, HTML, Flash, CSS, Ajax, SQL, J2EE.. etc. Also the components of a web based application are also more numerous, Web Application Server, HTTP Server, Database, Browser.. are tipical components but there could be more.. a client server app is tipical just what it says.. a client application and a Server application. My experience and personal preference is not web based .. web based is great for many things. But even though I am an IT Architect for a leading company that is completely emersed in Web Apps as the solution for everything... The cons are many still.. I do thing the technology will evolve and the cons will go away over time though.
Essentially the real limitations are only through of the platform, being the browser. If you have to account for all browsers in current use that can be a pain due to varying degrees of standards in each of them.
If have control of the which browser to use, that is everyone is on computers that you control on site, and say you install firefox on all of them, you could then leverage the latest Javascript and CSS standards to their fullest in your content delivery.
[edit] You could also look into options like the adobe integrated runtime or "AIR" as an option allowing you to code the front-end with traditional browser based options like xhtml/css/javascript, flash/flex and have the backend hooked up to your database online, only also providing functionality of a traditional desktop app at the same time.
The biggest difference and drawback I see with web applications is state management. Since the web is, by nature, stateless every thing you want to maintain has to be sent back and forth from the server with every request and response. How to efficiently store and retrieve it in a matter with respect to page size and performance is hard to do at times. Also the fact that there is no real standard (at least not that everyone adheres to) for browsers makes consistency really..........fun.
You need to have a network access to the server that you are going to have the web application on (if there are going to be multiple users for the application - which is typically the case).
Actually, there are more pros than cons - if you can give some details about your application, we could help a little more...
It completely depends on the requirements of your project. For the most part, there isn't much web applications cannot do these days. Admittedly, certain applications do belong on the desktop as browsers (while currently advancing, and rapidly), still are not quite there yet. From the advent of applications such as Google Docs, Gmail
There isn't much you -cannot- do on the web. If you're creating a World of Warcraft competitor however, the web is most certainly not the optimal solution. Again, unfortunately we'd need more insight on the application you're building for the camp. The best part about the web is that anyone with a browser can use your application.
Web applications delegate processing to a remote machine. Depending on the amount of processing, this can be a con. Consider a photo editor that's a web app.
Web applications also can't deal with a whole lot of data going back and forth to and from a client. You can watch video online.. when it's compressed. It will be awhile before we see any web-based video editing software.
Browser compatibility is also a hassle. You can't control the look-and-feel of the application 100%.
Vaibhav has a good point. What's your application?
A major one is down time for migrations... users will not expect the application to be down, ever, but realistically it will have to be down for major upgrades. When doing this with a desktop application, the user (or end-user systems admin) is in control of when upgrades happen; with an online app, they're not.
For applications which have large data, performance can be a major problem as you're storing a large number of users' data centrally, which means the IO performance will not be as good as it would be if you gave them all a laptop.
In general scalability gives problems for a server-based app. Desktop applications scale really well.
You can do an awful lot with a web-based app, but it is a lot easier to do certain things with a thick client:
Performance: You get simple access to the full power of the client's CPU.
Responsiveness: Interactivity is fast and easy.
Graphics: You can easily use graphics libraries such as DirectX and OpenGL to create fast impressive graphics.
Work with local files
Peer-to-peer
Deciding whether a web application is a good approach depends on what you are trying to achieve. However here are some more general cons of web applications:
Real integration with desktop apps (e.g. Outlook) is impossible
Drag and drop between your app and the desktop / other running apps
With a web application, there are more privacy concerns, when you are storing user data on your servers. You have to make sure that you don't loose/disclose it and your users have to be comfortable with the idea of storing that data on your servers.
Apart from that, there are many security problems, like Man-in-the-middle attacks, XSS or SQL injections.
You also need to make sure that you have enough computing power and bandwidth at hand.
"Ex. The back button can cause you some trouble."
You'll have to be specific on this. A lot of people make fundamental mistakes in their web applications and introduce bugs in how they handle transactions. If you do not use "Redirect after Post" (also known as Post-Redirect-Get, PRG design), then you've created a bug which appears as a problem with the back button.
A blanket statement that the back button in trouble is unlikely to be true. A specific example would clarify your specific question on this.
The back button really is not that much of an issue if you design your application correctly. You can use AJAX to manipulate parts of the current page, without adding items into the browser history (since the page itself wont change).
The biggest issue with designing web applications has to do with state, and the challenges that need to be programmed around. With a desktop application, state is easy to handle, you can leave a database connection opened, lock the record and wait for the user to make the changes and commit. With a web application, you could lock the record...but then what if the user closes the browser? These things must be overcome in the design of your application.
When designing a web application, make sure that each trip to the server "stands alone" and provides a complete answer. Always re-initialize your variables before performing any work and never assume anything. One of the challenges I ran into once was pulling "pages" of grid data back to the user. In a real busy system, with record additions/modifications happening in real time, the user navigation from page to page would vary greatly, sometimes even resulting in viewing the same set of a few records as new additions were added in-front of the query.

Resources