I have been doing some investigations into some light weight database Syncing tools to trial. The initial task we want to perform is a simple data sync from a few tables on a Sybase ASE database (15) to a SQL Server database (2008 R2). Timing wise, I'd like to keep my options open, but ultimately, I would like to have the ability to sync every minute or less.
I have been looking at SymmetricDS, which at face value seems to do exactly what I want it to. The drama is I have hit a couple of roadblocks on the Sybase side of things, which is proving to be very frustrating (Jumpmind support are assisting). It appears that Java has a problem with the default collation we have on our server, being HP-roman8. Unfortunately, to change this charset is way bigger then this project itself.
I have also started investigating Talend, but have hit a few roadblocks in relation to requiring older versions of drivers for Sybase and downgrading the installed version of Java.
Without having to go to Replication Server, does anyone have any suggestions on a relatively lightweight ETL or database Syncing tool that will do what I want? The biggest gotcha thus far is Sybase support - I really need something that will seamlessly work without having to hack too much.
Cheers
You should try uniVocity. It is a Java-based ETL framework that certainly can help you do what you need. You can use any JDBC driver, define your mappings with a few lines of code and have this working faster than a traditional ETL tool.
Have a read through its tutorial and also check out a few sample projects here
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).
Related
I've been searching for some time for a good solution to implement the idea of managing schema on an SQL Server Compact 3.5 database.
I know of several ways of managing schema on SQL Server Express, SQL Server Standard, SQL Server Enterprise, but the Compact Edition doesn't support the necessary tools required to use the same methodology.
Any suggestions/tips?
I should expand this to say that it is for 100+ clients with wrapperware software. As the system changes, I need to publish update scripts alongside the new binaries to the client. I was looking for a decent method by which to publish this without having to just hand the client a script file and say "Run this in SSMSE". Most clients are not capable of doing such a beast.
A buddy of mine disclosed a partial script on how to handle the SQL Server piece of my task, but never worked on Compact Edition. It looks like I'll be on my own for this.
What I think that I've decided to do, and it's going to need a "geek week" to accomplish, is to write some sort of a tool much like how WiX and NAnt works, so that I can just write an overzealous XML document to handle the work.
If I think that it is worthwhile, I'll publish it on CodePlex and/or The Code Project because I've used both sites a bit to gain better understanding of concepts for jobs I've done in the past, and I think it is probably worthwhile to give back a little.
Edit on 5/3/2010:
If someone is willing to "name" the project, I'll upload the dirty/nasty version that I've written for MS SQL to CodePlex so that maybe we can start hacking out a version of SQL Compact. Although, I think with the next revision of the initial application that I was planning, I'm going to be abandoning SQL Compact and just use XML Files for storage, as the software is being converted from an Installable package to being a Silverlight application. Silverlight just gives a better access strategy.
I am currently looking into Migrator.Net.
This allows you to write changes to your database, called migrations, directly in C#.
These migrations can contain everything from simple table additions/drops, column modifications, to complicated data update code.
When your application boots, it can verify what version the database is currently in and apply any migrations that are required to bring it up to date. All this is handled automatically. The code to run this update is as simple as:
Assembly asm = Assembly.Load("LocalModels.migration");
Migrator m = new Migrator("SqlServerCe", "Data Source=LocalModels.sdf", asm, false);
m.MigrateToLastVersion();
I am having a couple minor issues with the Compact support (it assumes the default schema is dbo). But I don't think it will be too difficult to fix them.
some random thoughts (not sure I can fully answer though)
the Microsoft Sync Framework is one option. I haven't had a chance to fully appreciate what it can do once you've deployed it after the initial first time (which seems to work fine). There's a MSDN site for it here
You can execute scripts on a mobile device, but not through something like SQL Management Studio, so in theory you could manage/maintain T-SQL scripts but the down side is that the T-SQL would be convoluted (to CE's supported statements) and I don't know a way to "automate" execution - but the Sync Framework might hold some answers..
If one of your key criteria is going to be working efficiently over a small pipe, the only real choice you have is to store a DB Schema Version (maybe somehow tied to the scripts checked into your CMS) and when an update is needed, the change scripts are sent over the wire and applied in order. You would probably want to keep a log in your DB as well of these scripts being applied so you can gracefully handle disconnects, reboots and other potentially nasty problems.
Is SQL Server Management Studio any use for you?
http://technet.microsoft.com/en-us/library/ms172933.aspx
Which of the Database servers would you people recommend for using in a moderate-to-large scale (will vary from customer to customer) application.
I know MS SQL but since the app will be developed using delphi the .net framework is kind of annoying to deploy aswell. Also how realiable is Interbase.
Thanks in advance.
Since I have been developing in Firebird for several years now, if I were in your situation, I would look no further. The fact that you can develop your own extension library (udf library) in Delphi makes it very attractive to start with. Some of the new SQL constructs like the List aggregator and Common Table Expressions are icing on the cake. Deployment is very light (under 5 MB), and the support is excellent including incredible tools like IBExpert. I used MS SQL Server in a big company I worked at a few years ago and I would choose Firebird over MS SQL Server.
Data Access to SQL Server has nothing to do with the .NET framework. Delphi can connect just fine to SQL Server.
In terms of database engines that work with delphi, another you can consider is the Sybase Advantage Database server
Interbase is as reliable as they come. In fact, I'd like to get a job as an Interbase DBA, because I could uses the sleep.
I would definitely recommend looking at Interbase by Embarcadero or the open-source sibling, Firebird. Both are very well supported by Delphi by means of native, high-performance db-access components and such.
Interbase is a totally rock-solid database - I would put it on a par with SQL Server or Oracle. It's totally capable of handling medium to high traffic for at least 50-100 users concurrently, no problem at all.
Another two worth competitor would be Elevate DB and Nexus DB. Both grew out of Delphi file-based database systems, and both have been around for quite some time (in one form or another) and are well proven, widely used, well tested.
Marc
This seems back-to-front to me. I would choose the database server first, then select the development environment that works best with the selected database.
SQL Server. For 2005, you can make your own SQL Server deployment application that automates installing of Windows Installer 3.1, .NET Framework 2.0, SQL Server 2005 then installing the database through generatated SQL script (Feel free to contact me if you need help with it, I'd gladly help you). If you still insist on not installing .Net framework which is 22MB, I'd say go with SQL Server 2000 as you can still upgrade it to 2005+ when you'd like to.
Also, I had been using Advantage Database Server for few years and it's alright, Few DLLs to copy and you're ready to go even if you'd like to switch to remote version.
Both Server and Client installations of MS SQL require the .NET framework on the target machine. That's about it. If we can use ms sql without .net on target machine then ms sql is fine.
I have being developing in Delphi for about 6 years using distinct editions of SQL Server, and this database server behaved very very well. All of them were critical applications.
So, I would recommend you to keep with it.If you don't want to use .NET with and keep SQL Server I would recommend an older version pre-2005.
This isn't really an answer, but some things you might like to consider before making your mind up.
What do you mean by moderate-to-large scale? Do you mean by the number of concurrent users or by the amount of data being stored?
Are you going to be supporting the database as well when it's deployed, or might the customer have their own DBAs and want to incorporate it into their infrastructure team's remit?
Do you need to build-in complex features like auditing or granular security that some databases might provide out-of-the-box? Do you need to be able to run an overnight batch process on the database in some controlled manner?
Do you need to be able to have robust high-volume transactional support, or is it more for just using as a sink for reporting data? i.e. is it more OLTP or OLAP?
Are you limited by cost? i.e. does it need to be free/open-source, or can you select paid-for vendor product?
Both Firebird & Interbase are great, but not generally well-received into an "enterprise" environment in my experience unless they're "hands-off" and embedded into the application. MS SQL Server is a "safe-bet" for most systems. There is a cost associated with it but, as you know, it does come with a complete toolset and very good support.
If you do go down the Firebird or Interbase route then do take a look at Jason Wharton's IBObjects library which provides a high-performance replacement for BDE when using these databases. http://www.ibobjects.com/
I'm looking for some spatial database features but can't upgrade to SQL Server 2008 at the moment. I've come across this open source mssqlspatial package which would provide most of the features I need however I'm hesitant to deploy it without knowing if it's production ready. I was wondering if anybody out there had deployed this package.
Thanks
Never heard of mssqlspatial, sorry. Have you considered PostGIS though? That'll give you an open source spatial database, and I believe it's quite well regarded. Never used it myself.
This isn't answer...just some additional info.
We just went to SQL Server 2008, but PostGIS would have been my first choice (wasn't my choice). As MarkJ says, it is well regarded.
You might want to checkout SharpmapV2. It's not a spatial database but you've got providers on just about everything including MsSqlSpatial:
http://sharpmapv2.googlecode.com/svn/trunk/SharpMap.Data.Providers/
And I'm seeing the same developers in both repositories (John Diss looks like a busy guy), so I think SharpMapv2 is more current:
http://mssqlspatial.codeplex.com/SourceControl/ListDownloadableCommits.aspx
Is it possible to use e.g. SQLite with PowerBuilder? I need an embedded open source database (no additional costs).
Like Bernard said, you'll need an ODBC driver, so as long as you're willing to go third party (if I understand the SQLite situation correctly), that should be no problem.
That said, if you have PowerBuilder, you have license to distribute the single-user SQL Anywhere run time engine. If no-cost is your only criteria, and you're only connecting locally, SQL Anywhere may be an option to evaluate. Not only is it an incredibly solid database, but there's a much larger base of documentation and experience connecting PowerBuilder to SQL Anywhere, so if you run into problems, you're more likely to get some help.
Good luck.
I don't believe that PowerBuilder contains a driver for native support to SQLite. But it definitely has a driver for ODBC, so that is always an option even if it isn't the most efficient one.
I used to use SQL Anywhere, but eventually ditched it for the reasons Joe Landau gave - can't change the schema using the distributable runtime engine.
I switched to Firebird, which has an embedded version, and that seems solid. The only issue is that the ODBC driver I'm using (Gemini), which seems to be the best one available, seems to have gone out of business. (I just checked - it seems to be available on other sites.) And you have to add the following to your PBODB*.INI file:
[Firebird]
PBSyntax='Firebird_SYNTAX'
PBNoCatalog='YES'
[Firebird_SYNTAX]
CreateTable='CREATE TABLE &TableName (::ColumnElement[::ColumnElement]...)'
ColumnElement='&ColumnName &DataType'
DropTable='DROP TABLE &TableName'
GetIdentity='Select gen_id(GEN_&TableName,0) from RDB$DATABASE'
I've been very happy with it. Using it for almost 2 years, with over 1,000 users, and no problems whatsoever. You can also easily switch to the Firebird server version if some users need that.
As noted, SQL Anywhere is available and solid. But it has a disadvantage--you can't change the schema using the run time engine. This makes it hard to, say, add a column to a db that you have distributed.
++ to the comments by DC on Firebird. One of the best free databases out there. I have used it for years for a PB application I sell to Law Firms.
Although I use the server version even if the target is a single workstation. Simplifies the deployment and the issue of adding workstations later if desired.
I use the standard Firebird ODBC driver at http://www.firebirdsql.org/index.php?op=files&id=odbc
There are two good GUI front database management tools that I hve used - IBOConsole and Flamerobin.
My company has a number of relatively small Access databases (2-5MB) that control our user assisted design tools. Naturally these databases evolve over time as data bugs are found and fixed and as the schema changes to support new features in the tools. Can anyone recommend a database diff tool to compare both the data and schema from one version of the database to the next? Any suggestions will be appreciated: free, open source, or commercial.
I use Red Gate Sql Compare for comparing schemas. It also has an interesting feature that allows you to save a snapshot of the schema which you can then use in later diffs. for example compare the schema of today with the schema of a month ago.
I use ApexSQL Diff. It is an excellent tool for doing just what you're describing...compare schema, compare data, generate change scripts. It not free, but it works well.
NOTE: ApexSQL Diff only works with SQL Server.
We never actually purchased it as we ended up using SQL Server 2005, but DBDiff seemed to do the trick: http://www.dkgas.com/downdbdiff.cgi
It works with any ODBC compatible DB.
I've used Total Access Detective in the past and it did the trick. It's a while ago though so you might want to investigate first...
If you're looking for a free alternative to Red Gate's most excellent SQL Compare, you might want to check SQLDBDigg made by SQLDBTools. It's what I used until I caved and bought SQL Compare.
It's not a perfect solution, but I often export both databases as txt/SQL files and then use a diff program, such as the one that comes with TortoiseSVN. You can then see all of the differences. It doesn't automatically create the SQL though to sync the dbs.
http://www.diffkit.org
Features
High performance, for large datasets (+10MM rows).
Very low memory overhead, even on very large datasets.
High quality-- comprehensive embedded regression test suite for the application/framework.
Java run everywhere (tm) — Linux, Solaris, OS X, Windows, etc.
Cross database-- Oracle, MySQL, DB2, and any JDBC datasource.
Command-line driven; no GUI needed; can run in headless environments.
XML configuration file driven.
Free Open Source Software.
Apache License, Version 2.0.
Clean Object Oriented Design make extension easy.
Easily embeddable as a Java library (jar).