I would like to have a SQL Server DB as a source into Kafka.
However, my SQL Server version is old (2012 Standard) and does not have the CDC feature, what are my options?
The 2 ways I've seen, from the Kafka documentation and blogposts, are to use the Debezium connector or the JDBC Connector. The content I read was from a 2018 post and so I wanted to check, now in 2022, if there are any new options that I'm missing.
Given the situation (SQL Server 2012), here are the choices I could think of:
Upgrade the DB to a later version, enable CDC, and work with the Debezium SQL Server Kafka Connect Source connector in true CDC
use the Confluent JDBC source connector
Are these the only 2 good options even in 2022?
(By good I mean please exclude telling me to write your own query-based CDC code).
Is the decision really just between:
ideal: I can upgrade the DB to a later SQL Server Version with CDC: then enable CDC and use the Debezium connector
not so ideal: I cannot upgrade the DB, no choice but to use the JDBC Source Connector
?
p.s I have referred to and read this question but that was from 2019 and also I wanted to know if upgrading and then using Debezium is a good idea
Related
Could anyone advise me how can sync data between SQL Server on premise to AWS using CDC or change tracking. I'm not sure,can I use SSIS for integration?
Thank you
The canonical AWS method for SQL Server-to-cloud CDC is Database Migration Services, it works with both self-managed SQL Server instances and RDS instances of SQL Server. Be aware it has some limitations and prerequisites for your SQL Server, so do an assessment up front that your scenario is applicable.
If you can't meet the prerequisites or constraints above, you can also consider using the popular open source tool Debezium, especially if you have to have a lot of custom hooks before or after the CDC, since it posts all the changes to a Kafka topic.
SSIS is an acceptable tool if you're only doing batch-level updates and don't need near-realtime replication and you want to fully control the synchronization, but there is a lot of overhead to developing and maintaining those types of packages. I wouldn't recommend it.
During a demo in a recent AWS webinar, a JAR file was specified as a driver for the Schema Conversion Tool to connect to an MS SQL Server db (I am providing a screenshot from the webinar). Why was JDBC chosen? Was this optional, and was a choice made by the person who presented the webinar? If so, what other options would be available as a driver?
From Schema Conversion Tool documentation:
For the AWS SCT to work correctly, you must install the JDBC drivers
for your source and target database engines.
I was wondering if I can use SSIS, as an ETL tool to extract data from Oracle 11g and load it to another Oracle 11g destination, and apply CDC. I have used SSIS CDC with MS SQL; however, I had to enable CDC on the database so I was wondering if I can do the same thing.
Your help is really appreciated.
You can follow the following Microsoft Docs Topics to learn more on Microsoft SQL Server 2012 Change Data Capture for Oracle by Attunity
https://social.technet.microsoft.com/wiki/contents/articles/7644.microsoft-sql-server-2012-change-data-capture-for-oracle-by-attunity-topics.aspx
https://learn.microsoft.com/en-us/sql/integration-services/change-data-capture/change-data-capture-service-for-oracle-by-attunity
This link contains helpful information on how to setup and use this component. so i think this is what you are looking for.
Also to work with oracle using SSIS 2012 you have to download the oracle connectors:
https://www.microsoft.com/en-us/download/details.aspx?id=29283
I chose to use Apache Mahout as my recommendation engine but at the same time due to some reasons it would be easier if I could store my data in a SQL Server db. Can mahout be connected with SQL Server without any problems ?
The documentation says that it can be connected with other db engines through JDB driver but I see all articles , books using mysql and also the data model supported are for mysql only.
How to convert MySQL to SQL Server databases:
SQL Import/Export Wizard through ODBC (http://www.mssqltips.com/sqlservertutorial/2205/mysql-to-sql-server-data-migration/)
SQL Server Migration Assistant (http://msdn.microsoft.com/en-us/library/hh313125(v=sql.110).aspx)
Here is the JDBC driver for SQL server:
JDBC Driver for SQL Server: http://msdn.microsoft.com/en-us/sqlserver/aa937724.aspx
Changing DB input format/driver in Hadoop cluster: http://blog.cloudera.com/blog/2009/03/database-access-with-hadoop/
There are also numerous example of using Mahout with an Azure Hadoop Cluster via HDInsight:
http://bluewatersql.wordpress.com/2013/04/12/installing-mahout-for-hdinsight-on-windows-server/
http://www.codeproject.com/Articles/620717/Building-A-Recommendation-Engine-Machine-Learning
I have just started my experiments with Mahout. I managed to run some book examples after replacing the in-memory data models with SQL92JDBCDataModel or SQL92BooleanPrefJDBCDataModel shipped with Mahout 0.9.
I passed an instance of SQLServerDataSource to constructors of those data models. This class is included into the Microsoft JDBC Drivers for SQL Server package (I used the version 4.1)
However, the SQL92JDBCDataModel documentaton states that it is "not optimized for performance".
I need to replicate my DB without adding it any columns (like the MS SQL replication does), I looked every where and every project I found was last updated 2-3 years ago (in the best case i found a project last version was from 2011).
The replicator needs to support MS SQL 2005 (2008 is not required now) over JDBC driver and it's also preferred that the project is still active and updated (bug fixing, features requests, support etc.)
if any one knows any replicator with these characteristics please help me !
just to be clear, the DB replicated is a MS SQL created by ActiveMQ 5.5.1 using Microsoft JDBC driver 3.0