I need to capture data changes in Hive and replicate it to oracle DB using IBM CDC 11.4
Is it supported?
If it is supported, Where can I download the agent?
Any tutorials or guides?
Related
I would like to have a SQL Server DB as a source into Kafka.
However, my SQL Server version is old (2012 Standard) and does not have the CDC feature, what are my options?
The 2 ways I've seen, from the Kafka documentation and blogposts, are to use the Debezium connector or the JDBC Connector. The content I read was from a 2018 post and so I wanted to check, now in 2022, if there are any new options that I'm missing.
Given the situation (SQL Server 2012), here are the choices I could think of:
Upgrade the DB to a later version, enable CDC, and work with the Debezium SQL Server Kafka Connect Source connector in true CDC
use the Confluent JDBC source connector
Are these the only 2 good options even in 2022?
(By good I mean please exclude telling me to write your own query-based CDC code).
Is the decision really just between:
ideal: I can upgrade the DB to a later SQL Server Version with CDC: then enable CDC and use the Debezium connector
not so ideal: I cannot upgrade the DB, no choice but to use the JDBC Source Connector
?
p.s I have referred to and read this question but that was from 2019 and also I wanted to know if upgrading and then using Debezium is a good idea
I was wondering if I can use SSIS, as an ETL tool to extract data from Oracle 11g and load it to another Oracle 11g destination, and apply CDC. I have used SSIS CDC with MS SQL; however, I had to enable CDC on the database so I was wondering if I can do the same thing.
Your help is really appreciated.
You can follow the following Microsoft Docs Topics to learn more on Microsoft SQL Server 2012 Change Data Capture for Oracle by Attunity
https://social.technet.microsoft.com/wiki/contents/articles/7644.microsoft-sql-server-2012-change-data-capture-for-oracle-by-attunity-topics.aspx
https://learn.microsoft.com/en-us/sql/integration-services/change-data-capture/change-data-capture-service-for-oracle-by-attunity
This link contains helpful information on how to setup and use this component. so i think this is what you are looking for.
Also to work with oracle using SSIS 2012 you have to download the oracle connectors:
https://www.microsoft.com/en-us/download/details.aspx?id=29283
We want to pull data from Redshift database into SQL Server. Currently, we are using SQL Workbench to analyze Redshift database. We referred the following link -
Connect Your Cluster By using SQL Workbench
Here we used - RedshiftJDBC41-1.1.17.1017 driver to connect.
Now we want to push this data to DWH in SQL Server 2016.
So what is the best way to accomplish this? There is not much information on net on how to pull data from Redshift.
Can we have SSIS component or something in SSDT/SSIS2015 to load this data?
Or can we have a middle layer of any tool (any ETL tools or Excel) which will dump this Redshift data into CSV or something which will be readable in SSIS?
Data from Amazon Redshift can be extracted and consequently loaded into almost any platform.
You have several options to move data from Redshift to SQL Server.
ETL Tool - You can use a commercial ETL tool. SSIS is a
perfectly legitimate way to extract from Redshift and to land the
data in SQL Server.
S3 Files - You can unload the data from Redshift into S3 buckets and
then use SSIS or bcp to copy data from buckets to your SQL Server
Local file system -- You can run the unload command to extract data
to local file system. You can have psql running locally in a server
inside of your network.
For information on how to connect SSIS and any other ETL or BI tool to Redshift please see this link.
Amazon Redshift provides ODBC Connectors which is not supported by any Sql Server BI Tools. MS BI stack only support OLE DB Providers and it is being provided by a third party vendor
I am trying to build SSAS cube sourcing data from Amazon RedShift.
I chose to use Apache Mahout as my recommendation engine but at the same time due to some reasons it would be easier if I could store my data in a SQL Server db. Can mahout be connected with SQL Server without any problems ?
The documentation says that it can be connected with other db engines through JDB driver but I see all articles , books using mysql and also the data model supported are for mysql only.
How to convert MySQL to SQL Server databases:
SQL Import/Export Wizard through ODBC (http://www.mssqltips.com/sqlservertutorial/2205/mysql-to-sql-server-data-migration/)
SQL Server Migration Assistant (http://msdn.microsoft.com/en-us/library/hh313125(v=sql.110).aspx)
Here is the JDBC driver for SQL server:
JDBC Driver for SQL Server: http://msdn.microsoft.com/en-us/sqlserver/aa937724.aspx
Changing DB input format/driver in Hadoop cluster: http://blog.cloudera.com/blog/2009/03/database-access-with-hadoop/
There are also numerous example of using Mahout with an Azure Hadoop Cluster via HDInsight:
http://bluewatersql.wordpress.com/2013/04/12/installing-mahout-for-hdinsight-on-windows-server/
http://www.codeproject.com/Articles/620717/Building-A-Recommendation-Engine-Machine-Learning
I have just started my experiments with Mahout. I managed to run some book examples after replacing the in-memory data models with SQL92JDBCDataModel or SQL92BooleanPrefJDBCDataModel shipped with Mahout 0.9.
I passed an instance of SQLServerDataSource to constructors of those data models. This class is included into the Microsoft JDBC Drivers for SQL Server package (I used the version 4.1)
However, the SQL92JDBCDataModel documentaton states that it is "not optimized for performance".
I have a Derby Database in Netbeans with connection string
jdbc:derby://localhost:1527/MyDatabase
Can this be used with ODBC? If so, how could I transform this or configure my Derby Database to be compliant with ODBC
The end goal is to get the Derby Database migrated to MySQL. Using the migrating wizard in MySQL Workbench appears to be the easiest way. However I do need ODBC connectivity.
Do you want to use ODBC because MySQL Workbench uses it to migrate database?
I migrated some databases between various engines and my favorite technology is to convert DDL schema (CREATE TABLE, CREATE VIEW etc) using specialized Python program. Then I use JDBC with getObject()/setObject() methods to copy data. You can see such copy database Jython program (Jython is a Python language that works using JVM and can use JDBC drivers) in my response to Blob's migration data from Informix to Postgres .
BTW Quick search shows that IBM have ODBC driver to Derby (they bought Informix that earlier bought Cloudscape): http://www.ibm.com/developerworks/data/library/techarticle/dm-0409cline2/
Use OpenDBCopy, which is an opensource database utility to migrate data from and to any database via JDBC connection.
You can copy table structures as well as data from any supported database.