We want to pull data from Redshift database into SQL Server. Currently, we are using SQL Workbench to analyze Redshift database. We referred the following link -
Connect Your Cluster By using SQL Workbench
Here we used - RedshiftJDBC41-1.1.17.1017 driver to connect.
Now we want to push this data to DWH in SQL Server 2016.
So what is the best way to accomplish this? There is not much information on net on how to pull data from Redshift.
Can we have SSIS component or something in SSDT/SSIS2015 to load this data?
Or can we have a middle layer of any tool (any ETL tools or Excel) which will dump this Redshift data into CSV or something which will be readable in SSIS?
Data from Amazon Redshift can be extracted and consequently loaded into almost any platform.
You have several options to move data from Redshift to SQL Server.
ETL Tool - You can use a commercial ETL tool. SSIS is a
perfectly legitimate way to extract from Redshift and to land the
data in SQL Server.
S3 Files - You can unload the data from Redshift into S3 buckets and
then use SSIS or bcp to copy data from buckets to your SQL Server
Local file system -- You can run the unload command to extract data
to local file system. You can have psql running locally in a server
inside of your network.
For information on how to connect SSIS and any other ETL or BI tool to Redshift please see this link.
Amazon Redshift provides ODBC Connectors which is not supported by any Sql Server BI Tools. MS BI stack only support OLE DB Providers and it is being provided by a third party vendor
I am trying to build SSAS cube sourcing data from Amazon RedShift.
Related
Is it possible to load data from S3 to SQL Server using Matillion ETL tool?
This is not a good use case for Matillion. The Matillion tool works by ingesting source data into a cloud database, and then running the transforms inside the database. There are 4 cloud databases supported: Snowflake, BigQuery, Redshift, and Synapse.
There are output components that can then output data to a target DB, but MSSQL is not currently supported (Azure SQL is).
I suggest looking at SSIS which has native support for MSSQL
Matillion ETL can help you to
Load data into its supported Cloud Data Warehouse (currently there are five options which are: Snowflake, Amazon Redshift, Google BigQuery, Microsoft Azure Synapse, and Delta Lake on Databricks)
Transform and integrate that data inside the Cloud Data Warehouse
MSSQL is not among the list of Matillion ETL supported Cloud Data Warehouses, so you can't use it to directly load data from S3 to MSSQL.
However you could use a Microsoft SQL Server Output component to transfer data into MSSQL, provided it went via one of the above listed supported Cloud Data Warehouses first.
I have read a few articles about moving data out of SAP BW and into SQL Server. I cant find any articles on moving the data from SQL Server to SAP BW, is it even possible and if so what would be the best way to handle this?
After searching on this topic, i found many link addressing this issue, in this answer i will try to summarize them all and to provide all links that can help you achieving your goal.
There are many way to import data from SQL Server into SAP BW:
(1) SAP BW DB Connect
With DB Connect, you can load data from a database system that is supported by SAP, by
linking a database to the BW as a source system, thereby creating a direct point of access to external relational database management systems (RDBMS).
Making metadata known to BW by generating a DataSource.
Transferring Data Using DB Connect
Create DataSource for DB Connect step by step
SAP BW 7.3: How to extract data from SQL Server using DB Connect
Note 512739 - BW external DB Connect for MS SQLServer
(2) SAP BO Services
Creating a Data Store by using SQL Server Database
(3) SSIS - Microsoft Connector for SAP BW
Microsoft has created a SSIS connector for SAP BW, which allows you to create connection To SAP BW, you can refer to the following links and official documentations:
How to load data into SAP BW 7.3 from SQL Server using SSIS
SQL Server 2016
The Microsoft Connector for SAP BW consists of a set of three components that let you extract data from, or load data into, an SAP Netweaver BW version 7 system.
The Microsoft Connector for SAP BW for SQL Server 2016 is a component of the SQL Server 2016 Feature Pack.
Microsoft Connector for SAP BW
Microsoft Connector for SAP BW Components
SQL Server 2014
The Microsoft Connector 1.1 for SAP BW consists of a set of three components that let you extract data from, or load data into, an SAP Netweaver BW version 7 system.
Microsoft Connector 1.1 for SAP BW
SQL Server 2008
Using SQL Server 2008 Integration Services with SAP BI 7.0
If you decide to go with SSIS, I found this white paper from SAP that contains a detailed tutorials:
How to use Microsoft Connector 1.0 in
MS SQL Server Integration Services packages with Open Hub services to export data from SAP BI to MS SQL
If this article didn't help, you can refer to tens of tutorials on how to create SSIS packages on the internet:
Steps to Create your First SSIS Package
SSIS Basics: Setting Up Your Initial Package
Create a new SSIS package/
Official integration services tutorials
(4) SSIS - Xtract IS BW Loader
This is a third-party tool created to allow SSIS to connect with SAP BW.
The Xtract IS Component Suite offers nine components to provide all kinds of SAP interface technologies. Implement your data flows in a secure and type-safe way with a graphical editor.
Xtract IS – Plug’n’Play for SAP and SQL Server Integration Services
Discussion
While searching for this issue, using SSIS to achieve that is not recommended because it may encounter some incompatibility issue, they recommended to use SAP BW standard anyDB source system or SAP BO Data services.
Issues example:
Loading Data from SSIS to SAP BW
Using SSIS to load data from SQL Server to SAP BW
SAP BW to SQl Server import and export wizard
On the other hand, SSIS guarantee the ease of use and there is thousands of topics and references for this technologies which will help you to achieve your goal faster than other technologies.
References and external links
How to load data from SQL Server to SAP BW using SSIS
Quick Tip: SSIS and SAP BW
-Setting up connection between SAP BW and SSIS
White paper - SAP BW transferring data with DB Connect
I guess the easiest way would be to use an ETL tool to extract data from MSSQL and insert into BW; specifically Microsoft's Integration Services (SSIS), which you likely already have with your SQL Server license.
SSIS is a visual tool specifically designed for ETL scenarios like what you described, and is easy to get started with, I'm guessing that this tutorial covers 90% of what you'll need to do.
However, you should also investigate if your license allows you to use the SSIS connector for SAP BW and test if it is a viable option to simplify writing to BW, as writing directly to BW's tables may not be an option (e.g. because of licensing restrictions and/or technical impact due to directly manipulating a closed system's database).
create the universe/ presentation layer using existing SQL Server DB. if there is no major transformation required ETL can be avoided
I need to build connectivity between Influx and a SQL Server database to transfer data from Influx to SQL Server periodically.
Ideally I would like to have a stored procedure in SQL Server which connects with Influx and gets the required data. This procedure will be scheduled using a SQL Server Agent Job.
Will appreciate any answers regarding the possibilities to achieve the requirement.
Start with the SSIS Data Streaming Destination. This will enable you to have a view in SQL Server that returns the results of running a package.
An SSIS package, in turn, can connect to data sources without loading drivers in the SQL Server process, or can run custom .NET code to interop, or orchestrate running external processes that generate files, and then return the file contents.
So whatever the mechanism of data extract from influxdb turns out to be, an SSIS package using the Data Streaming Destination can be an adapter for you to read that data from inside SQL Server.
I'd be happy if someone could help me.
I need to get data from PL sql developer oracle and import of sql server, I want to do automatic synchronization.
now I am exporting an Excel file from PL sql developer oracle, and load the file into sql server
The tool specifically built for this is Sql Server Integration Services (SSIS). You can create a data flow with source component using Oracle Driver and destination using Sql Server. You can even add transforms between the two to handle formatting and other tasks.
Here is a link to a series of videos on the topic. There is plenty of non - video materials as well if you prefer.
https://www.youtube.com/watch?v=LgB7zdyH0uI
I have a .dmp file (oracle data) and I have to import this file into SQL Server 2008 R2. I tried google but get no clear solution. Oracle is on other machine and SQL Server is on other machine. This .DMP file has only tables and data only nothing else.
Any body has any idea?
You can't get there from here. The files that the Oracle export utility (classic or DataPump) generate (which, by convention, frequently use the DMP extension) are proprietary binary files. They can only be consumed by the Oracle import utility (classic or DataPump) which will only allow you to load the data into another Oracle database.
You could load the DMP file into a new Oracle database but then you'll still need to move the data from Oracle to SQL Server. It may well be easier to ignore the DMP file and pull directly from the original Oracle database. There are a variety of tools that can be used to move data from an Oracle database to a SQL Server database. If you want SQL Server to control the process, you could SQL Server Integration Services (SSIS). You could also create a linked server in SQL Server that references the Oracle database and write queries against the Oracle database via that connection. If you wanted Oracle to push the data, you could also use the Oracle Transparent Gateway with Heterogeneous Services to create a database link from Oracle to SQL Server and issue SQL against the remote SQL Server database.
There is a nice StackOverflow thread on moving data from Oracle to SQL Server. The SSIS logic is extremely similar if you're pulling from Oracle to SQL Server or pushing from SQL Server to Oracle.
Oracle Dumps is not readable by SQL.
Simply it cannot be, But you have different solutions
SQL server integration services (SSIS)
Link between oracle and SQL (Oracle Gateway) but it works with SQL Ent.
Export the data from oracle in a delimited format and insert it into SQL, but it will takes time if data is huge.
When I faced with the same problem, I tried to investigate the format manually (in my case the dump file was generated by Oracle EXP). I found that:
Table definitions come as Oracle CREATE TABLE statement that can be converted into MS SQL format easily
Most kind of data ca be extracted quite easy (text goes "as is", numeric values are stored according to IEEE 754 format)
LOBs are stored in quite complicated way, I failed to recognize it
Then I found the tool that was able to do my migration task: https://www.convert-in.com/ord2mss.htm
Vendor said that it can migrate both exp and expdp to sql server, but I have tested it on EXP format only.