I need to build connectivity between Influx and a SQL Server database to transfer data from Influx to SQL Server periodically.
Ideally I would like to have a stored procedure in SQL Server which connects with Influx and gets the required data. This procedure will be scheduled using a SQL Server Agent Job.
Will appreciate any answers regarding the possibilities to achieve the requirement.
Start with the SSIS Data Streaming Destination. This will enable you to have a view in SQL Server that returns the results of running a package.
An SSIS package, in turn, can connect to data sources without loading drivers in the SQL Server process, or can run custom .NET code to interop, or orchestrate running external processes that generate files, and then return the file contents.
So whatever the mechanism of data extract from influxdb turns out to be, an SSIS package using the Data Streaming Destination can be an adapter for you to read that data from inside SQL Server.
Related
I have a question that might be very basic.
If I have a SSIS package where I have the Transfer DB task to transfer a database from a server with a SQL Server 2016 Instance to another server with an Instance SQL Server 2019,
Do I need to install SSIS for each instance on each server?
Do I need to create the catalog in both source and destination instances? or is it enough to create it only on the source instance? or it has to be on the destination?
Thanks!
SSIS is an ETL tool, so you'll install SSIS on the server that you want to designate as your ETL server. There are several common scenarios for where that server is located.
A stand-alone ETL server, which is neither the source nor the target of the current ETL job you're working on.
A separate ETL server that is an instance on the same server as either your current source or your current destination.
Your current source server
Your current destination server
You'll have to decide which scenario makes the most sense in your environment. At my current gig, we have a few dedicated ETL boxes, and we cluster the jobs on them by subject matter.
Previously, I worked in an environment where 10% of the ETL was importing data and 90% was manipulating it on one server. In that situation, it made sense to have the SSIS stuff all on the destination server so that we weren't moving data over the wire to transform it.
As your situation evolves, and you use more and more SSIS, you might change your mind. That's fine. It's not a trivial thing to change, but it's also not set in stone once you plunk SSIS down someplace.
This must be simple but I can't find the documentation. I have an ODBC connection to an Intersystems Cache' database that works fine in MS Access, but I can't see how to do the same in SQL Server (15.0.18930.0).
In Access, I did this: Blank Desktop Database, Create, External Data tab, ODBC Database, Link to the data source, (Select Data Source) --> Machine Data Source --> click on ODBC data source we already created. Then it gives me a list of tables to link to, and we're done.
The data source was System DSN, with various fields and a connection string like
DRIVER={InterSystems ODBC35};SERVER=xxxx;PORT=xxx;DATABASE=xxx;UID=xxx;PWD=xxx;
How is this done with SQL Server?
Thanks!
How is this done with SQL Server?
For SQL Server, write an SSIS package that uses the ODBC driver. Note that you may need to run the package in 32bit mode if the ODBC driver is 32bit. Then use that package to load a SQL Server table, or use an SSIS Data Streaming Destination to enable the SQL Server to dynamically run and read data from the SSIS package.
I have this scenario which has to be solved by SSIS:
I have multiple instances of the same DB (they may be on different servers)
I have one Master DB on a different server
now I need to transfer data from these different instances to Master DB using SSIS
My question is - where exactly SSIS needs to be installed? On Master DB server? On each of the source DB servers? Both?
It must be installed on the server it will be executed from, it can be a seperate server from master or clients or anyone of them.
Note that if you want to perform Bulk insert operation, then you should install it on master server.
Check the following links to get some insights:
The Data Loading Performance Guide
SQL Sever Destination vs OLE DB Destination
I have a .dmp file (oracle data) and I have to import this file into SQL Server 2008 R2. I tried google but get no clear solution. Oracle is on other machine and SQL Server is on other machine. This .DMP file has only tables and data only nothing else.
Any body has any idea?
You can't get there from here. The files that the Oracle export utility (classic or DataPump) generate (which, by convention, frequently use the DMP extension) are proprietary binary files. They can only be consumed by the Oracle import utility (classic or DataPump) which will only allow you to load the data into another Oracle database.
You could load the DMP file into a new Oracle database but then you'll still need to move the data from Oracle to SQL Server. It may well be easier to ignore the DMP file and pull directly from the original Oracle database. There are a variety of tools that can be used to move data from an Oracle database to a SQL Server database. If you want SQL Server to control the process, you could SQL Server Integration Services (SSIS). You could also create a linked server in SQL Server that references the Oracle database and write queries against the Oracle database via that connection. If you wanted Oracle to push the data, you could also use the Oracle Transparent Gateway with Heterogeneous Services to create a database link from Oracle to SQL Server and issue SQL against the remote SQL Server database.
There is a nice StackOverflow thread on moving data from Oracle to SQL Server. The SSIS logic is extremely similar if you're pulling from Oracle to SQL Server or pushing from SQL Server to Oracle.
Oracle Dumps is not readable by SQL.
Simply it cannot be, But you have different solutions
SQL server integration services (SSIS)
Link between oracle and SQL (Oracle Gateway) but it works with SQL Ent.
Export the data from oracle in a delimited format and insert it into SQL, but it will takes time if data is huge.
When I faced with the same problem, I tried to investigate the format manually (in my case the dump file was generated by Oracle EXP). I found that:
Table definitions come as Oracle CREATE TABLE statement that can be converted into MS SQL format easily
Most kind of data ca be extracted quite easy (text goes "as is", numeric values are stored according to IEEE 754 format)
LOBs are stored in quite complicated way, I failed to recognize it
Then I found the tool that was able to do my migration task: https://www.convert-in.com/ord2mss.htm
Vendor said that it can migrate both exp and expdp to sql server, but I have tested it on EXP format only.
What are the most reasonable ways to move table data from SQL Server to Oracle (on *nix) on a regular basis?
Using SQL Server Integration Services (SSIS) is likely your best bet. If you're not familiar with SSIS, the best way to try something out is to use the SQL Server Export Wizard and have it create an SSIS package for you. For example, if you go into SQL Server Management Studio and right-click on your database, then select Tasks->Export Data. From there click next until you get to the "Choose a Destination" step. Select the "Microsoft OLE DB Provider for Oracle" and click Properties to define your database connection. When you click through the wizard, on the Save and Execute page, make sure you check the checkbox labelled "Save SSIS Package", on the next screen specify where to save the SSIS package. Once you finish the Export Wizard, your data will have been exported and you will have an SSIS package that you can use as is, or go in and tweak it to do more specific things.
Once you have your SSIS package, you can schedule it by creating a SQL Server Agent Job.
Oracle Heterogeneous Connectivity / Database Gateways in conjunction with materialized view(s), PL/SQL, or Java
SSIS or DTS: both can be scheduled but require more than read-only access to SQL Server
Java (probably within Oracle but optionally at the OS) using ODBC or SQLJ to access SQL Server and, possibly, Oracle
SQL Server scheduled to export to CSV, Oracle scheduled to import from CSV
Any of the other ETL tools (e.g. Informatica, Cognos)
Any of the myriad languages that can access both databases (but would require maintaining a third environment to run the application within)
Scheduling:
Automatic / not reuqired with materialized views
Oracle DBMS_JOB / DBMS_SCHEDULER
OS-specific (cron, Windows Scheduled Tasks, etc)
In the case of SSIS, DTS, or CSV export, scheduled within SQL Server
Create a database link from Oracle to Sql Server (heterogeneous connectivity). You can use this link to retrieve the data from Sql Server with a simple select statement. If you want to schedule you can use a materialized view or dbms_scheduler.
An alternative is to put your data in a csv file, you can use an external table or sqlloader to load this data in the Oracle database.
Here is what I do:
Connect to SQL Server by Oracle SQL developer using this link:
https://kentgraziano.com/2013/01/14/tech-tip-connect-to-sql-server-using-oracle-sql-developer/
After you have added SQL jar, you will see SQL Server tab in the Connection window:
Then connect to the SQL Instance.
Then open SQL instance and choose database or table that you want to copy. Right click on any database/table, then click on "copy to oracle" there and choose the right user[database] at "Destination Connection Name " where you want to copy your tables.
You can also change some properties there.
Click "OK" and that's it.
Let me know in case of any issues.
You can have SQL Server interface with Oracle directly through SSIS (or DTS for 2k). It will provide ETL functionality and can be scheduled on a regular basis.
I had success creating a linked server (from within the Enterprise Manager, I think) to Oracle on the SQL Server side. Then I could use normal stored procedures on both sides to accomplish smaller data movement and updates in both directions. This approach can bypass the need to try and put something together outside the databases.
Try hard to use the latest possible Oracle client on the SQL Server side though. I recall some defects in the 10.2.0.2 client and getting the 10.2.0.4 client required your "official" Oracle registration or purchase number or something.
For big data moves (or maybe even moves/updates you want done on a daily or less frequent basis), definitely use one of the ETL tools. We had Informatica for our ETL processes, but if SSIS can pull off what you need, that's fine too.