Just wondering what is best practice to achieve:
We have stored procedure that runs in SQL Server and needs to do some calculations with a huge fact table to be stored on Netezza.
The flow:
Stored procedure will create temp tables on SQL Server
These will be sent to Netezza to be joined with the fact table
Calculations will be made in Netezza
Results will be passed back to SQL Server
What are the ways to pass the temp tables from SQL Server to Netezza?
Thanks
The Only way that I know of to get data in and out of Netezza is through flat files, or using the ODBC drivers to work with Netezza directly and hold a recordset in memerory and write to Netezza through an odbc connection..
Regardless of if you are on Windows or Unix, Netezza comes with client tools that you can use to connect to Netezza. Fyi, don't bother looking for them on the internet. You will have to get them from Netezza directly, or from the person that manages that relationship.
I would suggest looking into how I might be able to use the ODBC drives in SSIS to do the work for you. I'm not a pro at SSIS so I can't say I would know how to do that, but I would look into that first.
If I had to accomplish the task I would write something in C# to perform the following tasks.
Create flat files from sql server
Connect to Netezza Create external table that links to the flat file.
Call procedures in Netezza to do the work and generate the data for export in a temp table.
Export the new data to a flat file and import that back into sql server.
Now that I think about it you might also try the following, it is untested however. I wonder if you can create a linked table in sql server and an external table in Netezza that uses the same flat file. Baring a file lock, if they can, you can create a quasi-link to netezza from sql server.
To find out more about external tables in netezza. Look in the doc Netezza User Guide in chapter 5.
Netezza User Guide
Also, if you are interested in the coding side there is a very good link below to how to connect to netezza via c#.
Stack Post
I ended up using some of that post to build the method below to execute commands against Netezza.
OdbcDataReader GetReaderForCommand(string strCmd, string dbname)
{
var conn = new OdbcConnection();
conn.ConnectionString = "Driver={NetezzaSQL};servername=<servername>;port=5480;database="+dbname+"; username=<username>;password=<pwd>;";
OdbcDataReader rdr = null;
conn.Open();
System.Data.Odbc.OdbcCommand cmd = new System.Data.Odbc.OdbcCommand(strCmd, conn);
rdr = cmd.ExecuteReader();
return rdr;
}
Lastly, here are a couple of links that I would follow up with.
enzeecommunity.com - User base to search and ask questions of.
A free management studio to use with Netezza.
http://www.aginity.com/ProductivityTools/WorkbenchOverview.aspx -
Related
I need a bit advice how to solve the following task:
I got a source system based on IBM DB2 (IBMDA400) which has a lot of tables that changes rapidly and daily in structure. I must load specified tables from the DB2 into a MSSQL 2008 R2 Server. Therefore i thought using SSIS is the best choice.
My first attempt was just to add both datasources, drop all tables in MSSQL and recreate them with a "Select * Into #Table From #Table". But I was not able to get this working because I could not connect both OLEDB Connections. I also tried this with an Openrowset statement but the SQL Server does not allow that for security reasons and I am not allowed to change that.
My second try was to manually read the tables from the source and drop and recreate the tables with a for each loop and then load the data via the Data Flow Task. But I got stuck on getting the meta data from the Execute SQL Task... so i dont got the column names and types.
I can not believe that this is too hard to archieve. Why is there no "create table if not exist" checkbox on the Data Flow Task?
Of course i searched for the problem here before but could not find a solution.
Thanks in advance,
Pad
This is the solution i got at the end:
Create a File/Table which is used for selection of the source tables.
Important: Create a linked Server on your SQL Instance or a working Connectionstring for the OPENROWSET (i was not able to do so - i choosed the linked server)
Query source File/Table
Build a loop through the resultset
Use Variables and Script Task to build your query
Drop the destination table
Build another Querystring with INSERT INTO TABLE FROM OPENROWSET (or if you used linked Server OPENQUERY)
Execute this Statement
Done.
As i said above i am not quite happy with this but for now it should be ok. I will update this if i got another solution.
Working on a data data accuracy project. I have to find a way to compare data from a query from a SQL Server db with the data from a query from a MS Access db. The data on both db's should be identical, but sometimes there are errors. I have looked at data comparison tools but these seem to only be able to compare data from identical db vendors.
Is there a process that someone has used in the past to do this or an idea on how I might best approach this?
You can look at both data sets in Access, SQL, or Excel:
If the data set is small enough, I recommend Excel.
If you know SQL, you can export your Access data to text files, then do a Bulk Insert and get everything into SQL Server.
If you want to look at both data sets in Access, try this:
Go to your ODBC Data Source Administrator (searching for 'ODBC' from your Start menu should be sufficient)
Create a new System DSN connecting to your SQL Server db
Open your Access db (I'm using 2010, your version may be different)
Go to External Data->ODBC Database->Machine Data Source
Link to your tables of choice from your SQL Server
Query away!
I'm just getting into SQL-server 2008 R2 ssis , so forgive me if this is a basic question. I'm building an application using integration service.
Want to create (SSIS) package for importing an MS Access table into My-SQL database.
To create ssis , I don’t want to use Tasks...Import Data .
help me to import access table data to my-sql db.
If have any query plz ask ,thanks in advanced.
1 st you start with defining the connection managers.
You need one for MS Access and one for MySQL (I assume you use MySQL open source db engine as the destination).
Connecting to MySQL may be tricky. I know CozyRock's ODBC connection works with it: http://www.cozyroc.com/ssis/odbc-destination, but you can try creating an ODBC one as described here: http://blogs.msdn.com/b/mattm/archive/2008/03/03/connecting-to-mysql-from-ssis.aspx. Make sure both conn managers connect.
From this point on, drag and drop a new Data Flow Task (DFT).
Inside the add a source and consume your MS Access table, you may need to also add a Derived Column Transformation to convert your incoming columns to some other data types MySQL understands, and finally drag and drop the destination which will employ your MySQL conn. manager. Connect all. Map the columns appropriately and test. If works repeat the aforementioned in a 2nd DFT to cover the import of the other table.
I'm wanting to take data from a SQL Server table and populate a Oracle table. Right now, my solution is to dump the data into a Excel table, write a macro to create a sql file that I can load into Oracle. The problem with this is I want to automate this process and I'm not sure I can automate this.
Is there an easy way to automate populating a Oracle table with data from a SQL Server table?
Thanks in advance
I suppose it depends on your definition of "easy".
The most robust approach would be to either use heterogeneous connectivity in Oracle to create a database link to the SQL Server database and then pull the data from SQL Server or to create a linked server in SQL Server that connects to Oracle and then push the data from SQL Server to Oracle.
Yes. Take a look at MS SQL's SSIS which stands for SQL Server Integration Services. SSIS allows all sorts of advanced capabilities, including automated with Sql Server Jobs, for moving data between disparate data sources. In your case, connecting to Oracle can be achieved a variety of ways.
There are three ways to automate this:
1) You can do as Paul suggested and created an SSIS package that will do this and it can be scheduled via SQL Agent,
2) If you don't want to deal with SSIS, you can download the free SQL# (SQLsharp) CLR Library from http://www.SQLsharp.com/ and use the DB_BulkCopy Stored Procedure to do this in a T-SQL Stored Proc which can also be scheduled via SQL Agent. [note: I am the author of SQL#]
3) You can also set up a Linked Server from SQL Server to Oracle, but this has the draw-back of being a potential security hole. Of course, you could use an Oracle Login that only has write-access to that single table (or something similar to that).
There are lots and lots of ways to do it. Which you choose depends on your requirements.
Using Excel is fine if it's a one time thing.
If it's a once-in-a-while thing, then you could write a simple .NET app that uses a single DataSet and multiple DataAdapters to do the data dump. C# code example here.
if it's a regular thing, then you could put the above in a Schtasks task, or you could use SSIS. I think SSIS is an extra-cost option.
if the requirement is for "online access", then a linked database is probably appropriate.
I am using Enterprise Miner 6.2 and want to create a data source but my option is a SAS Table. How do I go about exporting SQL Server or Excel data into a SAS table?
SAS has many ways of connecting to and/or reading data from disparate sources. I haven't used Enterprise Miner, so I'm not sure which of SAS' methods are available to you directly from within EM, but it's likely there will be someone at your site who has some interface to Base SAS and who can help you/advise what data access products are installed and how you can use them.
For SQL Server data, SAS/Access to SQL Server or SAS/Access to OLE DB will allow you to read directly from SQL Server tables in place. Alternatively, someone could provide you with a dump of the data you need from the SQL Server database.
For Excel data, there are also SAS/Access products, but SAS also has native capabilities to read in the data if saved as, for example, a .csv or .txt file.
To help answer you further, perhaps can you come back with some details about what SAS products/interfaces are available to you?