I want to create an SSIS package with source as azure sql and destination as DB2 table.
I have to check if Id (int) in sql matches with Id (X12 basically varchar) in DB2, if it matches, I have to update name in db2 with the name in SQL
else insert new record in DB2.
Since, datatype of id does not match, I have to do a data type conversion- that is if id in sql is 1 .. I have to convert it to varchar with leading zeros - 00000000001
Can some one please provide a step by step approach?
and where exactly can I make this conversion.. in which task?
Related
I'm trying to work out a specific way to copy all data from a particular table (let's call it opportunities) and copy it into a new table, with a timestamp of the date copied into the new table, for the sole purpose of generating historic data into a database hosted in Azure Data Warehousing.
What's the best way to do this? So far I've gone and created a duplicate table in the data warehouse, with an additional column called datecopied
The query I've started using is:
SELECT OppName, Oppvalue
INTO Hst_Opportunities
FROM dbo.opportunities
I am not really sure where to go from here!
SELECT INTO is not supported in Azure SQL Data Warehouse at this time. You should familiarise yourself with the CREATE TABLE AS or CTAS syntax, which is the equivalent in Azure DW.
If you want to fix the copy date, simply assign it to a variable prior to the CTAS, something like this:
DECLARE #copyDate DATETIME2 = CURRENT_TIMESTAMP
CREATE TABLE dbo.Hst_Opportunities
WITH
(
CLUSTERED COLUMNSTORE INDEX,
DISTRIBUTION = ROUND_ROBIN
)
AS
SELECT OppName, Oppvalue, #copyDate AS copyDate
FROM dbo.opportunities;
I should also mention that the use case for Azure DW is million and billions of rows with terabytes of data. It doesn't tend to do well at low volume, so consider if you need this product, a traditional SQL Server 2016 install, or Azure SQL Database.
You can write insert into select query like below which will work with SQL Server 2008 +, Azure SQL datawarehouse
INSERT INTO Hst_Opportunities
SELECT OppName, Oppvalue, DATEDIFF(SECOND,{d '1970-01-01'},current_timestamp)
FROM dbo.opportunities
I have a SQL Server SSIS package that inserts data in a table according to data in another table. The thing is this job breaks the accents of the varchar data in the process. I suppose it has something to do with encoding.
Simplified, my package does the following :
Select data with an OLE DB Source through SQL command data access mode
SELECT id, name, lastname
FROM Client
<WHERE id = 1>
Insert the selected data with an OLE DB Destination through mapping to ClientCopy table.
I noticed that previewing the data in the first step was returning me the broken accents already. Because of this, the data inserted in ClientCopy obviously has those broken accents.
id name lastname
1 Andr‚ BriŠre
This same query when executed in SQL Server returns the data correctly so I'm a bit lost right now.
id name lastname
1 André Brière
Thanks for helping me out!
We have an Oracle table that contains archived data in the table format:
BLOB | BLOBID
Each BLOB is an XML file that contains all the business objects we need.
Every BLOB needs to be read, the XML parsed and put into 1 SQL Server table that will hold all data.
The table has 5 columns R | S | T | D | BLOBID
Sample XML derived from BLOB:
<N>
<E>
<R>33</R>
<S>1</S>
<T>2012-01-25T09:48:43.213</T>
<D>6.9534619e+003</D>
</E>
<E>
<R>33</R>
<S>1</S>
<T>2012-01-25T09:48:45.227</T>
<D>1.1085871e+004</D>
</E>
<E>
<R>33</R>
<S>1</S>
<T>2012-01-25T09:48:47.227</T>
<D>1.1561764e+004</D>
</E>
</N>
There are a few million BLOBs and we want to avoid copying all the data over as an XML column then to a table, instead we want to go BLOB to table in one step.
What is the best approach to doing this with SSIS/SQL Server?
The code below almost does what we are looking for but only in Oracle Developer and only for one BLOB:
ALTER SESSION SET NLS_TIMESTAMP_FORMAT='yyyy-mm-dd HH24:MI:SS.FF';
SELECT b.BLOBID, a.R as R, a.S as S, a.T as T, cast(a.D AS float(23)) as D
FROM XMLTABLE('/N/E' PASSING
(SELECT xmltype.createxml(BLOB, NLS_CHARSET_ID('UTF16'), null)
FROM CLOUD --Oracle BLOB Cloud
WHERE BLOBID = 23321835)
COLUMNS
R int PATH 'R',
S int PATH 'S',
T TIMESTAMP(3) PATH 'T',
D VARCHAR(23) PATH 'D'
) a
Removing WHERE BLOBID = 23321835 gives the following error ORA-01427: single-row subquery returns more than one row since there are millions of BLOBS. Even so is there a way to run this through SSIS? Adding the SQL to the OLE DB Source did not work for pulling the data from Oracle even for 1 BLOB and would result in errors.
Using SQL Server 2012 and Oracle 10g
To summarize, how would we go from a Oracle BLOB containing XML to SQL Server table with business objects derived from XML with SSIS?
I'm new to working with Oracle, any help would be greatly appreciated!
Update:
I was able to get some of my code to work in SSIS by modifying the Oracle Source in SSIS to use the SQL command code above minus the first line,
ALTER SESSION SET NLS_TIMESTAMP_FORMAT='yyyy-mm-dd HH24:MI:SS.FF';
SSIS doesn't like this line.
Error message with the ALTER SESSION line above included:
No column information was returned by the SQL Command
Would there be another way to format the date without losing data? I'll try experimenting more, possible using varchar(23) for the date instead of timestamp.
I have tried some examples but so far not working.
I have a Link Server (SQL Server 2014) to an Oracle 12C Database.
The table contain a datatype TIMESTAMP with data like this:
22-MAR-15 04.18.24.144789000 PM
When attempting to query this table in SQL Server 2014 via link server I get the following error using this code:
SELECT CAST(OracleTimeStampColumn AS DATETIME2(7)) FROM linkServerTable
Error:
Msg 7354, Level 16, State 1, Line 8
The OLE DB provider "OraOLEDB.Oracle" for linked server "MyLinkServer" supplied invalid metadata for column "MyDateColumn". The data type is not supported.
While the error is self explanatory, I am not certain how to resolve this.
I need to convert the timestamp to datetime2. Is this possible?
You can work around this problem by using OPENQUERY. For me, connecting to Oracle 12 from SQL 2008 over a linked server, this query fails:
SELECT TOP 10 TimestampField
FROM ORACLE..Schema.TableName
...with this error:
The OLE DB provider "OraOLEDB.Oracle" for linked server "ORACLE" supplied invalid metadata for column "TimestampField". The data type is not supported.
This occurs even if I do not include the offending column (which is of type TIMESTAMP(6). Explicitly casting it to DATETIME does not help either.
However, this works:
SELECT * FROM OPENQUERY(ORACLE, 'SELECT "TimestampField" FROM SchemaName.TableName WHERE ROWNUM <= 10')
...and the data returned flows nicely into a DATETIME2() field.
One way to solve the problem is to create a view in oracle server and convert the OracleTimeStampColumn compatible with sql server's datetime2datatype. You can change the time format to 24 hours format in oracle server's view and mark the field as varchar. Then you can convert the varchar2 column to datetime2 when selecting the column in SQL Server.
In Oracle Server
Create or Replace View VW_YourTableName As
select to_char(OracleTimeStampColumn , 'DD/MM/YYYY HH24:MI:SS.FF') OracleTimeStampColumn from YourTableName
In SQL Server
SELECT CAST(OracleTimeStampColumn AS DATETIME2(7)) FROM **linkServerVIEW**
I have one table in SQL server and 5 tables in Teradata.I want to join those 5 table in teradata with sql server table and store result in Teradata table.
I have sql server name but i dont know how to simultaneously run a query both on sql server and teradata.
i want to do this:
sql server table query
Select distinct store
from store_Desc
teradata tables:
select cmp_id,state,sde
from xyz
where store in (
select distinct store
from sql server table)
You can create a table (or a volatile table if you do not have write privileges) to do this. Export result from SQL Server as text or into the language of your choice.
CREATE VOLATILE TABLE store_table (
column_1 datatype_1,
column_2 datatype_2,
...
column_n datatype_n);
You may need to add ON COMMIT PRESERVE ROWS before the ; to the above depending on your transaction settings.
From a language you can loop the below or do an execute many.
INSERT INTO store_table VALUES(value_1, value_2, ..., value_n);
Or you can use the import from text using Teradata SQL Assistant by going to File and selecting Import. Then execute the below and navigate to your file.
INSERT INTO store_table VALUES(?, ?, ..., n);
Once you have inserted your data you can query it by simply referencing the table name.
SELECT cmp_id,state,sde
FROM xyz
WHERE store IN(
SELECT store
FROM store_table)
The DISTINCT function is most easily done on export from SQL Server to minimize the rows you need to upload.
EDIT:
If you are doing this many times you can do this with a script, here is a very simple example in Python:
import pyodbc
con_ss = pyodbc.connect('sql_server_odbc_connection_string...')
crs_ss = con_ss.cursor()
con_td = pyodbc.connect('teradata_odbc_connection_string...')
crs_td = con_td.cursor()
# pull data for sql server
data_ss = crs_ss.execute('''
SELECT distinct store AS store
from store_Desc
''').fetchall()
# create table in teradata
crs_td.execute('''
CREATE VOLATILE TABLE store_table (
store DEC(4, 0)
) PRIMARY INDEX (store)
ON COMMIT PRESERVE ROWS;''')
con_td.commit()
# insert values; you can also use an execute many, but this is easier to read...
for row in data_ss:
crs_td.execute('''INSERT INTO store_table VALUES(?)''', row)
con_td.commit()
# get final data
data_td = crs_td.execute('''SELECT cmp_id,state,sde
FROM xyz
WHERE store IN(
SELECT store
FROM store_table);''').fetchall()
# from here write to file or whatever you would like.
Is fetching data from the Sql Server through ODBC an option?
The best option may be to use Teradata Parallel Transporter (TPT) to fetch data from SQL Server using its ODBC operator (as a producer) combined with Load or Update operator as the consumer to insert it into an intermediate table on Teradata. You must then perform rest of the operations on Teradata. For the rest of the operations, you can use BTEQ/SQLA to store the results in the final Teradata table. You can also put the same SQL in TPT's DDL operator instead of BTEQ/SQLA and get it done in a single job script.
To allow use of tables residing on separate DB environments (in your case SQL-Server and Teradata) in a single select statement, Teradata has recently released Teradata Query Grid. But I'm not sure about exact level of support for SQL-Server and it will involve licensing hassle and quite a learning curve to do this simple job.