I am attempting to transform DateTime columns from PostgreSQL to DateTime columns in SQL Server.
Here are my options:
Advanced editor input date options
When I choose database time, it appears to give me the DB timestamp at the time that I query the data (I.E. 2022-10-13 00:00:00.000 - on the 13th of October and if I run it the next day 2022-10-14 00:00:00.000 for EVERY column).
When I choose DT_Date as the output and input data types - I get "1899-12-30" for every column.
What am I missing, why can't I get the data from the source as it is and just transform to my destination?
Because the source DateTime has "AM" and "PM" in the actual data, the only way that I've gotten this to work is to make my destination a VarChar and convert (in SSIS) from DateTime to string and then put the data away as a string.
Is there another option that will take the data as it sits and transform it to my destination in its original form?
Thanks!
Related
As part of my job duties, I'm responsible for extracting data from our vendor's Oracle 11g database, and loading it into our SQL Server 2016 database. I've been doing this successfully with SSIS and the Attunity Oracle connectors.
Today I was informed that there was a new column added to the existing Invoices table on the Oracle side. There was already a DATE column called Order Date, which contains valid date values with zero'd times, like 2017-12-25 00:00:00.
The new column is called Order Date Time and is also a DATE column. When I opened up the SSIS package and pulled up the Oracle source in my DFT, I previewed the data and found the values in Order Date Time to be 2432-82-75 50:08:01. I tried converting the column with CAST and all the TO_* functions, but the conversions either failed outright, or returned a string of zeros.
TO_CHAR("Order Date Time", 'YYYYMMDDHH24MISS')
yields 00000000000000
After a bit of Googling for "Oracle date value invalid", I'm now thinking that these DATE values are actually corrupted. Am I missing anything here? Is there some sort of special Oracle-specific technique for storing time values in a DATE column that I may not be aware of?
(And yes, it does bother me quite a bit that our vendor added another DATE column instead of just using the time portion of the existing Order Date column.)
Unfortunately, Oracle database engine allows inserting invalid date values, which leads to many problems especially when importing data to others database engines such as SQL Server.
To handle this issue, you have to implement the logic that fits your needs, as example:
You can exclude these records from you queries by filtering on acceptable date ranges: (WHERE date between ...)
You can Update records with invalid values by replacing with NULL
You can use a CASE statement in your query to replace values with NULL
I faced this issue one time while importing data to SQL Server from an Oracle data source, there was unacceptable date values, i decided to update all records where date are invalid and replace with NULL values before starting the import process.
There are many links related to this issue:
Detecting invalid values in the DB
How to identify invalid (corrupted) values stored in Oracle DATE columns
Corrupt date fields causing query failure in Oracle
Invalid Date in DATE Column SQLPlus VS SQLDeveloper
Ask Tom - date validation in oracle
Dealing with invalid dates
Error: Invalid date format
DB Connect; Oracle DB date field data is corrupt
Our business would be providing us a .csv file. One of the columns in the file would be in date format. Now as we know there are many date formats in Excel. The problem is that we need to check whether the date provided is a correct date. It could be in any format like ddmmyyyy, yyyymmdd, dd-mon-yyyy etc basically any format that Excel supports.
We are planning to first load the data in a staging area and the date field would be defined as varchar so that it can accept any data.
Now either using SSIS or via T-SQL, I need to check whether the date provided is actually a date and if it is I need to load it into a different table in YYYYMMDD format.
How do I go about doing the above?
Considering you have your excel data already loaded into a SQL Server table as varchar (you can easily do this using SSIS), something like this would work:
SELECT
case when ISDATE(YOUR_DATE) = 1 then CONVERT(int,YOUR_DATE,112) else null end as MyDate
FROM
YOUR_TABLE
I don't have access to a SQL Server instance at the moment and can't test the code above, so you may need to adapt to your needs, but this is the general idea.
You can also do further research on ISDATE and CONVERT functions in SQL Server. You should be able to achieve what you need combining them together.
Using SQL Server 2014 SSIS to import an vendor supplied Excel file through the Excel Source Data Flow. Two issues I'm having related to data conversion to the SQL table.
In the file is a text column that has prices (numeric values) in it I can't not get it to transform into a numeric field (decimal(8,2)) in SQL. I have used the Data Conversion data flow task converting it to DT_NUMERIC and it fails to process the field. I have also tried to let it go through the Data Conversion task and converted through a Derived Column casting the field to Numeric. Both fail, I'm at a loss as to how to get this into the database in a Decimal/Numeric format.
In the same file are three date fields with dates that look like 07/18/2015 in Excel. I have tried similarly with the Data Conversion and Derived column to get the date into the database as SQL date formats. I have cast the dates at DT_DBDATE and DT_DBDATE and DT_DBTIMESTANP and neither has worked I have also tried taking the month day and year and rearranging them into the SQL date format with Substring/left/right functions to split the string. Also to no avail.
Here is what I tried:
Excel Source ---> Data Conversion ----> Derived Column -----> OLE DB Destination
In the excel source it recognized the date as text, I leave that be in the data conversion to deal with it in the Derived Column where I have tried.
a. (DT_DBDATE)("20" + RIGHT(TRIM(sale_start),2) + "-" + LEFT(TRIM(sale_start),2) + "-" + SUBSTRING(TRIM(sale_start),4,2)) - I have done this with and without the trim with same results. I have also used Right(sale_start,4).
b. (DT_DBDATE) sale_start
The SQL table is data type DATE. I have also changed it to DATETIME and used DT_DBTIMESTAMP in place of DT_DBDATE above.
I can't change the file I'm receiving it needs to process into the database the way it comes from the vendor. Looking at the data in excel there seems to be no reason it wouldn't be ok.
Any direction on bringing this data in would be much appreciated.
2.
I was able to figure this out although I don't completely understand what the connection setting is doing. Similarly with a XML file this connection setting wasn't necessary although some version of a derived column was, I the above I believe in my XML import.
For the EXCEL Solution:
1) In the Excel File connection I added IMEX=1 to the end of the connection under properties. So the connection string looked like this:
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\SSIS\Test.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";
2) Used the following script in the derived column:
ISNULL([Copy of expected_date]) ? NULL(DT_DATE) : (LEN(TRIM([Copy of expected_date])) == 0 ? NULL(DT_DATE) : (DT_DATE)((DT_DBDATE)TRIM([Copy of expected_date])))
Thanks for taking the time to respond.
There is table with timestamp column in SQL Server 2008 R2. When I only add this column to my table I see values like this 0x00000000000007D1. I try to put data into it:
UPDATE test_time SET date3=
CONVERT(TIMESTAMP, CONVERT(datetime,'2002-08-20 14:00:00.000',120))
WHERE ogr_fid=1
But get error
Cannot update timestamp column
What's wrong here?
SQL Server's TIMESTAMP datatype has nothing to do with a date and time!
It's just a binary representation of a consecutive number - it's only good for making sure a row hasn't change since it's been read.
In never versions of SQL Server, it's being called RowVersion - since that's really what it is. See the MSDN docs on ROWVERSION:
Is a data type that exposes automatically generated, unique binary numbers within a database. rowversion is generally used as a mechanism
for version-stamping table rows. The
rowversion data type is just an incrementing number and does not
preserve a date or a time. To record a date or time, use a datetime2
data type.
So you cannot convert a string to a TIMESTAMP in SQL Server.
I am trying to import a column of dates from a spreadsheet in Excel 2003 into SQL Server 2005 using SSIS. I am in the UK so want dates formatted as dd/MM/yyyy.
Unfortunately, the column in the spreadsheet contains a mixture of dates stored as strings in dd/MM/yyyy (with Excel 'General' formatting) as well as dates using Excel 'Date' formatting dd/MM/yyyy (with locale 'English (United Kingdom)').
This is just the way it is and I can't expect the users to be able to sort this out themselves.
When looking at the spreadsheet, all of the dates visually appear correct i.e. dd/MM/yyyy.
I am trying to import the values into a varchar column in a holding table in the database. Then I run a stored procedure that copies these values into the proper table which contains a datetime column.
The problem is that the dates that are stored in the spreadsheet and use Date formatting get imported as MM/dd/yyyy into SQL Server and the dates stored as strings are getting imported as dd/MM/yyyy. I have IMEX=1 in the connection string.
Having dates using both formats in the same varchar column is obviously causing a problem when I try to put it into a datetime column, hence
SET DATEFORMAT MDY;
SET DATEFORMAT DMY;
are of no use.
Does anyone know why the SSIS package would import the seemingly correct dates in the Excel spreadsheet into SQL Server as MM/dd/yyyy anyway?
Is there anyway to force the SSIS package to import the dates as dd/MM/yyyy that will work with this combination of dates as strings and cells with date formatting applied?
Any other ideas?!
Many thanks,
Anthony
I think you have answered your own question. The import of date formatted cells are treated as dates and others as a string. Possibly you SQL server date setting is MM/dd/yyyy and hence the conversion.
Why don't you try adding a data conversion step in you SSIS package and convert everyting in the column into a single format - datetime or string. Then I am sure SQL server will handle all of them the same way.
Raj
What worked for me was to add IMEX=1 to the Excel connection string.
So it will look like this:
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\Folder1\Book1.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";