I have a data factory in Azure that has a pipeline to execute an SSIS package. An extract script is included in this package that will convert a date value from 2019-01-13 00:00:00 to 2019-01-13. After this, a logic app picks up the extract.txt file and sends it to sftp to drop into a folder.
As you can see from my extract script in SSMS converting a varchar data type to date (I receive date in Unix), for Arrival date the following value is given.
Script
Data viewer
However when I open my extract.txt file, the date shows as: 2019-01-13 00:00:00.0000000
Can anyone tell me why they think this may be happening? Thanks
When you map a date column to flat file it will include time even if it doesn't appear in data viewer or ssms.
SSIS solution
Instead of using dateadd() in the source, leave the column as it is, in SSIS use a derived column with the following expression:
LEFT((DT_WSTR,50)DATEADD("s", [start], (DT_DATE)"1970-01-01",10)
Output
2019-01-13
Based on the Cast (SSIS Expression) official documentation:
When a string is cast to a DT_DATE, or vice versa, the locale of the transformation is used. However, the date is in the ISO format of YYYY-MM-DD, regardless of whether the locale preference uses the ISO format.
SQL Server solution
Use the following SQL command:
select convert(varchar(10), dateadd(S, [start], '1970-01-01'),120)
Related
I am using SSIS to convert some oracle data to sql server. I found for Oracle date timestamp6 like this
26-DEC-82 12.00.00.000000000 AM
will cause the conversion in SSIS to fail
Error: Year, Month, and Day parameters describe an un-representable
I think it's because ssis don't know whether it's 2082 or 1982, so don't know how to convert.How can I convert the oracle dates to something with yyyy for the year part?
update: tried to_char function mentioned by Hadi. I can see the year now is 2682 (most of them). I added a pic showing with to_char and original column for plate_date and sold_date columns. As you can see most of the years are 26xx, two exceptions are 18xx. Can someone explain?
In the Oracle Source use an SQL Command to convert TimeStamp to nvarchar using TO_CHAR() function and use the universal data format yyyy-MM-dd HH:mm:ss:
TO_CHAR(SOURCECOLUMN, 'YYYY-MM-DD HH24:MI:SS')
And in SSIS data flow add a derived column with the following expression:
(DT_DATE)[SOURCECOLUMN]
Or add a data conversion transformation and convert the column to date data type.
In SQL Server, the datatype "timestamp" is nothing to do with dates or times.
Microsoft have renamed their old "timestamp" datatype to "rowversion" because it is just an 8-byte number that is used to record a sequence of “row changed” events.
On the other hand Oracle's "timestamp" really is about time because Oracle's "timestamp" extends their "date" datatype. more here.
Unfortunately, SQL Server still recognises "timestamp" as a valid datatype name.
So, I suspect that your error message may have something to do with the timestamp-timestamp homonym.
When I use T-SQL to convert a datetime into dd.mm.yyyy for an csv output using SSIS, the file is produced with a dd-mm-yyyy hh:mm:ss which is not what i need.
I am using:
convert(varchar,dbo.[RE-TENANCY].[TNCY-START],104)
which appears correct in SSMS.
Which is the best way to handle the conversion to be output from SSIS?
Not as simple as i thought it would be.
It works for me.
Using your query as a framework for driving the package
SELECT
CONVERT(char(10),CURRENT_TIMESTAMP,104) AS DayMonthYearDate
I explicitly declared a length for our dd.mm.yyyy value and since it's always going to be 10 characters, let's use a data type that reflects that.
Run the query, you can see it correctly produces 13.02.2019
In SSIS, I added an OLE DB Source to the data flow and pasted in my query
I wired up a flat file destination and ran the package. As expected, the string that was generated by the query entered the data flow and landed in the output file as expected.
If you're experiencing otherwise, the first place I'd check is double clicking the line between your source and the next component and choose Metadata. Look at what is reported for the tenancy start column. If it doesn't indicate dt_str/dt_wstr then SSIS thinks the data type is date variant and is applying locale specific rules to the format. You might also need to check how the column is defined in the flat file connection manager.
The most precise control on output format of the date can be achieved by T-SQL FORMAT(). It is available since SQL Server 2012.
It is slightly slower than CONVERT() but gives desired flexibility
An example:
SELECT TOP 4
name,
FORMAT(create_date, 'dd.MM.yyyy') AS create_date
FROM sys.databases;
name create_date
--------------------
master 08.04.2003
tempdb 12.02.2019
model 08.04.2003
msdb 30.04.2016
p.s. take into account that FORMAT() produces NVARCHAR output, which is different from your initial conversation logic, therefore extra cast to VARCHAR(10)) perhaps will be necessary to apply
Our business would be providing us a .csv file. One of the columns in the file would be in date format. Now as we know there are many date formats in Excel. The problem is that we need to check whether the date provided is a correct date. It could be in any format like ddmmyyyy, yyyymmdd, dd-mon-yyyy etc basically any format that Excel supports.
We are planning to first load the data in a staging area and the date field would be defined as varchar so that it can accept any data.
Now either using SSIS or via T-SQL, I need to check whether the date provided is actually a date and if it is I need to load it into a different table in YYYYMMDD format.
How do I go about doing the above?
Considering you have your excel data already loaded into a SQL Server table as varchar (you can easily do this using SSIS), something like this would work:
SELECT
case when ISDATE(YOUR_DATE) = 1 then CONVERT(int,YOUR_DATE,112) else null end as MyDate
FROM
YOUR_TABLE
I don't have access to a SQL Server instance at the moment and can't test the code above, so you may need to adapt to your needs, but this is the general idea.
You can also do further research on ISDATE and CONVERT functions in SQL Server. You should be able to achieve what you need combining them together.
I am sorry to post what seems a very simple issue but I cannot find an answer and I am wasting days (not just hours at this point). I am fairly new to SSIS and it is just kicking my backside.
Background:
Pretty straightforward SSIS Package to import an Excel sheet into a Staging table in SQL Server. Since I do not want to mislead anyone by using the wrong nomenclature, I will refer to the Excel source as Excel and the SQL Server table as the Target Table.
This package HAS worked before. However, it is now failing because of data truncation for a Date Column. The Excel Column has been formatted as a DATE (and I have tried a few different format options within DATE). The target column is also a DATE column (NOT datetime). The data in Excel is predominantly empty cells with a few sporadic values. I think the errors started when the dates started appearing in the data (rather than just blanks).
I have tried using the Advanced Editor for both sides (Excel & Target) and tried numerous data type settings all around but I keep getting the same failure. I suspect that it is now pretty messed up with the various tests that I have done.
I have also tried adding a Data Conversion transform for the Date field “date[DT_DATE]” – that did not work. AND, I have tried creating a Derived Column - first based on the Excel column and then on the Transformed column. All of those attempts have failed.
Questions:
1) What is the best practice for importing Excel data into SQL Server for DATE Columns?
2) Since this is two very mature Microsoft Apps (Excel & SQL Server) working together, it seems like it should be simple. This leads me to believe that I must be missing some basic concepts here. Can anyone set me straight?
3) How do all of you get an Excel date into SQL Server?
4) What is the trick for synchronizing columns after making edits?
Thanks for any insights you can provide. Sorry to bother you all with what seems pretty simple.
David
Personally I don't think there is a best practice for excel dates, it is always a pain for me.
If you can format the excel file try changing it to 'Text'. it will import as Unicode and not a date. If not, try and convert the column in a "Data Conversion
" task to Unicode
after that is done, you would need to use a "Derived Column" task. Build the date in the format you want.
example for source MM/dd/yyyy hh:mm:ss
Build to be yyyy-MM-dd
SUBSTRING(datecolumn,7,4)+ "-" + SUBSTRING(datecolumn,1,2)+ "-" +SUBSTRING(datecolumn,4,2)
Might be crude, but saves my sanity.
If the date looks something like m/d/yyyy not including 2 values when Jan or something, you will add a few things like this for the month part.
RIGHT("0" + SUBSTRING(datecolumn,1,FINDSTRING(datecolumn,"/"1)-1),2)
Good luck
The main problem when importing Data from excel worksheets is that excel is that each column in excel can have multiple data types or formats, so the same column can contains Dates and Numbers and text or dates with differents formats (Some formats cannot be converted implicitly to dates in SSIS).
If all date value are stored as date (not Text), The best practice to Import dates from an excel worksheet is to convert DATE to Number format "0.000000000" (in excel it is called Serial DateTime) from excel or programmatically using a library like Microsoft.Office.Interop.Excel
You can refer to this Link but use the following:
xlCells.NumberFormat = "0.0000000"
Then in SSIS package use a script component to convert it again to Date using DateTime.FromOADate() Function
*Assuming that inColumn is the Date column with a numeric type, add an output column outColumn of type DT_DBTIMESTAMP or DT_DATE and use the following code:
If Not Row.inColumn_IsNull Then
Row.OutColumn = DateTime.FromOADate(CDbl(Row.inColumn))
Else
Row.OutColumn_IsNull = True
End If
Note: When converting column to Number Format, you ignored all formats but still have the date value
To read more about DateTimes in Excel you can refer to this Link
To read more about Date time formats that can be implicitly converted to date in SSIS follow SSIS Source Format Implicit Conversion for Datetime
I'm importing dates from a German SQL Server table into a German Excel file via the built-in Excel connection tool.
However the date format is just like in the SQL Server: 2012-08-08 but I want to display: 08.08.2012. When I double-click inside a cell it will recognize the German date formatting but of course I would like to have that format for the entire column without having to manually change it.
I also need to be able to use these dates for calculations.
Do I need to change something in SQL Server or how do I make this work?
Thank you.
The problem is that Excel does not recognise the SQL Server Date type. Cast the date to a Smalldatetime or Datetime, then import and format.
As long as it's recognized ad a date, you can set the formatting of the entire column to a custom one and set the value of the custom format to dd.mm.yyyy (or mm.dd.yyyy, depending on which you want).
Not a programming answer, but it will get you the display you want just by setting the custom format of the whole column.