I am using a flat file data provider in SSIS to import data from an external system. I don't have any control over the file, it is pushed out on a weekly basis, and I pick it up from a common folder.
The first two columns of the CSV are dates. Part of the way through the file, the date format has changed from date to numeric as follows:
Service_Date, Event_Datetime
2018-04-30,2018-04-30 21:18
43220,43220.92412
As you can see, the format changed from date to numeric. Other date columns not shown here also have changed.
Obviously, this is breaking the data flow task.
Aside from going into Excel and saving the CSV with the proper column format, is there any way within SSIS can convert on the fly so that the job doesn't fail and require manual intervention?
These data values 43220,43220.92412 are called date serials, you can get the date value in many approaches:
(1) Using A derived Column
You can convert this column to float then to datetime using a derived column:
(DT_DATE)(DT_R8)[dateColumn]
References
convert Excel Date Serial Number to Regular Date
Is there a better way to parse [Integer].[Integer] style dates in SSIS?
(2) Using a script component
You can use DateTime.FromOADAte() function, as example: (code in VB.NET)
If Row.ServiceDate_IsNull = False AndAlso String.IsnullOrEmpty(Row.ServiceDate) Then
Dim dblTemp as Double
If Double.TryParse(Row.ServiceDatemdblTemp) Then
Row.OutputDate = DateTime.FromOADate(dblTemp)
Else
Row.OutputDate = Date.Parse(Row.ServiceDatemdblTemp)
End
End If
Reference
SSIS Script Task - VB - Date is extracting as INT instead of date string
I was able to solve the problem using a variation of the derived column. This expression would catch the column obviously formatted as a date, and convert it to a date, otherwise it converts the date serial to a float first, then to a date
FINDSTRING(Date_Service,"-",1) != 0 ? (DT_DATE)Date_Service : (DT_DATE)(DT_R8)Date_Service
Related
I have a situation where I am getting dates in two separate formats, MM/dd/yyyy & yyyy-dd-MM, AND there might be even more different formats as well in csv which will be obviously in string.
Below are the data which currently come as String from CSV-
1/14/2022 0:00
2021-12-31 00:00:00
I am using a Dataflow task in ADF to load the data into Azure SQL where the default format it uses should be yyyy-MM-dd HH:mm:ss.
how can I do this?
ok, i managed to build a quick demo.
Main idea of my solution:
you need to differentiate between valid rows and rows that needs to be modified.
in order to do so, i used case condition.
the idea is to add a derived column with a name 'Date' and modify only needed rows.
Input Data:
i created a csv file and saved my data as a dataset in ADF.
ADF:
In source, i select my dataset as an input.
in a derived column activity:
added a new derived column with a name 'Date' , value :
case(contains(split(Date,''),#item=='/'), toString(toTimestamp(Date,'MM/dd/yyyy H:mm'),'yyyy-MM-dd HH:mm:SS'), Date)
in toTimestamp method, i added first the dateFormat of my input Date and in toString the desired format that i want to cast the date to it.
Output:
P.s
You can cast all possible date formats that will appear in your data in that way.
you can read more about it here:
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-expressions-usage#toTimestamp
I am loading a flat file to a database table, and need to change the format of the date from YYYY-MM-DD in the flat file, to MM/DD/YYYY in the database table. I tried using the following statement in Derived Columns as shown below, but not sure how to configure the statement, so I got an error message stating that SSIS could not parse the expression.
Derived Column Name: EFF_DATE
Derived Column: Replace EFF_DATE
Expression: TOKEN( MONTH([EFF_DATE]),"//|",DAY([EFF_DATE]),"//|",YEAR([Copy of EFF_DATE]) )
DATA TYPE: databasetimestamp[DT_DBTIMESTAMP]
Can anyone help me determine how to change the format of the column in Derived Column? Otherwise, please let me know if there is another way to do it. Thank you.
This question was different from the last one. In the last question, the date column was data type DateTime. But in this question, the date is a string, and when I used the Derived Column to change the date from YYYY-MM-DD to MM/DD/YYYY, it kept the leading zeroes in MM and DD. The issue then became, not just changing the date format, but also removing the leading zeroes from the Month and Day.
However, I researched and came up with a better solution in SSIS for changing the date value with data type string, as the database I am working with stores the date in that format.
I removed the Derived Column from my Data Source Task, and added an Execute SQL Task in the Control Flow, then added the following Update statement which not only changes the format from YYYY-MM-DD to MM/DD/YYYY, but also removes the leading zeroes from Month and Day. The CONCAT function I used the sample SQL below changes the format from YYYY-MM-DD to MM/DD/YYYY, while the Convert function changes the MM and DD values to data type INT which removed any leading Zeros. This solution allowed the date to remain a string, as that was the table format I had to work with.
UPDATE [StagingTable]
SET START_DATE =
CONCAT( CONVERT(INT, SUBSTRING(START_DATE, 6,2)), '/', CONVERT(INT, RIGHT(START_DATE, 2)),'/', LEFT(START_DATE,4) )]
Thanks to everyone for their comments, as it helped me to think outside the box and determine this solution.
I am trying to load some data from a .csv file into my SQL Server. There is a column for date which has datatype Unicode (WSTR) in the .csv file and the column for storing that date in SQL Server is of Datetime data type.
When I used DATA CONVERSION transformation to convert WSTR data to DBTIMESTAMP data, it got changed but with an error that it interchanged the month and date which gives me the wrong date.
The date should be like 2019-09-03 (for 3rd Sep 2019), but I get 2019-03-09.
Please suggest what the issue is that I am facing?
Problem
This may occurs when converting a string to a date value without specifying the date format. Reffering to the SSIS data conversion transformation official documentation:
If you are converting data to a date or a datetime data type, the date in the output column is in the ISO format, although the locale preference may specify a different format.
Assume that the data is stored in the csv file with the following date format MM/dd/yyyy and the default date format in the regional settings is dd/MM/yyyy then date values will not be converted properly.
Solution
In order to specify the date format you have to use a Script Component or a Derived column.
Derived Column
You have to reorder the date part manually, you can use TOKEN() and TOKENCOUNT() function to do that, the following expression convert dd/MM/yyyy format into yyyy-MM-dd universal format:
TOKEN([DateColumn],"/",3) + "-" + RIGHT("0" + TOKEN([DateColumn],"/",2),2) + "-" +RIGHT("0" + TOKEN([DateColumn],"/",1),2)
Script Component
You can use DateTime.ParseExact() function to parse a date based on a specific format:
DateTime.ParseExact(Row.DateColumn,"MM/dd/yyyy",System.Globalization.CultureInfo.InvariantCulture");
I have date values on an excel sheet in the dd/mm/yyyy format. Initially I catch those values as text (.Range(A2).Text)
Obviously the insert statement fails since the date values do not display in the mm/dd/yyyy format.
I have tried parsing the date values and assign them to a date variable but the Insert statement still failed. .net displays the date like that: #mm/dd/yyyy#. Is it the hash tag that generates the issue?
If I parse and then pass back to another string variable the value still displays in the dd/mm/yyyy format.
How can I get string values in the mm/dd/yyyy format?
Dates in SQL Server are stored in a neutral, binary format. When you insert a row in SQL Server, best practices say to use yyyymmdd, as in:
INSERT INTO MyTable(MyDate) VALUES ('20180801')
By default, SQL will parse the string (in the values clause above) according to the current locale settings. That is why the above, ISO format is preferred since it is unambiguous.
I am using Power Query to access my PostgreSQL database and filter my data by certain date parameters. However, for one of my database tables the date format is YYYYMM (201510).
Is it possible to convert this format to an actual date format?
Power Query recognizes YYYY-MM or YYYYMMDD as valid date formats, but not YYYYMM. Here's a solution inserting a hyphen then changing types:
Split the text by number of characters, 4
Delete the automatic number type inference step.
Merge the columns using a custom separator -
Change type to date
Here's a simple example:
let
Source = Csv.Document("201510
201502"),
SplitColumnByPosition = Table.SplitColumn(Source,"Column1",Splitter.SplitTextByPositions({0, 4}, false),{"Column1.1", "Column1.2"}),
MergedColumns = Table.CombineColumns(SplitColumnByPosition,{"Column1.2", "Column1.1"},Combiner.CombineTextByDelimiter("-", QuoteStyle.None),"Merged"),
ChangedType = Table.TransformColumnTypes(MergedColumns,{{"Merged", type date}})
in
ChangedType
Please try:
=DATE(LEFT(A1,4),RIGHT(A1,2),1)
select to_date('201510', 'YYYYMM');
to_date
------------
2015-10-01
In this case I prefer to create a new custom column and delete the original column afterwards. In your case the formula for the custom column would look like this:
=Text.Range([DateColumn],0,4) & "-" & Text.Range([DateColumn],4,2) & "-01"
After adding the custom column you can change it's format to Date.
For further reference check the M Formula reference: https://msdn.microsoft.com/en-us/library/mt211003.aspx