Is there any way to check Date(like isDate function in TSQL) column in SSIS package derived column expression after extraction from Sourcefile before loading to target dtabase?
Thanks
there is no built in function but you can run a script task and use vb.net code to check if the column is a date and operate on it as you wish...
I had a similar issue. I had a date/time in a text file, but the field had a dash between the date and the time. I created a derived column to do a replace on the dash:
REPLACE([TimeField], "- ", "")
I then added a convert column to convert the field to a date. I chose to ignore errors. I then added another Derived Column to check if the converted field was NULL, indicating that it could not convert the value. If it was, I had it set to the current date.
There is a a data conversion task you can drop in. Then redirect the rows as needed, either failing the import entirely or redircting the rows that don't work.
Or you could try a conditional split wher eyou cast the field to a date data type and then send the failures along another path (either deleting the records or nulling out the field would be the common action.)
See also http://www.sqlis.com/sqlis/post/Expression-Date-Functions.aspx > "IsDate workaround" for a technique that can be adapted
You can check whether your variable has a date or not using a conditional statement like this:
testDateVariable?true:false
For example, if date > 2 then it is true (and put the date, or format the date as you wish). If it is false, put null (you replace true with the date format and false with null).
All this is in a drived column in SSIS.
Related
I have a String field in a Dataset in (mmddyyyy) format.
I am trying to convert it into a Date field in SSRS.
I already tried using the below command but I am getting error.
CDate(Fields!LocalTXNDate.Value)
Can anyone please suggest.
While Larnu is correct, the way to do it is to correct the database, sometimes we lowly report makers have no say in making these changes - much less getting a DBA to do it in a reasonable amount of time.
If you can't change the data to be correct, the easiest way to convert and use the field as a date is to add a Calculated Field to the Dataset. Open the dataset properties, click on the Fields tab, Add a Calculated field.
For the Expression, use string functions to parse the field into a generic date format and then CDATE to convert to a date type. Then use the new field for dates. You could also use this in your text box if it's not being reused but it's easier to manipulate the Calculated field.
=CDATE(
RIGHT(Fields!LocalTXNDate.Value, 4) & "-" &
LEFT(Fields!LocalTXNDate.Value, 2) & "-" &
MID(Fields!LocalTXNDate.Value, 3, 2)
)
The problem here isn't SSRS but your data, and that you are using a string based data type to store the data. You need to fix the problem at the source, not at the report level.
The string format you have chosen, MMddyyyy isn't a format that is recognised by default in any of the languages in SQL Server, nor if you explicitly use SET DATEFORMAT, nor does it appear as a style. SET DATEFORMAT MDY; SELECT CONVERT(date,'11172022'); will fail. Therefore you'll need to first do some string manipulation on the data first to be an unambiguous format (yyyyMMdd):
UPDATE YT
SET YourDateColumn = CONVERT(varchar(8),V.DateValue,112)
FROM dbo.YourTable YT
CROSS APPLY (VALUES(TRY_CONVERT(date,CONCAT(RIGHT(YT.YourDateColumn,4),LEFT(YT.YourDateColumn,4)))))V(DateValue);
For any bad values you have, such as '17112022' this will UPDATE the value to NULL; as such you may want to create a new column for the new value, or perhaps a new column to store the value of dates that couldn't be converted.
After you've changed the value to an unambiguous format, then you can ALTER the column:
ALTER TABLe dbo.YourTable ALTER COLUMN YourDateColumn date NULL;
Note that if you have any constraints, you will need to DROP those first, and then reCREATE them afterwards.
Now that the data type of the column is correct, you need not do anything in SSRS, as the data type is correct.
Requirement:
Need the file to be exported as below format, where gender, age, and interest are columns and value after : is data for that column. Can this be achieved while using Snowflake, if not is it possible to export data using Python
User1234^gender:male;age:18-24;interest:fishing
User2345^gender:female
User3456^age:35-44
User4567^gender:male;interest:fishing,boating
EDIT 1: Solution as given by #demircioglu
It displays as NULL values instead of other column values
Below the EMPLOYEES table data
When I ran below query
SELECT 'EMP_ID'||EMP_ID||'^'||'FIRST_NAME'||':'||FIRST_NAME||';'||'LAST_NAME'||':'||LAST_NAME FROM tempdw.EMPLOYEES ;
Create your SQL with the desired format and write it to a file
COPY INTO #~/stage_data
FROM
(
SELECT 'User'||User||'^'||'gender'||':'||gender||';'||'age'||':'||age||';'||'interest'||':'||interest FROM table
)
file_format = (TYPE=CSV compression='gzip')
File format here is not important because each line will be treated as a field because of your delimiter requirements
Edit:
CONCAT function (aliased with ||) returns NULL if you have a NULL value.
In order to eliminate NULLs you can use NVL2 function
So your SQL will have series of NVL2s
NVL2 checks the first parameter and if it's not NULL returns first expression, if it's NULL returns second expression
So for User column
'User'||User||'^' will turn into
NVL2(User,'User','')||NVL2(User,User,'')||NVL2(User,'^','')
P.S. I am leaving up to you to create the rest of the SQL, because Stackoverflow's function is to help find the solution, not spoon feed the solution.
No, I do not believe multiple delimiters like this are supported in Snowflake at this time. Multiple byte and multiple character delimiters are supported, but they will need to be specified as the same delimiter repeated for either record or line.
Yes, it may be possible to do some post-processing or use Python scripts to achieve this. Or even SQL transformative statements. This is not really my area of expertise so if someone has an example for you, I'll let them add to the discussion.
The following instruction aims at using a specific format to import DATEs
alter session set DATE_INPUT_FORMAT = 'YYYY-MM-DD HH24:MI:SS.FF';
However, it seems to have no effect on the following:
copy into schema.table
from s3://bucket/file.parquet
credentials=(aws_key_id='...' aws_secret_key='...')
match_by_column_name=case_insensitive
file_format=(type=parquet);
Which results in errors like below:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 100071 (22000):
Failed to cast variant value "2020-06-16 00:00:00.000" to DATE
When a column in the imported Parquet file has a format as specified above for a date field.
This really sounds like a bug, as the above COPY INTO scenario should in theory be a typical use case for altering the DATE_INPUT_FORMAT.
Is there a way to address this?
The DATE_INPUT_FORMAT should affect the copy command. The documentation talks about not supporting a timestamp from a variant column on a date conversion.
Although TO_DATE accepts a TIMESTAMP value, it does not accept a TIMESTAMP inside a VARIANT.
I need to set a format for a column when the column "KPI" has a value with '%' I need that the value has the format for a percentage otherwise the currency format, so I have something like this:
=IIF(InStr(Fields!KPI.Value,"%")>0,Format(Fields!DIA.Value, "P"),Format(Fields!DIA.Value, "C"))
That expression works properly, but when I try to export that report to excel, that column is not summarizing when I select several columns as you will see in this picture:
As you can see excel is recounting but not summarizing, I have tried formatting each field from SQL Server and with this formula:
=IIF(InStr(Fields!KPI.Value,"%")>0,Format(Fields!DIA.Value, "###.#%"),Format(Fields!DIA.Value, "###,###,###,###.##"))
But the same happend when I export that report to excel, is there another way to handle this?
In order to make sure that Excel interprets a value as numeric, you can use the RenderFormat global variable to detect when the report is being rendered for Excel and not include the problematic formatting characters:
=IIF(InStr(Fields!KPI.Value,"%")>0,Format(Fields!DIA.Value, "###.#%"),IIF(OR(Globals!RenderFormat.Name="EXCELOPENXML",Globals!RenderFormat.Name="EXCEL"),Fields!DIA.Value,Format(Fields!DIA.Value, "###,###,###,###.##")))
If its critical that you display a thousands separator in your generated output you can use the Language global variable to detect the user's region and use the appropriate character. That check would take the general form Globals.Lanuage="en-US", depending on what particular regions are involved.
Here's an example of using the RenderFormat.Name variable to check the renderer in VS2013:
When doing a preview of the report, the format will show that the render type is RPL:
I'm importing thousands of csv files into an SQL DB. They each have two columns: Date and Value. In some of the files, the value column contains simply a period (ex: "."). I've tried to create a derived column that will handle any cell that contains a period with the following code:
FINDSTRING((DT_WSTR,1)[VALUE],".",1) != 0 ? NULL(DT_R8) : [VALUE]
But, when the package runs it gets the following error when it reaches the cell with the period in it:
The data conversion for column "VALUE" returned status value 2 and status text
"The value could not be converted because of a potential loss of data".
I'm guessing there might be an escape character that I'm missing in my FINDSTRING function but I can't seem to find what that may be. Does anyone have any thoughts on how I can get around this issue?
Trying to debug things like this is why I always advocate adding many Derived Columns to the Data Flow. It's impossible to debug that entire expression. Instead, first find the position of the period and add that as a new column. Then you can feed that into the ternary operation and bit by bit you can add data viewers to ensure you are seeing what you expect to see.
Personally, I'd take a different approach. It seems that you'd like to make any columns that are . into a null of type DT_R8.
Add a derived column, TrimmedValue and use this expression to remove any leading/trailing whitespace and then
RTRIM(LTRIM(Value))
Add a second derived column component, this time we'll add column MenopausalValue as it will remove the period. Use this expression
(TrimmmedValue == ".") ? Trimmedvalue : NULL(DT_WSTR, 50)
Now, you can add your final Derived Column wherein we convert the string representation of Value to the floating point representation.
IsNull(MenopausalValue) ? NULL(DT_R8) : (DT_R8) MenopausalValue
If the above shows an error, then you need to apply the following version as I can never remember the evaluation sequence for ternary operations that change type.
(DT_R8) (IsNull(MenopausalValue) ? NULL(DT_R8) : (DT_R8) MenopausalValue)
Examples of breaking these operations into many steps for debugging purposes
https://stackoverflow.com/a/15176398/181965
https://stackoverflow.com/a/31123797/181965
https://stackoverflow.com/a/33023858/181965
You can do it like this:
TRIM(Value) == "." ? NULL(DT_R8) : (DT_R8)Value