I have a field JSONStructure which has NT_TEXT Data Type and have to do a replace function on that.But, It looks like I cannot do a replace function on column having DT_NText DataType. I tried using Data conversion in SSIS But my JSONStructure can have more than 8000 characters and It is not working.
Can someone suggest me the best way to do it.
Thanks in Advance.
I think you'll need to use a Script Component, acting as a Transformation, and you'll need to specify that the column is read/write and then use C#/VB.NET string methods to perform the string manipulation
Related
How to add leading zeros in ADF data flow from the expression builder
For example – have column with numeric value as “000001” but it is coming as 1 only in SQL DB , if I put in entire value in single quotes it is coming but I need dynamic way of implementation with out hard coding.
I agree with #Larnu's comments that even if we give 00001 to an int type column it will give as 1 only.
So, we have to give those in single quotes ('00001') to use like that or import the incoming data as string instead of int.
As you are using ADF dataflow, if you want to use the 00001, you can generate those using derived column transformation from SQL source. But this depends on your requirement like how your leading 0's varies. So, use according to it.
Sample demo:
concat('0000', toString(id))
Result:
Use that column as per your requirement, after that you can convert it back to the same input by toInteger(id).
I am facing a strange issue while using SSIS "Data Conversion component" to convert string to decimal datatype. I use SSIS 2016.
The source data input has values of mixed data types- string, integer, decimal and is defined as varchar in the flat file source. The target data type expected is numeric. When explicit type conversion happens from string to decimal, we expect the alphanumeric values to get rejected to error table and only the numeric values to pass through.
Instead, we are seeing some alphanumeric values shedding the characters in the value and passing through successfully with no error.
Examples: Value "3,5" converted to 35
Value "11+" converted to 11
We do not have control over source data and will not be able to replace char data before passing data into Data conversion component.
We have tried the below steps as a workaround and it has worked.
i.e,
First Data Conversion from DT_STR to DT_NUMERIC
Capture error rows that fail the above conversion
Second Data Conversion from DT_NUMERIC to DT_DECIMAL
But as the source data is not reliable, we may have to apply this workaround wherever there are numeric fields (int types & deicmals) which is not a friendly solution.
So checking with you all to understand if there is an easier and better solution tried out by anyone.
I did not expect this result, but I tried an expression task and it worked for DT_DECIMAL:
(DT_DECIMAL,1)"11+" -- evaluates to 11.0
But it does not work for DT_NUMERIC. SSIS won't allow a direct numeric result, but it can be nested inside a cast to DT_DECIMAL. Just to demonstrate that, in an expression task even this "numerically valid" cast would not be permitted, because the output simply can't be of type DT_NUMERIC:
(DT_NUMERIC, 3, 0)123
But this is permitted:
(DT_DECIMAL,0)((DT_NUMERIC, 3, 0)123)
So as long as you are happy to specify a precision and scale big enough to hold your data during the "validity" check done by DT_NUMERIC, and then cast it from there to DT_DECIMAL, all in a derived column transform, then DT_NUMERIC seems to enforce the strict semantics you want.
SSIS allows this:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11")
But not either of these:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11+")
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"3,5")
#billinkc Sorry for not responding to you earlier.
We are working under some restrictions:
(1) All we want to do is capture datatype issues in input data, so we wanted to harness the capability of SSIS Data Conversion Component in SSIS.
(2) DBA doesn't want us to use SQL for type conversions, so we are required to do these conversions between flat file source and flat file destination using SSIS.
(3) We are required to capture the type conversion errors at every step of conversion into an error output file with error column name and error description, to be used later. So we cannot remove char data in the field before passing it to Data Conversion component.
#allmhuran - We have used Derived column task before Data Conversion component to replace unnecessary characters in one of the other fields, but using the same for type conversion makes achieving (3) difficult. Because error output from Derived column task and Data Conversion component cannot be redirected to the same error output file.
We can completely ignore Data Conversion component and use only Derived column task to do all type conversions, whether single or nested. I am trying this and the error descriptions do not always look good, but the cons of the former method can be overcome. I will try this out!
I want a function to convert a varchar to hexadecimal in sql server. Please help me.
P.S. I will use unhex() in Hive to try to get the original value back. This is because my data contains special characters and backslash and the HIVE external table does not recognise it
You can try to use CONVERT function:
SELECT CONVERT(VARBINARY(MAX), 'myText')
Output:
0x6D7954657874
you can use
select try_convert(varbinary,varcharcolumn)
You should use try_convert to avoid errors when the conversion fails.. Also, varbinary(n) is a better alternative to varbinary(max) as the latter would set the page size to 2GB, which might be excessive.
Hope this helps!
i am currently having some difficulties trying converting a Unicode character string (DT_WSTR) into an INT
this is what i have
(Note most writings are in German, but i think someone who has worked with SSIS will unterstand anyways)
Data Type selected: A four-byte, unsigned integer.but all it does is fail terribly
Try Derive Column component instead of Data Type Conversion.
I have a string like
`...<..../><... id='someID' .../><...../>....`
(the total length of that string is more than 15k chars, it's an XML form definition)
Inside of that string I have the someID value. I want to put after the element containing that value a new string:
...<..../><... id='someID' .../><my_new_string><...../>....
I tried to split that long string basing on the someID value, but that approach is too slow. How can I achieve that on the other way ?
Or maybe is it possible to select the substring <... id='someID' .../> ?
SQL server can work with XML. You do not need to use substring.
A simular problem was solved on this page: xml.modify
Have you tried using Replace? For example:
REPLACE(yourString, yourPattern, yourPattern + newString);
Using your example, it would look something like:
REPLACE('...<..../><... id='someID' .../><...../>....',
'<... id=''someID'' .../>',
'<... id=''someID'' .../><my_new_string>');
Please notice I escaped the ' characters around "someID".
Best regards.
It isn't clear what data type your string is: xml or (n)varchar? For xml, you can use the various data type methods; for (n)varchar the STUFF() function inserts one string into another string.