IronPython try/except not catching error - try-catch

# the lines below are iterating through TIBCO Spotfire tables, getting the columns and the column properties, one of which is the column's "Calculated Expression" property
for tb in Document.Data.Tables:
for col in tb.Columns:
# initialize column variable
ces = ""
# get calculated expression and cast to string
try:
ces = col.Properties.CalculatedExpression.ToString()
exception:
ces="Error:Calculated Expression Not Read"
I know in advance that some of Calculated Expressions have unicode characters in them (can't do anything about that) so, I'm trying to "catch" those issues and simply write out the error.Then go to the next column.But, I continue to get the following error complaining about the greater than or equal to symbol:
System.Text.EncoderFallbackException: 'ascii' codec can't encode character '\u2264' in position 174

Doesn't explain it but what if you try:
for tb in Document.Data.Tables:
for col in tb.Columns:
# initialize column variable
ces = ""
# get calculated expression and cast to string
try:
ces = str(col.Properties.CalculatedExpression)
exception:
ces="Error:Calculated Expression Not Read"
instead?
If that doesn't work look into the unicode() function.
str() docs - https://ironpython-test.readthedocs.org/en/latest/library/functions.html#str
unicode() docs - https://ironpython-test.readthedocs.org/en/latest/library/functions.html#unicode
I'd test it myself if there was example data given.

Related

SSIS Expression cannot be evaluated -- why not?

I have variable in an SSIS package name User::FileFullPathLDR.
From this variable at run time of the SSIS package I want to extract the file name from FileFullPathLDR and put it in another variable called User::FileName.
I have tried all types of combinations to try and get this code to resolve in Expression Builder to resolve except standing on my head with incense burning saying OM 4 million times, and nothing seems to work.
I put the following expression in Expression Builder for the variable User::FileName:
REVERSE(SUBSTRING(REVERSE(#[User::FileFullPathLDR]),1,FINDSTRING(REVERSE(#[User::FileFullPathLDR] ),"\\",1)-1))
From the full path I expect to get the filename similar to: LDRFile01242019.txt.
But I keep getting the following error when expression builder parses this statement:
The length -1 is not valid for function "Substring". The length parameter cannot be negative. Change the length parameter to zero or a positive value.
I tested this expression:
REVERSE(SUBSTRING(REVERSE(#[User::FileFullPathLDR]),1,FINDSTRING(REVERSE(#[User::FileFullPathLDR]),"\\",1)-1))
against this test string:
C:\Folder1\Folder2\FileName.TXT
and it returned
FileName.TXT
There was no error
I tested it by creating a variable and defining an expression for it, then evaluating that expression.
You can try the following expression:
RIGHT(#[User::FileFullPathLDR],FINDSTRING(REVERSE(#[User::FileFullPathLDR]),"\\",1)-1)
Also try to add a conditional operator to avoid bad values errors:
FINDSTRING(REVERSE(#[User::FileFullPathLDR]),"\\",1) == 0 ? "" : RIGHT(#[User::FileFullPathLDR],FINDSTRING(REVERSE(#[User::FileFullPathLDR]),"\\",1)-1)
References
SSIS Expression to get filename from FilePath
SSIS Expression to Get File Name From Full Path
The part FINDSTRING(#[User::FileFullPathLDR],"\\",1) returns 0 if FileFullPathLDR does not contain a \. Since you are subtracting -1, you might end up with a negative value if your string doesn't match the pattern, or if the variable is set at runtime (you might have an empty string during validation).
If you need it to work with an empty string as well, you could add a \\ in front of it if there is no \\ present yet, using something like FINDSTRING(#[User::FileFullPathLDR],"\\",1) == 0 ? "\\"+ #[User::FileFullPathLDR] : #[User::FileFullPathLDR]
So the whole thing will be:
REVERSE(
SUBSTRING(
REVERSE(
#[User::FileFullPathLDR]),1,FINDSTRING(REVERSE(FINDSTRING(#[User::FileFullPathLDR],"\\",1) == 0 ? "\\"+ #[User::FileFullPathLDR] : #[User::FileFullPathLDR])
,"\\",1
)-1
)
)
So, if there is no \ present, it will just return the string itself.

SSIS Derived Column - Text in Numeric Field is not converting

I'm importing thousands of csv files into an SQL DB. They each have two columns: Date and Value. In some of the files, the value column contains simply a period (ex: "."). I've tried to create a derived column that will handle any cell that contains a period with the following code:
FINDSTRING((DT_WSTR,1)[VALUE],".",1) != 0 ? NULL(DT_R8) : [VALUE]
But, when the package runs it gets the following error when it reaches the cell with the period in it:
The data conversion for column "VALUE" returned status value 2 and status text
"The value could not be converted because of a potential loss of data".
I'm guessing there might be an escape character that I'm missing in my FINDSTRING function but I can't seem to find what that may be. Does anyone have any thoughts on how I can get around this issue?
Trying to debug things like this is why I always advocate adding many Derived Columns to the Data Flow. It's impossible to debug that entire expression. Instead, first find the position of the period and add that as a new column. Then you can feed that into the ternary operation and bit by bit you can add data viewers to ensure you are seeing what you expect to see.
Personally, I'd take a different approach. It seems that you'd like to make any columns that are . into a null of type DT_R8.
Add a derived column, TrimmedValue and use this expression to remove any leading/trailing whitespace and then
RTRIM(LTRIM(Value))
Add a second derived column component, this time we'll add column MenopausalValue as it will remove the period. Use this expression
(TrimmmedValue == ".") ? Trimmedvalue : NULL(DT_WSTR, 50)
Now, you can add your final Derived Column wherein we convert the string representation of Value to the floating point representation.
IsNull(MenopausalValue) ? NULL(DT_R8) : (DT_R8) MenopausalValue
If the above shows an error, then you need to apply the following version as I can never remember the evaluation sequence for ternary operations that change type.
(DT_R8) (IsNull(MenopausalValue) ? NULL(DT_R8) : (DT_R8) MenopausalValue)
Examples of breaking these operations into many steps for debugging purposes
https://stackoverflow.com/a/15176398/181965
https://stackoverflow.com/a/31123797/181965
https://stackoverflow.com/a/33023858/181965
You can do it like this:
TRIM(Value) == "." ? NULL(DT_R8) : (DT_R8)Value

SSIS Flat File with Numeric DataType with Spaces

I'm using SSIS to load a fixed length Flat File into SQL.
I have a weight field that has been giving me trouble all day.
It has a length of 8 with 6 DECIMAL POSITIONS IMPLIED (99V990099).
The problem i'm having is when it isn't populated and has 8 spaces.
Everything i try gets an error:
"Invalid character value for cast specification"."
OR
"Conversion failed because the data value overflowed the specified type.".
OR
Data conversion failed.
The data conversion for column "REL_WEIGHT" returned status value 2 and status text
"The value could not be converted because of a potential loss of data.".
I've tried declaring it as DT_String & DT_Numeric.
I've tried many variations of:
TRIM([REL_WEIGHT])=="" ? (DT_STR,8,1252)NULL(DT_STR,8,1252) : REL_WEIGHT
ISNULL([REL_WEIGHT]) || TRIM([REL_WEIGHT]) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)[REL_WEIGHT]
TRIM(REL_WEIGHT) == "" ? (DT_NUMERIC,8,6)0 : (DT_NUMERIC,8,6)REL_WEIGHT
But nothing seems to work.
Please someone out there have the fix for this!
I think you may be running afoul of the following point, explained nicely at http://vsteamsystemcentral.com/cs21/blogs/applied_business_intelligence/archive/2009/02/01/ssis-expression-language-and-the-derived-column-transformation.aspx:
You can add a DT_STR Cast statement to the expression for the MiddleName, but it doesn't change the Data Type. Why can't we change the data type for existing columns in the Derived Column transformation? We're replacing the values, not changing the data type. Is it impossible to change the data type of an existing column in the Derived Column? Let's put it this way: It is not possible to convert the data type of a column when you are merely replacing the value. You can, however, accomplish the same goal by creating a new column in the Data Flow.
I've solved this on past occasions by loading the data from the flat file as strings, and then deriving a new column in a Derived Column transformation which is of numeric type. You can then perform the appropriate trimming, validation, casting, etc. in the SSIS expression for that new column in the transformation.
Here, I found an example SSIS expression I used at one point to derive a time value from a 4-digit string:
(ISNULL(Last_Update_Time__orig) || TRIM(Last_Update_Time__orig) == "") ? NULL(DT_DBTIME2,0) : (DT_DBTIME2,0)(SUBSTRING(TRIM(Last_Update_Time__orig),1,2)+":"+SUBSTRING(TRIM(Last_Update_Time__orig),3,2)+":00")
There has to be a better way to do it, But i found a way that works.
Create a Derived Column Expression:
TRIM(REL_WEIGHT) == "" ? (DT_STR,9,1252)"0.0000000" : (DT_STR,9,1252)(LEFT(REL_WEIGHT,2) + "." + RIGHT(REL_WEIGHT,6))
THEN Create a Data Conversion Task to change it to Numeric and set scale to 6.
And then Map the [Copy of NewField] to my SQL table field set up as Decimal(8,6).
I don't know how the performance will be of that when loading a million records, probably not the best. If someone knows how to do this in a better way performance wise please let me know.
Thanks,
Jeff

CONCATENATE syntax error "unable to interpret text-cb1"

I've been trying to make a dynamic col for a select. it's just for learning.
I've made a selection screen with some select-options and checkbox parameters. whenever i have a checked checkbox i want to concatenate a string to my lineselection var.
lineselect = ' CARRID CONNID'.
SELECTION-SCREEN BEGIN OF BLOCK block1 WITH FRAME TITLE text-001.
[...]
SELECTION-SCREEN END OF BLOCK block1.
IF cbcofr EQ 'X'. "where cbcofr is checkbox
CONCATENATE text-cb1 INTO lineselect SEPARATED BY space. "where text-cb1 is 'CONTRYFR
ENDIF.
When i check for error the compiler just says "Unable to interpret "text-cb1". possible cause: incorrect spelling or comma error."
Is not about text-cb1 , i've tried with string 'COUNTRYFR' and says the same thing. I don't get where my error is.
The syntax for concatenate is as follows:
CONCATENATE c1 c2 [... cn] INTO targetc [SEPARATED by sep].
or
CONCATENATE lines of itab into targetc [SEPARATED by sep].
As you have already noted, you need at least two source variables to concatenate.
Full documentation can be found here
As of Netweaver release 7.02 you can also do this:
targetc = c1 && [c2 ... && cn].
In this case, you lose the "separator" functionality, though.

Inserting an array column into the db in Yii

I needed to insert an array field into a database and I was pleased to notice that PostGreSQL had that functionality. But now I am not able to insert the data using the tables active record.
I have tried the below calls with no success
$active_record->array_column = $_array_of_values;
which gives me the exception
Exception Raised:CDbCommand failed to execute the SQL statement: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: array value must start with "{" or dimension information
I have also tried this using
foreach($_array_of_values as $value){
$active_record->array_column[] = $value;
}
which tells me
Indirect modification of overloaded property FeatureRaw::$colors_names has no effect
Can anyone help me with this?
Thanks!
Data must be inserted in the form (text representation of an ARRAY):
INSERT INTO tbl (arr_col) VALUES ('{23,45}')
Or:
INSERT INTO tbl (arr_col) VALUES ('{foo,"bar, with comma"}')
So you need to enclose your array values in '{}' and separate them with comma ,. Use double quotes "" around text values that include a comma.
I listed more syntax variants to insert arrays in a related answer.
For those who also have same problem:
I didn't check the Yii1 behavior, but in Yii2 you simply can insert array as properly formed string as Erwin Brandstetter mentioned in his comment:
$activeRecord->arrayField = '{' . implode(',',$array_values) . '}';
Of course you need to make additional efforts when your $array_values has strings with commas, etc. And you still need to convert value back to array after you load ActiveRecord.
You can make these conversions in ActiveRecord's beforeSave() and afterLoad() and you will not need to convert values manually.
UPD. Recently I made a simple behavior for Yii2 to use array fields with ActiveRecord without manual field building: kossmoss/yii2-postgresql-array-field. It is more generalized way to solve the problem and I hope it will help. For those who use Yii1: you can investigate the package code and create your own solutuion compatible with your framework.

Resources