Here is a screenshot of the console output after executing the sqlUpdate() function. As you can see, it fails at the first row of the data frame.
I suspect that the problem has to do with the introduced_at list component of my Bills.df data frame.
Bills.df$introduced_at <- as.Date(Bills.df$introduced_at, "%Y-%m-%d")
Before this statement, introduced_at is of type character. But this changes it to double. I understand that this may have something to do with the R internals, since it uses C data types to store various R data structures. Maybe it's the wrong expression for data type conversion?
Additionally, if this is a Date type issue, I'd also like to know how to properly convert the updated_at type to Datetime to prevent a similar issue from occurring. The structure of updated_at is as follows: 2015-09-01T10:19:13-04:00. I'm not comfortable with as.POSIXlt and asPOSIXct so I'm not sure which function is appropriate.
"updated_at": "2015-09-01T10:19:13-04:00"
Related
We are using a software that has limited Oracle capabilities. I need to filter through a CLOB field by making sure it has a specific value. Normally, outside of this software I would do something like:
DBMS_LOB.SUBSTR(t.new_value) = 'Y'
However, this isn't supported so I'm attempting to use CAST instead. I've tried many different attempts but so far these are what I found:
The software has a built-in query checker/validator and these are the ones it shows as invalid:
DBMS_LOB.SUBSTR(t.new_value)
CAST(t.new_value AS VARCHAR2(10))
CAST(t.new_value AS NVARCHAR2(10))
However, the validator does accept these:
CAST(t.new_value AS VARCHAR(10))
CAST(t.new_value AS NVARCHAR(10))
CAST(t.new_value AS CHAR(10))
Unfortunately, even though the validator lets these ones go through, when running the query to fetch data, I get ORA-22835: Buffer too small when using VARCHAR or NVARCHAR. And I get ORA-25137: Data value out of range when using CHAR.
Are there other ways I could try to check that my CLOB field has a specific value when filtering the data? If not, how do I fix my current issues?
The error you're getting indicates that Oracle is trying to apply the CAST(t.new_value AS VARCHAR(10)) to a row where new_value has more than 10 characters. That makes sense given your description that new_value is a generic audit field that has values from a large number of different tables with a variety of data lengths. Given that, you'd need to structure the query in a way that forces the optimizer to reduce the set of rows you're applying the cast to down to just those where new_value has just a single character before applying the cast.
Not knowing what sort of scope the software you're using provides for structuring your code, I'm not sure what options you have there. Be aware that depending on how robust you need this, the optimizer has quite a bit of flexibility to choose to apply predicates and functions on the projection in an arbitrary order. So even if you find an approach that works once, it may stop working in the future when statistics change or the database is upgraded and Oracle decides to choose a different plan.
Using this as sample data
create table tab1(col clob);
insert into tab1(col) values (rpad('x',3000,'y'));
You need to use dbms_lob.substr(col,1) to get the first character (from the default offset= 1)
select dbms_lob.substr(col,1) from tab1;
DBMS_LOB.SUBSTR(COL,1)
----------------------
x
Note that the default amount (= length) of the substring is 32767 so using only DBMS_LOB.SUBSTR(COL) will return more than you expects.
CAST for CLOB does not cut the string to the casted length, but (as you observes) returns the exception ORA-25137: Data value out of range if the original string is longert that the casted length.
As documented for the CAST statement
CAST does not directly support any of the LOB data types. When you use CAST to convert a CLOB value into a character data type or a BLOB value into the RAW data type, the database implicitly converts the LOB value to character or raw data and then explicitly casts the resulting value into the target data type. If the resulting value is larger than the target type, then the database returns an error.
I am facing a strange issue while using SSIS "Data Conversion component" to convert string to decimal datatype. I use SSIS 2016.
The source data input has values of mixed data types- string, integer, decimal and is defined as varchar in the flat file source. The target data type expected is numeric. When explicit type conversion happens from string to decimal, we expect the alphanumeric values to get rejected to error table and only the numeric values to pass through.
Instead, we are seeing some alphanumeric values shedding the characters in the value and passing through successfully with no error.
Examples: Value "3,5" converted to 35
Value "11+" converted to 11
We do not have control over source data and will not be able to replace char data before passing data into Data conversion component.
We have tried the below steps as a workaround and it has worked.
i.e,
First Data Conversion from DT_STR to DT_NUMERIC
Capture error rows that fail the above conversion
Second Data Conversion from DT_NUMERIC to DT_DECIMAL
But as the source data is not reliable, we may have to apply this workaround wherever there are numeric fields (int types & deicmals) which is not a friendly solution.
So checking with you all to understand if there is an easier and better solution tried out by anyone.
I did not expect this result, but I tried an expression task and it worked for DT_DECIMAL:
(DT_DECIMAL,1)"11+" -- evaluates to 11.0
But it does not work for DT_NUMERIC. SSIS won't allow a direct numeric result, but it can be nested inside a cast to DT_DECIMAL. Just to demonstrate that, in an expression task even this "numerically valid" cast would not be permitted, because the output simply can't be of type DT_NUMERIC:
(DT_NUMERIC, 3, 0)123
But this is permitted:
(DT_DECIMAL,0)((DT_NUMERIC, 3, 0)123)
So as long as you are happy to specify a precision and scale big enough to hold your data during the "validity" check done by DT_NUMERIC, and then cast it from there to DT_DECIMAL, all in a derived column transform, then DT_NUMERIC seems to enforce the strict semantics you want.
SSIS allows this:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11")
But not either of these:
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"11+")
(DT_DECIMAL,0)((DT_NUMERIC, 2, 0)"3,5")
#billinkc Sorry for not responding to you earlier.
We are working under some restrictions:
(1) All we want to do is capture datatype issues in input data, so we wanted to harness the capability of SSIS Data Conversion Component in SSIS.
(2) DBA doesn't want us to use SQL for type conversions, so we are required to do these conversions between flat file source and flat file destination using SSIS.
(3) We are required to capture the type conversion errors at every step of conversion into an error output file with error column name and error description, to be used later. So we cannot remove char data in the field before passing it to Data Conversion component.
#allmhuran - We have used Derived column task before Data Conversion component to replace unnecessary characters in one of the other fields, but using the same for type conversion makes achieving (3) difficult. Because error output from Derived column task and Data Conversion component cannot be redirected to the same error output file.
We can completely ignore Data Conversion component and use only Derived column task to do all type conversions, whether single or nested. I am trying this and the error descriptions do not always look good, but the cons of the former method can be overcome. I will try this out!
I´m using flink to load a csv file to a dataset of pojos, defined through a scala case class, using readCsvFile method, and I have a problem that i cannot solve.
When in the csv there is a record with some format error in any of its fields it is discarded, and I assume that the only way to keep those records is to type them all as String and do the validations myself.
The problem is that if the last field after the delimiter is empty, the record is discarded by default, I think because it is considered as not having the expected number of fields it should, and it is not possible to handle this record error, while if the empty value if in any of the previous fields there is no problem.
Example
field1|field2|field3
a||c
a|b|
In this example, first record is returned by readCsvFile method but not the second.
Is this behaviour right? and there is any walk around to get the record?
Thanks
Case classes and tuples in Flink do not support null values. Therefore, a||c is invalid if the empty field is not a String. I recommend to use the RowCsvInputFormat in this case. It supports nulls and generic rows can be converted to any other class in a following map operator.
The problem is that, as you say, if the field is a String, the record should be valid even if it is null, and this doesn't happen when the null value is in the last field. The behaviour is different depending on the position.
I will try also with RowCsvInputFormat as you recommend.
Thanks
I'm attempting to create a filter on a tablix in Report Builder 3.0. I click on the Tablix, choose Properties and then click on Filters. Clicked on Add and created my filter:
Expression [SYSTEM_CODE]
Operator =
Value 1005
I don't get an error message but when I run the report, nothing is showing when I know there's valid data for that system code. It doesn't matter what value I choose, I don't see anything when I run the report. It also doesn't matter if I enclose the value in single or double quotes. There is no grouping on this report so there's no filter there either.
I'm pulling my hair out because this should fairly simple.
I had a similar issue and maybe what I found will work for you. I created a simple query to show you what I found (note the Convert statement is intentional here):
Select getdate() as "MyDate", CONVERT(Decimal(12,0), 1005) As "System_Code"
I then created a filter for the System_Code value. Note the Float value in the Type field that you cannot change.
Screen shot of disabled Type field
When I ran the report, I got this error:
The processing of FilterExpression for the tablix ‘Tablix1’ cannot be
performed. Cannot compare data of types System.Decimal and
System.Double. Please check the data type returned by the
FilterExpression.
I know, it says Decimal to Double (I blame Microsoft as this baffled me too). This is happening because this type of parameter can only process Integer and Float values. Report Builder doesn't let you change them as it infers this type from the dataset (based on my own experience). If you set the query to return an Integer, the filter will work:
Select getdate() as "MyDate", CONVERT(Integer, 1005) As "System_Code"
However, checking the Type field will still be disabled, and may show as float. Selecting another value in the drop down expression field then selecting back to the System Code value will change the Type value from Float to Integer.
The funny thing is, if you try another Convert data type value, i.e. BigInt, you will get that you cannot convert from Int64 to Float (/sigh Microsoft).
The closest I could get to seeing why this is happening, after many hours of searching, was this page on Microsoft's site which discusses that the value expected is an Integer or a Float. Even though I am not performing a Sum function, the effect appears the same.
Sum Function (Report Builder)
I have a report connected to BI cube. In the report there is a parameter with the datasat from the cube as well. This parameter correctly appears as a multi-select drown down in the report viewer, and the values are correctly populated.
However, when I am calling the report using javascript and passing the report parameters as a post request, there is an interesting behaviour.
If in the parameter, I pass a value that is not included in the list, the report crashes with the following error:
Default value or value provided for the report parameter 'Name' is not a valid value. (rsInvalidReportParameter)
Is there a way to make SSRS simply skip the invalid values rather than throw an error?
Without knowing more on what is 'acceptable' for the parameter value this question is nearly impossible to answer. In SSRS the first level of acceptance of a parameter is the type which is the common 'text' (string of equivalent in code), integer or DateTime. If you are passing a parameter to the report NOT in the format it wants it will bomb. That should be as designed.
If you limit your scope of 'available' values of a parameter by attaching the parameter to a dataset stating: "Get values from Dataset". If you give the parameter a value NOT in this collection it will also bomb. This would be the argument equivalent of selecting 6 when your choices are 1-5. And getting the standard .NET equivalent of 'argument is outside the bounds of the array.' That is also by design.
For debugging I would suggest trying to set the value in SSRS as the 'default' statically and if it can accept it then the issue is in the transfer and the type. If you can and then cannot from the web pass in via javascript or whatever method you choose it is the type being passed in. In such cases I have passed in a string and then transferred it's type in SSRS dynamically with a cast.