I'm trying to export data from SQL Server to flat files.
The data includes a bit column 0/1 and I need it as is but it get exported as Boolean TRUE/FALSE which is causing ETL bulk insert to fail.
I tried changing the data mapping to single byte integer, float, numeric, string, text hoping to get a simple 0/1 but nothing worked.
Any pointers on how to solve this are appreciated (other than replacing text within the resulted file).
Add a IIF in your select to convert Boolean to 0/1 to export:
SELECT Name, Code
,IIF(EuroZone = 1, 1, 0) AS EuroZone
,IIF(Visible= 1, 1, 0) AS Visible
FROM your_table
The problem is, SQL Server has a bit data type. It does not have a boolean, while SSIS
uses .Net data types, not SQL data types. It supports boolean, not bit. Though it has built-in conversions, to resolve problems like that. So in my opinion, you need to use a derived column to solve that.
Related
I am trying to import "Financial data" from Excel files in to sql table. Problem I am facing is that My ssis package is incrementing decimal values. e.g -(175.20) from Excel is being loaded as "-175.20000000000005" in SQL.
I am using nVArChar (20) in destination SQL table. Images attached. What's the best data type in destination table. I have done a lot of reading and people seem to suggest decimal data but Package throws error for Decimal data type.Need help please.
Ended up changing the Data type to "Currency" in my SQL destination. Then added a data conversion task to change "DT_R8" data type from excel source to "currency[DT_CY]. This resolved the issue. could have used decimal or Numeric (16,2)data type in my destination as well but then i just went ahead with currency and it worked.
You could use a Derived Column Transformation in your Data Flow Task, with an expression like ROUND([GM],2) (you might need to replace GM with whatever your actual column name is).
You can then go to the Advanced Editor of the Derived Column Transformation and set the data type to decimal with a Scale of 2 on the 'Input and Output Properties' tab (look under 'Derived Column Output').
You'll then be able to use a decimal data type in your SQL Server table.
This may be a stupid question but I must ask since I see it a lot... I have inherited quite a few packages in which developers will use the the Data Conversion transformation shape when dumping flat files into their respective sql server tables. This is pretty straight forward however I always wonder why wouldn't the developer just specify the correct data types within the flat file connection and then do a straight load into the the table?
For example:
Typically I will see flat file connections with columns that are DT_STR and then converted into the correct type within the package ie: DT_STR of length 50 to DT_I4. However, if the staging table and the flat file are based on the same schema - why wouldn't you just specify the correct types (DT_I4) in the flat file connection? Is there any added benefit (performance, error handling) for using the data conversion task that I am not aware of?
This is a good question with not one right answer. Here is the strategy that I use:
If the data source is unreliable
i.e. sometimes int or date values are strings, like when you have the literal word 'null' instead of the value being blank. I would let the data source be treated as strings and deal with converting the data downstream.
This could mean just staging the data in a table and using the database to do conversions and loading from there. This pattern avoid the source component throwing errors which is always tricky to troubleshoot. Also, it avoids having to add error handling into data conversion components.
Instead, if the database throws a conversion error, you can easily look at the data in your staging table to examine the problem. Lastly, SQL is much more forgiving with date conversions than ssis.
If the data source is reliable
If the dates and numbers are always dates and numbers, I would define the datatypes in the connection manager. This makes it clear what you are expecting from the file and makes the package easier to maintain with fewer components.
Additionally, if you go to the advanced properties of the flatfile source, integers and dates can be set to fast parse which will speed up the read time: https://msdn.microsoft.com/en-us/library/8893ea9d-634c-4309-b52c-6337222dcb39?f=255&MSPPError=-2147217396
When I use data conversion
I rarely use the data conversion component. But one case I find it useful is for converting from / to unicode. This could be necessary when reading from an ado.net source which always treats the input as unicode, for example.
You could change the output data type in the flat file connection manager in Advanced page or right click the source in Data flow, Advanced editor to change the data type before loading it.
I think one of the benefit is the conversion transformation could allow you output the extra column, usually named copy of .., which in some case, you might use both of the two columns. Also, sometimes when you load the data from Excel source, all coming with Unicode, you need to use Data conversion to do the data TF, etc.
Also, just FYI, you could also use Derived Column TF to convert the data type.
UPDATE [Need to be further confirmed]:
From the flat file source connection manager, the maximum length of string type is 255, while in the Data Conversion it could be set over 255.
I am building a Data Flow Task in an SSIS package that pulls data in from an OLE DB Source (MS Access table), converts data types through a Data Conversion Transformation, then routes that data to an OLE DB Destination (SQL Server table).
I have a number of BIT columns for flag variables in the destination table and am having trouble with truncation when converting these 1/0 columns to (DT_BYTES,1). Converting from DT_WSTR and DT_I4 to (DT_BYTES,1) results in the same truncation, and I have verified that it is happening at that step through the Data Viewer.
It appears that I need to create a derived column similar to what is described in the answers to the question linked below, but instead of converting to DT_BOOL, I need to convert to (DT_BYTES,1), as casting from DT_BOOL to DT_BYTES is apparently illegal?
SSIS Converting a char to a boolean/bit
I have made several attempts at creating a derived column with variations of the logic below, but haven’t had any luck. I am guessing that I need to use Hex literals in the “1 : 0” portion of the expression, but I haven’t been able to find valid syntax for that:
(DT_BYTES,1)([Variable_Name] == (DT_I4)1 ? 1 : 0)
Am I approaching this incorrectly? I can’t be the first person to need to insert BIT data into a SQL Server table, and the process above just seems unnecessarily complex to me.
I'm trying to use SqlBulkCopy.WriteToServer to insert data in a certain table of an SQL database. To do so, I have a DataTable which is populated with records I need to save on the database. Basically my code is based on the Microsoft's example provided for the method SqlBulkCopy.WriteToServer.
The problem is that I have two DateTime fields in the SQL table, and i don't know how to represent them while defining the columns of the DataTable. I tryed both System.String and System.DateTime, but after the code has been executed, he says he can not convert a String type to DateTime. DataTable columns are defined in the following way (code taken from the example linked above):
Dim productID As DataColumn = New DataColumn()
productID.DataType = System.Type.GetType("System.Int32")
How can i do that? What is the correct type to use for a DataTable's column corresponding to an SQL DateTime field?
Previously, I used an SQL command to map every field, for example:
' Fields initialization
SqlCmd.Parameters.Add("#Field1", SqlDbType.DateTime)
[...]
SqlCmd.Parameters.Add("#FieldN", SqlDbType.NChar, 255)
' After opened the transaction
SqlCmd.Parameters("#Field1").Value = MyDateTimeSavedInAString
[...]
SqlCmd.Parameters("#FieldN").Value = "NTHVALUE"
Thanks in advance.
[UPDATE1] The DateTime column now works, but the same kind of error is given by another column which will be saved on a time field in the SQL Server's table. What kind of VB .NET type i should use to map the DataTable column with the one of SQL Server marked as time?
[UPDATE2] I'm trying to use an SQL table with every field set to nvarchar data type, but it still gives the same error. In fact he says that it is impossible to convert the String type of the origin column in the nvarchar type of the destination column.
Use DateTime - DateTime conversion works like a charm. SqlBulkCopy does not want a lot of modifications in the data - it bypasses most of SQL Server's processing for raw performance.
And you can avoid using a DataTable - it takes about an hour or two to write your own object wrapper ;) DataTables are not exactly efficient.
And try to wrap it up more- SqlBUlkCopy is terrible code in that it puts an exclusive lock on the target table. I have my own wrapper creating a temporary table, bulk copying into this and then using a simple SELECT INTO to move the data to the final table in a short atomic operation.
And be aware - below around 1000 lines it makes no sense to use SqlBulkCopy. High overhead. Rather create a long multi line insert statement.
I am using SQL Server Integration Services 2008 to put a certain boolean value in the database, and originally I used the SSIS type of DT_BOOL (boolean).
However, in the database, the boolean value is being stored as a tinyint field (for legacy reasons). A boolean value of TRUE would be saved in the database as 255 (all bits on), instead of the more traditional value of 1.
Is there any way to force a DT_BOOL value of TRUE to map to 1 instead of 255? Or do I need to change the SQL field to bit, or use a numeric SSIS type like DT_UI1?
Well you could add a further transformation task into your SSIS package to transform the DT_BOOL value of TRUE to 1 and an SSIS quuivalent of the SQL Server data type tinyint but of course this adds an additional step to your processing.
You would be far better off modifying your database data type to the more appoprriate bit data type in my opinion if this is indeed an option to you.