I have a complicated ETL in SQL Server where part of the pipeline populates multiple fields from one DB to the next. The original problem is that these fields were designated as BIGINT in the destination table and Decimal(18,2) in the source table. The Figures would transfer, but only digits left of the decimal.
I have the ETL worked out so that the destination table has the same Decimal(18,2)data type, but whenever I execute the package all figures to the left of the decimal will populate but all figures to the right are 0 when it should reflect the CENTS
Current figures look like this
326.00
556.00
751.00
They should look like this
326.54
556.89
751.44
This is SISS packages designed in Visual Studio. How do I get the 00's in the destination table to reflect the numbers in the source table?
Related
Visual Studio 2015; Source: OLEDB (SQL2012); Dest: FlatFile (csv)
I have stored procedures that I need to pull data from, that I don't have control of, that have output of float data types. In SSMS the data looks something like 3.45985, but in SSIS the output csv file has the data like 3.45989999999 (The numbers are made up, but you get the point).
I do understand how float works, and understand that float is not a precise data type -- but I'm not in control of the stored procedure. Where I am struggling is how to work-around it. I've read that you can set the output to string in the flat file destination, but it doesn't seem to be working. I've also tried taking the output from the stored procedure and putting the results into a table variable and have had success formatting like I need. However some of the stored procedures already are doing some inserts into a temp table and when I try to store the results into a table variable I'm getting an error. So, that solution is really not working for me.
My business users are used to these being like 3.45985 in a legacy application. I'm re-writing these to all be done via SSIS to automate the extracts, and not have to support the legacy business line application.
You can use a Derived Column Transformation to round the number up and cast it as a decimal, with an expression such as:
(DT_DECIMAL, 5) ROUND([ColumnName], 5)
I am trying to import "Financial data" from Excel files in to sql table. Problem I am facing is that My ssis package is incrementing decimal values. e.g -(175.20) from Excel is being loaded as "-175.20000000000005" in SQL.
I am using nVArChar (20) in destination SQL table. Images attached. What's the best data type in destination table. I have done a lot of reading and people seem to suggest decimal data but Package throws error for Decimal data type.Need help please.
Ended up changing the Data type to "Currency" in my SQL destination. Then added a data conversion task to change "DT_R8" data type from excel source to "currency[DT_CY]. This resolved the issue. could have used decimal or Numeric (16,2)data type in my destination as well but then i just went ahead with currency and it worked.
You could use a Derived Column Transformation in your Data Flow Task, with an expression like ROUND([GM],2) (you might need to replace GM with whatever your actual column name is).
You can then go to the Advanced Editor of the Derived Column Transformation and set the data type to decimal with a Scale of 2 on the 'Input and Output Properties' tab (look under 'Derived Column Output').
You'll then be able to use a decimal data type in your SQL Server table.
I have a flat file source that grabs a decimal value for the column Note Amount. The Source system specifies that this is a DECIMAL(12,2) file. I have the column in SSIS set to be DT_Decimal
The particular Flat file has a value of 122735.13 for one of it's amounts. However if I place a data viewer on the flow after the source it seems to be cutting the decimal off.
Any idea why this is happening? For reference I'm going to a Decimal(12,2) SQL Server 2012 Database.
Well it was pointed out that I needed to add the Scale that was missing.
Or even better do as this suggests and switch to numeric where I can set precision and scale.
I ended up with reading it as text and alterering the table field to decimal before and after import. No other way let me keep the deciaml point in SSIS 2013.
I have a table in my source data base with 100 columns on it .Where i use ssis package to load data from source data base to destination data base table.
Where table in source and destination table are same .But in some times in Destination table address fields will be changed and or new data types are added.
So how can i find new columns are changed data types in destination table while compared with the source table.
Is there any stored procedure to find missing columns or address fields and changed data types.
i can check manually but its killing my time, where i have 50 tabels where each table consists of 100 to 200 columns.
Please some one help me to find ?
You can grab all the table names from sys.tables and all the column names from sys.columns.
You can write code to perform this operation on the destination and source databases; However, you might waste alot of time re-inventing the wheel.
http://www.red-gate.com/products/sql-development/sql-compare/
Just get a trial version of SQL-Compare from the sales team at red gate. If you like it, just buy it. It only costs $495.
Probably a-lot less money than trying to write something yourself!
I have a desktop application through which data is entered and it is being captured in MS Access DB. The application is being used by multiple users(at different locations). The idea is to download data entered for that particular day into an excel sheet and load it into a centralized server, which is an MSSQL server instance.
i.e. data(in the form of excel sheets) will come from multiple locations and saved into a shared folder in the server, which need to be loaded into SQL Server.
There is a ID column with IDENTITY in the MSSQL server table, which is the primary key column and there are no other columns in the table which contains unique value. Though the data is coming from multiple sources, we need to maintain single auto-updating series(IDENTITY).
Suppose, if there are 2 sources,
Source1: Has 100 records entered for the day.
Source2: Has 200 records entered for the day.
When they get loaded into Destination(SQL Server), table should have 300 records, with ID column values from 1 to 300.
Also, for the next day, when the data comes from the sources, Destination has to load data from 301 ID column.
The issue is, there may be some requests to change the data at Source, which is already loaded in central server. So how to update the data for that row in the central server as the ID column value will not be same in Source and Destination. As mentioned earlier ID is the only unique value column in the table.
Please suggest some ides to do this or I've to take up different approach to accomplish this task.
Thanks in advance!
Krishna
Okay so first I would suggest .NET and doing it through a File Stream Reader, dumping it to the disconnected layer of ADO.NET in a DataSet with multiple DataTables from the different sources. But... you mentioned SSIS so I will go that route.
Create an SSIS project in Business Intelligence Development Studio(BIDS).
If you know for a fact you are just doing a bunch of importing of Excel files I would just create many 'Data Flow Task's or many Source to Destination tasks in a single 'Data Flow Task' up to you.
a. Personally I would create tables in a database for each location of an excel file and have their columns map up. I will explain why later.
b. In a data flow task, select 'Excel Source' as the source file. Put in the appropriate location of 'new connection' by double clicking the Excel Source
c. Choose an ADO Net Destination, drag the blue line from the Excel Source to this endpoint.
d. Map your destination to be the table you map to from SQL.
e. Repeat as needed for each Excel destination
Set up the SSIS task to automate from SQL Server through SQL Management Studio. Remember you to connect to an integration instance, not a database instance.
Okay now you have a bunch of tables right instead of one big one? I did that for a reason as these should be entry points and the logic to determinate dupes and import time I would leave to another table.
I would set up another two tables for the combination of logic and for auditing later.
a. Create a table like 'Imports' or similar, have the columns be the same except add three more columns to it: 'ExcelFileLocation', 'DateImported'. Create an 'identity' column as the first column and have it seed on the default of (1,1), assign it the primary key.
b. Create a second table like 'ImportDupes' or similar, repeat the process above for the columns.
c. Create a unique constraint on the first table of either a value or set of values that make the import unique.
c. Write a 'procedure' in SQL to do inserts from the MANY tables that match up to the excel files to insert into the ONE 'Imports' location. In the many inserts do a process similar to:
Begin try
Insert into Imports (datacol1, datacol2, ExcelFileLocation, DateImported) values
Select datacol1, datacol2, (location of file), getdate()
From TableExcel1
End try
-- if logic breaks unique constraint put it into second table
Begin Catch
Insert into ImportDupes (datacol1, datacol2, ExcelFileLocation, DateImported) values
Select datacol1, datacol2, (location of file), getdate()
From TableExcel1
End Catch
-- repeat above for EACH excel table
-- clean up the individual staging tables for the next import cycle for EACH excel table
truncate TableExcel1
d. Automate the procedure to go off
You now have two tables, one for successful imports and one for duplicates.
The reason I did what I did is two fold:
You need to know more detail than just the detail a lot of times like when it came in, from what source it came from, was it a duplicate, if you do this for millions of rows can it be indexed easily?
This model is easier to take apart and automate. It may be more work to set up but if a piece breaks you can see where and easily stop the import for one location by turning off the code in a section.