How to append specific data in flatfile destination using SSIS package loop iterate?
Source: emp (SQL Server) sample data. In this table I have 1 million records
I need to load specific empid information into flat file destination using loop iterate.
Sample data
empid | Ename
------+---------
1 | ab
2 | dg
3 | hg
Load the specific empid information maintain in empmaster table.empids values frequenctly changing in empmaster table
Table: Empmaster
Empid
-----
1
2
5
6
Destination: text files emp.txt
empid | Ename
-------+--------
1 | ab
2 | dg
I tried like this:
Step 1: create 2 variables empid (datatype varchar) and empidresult (datatype object)
Step 2: drag and drop the ExecuteSQL task and using oledb connection and SQL command is: select distinct empdid from empmaster and resultset is: fullresultset and resultset variable select the empidresult
Step 3: drag and drop the foreachloopcontainer and container type is foreachadoenumerator adoobjectsource variable is: empidresult and variable mapping is: empid
Step 4: inside container drag and drop the dataflow task and using oledb source connection SQL statement with parameters
select empid,ename from emp where empid =? and parameter mapping to empid
Step 5: drag and drop flatfiledestination and configure emp.txt file
here I select the columns names in the first data row unchecked and overwrite data in the file unchecked and executed the package i can append the data for required empids data but header is missing
I tried another methods empmaster records i have converts rows to columns and result store into object with commasepareate buts its supporting only up to 9000 records
if master table have more than 10,000 records then its not working
I need first row as header then remain each loop iterate should be data append in destination files.
How to achive this task in sql server?
what's wrong with a data flow sourced by:
SELECT empid,ename
FROM emp
JOIN Empmaster on emp.empid = Empmaster.empid
and then just run that into a flatfile output.
Related
I have a requirement to read the data rows from one source table and insert different rows in another table
I have tried reading the rows from the table and inserting is working fine using data flow in SSIS
But based on the source rows(split source and insert into 2 rows in destination) based on Common ID
i.e Source System ID and Source System Key in one row and targetSystemID and Target system key in another row with same commonID,RefID
Example:
Source table columns & rows
RefID SourceSystemID SourceKey TargetSystemID TargetKey CommonID KeyDate
3 1 9BFG06E8AD 2 001351913001 12345 20-02-2010
4 1 8KIJ67ASG9 2 002352014005 45678 20-02-2010
Required destination column rows
DataID(AutoIncrement) RefID SystemID SystemKey CommonID DateCreated(using getdate())
1 3 1 9BFG06E8AD 12345 getdate()
2 3 2 001351913001 12345 getdate()
3 4 1 8KIJ67ASG9 45678 getdate()
4 4 2 002352014005 45678 getdate()
Can anyone assist please?
You can use UNION ALL in SSIS.
There are two ways to use UNION ALL in SSIS
1. Using SQL Query within SSIS
Use this query as a source
SELECT Row_Number() over(order by RefId) AS DataID,*
FROM(
SELECT RefID, SourceSystemID As SystemId, SourceKey, CommonID, GetDate() as CreatedDate
FROM sample
Union All
SELECT RefID, TargetSystemID, TargetKey, CommonID, GetDate() as CreatedDate
FROM sample
) AS T
Order by RefId,SystemId
;
2. Using SSIS Package Object
Create two different OLE-DB Sources inside the Dataflow Task
like this (Though, Its sources are different but Instead of Excel & CSV, put two OLE-DB Source) and select only required columns in source one and two.
Please refer this link to use Union-All in SSIS
EDITED: Check this fiddle to match your answer!
Table emp with columns:
id | name | sal | deptno | location
One user is inserting/updating 20 million records into that table.
Another user is retrieving data using select statement from the same emp table.
insert / update statement will take 2 hours and at the same time another user retrieve data in the mid of insertion/update process
How many records will come for execute the select statement in SQL?
How to check or what steps we need to follow to achieve this task in SQL Server?
I have a data set which is of following format in excel
Col A Col B WW1 WW2 WW3.... WW51 Ww52
The columns A & B has text values and WW1 to WW52 has numeric values. This data is unpivoted to the following format into a SQL staging table
Col A Col B CurrentWW WorkWeeks Qty
The WorkWeeks column will have 52 workweek values including the current work week in the format yyyyww and Qty as numeric. A sample data set will look like as follows :
ColA ColB CurrentWW WorkWeeks Qty
US01 ABC 201717 201717 1
US01 EFG 201717 201718 1
US01 HIJ 201717 201719 2
and so on.. until the 52nd workweek starting from 201717.
How can I achieve this in SSIS?
I have tried the following options - Lookup,Merge Join,Union all.. but it joins with the first row only. I know Im very close but still missing something. Any help would be appreciated.
What is your end result (flat file, Excel file, sql table, etc)? You could always load the Excel file into another staging table and then join the two tables using SQL. That may help reduce the complexity since I know joining two sources with SSIS can sometimes be a pain.
I would like to do data mining. But my data is not useful.
my table structure is something like:
date customerid age residence prosubsclassid productid
----------------------------------------------------------------------------
21.11.2001 123232323 a b 2099 23232322
amount asset sales
------------------------
4 34 56
Now I have to show the data in this way:
prosubsclassid 130207 130208 130209
------ ------ ------
1413232 1 3 1
3435545 2 1 2
3534344 3 1 sum(amount)
Column(customerid)
I want to convert to tabular form in my data.
There is no automatic way to do this. There are support for pivot in SQL server, but the columns still needs to be specified.
Depending on if you want to have a text report or a table with dynamic columns I would group the data per date and prosubsclassid and then using a cursor to build the data.
If you want a dynaic table build a dynamic sql query based on the grouped data and run Exec.
If you want text report, just concatenate the string data the way you want per line into
a temp table with one textcolumn and when you are done, select the table.
I have the following table ,
Database Table:
BatchID BatchName Chemical Value
----------------------------------------------
BI-1 BN-1 CH-1 1
BI-2 BN-2 CH-2 2
----------------------------------------------
I need to display the following table.
BI-1 BI-2
BN-1 BN-2
-----------------------------------------
CH-1 1 null
------------------------------------------
CH-2 null 2
------------------------------------------
Here BI-1,BN-1 are two rows in a single columns i need to display chemical value as row of that.Could Please help me to solve this problem.
I tried it in Pivot table but i unable to get this.
So is there any chance in Reporting Server MDX.
First, you need to create a cube with Analysis Services (MDX only works against multidimensional data sources)
Then, assuming you have a cube "MYCUBE" with "Batch" and "Chemical" dimensions and a "Value" measure, the query would be something like this (of course, you could select just the members you need instead of all the members in the dimensions):
SELECT
{[Batch].members * [Measures].[Value]} on columns,
{[Chemical].members} on rows
FROM [MYCUBE]