First of all I have never attempted something like this in SSIS and I am very new to SSIS package development.
I need to build a component in my package that will run through a table of data (say 80 rows) and set a field titled DisplayOrder to the auto incremented number. The catch is that one of the records HAS to be set to 0 and then the rest of he records set to the auto incremented number.
In regards to code, I am not even sure what code to attach to this question or even what screenshots.
I finally figured it out and there is no need for a loop.
Create a SQL Task to clear the linked Table.
Script Used
DELETE FROM [Currency].[ExchangeRates]
Create a SQL Task to clear the main table.
Script Used
DELETE FROM [Currency].[CurrencyList]
Load the values into the main table.
Actions Used
Load values from XML Source
Dump values to [ExchangeRates] Table
Create a SQL Task to load the Values from the main table to the linked table.
Script Used
INSERT INTO [Currency].[CurrencyList] (CurrencyCode, CurrencyName, ExchangeRateID, DisplayOrder) SELECT [er].[TargetCurrency] AS [CurrencyCode], [er].[TargetName] AS [CurrencyName], [er].[ID] AS [ExchangeRateID], ROW_NUMBER() OVER (ORDER BY [ER].[TargetName]) AS [DisplayOrder] FROM [Currency].[ExchangeRates] AS [er] ORDER BY [CurrencyName]
Create a SQL Task to load a new record to the main table for use as DisplayOrder 0.
Script Used
INSERT INTO [Currency].[ExchangeRates] ([Title], [Link], [Description], [PubDate], [BaseCurrency], [TargetCurrency], [TargetName], [ExchangeRate]) VALUES ('1 USD = 1 USD','http://www.floatrates.com/usd/usd/','1 U.S. Dollar = 1 U.S. Dollar',(SELECT TOP 1 [PubDate] FROM [Currency].[ExchangeRates]),'USD','USD','United States Dollar','1')
Create a SQL Task to reference the newly created record from the main table.
Script Used
INSERT INTO [Currency].[CurrencyList] (CurrencyCode, CurrencyName, ExchangeRateID, DisplayOrder) SELECT [er].[TargetCurrency] AS [CurrencyCode], [er].[TargetName] AS [CurrencyName], [er].[ID] AS [ExchangeRateID], 0 AS [DisplayOrder] FROM [Currency].[ExchangeRates] AS [er] WHERE [er].[TargetCurrency] = 'USD'
Related
Consider the code below in a unit test, where I add a new Tag object in a pre-populated SQLite database.
#Test // Line 1
public void add() {
Tag tagToAdd = new Tag("Tall");
Tag addedTag = this.tagDao.add(tagToAdd);
assertNotNull(addedTag);
assertEquals(3L, addedTag.getId()); // Line 6
assertEquals(tagToAdd.getTag(), addedTag.getTag());
List<Tag> tags = this.tagDao.get();
assertEquals(3, tags.size());
}
On line 6, I expect the ID of the Tag to be 3, because the field is an AUTOINCREMENT and the test is initialized with a database already containing 2 Tags. This works fine every time I run the test and the ID is always 3.
Now, I am integrating flyway to the project. Every time I run the test, the AUTOINCREMENT starts from the value of the last run, so the Tag ID increments by 1 every run, and the test fails.
Any idea on how I can get flyway to always reset the database to a brand new state, and reset the AUTOINCREMENT value ? I could write a query to do it manually, but this is not maintainable.
What I have tried so far ?
Integrate #FlywayTest, as this executes flyway task clean
Defined a FlywayMigrationStrategy bean, which contains flyway.clean()
Set spring.flyway.clean-on-validation-error to true in my application.properties (that said, there was no change in my sql, so not sure if this changed anything)
-- Edit
My 1st migration script contains the below.
DROP TABLE IF EXISTS Tag;
CREATE TABLE Tag(
id INTEGER PRIMARY KEY AUTOINCREMENT,
tag VARCHAR(255) NOT NULL UNIQUE,
createdDate TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
modifiedDate TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
If I understood everything correctly - you have a database and a table in this database which is created once and the same table is used for tests every time - you just delete rows from the table (without removing it) when tests are completed (or before starting next tests) and flyway just inserts two tags into this table every time you run the tests.
If that's right - you can just reset sequence in SQLite to set it back to 1 so next inserted row will be inserted with this id. You can do it by running the following query:
UPDATE `sqlite_sequence` SET `seq` = 1 WHERE `name` = 'tags_table_name';
Alternatively, you can set seq to 0 - this value is incorrect so SQLite will use next available correct value (if there are no rows in the table - it will be one, if there are some values - it will first available number).
Yet another possibility is just to delete your table after tests and recreate it before running next tests - as it is a database and table just for tests - it should work correctly. This way you have your sequence counter set back to value 1 each time. I would actually go this way until you have really good reason not to delete the table.
I have a table which stores all of my customers and their invoices (less than 5k total), I want to to use a foreach loop container to write each one of these (customers) to their own file listing their own invoices.
I have used a foreach loop container to read/load/write files before so I understand that part but how do I apply the foreach loop on the AccountNumber as the enumerator?
For each file, I only want that customers info.
My table:
AccountNumber InvoiceNumber OriginalCharge
A255 2017-11 225.00
A255 2017-12 13.50
A255 2018-01 25.00
D870 2017-09 7.25
D870 2017-10 10.00
R400 2016-12 100.00
R400 2017-03 5.00
R400 2017-04 7.00
R400 2017-09 82.00
So this would produce 3 files and would include the invoices/original charge for the given customers.
File 1 = Customer A255
File 2 = Customer D870
File 3 = Customer R400
Or should I approach this differently?
Environment: SQL Server 2014
SSIS-2012
Thanks!
You'll need to apply a few different recipes to make this work.
Dynamic file name
Source query parameterization
Shredding record set
Assumptions
You have three SSIS Variables:
CurrentAccountNumber String (initial value of A255)
rsAccountNumbers Object
FileNameOutput String EvaluateAsExpression = True "C:\\ssisdata\output\\" + #[User::CurrentAccountNumber] + ".txt"
The package would look something like
[Execute SQL Task] -> [Foreach (Ado.net) Enumerator] -> [Data Flow Task]
Execute SQL Task
Set the resultset type to Full
Your source query would be SELECT DISTINCT AccountNumber FROM dbo.Invoices;
In the Results tab, assuming OLE DB Connection Manager, click add result button and use a "name" of 0 and the variable becomes User::rsAccountNumbers
Foreach (Ado.net) Enumerator
Set your enumerator type as Ado.NET and single table. Use the variable User::rsAccountNumbers and assign the zeroeth element to our variable CurrentAccountNumber
Run the package as is to verify the Execute SQL Task is returning a resultset that the Foreach can shred. Observe that each loop in the enumerator results in the value of our Variable FileNameOutput changing (C:\ssisdata\output\A255.txt, C:\ssisdata\output\D870.txt, etc)
Data flow task
This a simple flow
[OLE DB Source] -> [Flat File Destination]
Configure your OLE DB Source to be a Query SELECT * FROM dbo.Invoices WHERE D.AccountNumber = ?;
Click the Parameter button. Configure the name 0 to be #[User::CurrentAccountNumber]
Flat File Destination - Connect the Source to the destination, create a new
Flat File Connection Manager and connect the columns.
Dynamic file name
The final piece will be to edit the Flat File Connection manager created above to use the variable FileNameOutput instead of the hard coded value you indicated. Right click on the Flat File Connection manager and select Properties. In the resulting properties window, find the Expressions property and click the ellipses (...) In the lefthand window, find ConnectionString and in the righthand window, use #[User::FileNameOutput]
F5 and the package should fire up and generate an output file per account number.
I have a page that ask of users opinion about a topic. Their responses are then saved into a table. What I want to do is to check how many users selected an option 1,2,3 and 4.
What I have now are multiple T-SQL queries that run successfully but I believe there is a simplified version of the code I have written. I would be grateful if someone can simplify my queries into one single query. Thank you.
here is sample of data in the database table
enter image description here
$sql4 = "SELECT COUNT(CO) FROM GnAppItms WHERE CO='1' AND MountID='".$mountID."'";
$stmt4 = sqlsrv_query($conn2, $sql4);
$row4 = sqlsrv_fetch_array($stmt4);
$sql5="SELECT COUNT(CO) FROM GnAppItms WHERE CO='2' AND MountID='".$mountID."'";
$stmt5=sqlsrv_query($conn2,$sql5);
$row5=sqlsrv_fetch_array($stmt5);
$sql6="SELECT COUNT(CO) FROM GnAppItms WHERE CO='3' AND MountID='".$mountID."'";
$stmt6=sqlsrv_query($conn2,$sql6);
$row6=sqlsrv_fetch_array($stmt6);
$sql7="SELECT COUNT(CO) FROM GnAppItms WHERE CO='4' AND MountID='".$mountID."'";
$stmt7=sqlsrv_query($conn2,$sql7);
$row7=sqlsrv_fetch_array($stmt7);
You can do it by using group by in sql server
example :
create table a
(id int,
mountid nvarchar(100),
co int,
)
insert into a values (1,'aa',1)
insert into a values (2,'aa',2)
insert into a values (3,'aa',1)
insert into a values (4,'aa',2)
insert into a values (5,'aa',3)
Query
select co,count(co)as countofco from a
where mountid='aa'
group by
co
result
co countofco
1 2
2 2
3 1
Note : Beware of SQL injection when you are writing a sql query, so always use parametrized query. You can edit the above example code and make it as a parametrized query for preventing sql injection
I am trying to use Dapper support my data access for my server app.
My server app has another application that drops records into my database at a rate of 400 per minute.
My app pulls them out in batches, processes them, and then deletes them from the database.
Since data continues to flow into the database while I am processing, I don't have a good way to say delete from myTable where allProcessed = true.
However, I do know the PK value of the rows to delete. So I want to do a delete from myTable where Id in #listToDelete
Problem is that if my server goes down for even 6 mintues, then I have over 2100 rows to delete.
Since Dapper takes my #listToDelete and turns each one into a parameter, my call to delete fails. (Causing my data purging to get even further behind.)
What is the best way to deal with this in Dapper?
NOTES:
I have looked at Tabled Valued Parameters but from what I can see, they are not very performant. This piece of my architecture is the bottle neck of my system and I need to be very very fast.
One option is to create a temp table on the server and then use the bulk load facility to upload all the IDs into that table at once. Then use a join, EXISTS or IN clause to delete only the records that you uploaded into your temp table.
Bulk loads are a well-optimized path in SQL Server and it should be very fast.
For example:
Execute the statement CREATE TABLE #RowsToDelete(ID INT PRIMARY KEY)
Use a bulk load to insert keys into #RowsToDelete
Execute DELETE FROM myTable where Id IN (SELECT ID FROM #RowsToDelete)
Execute DROP TABLE #RowsToDelte (the table will also be automatically dropped if you close the session)
(Assuming Dapper) code example:
conn.Open();
var columnName = "ID";
conn.Execute(string.Format("CREATE TABLE #{0}s({0} INT PRIMARY KEY)", columnName));
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = ids.Count;
bulkCopy.DestinationTableName = string.Format("#{0}s", columnName);
var table = new DataTable();
table.Columns.Add(columnName, typeof (int));
bulkCopy.ColumnMappings.Add(columnName, columnName);
foreach (var id in ids)
{
table.Rows.Add(id);
}
bulkCopy.WriteToServer(table);
}
//or do other things with your table instead of deleting here
conn.Execute(string.Format(#"DELETE FROM myTable where Id IN
(SELECT {0} FROM #{0}s", columnName));
conn.Execute(string.Format("DROP TABLE #{0}s", columnName));
To get this code working, I went dark side.
Since Dapper makes my list into parameters. And SQL Server can't handle a lot of parameters. (I have never needed even double digit parameters before). I had to go with Dynamic SQL.
So here was my solution:
string listOfIdsJoined = "("+String.Join(",", listOfIds.ToArray())+")";
connection.Execute("delete from myTable where Id in " + listOfIdsJoined);
Before everyone grabs the their torches and pitchforks, let me explain.
This code runs on a server whose only input is a data feed from a Mainframe system.
The list I am dynamically creating is a list of longs/bigints.
The longs/bigints are from an Identity column.
I know constructing dynamic SQL is bad juju, but in this case, I just can't see how it leads to a security risk.
Dapper request the List of object having parameter as a property so in above case a list of object having Id as property will work.
connection.Execute("delete from myTable where Id in (#Id)", listOfIds.AsEnumerable().Select(i=> new { Id = i }).ToList());
This will work.
i just created a java file to parse a csv files and saved them into an oracle database.. but i need a field ID which acts as a primary key.. and i am a bit confused abt looping..
I think all you need to do is utilize a sequence (as suggested by Ronnis)
as such
CREATE SEQUENCE FIELD_ID_SEQ START WITH 1 INCREMENT BY 1 NOCYCLE NOCACHE;
/*NOTE THE SEQUENCE, WHILE INCREMENTING, IS NOT GUARANTEED TO BE 1,2,3,4...N ->expect gaps in the #*/
Now either in your java app where you are saving the data:
"INSERT INTO TABLE_OF_CSV(FIELD_ID, FIELD_COLA, FIELD_COLB) VALUES(FIELD_ID_SEQ.NEXTVAL, ?,?);"
OR
Now if you are using a procedure (or a procedure within a package) you can do this (note this returns the primary key back to the calling app)
create procedure insertIntoCSVTable(pCOLA IN TABLE_OF_CSV.FIELD_COLA%TYPE
, pCOLB IN TABLE_OF_CSV.FIELD_COLB%TYPE
, pFIELD_ID OUT TABLE_OF_CSV.FIELD_ID%TYPE)
AS
BEGIN
INSERT INTO TABLE_OF_CSV(FIELD_ID, FIELD_COLA, FIELD_COLB)
VALUES(FIELD_ID_SEQ.NEXTVAL, pCOLA, pCOLB)
RETURNING FIELD_ID
INTO pFIELD_ID
;
END insertIntoCSVTable;
no looping required assuming you are already looping in your java code (assuming a row-by-row insert)
OR
You may use a trigger to insert a new value into the table:
create or replace
TRIGGER TABLE_OF_CSV_TRG BEFORE INSERT ON TABLE_OF_CSV
FOR EACH ROW
BEGIN
<<COLUMN_SEQUENCES>>
BEGIN
IF :NEW.FIELD_ID IS NULL THEN
SELECT FIELD_ID_SEQ.NEXTVAL INTO :NEW.FIELD_ID FROM DUAL;
END IF;
END COLUMN_SEQUENCES;
END;