I need to integrate two similar databases into third DB3. DB3 is almost the same as DB1.
First database DB1:
Addresses table with: primary key AddressId
People table with: primary key PersonId , foreign key AddressId
Second database DB2:
It is pretty similar, but in other language
Data from DB1 to DB3 flows smoothly, table after table. For example I have 1000 records in DB3 table named Addresses from DB1 and 1000 records in table named People from DB1.
Let's suppose Person with number 30 in DB3 (after transfering from DB1) has the IdAddress number 20.
Address with number 40 in DB2 has the ID number 1040 in DB3 and the Person has ID number 30 in DB2 and 1030 in DB3.
While transferring table People from B2 to B3 we need to know the address ID is not 40 but 1040.
I'm trying to use lookup to find existing record, but I'm newbie in SSIS VS designer. Could you help me? How can I resolve this problem?
Suggestions
You can do this using Lookup Transformation component as you mentioned, but first you have to:
Select the basic information of each table that can distinguish each logical entity. Example if talking about Persons you can choose the Fullname+ Mothername + Date Of Birth , ...
After selecting this attributes you have to add a Lookup Transformation
Map thiese columns between Source and Lookup table
Select the ID column (from Lookup table) as Output and rename the column to be NewID
Select Ignore failure option to handle non matches situation
After doing these steps, if the same person was inserted previously you will get the ID in NewID column, else NewID is NULL
Additional Information
Microsoft Docs - Lookup Transformation
UNDERSTAND SSIS LOOKUP TRANSFORMATION WITH AN EXAMPLE STEP BY STEP
SSIS Lookup Transformation
Lookup Transformation in SSIS
Related
I have a task to Generate a Derived Column (RestrictionID) from a table and add it to a source table if 6 columns on both tables match. The 6 columns include (Country, Department, Account .etc) I decided to go with the SSIS Lookup and generate the Derived column when there's a match. This ID also has a time and amount Limit. Once all the records have ID's, I'm supposed to calculate a running total based on the ID to enforce the limits which is the easy part.
The only problem is this Lookup table changes almost daily and any or all of the 6 columns can have NULLS. Even the completely null rows have an Id. Nulls mean the Restriction is open. eg. If the Country column on one record on the lookup table is null, then the ID of that record can be assigned to records with any country on the source. If one row on the lookup has all null columns, then this is completely open and all records on the source qualify for that ID. The Source table doesn't have NULLS.
Please assist if possible
Thanks
If NULL means any and ignore column in lookup then add this to your where:
use a stored proc and pass your values in and return:
select lookup.ID
from lookup
where #Country = isnull(lookup.Country,#Country) //If lookup inull then it refers to itself creating a 1=1 scenario
and #Department = isnull(lookup.Department,#Department)
and ...
i have problem to Transfer data from one sqlserver 2008 r2 to another sql server 2012 databases with different schema, here is some different scenario,
database 1
database 1 with tables Firm and Client, these both have FirmId and ClientId primary key as int datatype,
FirmId is int datatype as reference key used in Client table.
database 2
database 2 with same tables Firm and Client, these both have FirmId and ClientId but primary key as uniqueidentifier,
FirmId is uniqueidentifier datatype as reference key used in Client table.
problem
the problem is not to copy data from 1 database table to 2 database table, but the problem is to maintain the reference key's Firm table into Client table. because there is datatype change.
i am using sql server 2008 r2 and sql server 2012
please help me to resolve / find the solution, i really appreciate your valuable time and effort. thanks
I'll take a stab at it even if I am far from an expert on SQLServer - here is a general procedure (you will have to repeat it for all tables where you have to replace INT with UID, of course...).
I will use Table A to refer to the parent (Firm, if I understand your example clearly) and Table B to refer to the child (Client, I believe).
Delete the relations pointing to Table A
Remove the identity from the id column of Table A
Create a new column with Uniqueidentifier on Table A
Generate values for the Uniqueidentifier column
Add the new Uniqueidentifier column in all the child tables (Table B)
Use the OLD id column to map your child record & update the new Uniqueidentifier value from your parent table.
Drop all the id columns
Recreate the relations
Having said that, I just want to add a warning to you: converting to UID is, according to some, a very bad idea. But if you really need to do that, you can script (and test) the above mentioned procedure.
I have a log table in SQL Server. This table is populated by 2 different sources - UI and goanywhere. This table has a log id column which will have sequence numbers for each entered record. What is the way to populate this column with correct sequence of numbers when the population is from 2 different channels. Suggestions please
In SQL Server, use an INT IDENTITY column in your table:
CREATE TABLE dbo.Log
(
LogID INT NOT NULL IDENTITY(1,1),
.... other columns...
)
and when you insert rows into this table, just don't specify the LogID column in your list of columns to insert into - SQL Server will automatically add values to it.
See the MSDN documentation on IDENTITY for more detail.
Use auto increment feature of mysql . It does not matter how many channels are there.
I am developing a system in which I have a table Employees with multiple columns related to employees. I have a column for the JobTitle and another column for Department.
In my current design, the JobTitle & the Department columns are compound foreign keys in the Employees table and they are linked with the Groups table which has 2 columns compound primary key (JobTitle & Department) and an extra column for the job description.
I am not happy about this design because I think that linking 2 tables using 2 compound varchar columns is not good for the performance, and I think it would be better to have an Integer column (autonumber) JobTitleID used as the primary key in the Groups table and as a foreign key in the Employees table instead of the the textual JobTitle & the Department columns.
But I had to do this because when I import the employees list (Excel) into my Employees table it can just be directly mapped (JobTitle --> JobTitle & Department --> Department). Otherwise if I am using an integer index as primary key I would have then to manually rename the textual JobTitle column in the excel sheet to a number based on the generated keys from the Groups table in order to import.
Is it fine to keep my database design like this (textual compound primary key linked with textual compound foreign key)? If not, then if I used an integer column in the Groups table as primary key and the same as a foreign key in the Employees table then how can I import the employees list from excel directly to Employees table?
Is it possible to import the list from Excel to SQL Server in a way that the textual JobTitle from the excel sheet will be automatically translated to the corespondent JobTitleID from the Groups table? This would be the best solution, I can then add JobTitleID column in the Groups table as a primary key and as a foreign key in the Employees table.
Thank you,
It sounds like you are trying to make the database table design fit the import of the excel file which is not such a good idea. Forget the excel file and design your db tables first with correct primary keys and relationships. This means either int, bigint or guids for primary keys. This will keep you out of trouble unless you absolutely know the key is unique such as in a SSN. The when you import, then populate the departments and job titles into their respective tables creating their primary keys. Now that they are populated, add those keys to the excel file that can be imported into the employees table.
This is just an example of how I would solve this problem. It is not wrong to use multiple columns as the key but it will definitely keep you out of harms way if you stick with int, bigint or guids for your primary keys.
Look at the answer in this post: how-to-use-bulk-insert...
I would create a simple Stored Procedure that imports your excel data into a temporary unrestricted STAGING table and then do the INSERT into your real table by doing the corresponding table joins to get the right foreign keys and dump the rows that failed to import into an IMPORT FAIL table. Just some thoughts...
I have a problem please help me.
suppose I have three databases Db1,Db2,DbCenter and tbl_country exists in all of databases.
tbl_country in Db1 has following records:
tbl_country
Id Name
1 US
2 Germany
tbl_country in Db2 has following records:
tbl_country
Id Name
1 Australia
2 Italy
and the merged records of tbl_country in Db1 and Db2 must be merged in DbCenter,
so tbl_country in DbCenter has following entries:
tbl_country
Id Name
1 Us
2 Germany
3 Australia
4 Italy
the "id" column in all of tables is primary key and identity.
What should I do to the records of two Db1 and Db2 be added in the last of tbl_country of DbCenter.
I'm using transactional replication. set Db1 and Db2 as publishers and specify DbCenter as subscriber and in article properties of Db2 set the action of "Action if name in use" to "Keep existing object unchanged" but it delete the records in DbCenter and substitute them with new ones when the records in "id" column are the same.
Db1 and Db2 are Sql server 2000 version and Db3 is Sql server 2008R2.
Maintain the original key values in the DbCenter database, along with metadata that specifies which database it came from. Then generate a surrogate key that will be used as the PK. That way you both have unique references for each Country records and a way to link back to the source information.
For example:-
ID_SK SOURCE_ID SOURCE_DB NAME
1 1 Db1 US
2 2 Db1 Germany
3 1 Db2 Australia
4 2 Db2 Italy
If Db1 and Db2 have overlapping primary key values, I don't think there is any way to do this. If you have control over the schema, and are able to change it, you might want to change the primary keys to the uniqueidentifier type (instead of int) which is guaranteed to be globally unique.