Situation
In my new role, I need to check the ETL logic that is populating several of our tables. We deploy SSIS projects to manage our ETL workflow. Often we get feedback that a specific column may not have the expected data. In that situation, I will examine the ETL logic to make sure we are bringing the data correctly.
Complication
It is very time-consuming for me to identify which SSIS projects populate certain tables. We have many SSIS packages.
Question
Can I write a query to search the Integration Services Catalog (SSIS Projects) for a table reference?
Our SSIS Projects are stored in a separate database in a folder called Integration Services Catalog. Packages are not stored in the msdb system database.
SELECT
f.NAME AS FolderName
,sib.description
,sib.name as ProjectName
,sib.created_time
,sib.project_id
FROM [SSISDB].[internal].[projects] SIB
INNER JOIN internal.folders F ON F.folder_id = SIB.folder_id
I have found information about the projects but I can't see any XML, or code to search for table references. Also, I know I could search the SSIS project with Powershell, but it seems these are all stored on the server and I have no idea how to find the files to search from PowerShell.
We use SQL Server 2016.
The table references will be inside the XML of the SSIS Package. If you right click on a .dtsx and select edit with notepad++, this will open the xml and allow you to see what is being called. A quick way of doing this to go into you windows search box, type in 'Indexing Options', add the folder where your SSIS solutions are being held, go to advanced options select 'file types' and change the type of index to 'Index Properties and File Options'.
Now when you go to the Folder and type in the table name of the offending column the affected packages will appear. The only thing with this method is if you have some EXECUTE SQL TASKS that are just EXEC usp...xyz. It won't find the table name. You would then need to index the .sql of the saved stored procedures to see if that table name is referenced.
Hope this helps.
Related
I have database in SQL Server called 'XYZ'. Now I want to change it to 'ABC'.
The problem is that my SSRS reports and SSIS packages are connected to the XYZ.
Everything that I have build SSRS reports & SSIS is now live, users using this Reports 24/7.
Is there any way to rename database with minimum/without any server/database downtime?
Thanks
Here's a Rube Goldberg approach:
Create a new, empty database that has the name you're ultimately intending on renaming your current database to (in your example "ABC")
Create a synonym in your new database for every object referenced by your SSIS packages and SSRS reports that uses a three-part name as the target. For example: create synonym [ABC].[dbo].[myTable] for [XYZ].[dbo].[myTable]
Update your packages and reports to point to the new database.
Under cover of darkness, rename ABC to ABC_drop and XYZ to ABC.
Drop ABC_drop.
It doesn't eliminate downtime, but does give you time to update all of the report and ETL package references. The rollback is also simple before step 5.
I have been told an entity called table_loader in a database, database_1 in SSMS (version 2008 R2) exists and needs to be fixed. It is not obviously a stored proc. It's purpose is to convert an excel spreadsheet to table data. Is there any easy way to search a database for an entity name in SSMS.
The find function appears to work only with text SQL files as opened in SSMS.
Since originally posting I have found out from a colleague that this entity is a DTS package; however, I believe searching a database for a name is still a useful thing to be able to do, especially if you don't know what "layer" the entity is in with respect to the database folder structure.
Thanks.
A great, free, tool is Red-Gate SQL Search. It lets you search for just about any object from SSMS in a very user-friendly manner. http://www.red-gate.com/products/sql-development/sql-search/. You just type in the object name and it will search across databases and object types and display what it finds. I like it because it also searches within sproc text and such which can be very helpful, depending on what you're looking for.
If you open up a query window in SSMS you can use the below SQL to do a wildcard search:
USE [dbname]
SELECT * FROM sysobjects WHERE name like '%table_loader%'
This thread has got some good queries and lists the xtype meanings (sproc, table, key etc):
How do I get list of all tables in a database using TSQL?
I want to dump one SQL Server database - get all SQL code necessary to create a similar database. I have full online rights to DatabaseA, I can feed it with SQL and get the results back in rows in a table.
I do not have the possibility to use Enterprise Manager, any applications, utilities or the like. I can only feed it with pure SQL.
What I am after is SQL code, like CREATE TABLE and so on. So that I just can paste this into a query and voila - tables, procedures, functions are created in DatabaseB.
I will not copy the data.
This partly does what I want, it gives me procedures and functions:
Select object_Name(object_ID),definition from sys.SQL_Modules
But not for tables.
You can use the command line or you can create a stored procedure to create a back up, then use that backup to create a new database. I have used the command line often. Here is a previous Stack question that has a command line example and a link to a stored procedure example.
You can generate scripts in SQL Server Management Studio for an entire database or specific database objects.
To do this, right click the database then select Tasks then select Generate Scripts.
It will then open a wizard which will give you the option to choose to script the full database or just specific database objects.
SSIS seems to insist on logging to the system table SYSSSISLOG. Is there a way to make it use a different table?
I want each package to log to a different table.
Quick answer is the same as John Sansom's answer: When logging is used, it creates a table and a stored proc (name varies with version between 2005 and 2008) The stored proc can be modified to do whatever you want. If the stored proc is removed Sql server re-creates it, but if the stored proc is there, Sql server assumes it is OK and leaves it alone. This allows you to modify the stored proc to write to whatever table/tables you want.
Well, you can query that huge-ass log table with something like this:
--first, we identify the packages
;with DetectedPackages as (
select source, s.executionid
from dbo.sysssislog as s
where event = 'PackageStart'
group by source, s.executionid
)
--then we use those executionids to display results
select * from dbo.sysssislog as s
join DetectedPackages dp on s.executionid = dp.executionid
where dp.source = 'PackageName'
And if you want to encapsulate every package in a view, now you know how to do that.
Take a look at the following article over on SQL Server Central, you may need to register but it's free to do so and you will find the site to be excellent SQL Server resource.
The article details how to implement a custom Log Provider that redirects the SSIS log output to another table. Using this implementation as your framework you could extend it to meet your requirements.
SSIS Custom Logging the Easy Way
The above is quite correct however not written well. When you specify your logging in SSIS you can log to a specific data provider IE SSIS Log provider for SQL Server. When you point this to a specific database it will create a [dbo].[sysssislog] table under the System Tables folder in your database. If you navigate in SSMS to your database and programmability -> Stored Procedures there will be a procedure called [dbo].[sp_ssis_addlogentry] this will insert log entries from SSIS. You can repoint this stored procedure to point to the table you want to log to instead of the one generated by SSIS within your database.
I would like to copy a table from one database to another. I know you can easily do the following if the databases are on the same SQL Server.
SELECT * INTO NewTable FROM existingdb.dbo.existingtable;
Is there any easy way to do this if the databases are on two different SQL Servers, without having to loop through every record in the original table and insert it into the new table?
Also, this needs to be done in code, outside of SQL Server Management Studio.
Yes. add a linked server entry, and use select into using the four part db object naming convention.
Example:
SELECT * INTO targetTable
FROM [sourceserver].[sourcedatabase].[dbo].[sourceTable]
If it’s only copying tables then linked servers will work fine or creating scripts but if secondary table already contains some data then I’d suggest using some third party comparison tool.
I’m using Apex Diff but there are also a lot of other tools out there such as those from Red Gate or Dev Art...
Third party tools are not necessary of course and you can do everything natively it’s just more convenient. Even if you’re on a tight budget you can use these in trial mode to get things done….
Here is a good thread on similar topic with a lot more examples on how to do this in pure sql.
SQL Server(2012) provides another way to generate script for the SQL Server databases with its objects and data. This script can be used to copy the tables’ schema and data from the source database to the destination one in our case.
Using the SQL Server Management Studio, right-click on the source database from the object explorer, then from Tasks choose Generate Scripts.
In the Choose objects window, choose Select Specific Database Objects to specify the tables that you will generate script for, then choose the tables by ticking beside each one of it. Click Next.
In the Set Scripting Options window, specify the path where you will save the generated script file, and click Advanced.
From the appeared Advanced Scripting Options window, specify Schema and Data as Types of Data to Script. You can decide from here if you want to script the indexes and keys in your tables. Click OK.
Getting back to the Advanced Scripting Options window, click Next.
Review the Summary window and click Next.
You can monitor the progress from the Save or Publish Scripts window. If there is no error click Finish and you will find the script file in the specified path.
SQL Scripting method is useful to generate one single script for the tables’ schema and data, including the indexes and keys. But again this method doesn’t generate the tables’ creation script in the correct order if there are relations between the tables.
Microsoft SQL Server Database Publishing Wizard will generate all the necessary insert statements, and optionally schema information as well if you need that:
http://www.microsoft.com/downloads/details.aspx?familyid=56E5B1C5-BF17-42E0-A410-371A838E570A
Generate the scripts?
Generate a script to create the table then generate a script to insert the data.
check-out SP_ Genereate_Inserts for generating the data insert script.
Create the database, with Script Database as... CREATE To
Within SSMS on the source server, use the export wizard with the destination server database as the destination.
Source instance > YourDatabase > Tasks > Export data
Data Soure = SQL Server Native Client
Validate/enter Server & Database
Destination = SQL Server Native Client
Validate/enter Server & Database
Follow through wizard