Create procedure COPY INTO statement with column names from external storage - sql-server

I would like to use the COPY INTO statement to copy a table with it's column names from an external storage.
Would like to achieve it using a generic procedure which can use for different tables.
Here below you find a draft. The input parameters could be temp_file_path, schema, table name and list of column names of destination table.
Variable temp_file_path is location of files in Azure data lake(dl).
The variable on column list is a placeholder. Need to know how we can implement it.
Other suggestions are welcome.
CREATE PROC [copy_into_sql_from_dl_columns]
#temp_file_path [VARCHAR](4096)
, #dest_schema [VARCHAR](255)
, #dest_table [VARCHAR](255)
, #dest_columns [VARCHAR](255)
AS
IF #temp_file_path IS NULL OR #dest_schema IS NULL OR #dest_table IS NULL OR #dest_columns IS NULL
BEGIN
PRINT 'ERROR: You must specify temp_file_path, destination schema, table and column names.'
END
ELSE
BEGIN
DECLARE #dest_temp_table AS VARCHAR(4096)
SET #dest_temp_table = '['+#dest_schema + '].[' + #dest_table + ']'
-- set target column names into a target temp variable #dest_temp_columns_list
DECLARE #copy_into_query AS VARCHAR(8000)
SET #copy_into_query = 'COPY INTO ' + #dest_temp_table + ' ('+ #dest_temp_columns +')'+' FROM ''' #temp_file_path + ''' WITH (FILE_TYPE = ''parquet'', AUTO_CREATE_TABLE = ''OFF'' ) ';
EXEC (#copy_into_query)
END
GO
Environment is Azure cloud synapse
DB is SQL dedicated pool (Azure Synapse Analytics)

Can you use SELECT INTO?
SELECT *
INTO newtable [IN externaldb]
FROM oldtable
WHERE condition;

Create procedure COPY INTO statement with column names from external storage
I repro'd this and below is the approach.
Two sample parquet files are taken in azure data lake storage. This is considered as source.
Similarly, Target tables are created with the same structure as source files in the dedicated SQL pool.
A lookup table is created in dedicated SQL pool to store the details like File Path of source data lake, Schema and table name of target and column names of target table.
create table lkp_table
(temp_file_path VARCHAR(255),
dest_schema VARCHAR(100),
dest_table varchar(100),
dest_columns varchar(100) )
Then Stored procedure script is written in dedicated SQL pool to copy data from external storage to dedicated SQL pool. (Few changes are made in the Q's stored procedure script).
Create PROC [copy_into_sql_from_dl_columns]
#temp_file_path [VARCHAR](4096)
, #dest_schema [VARCHAR](255)
, #dest_table [VARCHAR](255)
, #dest_columns [VARCHAR](255)
AS
IF #temp_file_path IS NULL OR #dest_schema IS NULL OR #dest_table IS NULL OR #dest_columns IS NULL
BEGIN
PRINT 'ERROR: You must specify temp_file_path, destination schema, table and column names.'
END
ELSE
BEGIN
DECLARE #dest_temp_table AS VARCHAR(4096)
SET #dest_temp_table = '['+#dest_schema + '].[' + #dest_table + ']'
-- set target column names into a target temp variable #dest_temp_columns_list
DECLARE #copy_into_query AS VARCHAR(8000)
SET #copy_into_query = 'COPY INTO ' + #dest_temp_table + ' ('+ #dest_columns +')'+' FROM ''' + #temp_file_path + ''' WITH (FILE_TYPE = ''parquet'', AUTO_CREATE_TABLE = ''OFF'' ) ';
EXEC (#copy_into_query)
END
Then, In Azure Synapse/data factory, Lookup activity is taken and dataset for the lookup activity is referred to lookup table that is created in dedicated SQL pool.
Then For-each activity is created and in items of for-each activity,
#activity('Lookup1').output.value is given.
Then inside foreach activity, Execute stored procedure activity is added
Linked Service is selected and Stored procedure name is given.
Import Parameter is selected and value is given in the parameters
dest_columns: #item().dest_columns
dest_schema: #item().dest_schema
dest_table: #item().dest_table
temp_file_path: #item().temp_file_fath
Once pipeline is run, all files from ADLS are copied to Dedicated SQL pool.

Related

Table already exists in a new SQL Server database

I'm writing a script to create a bunch of tables in SQL Server. As I write the script, I want to create and delete the database. The problem is that the I get an error saying that the object already exists.
Here is my example script
DECLARE #db_name varchar(20);
DECLARE #command varchar(100);
SET #db_name='testdb';
SET #command='DROP DATABASE ' + #db_name
IF EXISTS(SELECT * FROM sys.databases WHERE name=#db_name)
exec(#command)
SET #command='CREATE DATABASE ' + #db_name
EXEC(#command)
--listing databaes
SELECT name from master.dbo.sysdatabases
-- using our database
SET #command='USE ' + #db_name
EXEC(#command)
PRINT 'listing tables'
SET #command = 'SELECT table_name FROM ' + #db_name + '.INFORMATION_SCHEMA.TABLES WHERE table_type = "base TABLE"'
EXEC(#command)
CREATE TABLE stuff(
name VARCHAR(30) PRIMARY KEY NOT NULL,
weight INT,
quantity INT)
and the output I get is
name
------------
master
tempdb
model
msdb
testdb
SW
(6 rows affected)
listing tables
table_name
Error:
Msg 2714, Level 16, State 6, Server Orange, Line 22
There is already an object named 'stuff' in the database.
I run this on a Linux mint machine, a freshly installed SQL Server, and I use sqlcmd. I guess I can put a drop/delete command before the creating the table, but this shouldn't happen to begin with. What is going on here?
When you execute a USE statement from dynamic SQL, the database context reverts back to the original database context (master?) when the executed batch completes. You'll need to add a USE to the CREATE TABLE script and execute it using dynamic SQL too:
SET #command = N'USE' + QUOTENAME(#db_name) + N';
CREATE TABLE stuff(
name VARCHAR(30) PRIMARY KEY NOT NULL,
weight INT,
quantity INT);
';
bind the create table statement inside a object existence check. like this
IF OBJECT_ID('stuff') IS NULL
BEGIN
CREATE TABLE stuff(
name VARCHAR(30) PRIMARY KEY--NOT NULL is not needed as the primary key does not allow NULL,
weight INT,
quantity INT)
END

How to check existence of a table from a different sql db?

I have db A and db B. At the beginning of a stored procedure I want to back up all rows from B.mytable to B.mytablebackup. The rest of the stored procedure runs against tables on db A (which gathers data and writes it to B.mytable).
So I check to see if B.mytablebackup exists
IF EXISTS(SELECT 1 FROM B.dbo.mytablebackup)
and if it does, the stored procedure does an
INSERT INTO B..mytablebackup SELECT * FROM B..mytable
If it doesn't exist it does a
SELECT * INTO B..mytablebackup from B..mytable
But when I execute the stored procedure I get the error
There is already an object named 'mytablebackup' in the database
I added a Print statement and execution is taking the "does not exist" branch of the IF.
What am I doing wrong?
For SQL Server, you should use system view sys.tables to check if table exists.
IF EXISTS(SELECT 1 FROM B.sys.tables WHERE name = 'mytablebackup')
OBJECT_ID can be used too:
IF OBJECT_ID('B.dbo.mytablebackup') IS NOT NULL
You can directly check from the given DB,SCHEMA and TABLE parameters (For dynamic database, schema and table use)
DECLARE #targetdatabase NVARCHAR(MAX),
#SchemaName NVARCHAR(MAX),
#TableName NVARCHAR(MAX)
DECLARE #TempTableName NVARCHAR(MAX) = QUOTENAME(#targetdatabase) + '.' +
QUOTENAME(#SchemaName) + '.' + QUOTENAME(#TableName)
IF OBJECT_ID(#TempTableName) IS NULL
BEGIN
PRINT #TempTableName
END

SSIS : how to convert the source column from ID to Value

I'm creating a SSIS package to load data from a CSV file to SQL table. The sample CSV file is
EMP_ID,EMP_NAME,DEPT_ID,MANAGER_ID,SALARY
1801,SCOTT,20,1221,3000
1802,ALLEN,30,1221,3400
I need to load data into a SQL Server table, but while loading I need to load Department Name and Manager Name instead of their IDs. So I need to convert the CSV source to
1801,SCOTT,FINANCE,JOHNSON,3000
1802,ALLEN,HR,JOHNSON,3400
The values for Department Name and Manager name come from the SQL Server database only. But how do I query and convert ID to text values?
I'm new to SSIS, please suggest how can I achieve this.
Thanks
John
CREATE PROCEDURE [dbo].[BulkInsert]
(
-- Declare Parameters here for your CSV file
)
AS
BEGIN
SET NOCOUNT ON;
declare #query varchar(max)
CREATE TABLE #TEMP
(
[FieldName] [int] NOT NULL ,
[FieldName] int NOT NULL,
)
SET #query = 'BULK INSERT #TEMP FROM ''' + PathOfYourTextFile + ''' WITH ( FIELDTERMINATOR = '','',ROWTERMINATOR = ''\n'')'
--print #query
--return
execute(#query)
BEGIN TRAN;
MERGE TableName AS Target
-- Now here you can get the value Department Name and Manager Name by using Target.Id --in the table from where you mant to get the value of the Manager Name
USING (SELECT * FROM #TEMP) AS Source
ON (Target.YourTableId = Source.YourTextFileFieldId)
-- In the above line we are checking if the particular row exists in the table(Table1) then update the Table1 if not then insert the new row in Table-1.
WHEN MATCHED THEN
UPDATE SET
Target.SomeId= Source.SomeId
WHEN NOT MATCHED BY TARGET THEN
-- Insert statement
The above code is just an example for you by taking the help from this you can edit in your code. And one more important thing for you, Bulk Insert is one of the great way to save the CSV files. So try to use this..:)
In SSIS package from Data Flow tab use LOOKUP process from the Toolbox. You'll specify the table to get your string values from and which columns to use for the join and the column to substitue your IDs with.

Dynamically named temp table returns "invalid object name" when referenced in stored procedure

When I run the following code, I get an "invalid object name" error, any idea why?
I need to create a dynamically named temp table to be used in a stored procedure.
DECLARE #SQL NVARCHAR(MAX)
DECLARE #SessionID NVARCHAR(50)
SET #SessionID = 'tmp5l7g9q3l1h1n5s4k9k7e'
;
SET
#SQL = N' CREATE TABLE #' + #SessionID + ' ' +
N' (' +
N' CustomerNo NVARCHAR(5), ' +
N' Product NVARCHAR(3), ' +
N' Gross DECIMAL(18,8) ' +
N' )'
;
EXECUTE sp_executesql #SQL
;
SET
#SQL = N' SELECT * FROM #' + #SessionID
;
EXECUTE sp_executesql #SQL
Thanks!
WHY MESS WITH THE NAMES? Let SQL Server will manage this for you:
Temporary Tables in SQL Server
from the above link:
If the same routine is executed simultaneously by several processes,
the Database Engine needs to be able to distinguish between the
identically-named local temporary tables created by the different
processes. It does this by adding a numeric string to each local
temporary table name left-padded by underscore characters. Although
you specify the short name such as #MyTempTable, what is actually
stored in TempDB is made up of the table name specified in the CREATE
TABLE statement and the suffix. Because of this suffix, local
temporary table names must be 116 characters or less.
If you’re interested in seeing what is going on, you can view the
tables in TempDB just the same way you would any other table. You can
even use sp_help work on temporary tables only if you invoke them from
TempDB.
USE TempDB
go
execute sp_Help #mytemp
or you can find them in the system views of TempDB without swithching
databases.
SELECT name, create_date FROM TempDB.sys.tables WHERE name LIKE '#%'
You are doing it wrong!
Try:
exec(#SQL)
instead of:
EXECUTE sp_executesql #SQL
To use sp_executesql the variable must be inside #SessionID the quotes and it must be provided has input parameter. Check this for a full example!
You've to be aware that Dynamic SQL is a good port for SQL injections!
This syntax works
CREATE TABLE #SessionID (CustomerNo NVARCHAR(5), Product NVARCHAR(3), Gross DECIMAL(18,8));
Select COUNT(*) from #SessionID;
Drop Table #SessionID;

How to export a SQL Server 2008 Database Diagram to another DB?

I use the handy Database Diagramming tool in SQL Server 2008 for creating and managing relationships. I have exported the sourceDB to the destinationDB but the diagram doesn't come across.
I am looking around trying to figure out how to export just the diagram I have in one database to another... This online KB article fails since select * from dtproperties doesn't exist anymore.
#Ash I was having the same problem. Here's what we did to get around it...
It seems that System Diagrams are stored within the "sysdiagrams" table. So the first thing you need to do is determine the diagram_id of the Diagram you wish to copy. Run the following query to list them all. ** Note you need to replace "SourceDB" with the name of your database.
-- List all database diagrams
SELECT * FROM [SourceDB].[dbo].sysdiagrams
Then you can use INSERT to duplicate the diagram from one database to another as follows. ** Note again replace "SourceDB" with the name of the Database containing the existing diagram and "DestinationDB" with the name of the Database you wish to copy to. Also #SourceDiagramId should be set to the id retrieved above.
-- Insert a particular database diagram
DECLARE #SourceDiagramId int = 1
INSERT INTO [DestinationDB].[dbo].sysdiagrams
SELECT [name],diagram_id , version,definition from [SourceDB].[dbo].sysdiagrams
WHERE diagram_id = #SourceDiagramId
Then you need to set the "principal_id" to 1 manually.
-- Update the principal id (no idea why, but it set the owner as some asp_net user
UPDATE [DestinationDB].[dbo].sysdiagrams
SET principal_id = 1
This worked for us it seems pretty hacky especially since the Diagram is stored entirely in a single binary field "definition".
Answer comes from:
http://www.dotnetspider.com/resources/21180-Copy-or-move-database-digram-from-for.aspx
This generates an import string:
SELECT
'DECLARE #def AS VARBINARY(MAX) ; ' +
'SELECT #def = CONVERT(VARBINARY(MAX), 0x' + CONVERT(NVARCHAR(MAX), [definition], 2) + ', 2) ;' +
' EXEC dbo.sp_creatediagram' +
' #diagramname=''' + [name] + ''',' +
' #version=' + CAST([version] AS NVARCHAR(MAX)) + ',' +
' #definition=#def'
AS ExportQuery
FROM
[dbo].[sysdiagrams]
WHERE
[name] = '' -- Diagram Name
Next, you run the generated string in other DB.
As PROCEDURE:
-- =============================================
-- Author: Eduardo Cuomo
-- Description: Export Database Diagrama to SQL Query
-- =============================================
CREATE PROCEDURE [dbo].[Sys_ExportDatabaseDiagram]
#name SYSNAME -- Diagram Name
AS
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
SELECT
'DECLARE #def AS VARBINARY(MAX) ; ' +
'SELECT #def = CONVERT(VARBINARY(MAX), 0x' + CONVERT(NVARCHAR(MAX), [definition], 2) + ', 2) ; ' +
' EXEC dbo.sp_creatediagram' +
' #diagramname=''''' + [name] + ''''',' +
' #version=' + CAST([version] AS NVARCHAR(MAX)) + ',' +
' #definition=#def'
AS ExportQuery
FROM
[dbo].[sysdiagrams]
WHERE
[name] = #name
You can get rid of the UPDATE statement by fixing your INSERT statement - specifically the select portion. You are inserting the diagram_id column into the principal_id column (diagram_id is an identity).
Change it to:
DECLARE #SourceDiagramId int = 1
INSERT INTO [DestinationDB].[dbo].sysdiagrams
SELECT [name],principal_id,version,definition from [SourceDB].[dbo].sysdiagrams
WHERE diagram_id = #SourceDiagramId
And presto, it's all in there right the first time.
As in C Isaze answer, there are three simple steps:
1- Create the same number of "dummy" diagrams in the target server where you want to copy the diagrams
2- Add the target server as a Linked Server in the source server
3- run this script on source server
update [LINKEDSERVER].TARGETDB.[dbo].sysdiagrams set [definition]=
(SELECT [definition] from SOURCEDB.[dbo].sysdiagrams WHERE diagram_id = 1)
where diagram_id=1
If the databases are in different servers, there may be permission issues.
To copy the sysdiagrams, create the same number of "dummy" diagrams in the target server where you want to copy the diagrams, add the target server as a Linked Server in the source server and then run the script:
SELECT * from [LINKEDSERVER].TARGETDB.[dbo].sysdiagrams
SELECT * from SOURCEDB.[dbo].sysdiagrams
update [LINKEDSERVER].TARGETDB.[dbo].sysdiagrams set definition=
(SELECT definition from SOURCEDB.[dbo].sysdiagrams WHERE diagram_id = 1)
where diagram_id=1
-- the first 2 select commands will confirm that you are able to connect to both databases
-- then change the id as required to copy all the diagrams
There's a tool for exporting the diagrams to file and back into a database that you can find here: https://github.com/timabell/database-diagram-scm/
You'd be able to use this by pointing it at your original database and doing an export, and then pointing at your target database and doing an import.

Resources