Good Day,
I am attempting to use SSIS to bulk insert XML files into a sql database. Within the Bulk Insert Task editor there are two options beneath the Connection section of the window beneath heading Format, they are "Specify" and "Use File". "Specify" appears to speak to traditional files, and I am thinking this is not applicable to xml files (?). The other option is "Use File", to exercise this option what would I need to do in relation to my source file?
Thank you.
I was able to achieve my goal of bulk inserting all xml files from a directory using the following script within a "Execute SQL Task" task. Ensure that within your "Execute SQL Task" that option ByPassPrepare is set to "True". Also on the Parameter Mapping section your parameter name MUST be preceded by an "#" character. You cannot call your variable in your sql by the name you just defined, a question mark must be used. If you are making multiple variable calls in the same script you have to adjust how you are calling the question marks. Within the For Each loop container where the execute sql task resides go to the Collection section, define your folder where the files reside that you want to load (Folder), and define your files (Files) as *.xml. Within the "Parameter Mapping" section register the user defined variable that contains the file path (Remember must contain an actual file in the variable definition).
declare #sql nvarchar(max);
set #sql = '
INSERT INTO testXMLwithOpenXML(XMLData, LoadedDateTime)
SELECT CONVERT(XML, BulkColumn) AS BulkColumn, GETDATE()
FROM OPENROWSET(BULK ''' + ? + ''', SINGLE_BLOB) AS x;'
exec(#sql)
Related
I want to automate uploading certain files into my SQL Server every day, but each file has a different name.
I was researching and found a simple way was to schedule a bulk insert statement to run every day, but I don't know how to implement the file name change in the query. I'm not too familiar with using Windows command prompt but I'm open to using that as a solution.
The file name change is something like mmddyyyyfile with the mmddyyyy part changing to correspond with the day's date.
We use this technique in our system bulk loads when we have regular file extracts to import, similar to what you describe in your situation. If you have access to and are willing to use xp_cmdshell (which is sounds like you are) then doing something like this allows for dynamic filenames and you don't have to worry about what your date pattern is:
SET NOCOUNT ON;
DECLARE #cmdstr VARCHAR(1024) = 'dir c:\upload /B'; --set your own folder path here
DECLARE #FileName VARCHAR(1024);
DROP TABLE IF EXISTS #CmdOutput;
CREATE TABLE #CmdOutput (CmdOutput varchar(1024));
INSERT #CmdOutput EXEC master..xp_cmdshell #cmdstr;
DECLARE FILES CURSOR FAST_FORWARD FOR
SELECT CmdOutput
FROM #CmdOutput
WHERE CmdOutput IS NOT NULL;
OPEN FILES;
FETCH NEXT FROM FILES INTO #FileName;
WHILE ##FETCH_STATUS = 0
BEGIN
/*
Use dynamic SQL to do your bulk load here based on the value of #FileName
*/
FETCH NEXT FROM FILES INTO #FileName;
END;
CLOSE FILES;
DEALLOCATE FILES;
DROP TABLE #CmdOutput;
This will blindly take any file in the folder and include it in the cursor iterations. If the folder containing your .csv files will have something else that you don't then you can easily add filtering to the WHERE clause that defines the cursor to limit the files.
Finally, the obligatory warning about enabling and using xp_cmdshell on your SQL Server instance. I won't go into the details about that here (there is ample information that can be searched out), but suffice to say it is a security concern and needs to be used with the understanding of risks involved.
I am running the following code and I am getting a syntax error near '.TableReference' error, the code use to work then I did something and now I have this error and I can't seem to find the issue
Through troubleshooting I have narrowed the code issue to the ' FROM ' + #TableName section but it appears to be good code.
BEGIN
--SET NOCOUNT ON;
DECLARE #TableName AS NVARCHAR(MAX) --The Fully qualified database name
DECLARE #Ref AS NVARCHAR(MAX) --The name of the Table we are processing
DECLARE #TempTab AS NVARCHAR(MAX) --the temporary table we are subjecting to the tortures of this process
DECLARE #TempQuery AS NVARCHAR(MAX) --Query to move all data into the temporary table
--This selects the first record in the Website Request Table which hasn't been processed and passes it into the TempTab variable
SET #NDTRef = (SELECT TOP 1 Reference from dbo.WebRequestTable Where Processing IS NULL)
SET #TableName = 'Processing.dbo.'+#NDTRef
Set #TempTab = 'TEMP' + #NDTRef
SET #TempQuery = 'SELECT * INTO '+ #TempTab +' FROM ' + #TableName
EXEC sp_sqlexec #TempQuery;
END
Any help would be appreciated it is a stand alone instance of SQL Server 2019 and the code is a part of a stored procedure but the rest of the code runs off the temporary table created in this block
After suggestions I put in a print statement regarding the #TempQuery when put straight after and the EXEC removed the output is
SELECT * INTO TEMP2294690 FROM Processing.dbo.2294690
With the EXEC back in play I get the error
Msg 102, Level 15, State 1, Line 17 Incorrect syntax near '.2294690'.
The print output after the EXEC shows:
SELECT * INTO TEMP2294690 FROM Processing.dbo.2294690
The Table 2294690 exists in the database Processing the Temp2294690 is a table that should be created by this block but it isn't being created
In SQL Server, regular Identifiers must begin with a letter, an underscore (_), at sign (#) or the number sigh (#).
(There are other rules as well, but this is the one relevant to the question...)
Identifiers that don't follow the rules of regular identifiers can be only used if they are enclosed in square brackets ([]) or double quotation marks (").
The best way to handle identifiers when creating dynamic SQL statements is to use the built in QUOTENAME function - this way you can make sure your query doesn't break even if the identifier doesn't follow the rules of regular identifiers.
So your SQL should look like this:
SET #TableName = '[Processing].[dbo].'+ QUOTENAME(#NDTRef)
SET #TempTab = 'TEMP' + #NDTRef
SET #TempQuery = 'SELECT * INTO '+ QUOTENAME(#TempTab) +' FROM ' + #TableName
That being said, you should also probably check my blog post entitled The do’s and don’ts of dynamic SQL for SQL Server where you can find some more information about how to safely create dynamic SQL.
You know that if the view is temporary only you can see it and if you close your session user the view is deleted and dont save try with create view dont temp and drop after you read.
If all previus you have it in your mind you could try this for see if the view has created --> its a extract of Microsoft official pagge:
VIEW_METADATA
Specifying the instance of SQL Server will return the DB-Library, ODBC, and OLE DB APIs the metadata information about the view instead of the base tables when you request the browse mode metadata for a query that references the view. Browse mode metadata is additional metadata that the SQL Server instance returns to these client-side APIs. This metadata enables client-side APIs to implement updateable client-side cursors. Browse mode metadata includes information about the base table to which the columns in the result set belong.
I need to move files from one blob to another. Not copy. I need to move; meaning when I move from A to B then files go from A to B and nothing is there in A. Which is so basic but not possible in Azure Blob. Please let me know if its possible. I am using it from SQL Server using AzcopyVersion 10.3.2
Now because of this, I need to copy files from A to B and then remove files form A. There are 2 problems.
1) I only want certain files to go from A to B.
DECLARE #Program varchar(200) = 'C:\azcopy.exe'
DECLARE #Source varchar(max) = '"https://myblob.blob.core.windows.net/test/myfolder/*?SAS"'
DECLARE #Destination varchar(max) = '"https://myblob.blob.core.windows.net/test/archive?SAS"'
DECLARE #Cmd varchar(5000)
SELECT #Cmd = #Program +' cp '+ #Source +' '+ #Destination + ' --recursive'
PRINT #cmd
EXECUTE master..xp_cmdshell #Cmd
So When I type myfolder/* then it will take all the files. When I try myfolder/*.pdf, it says
failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A
When I try myfolder/%2A.pdf OR myfolder/%2Apdf it still gives the error.
INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.
But the destination folder is already there. And in the log file it says,
RESPONSE Status: 403 This request is not authorized to perform this operation.
For azcopy version 10.3.2:
1.Copy specific files, like only copy .pdf files: you should add --include-pattern "*.pdf" to your command. And also remember for the #Source variable, remove the wildcard *, so your #Source should be '"https://myblob.blob.core.windows.net/test/myfolder?SAS"'.
The completed command looks like this(please change it to meet your sql cmd):
azcopy cp "https://xx.blob.core.windows.net/test1/folder1?sas" "https://xx.blob.core.windows.net/test1/archive1?sas" --include-pattern "*.pdf" --recursive=true
2.For delete specific blobs, like only delete .pdf files, you should also add --include-pattern "*.pdf" to your azcopy rm command.
And also, there is no move command in azcopy, you should copy it first => then delete it. You can achieve this with the above 2 commands.
As I am a beginner in SQL Server and my scripting is not very polished yet. I need suggestions on the below issue.
I receive files from a remote server to my machine (around 700/day) as follows :
ABCD.100.1601310200
ABCD.101.1601310210
ABCD.102.1601310215
Naming Convention:
Here the first part 'ABCD' remains the same, middle part is a sequence id which is in incremental order for every file. The last part is time stamp.
File structure
The file does not have any specific extension but can be opened with notepad/excel. Therefore can be called as flat file. Each files consist of 95 columns and 20000 rows fixed with some garbage value on top 4 and bottom 4 rows of column 1.
Now, I need to make a database in SQL server where I can import data from these flat files using a scheduler. Suggestion needed.
There are probably other ways of doing this, but this is one way:
Create a format file for your tables. You only need to create it once. Use this file in the import script in step 2.
Create an import script based on OPENROWSET(BULK '<file_name>', FORMATFILE='<format_file>'
Schedule the script from step 2 in SQL Server to run against the database you want the data imported in
Create the format file
This creates a format file to be used in the next step. The following script creates a format file in C:\Temp\imp.fmt based on an existing table (replace TEST_TT with the database you are importing to). This creates such a format file with a , as field seperator. If the files have tab as seperator, remove the -t, switch.
DECLARE #cmd VARCHAR(8000);
SET #cmd='BCP TEST_TT.dbo.[ABCD.100.1601310200] format nul -f "C:\Temp\imp.fmt" -c -t, -T -S ' + (SELECT ##SERVERNAME);
EXEC master..xp_cmdshell #cmd;
Before executing this you will to reconfigure SQL Server to allow the xp_cmdshell stored procedure. You only need to do this once.
EXEC sp_configure 'show advanced options', 1
GO
RECONFIGURE
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
import script
This script assumes:
The files need to be imported to separate tables
The files are located in C:\Temp
The format file is C:\Temp\imp.fmt (generated in the previous step)
SET NOCOUNT ON;
DECLARE #store_path VARCHAR(256)='C:\Temp';
DECLARE #files TABLE(fn NVARCHAR(256));
DECLARE #list_cmd VARCHAR(256)='DIR ' + #store_path + '\ABCD.* /B';
INSERT INTO #files EXEC master..xp_cmdshell #list_cmd;
DECLARE #fullcmd NVARCHAR(MAX);
SET #fullcmd=(
SELECT
'IF OBJECT_ID('''+QUOTENAME(fn)+''',''U'') IS NOT NULL DROP TABLE '+QUOTENAME(fn)+';'+
'SELECT * INTO '+QUOTENAME(fn)+' '+
'FROM OPENROWSET(BULK '''+#store_path+'\'+fn+''',FORMATFILE=''C:\Temp\imp.fmt'') AS tt;'
FROM
#files
WHERE
fn IS NOT NULL
FOR XML PATH('')
);
EXEC sp_executesql #fullcmd;
I using ssis package.I want insert flat file source (text file) to sql.
Addres of text file is dynamic so i define variable for path.I have the sp like this:
CREATE PROCEDURE [dbo].[Insert_FileMaster]
#FILE_PATH nVARCHAR(MAX)
,#id int OUTPUT
AS
BEGIN
insert into [dbo].[FileMaster] ([FM_Name])
values(#FILE_PATH))
set #id = ##IDENTITY
END
I want exec this sp With variable parameter.
this is my package:
Which ssis tool should i use?and how to get output from sp (return parametert must be use in another sp in package)?
You will want to add an Execute SQL Task before your Data Flow Task (this will be at the Control Flow level).
You will need to configure the Execute SQL task as described in this answer.
Insert a single row and return its primary key