I need to move files from one blob to another. Not copy. I need to move; meaning when I move from A to B then files go from A to B and nothing is there in A. Which is so basic but not possible in Azure Blob. Please let me know if its possible. I am using it from SQL Server using AzcopyVersion 10.3.2
Now because of this, I need to copy files from A to B and then remove files form A. There are 2 problems.
1) I only want certain files to go from A to B.
DECLARE #Program varchar(200) = 'C:\azcopy.exe'
DECLARE #Source varchar(max) = '"https://myblob.blob.core.windows.net/test/myfolder/*?SAS"'
DECLARE #Destination varchar(max) = '"https://myblob.blob.core.windows.net/test/archive?SAS"'
DECLARE #Cmd varchar(5000)
SELECT #Cmd = #Program +' cp '+ #Source +' '+ #Destination + ' --recursive'
PRINT #cmd
EXECUTE master..xp_cmdshell #Cmd
So When I type myfolder/* then it will take all the files. When I try myfolder/*.pdf, it says
failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A
When I try myfolder/%2A.pdf OR myfolder/%2Apdf it still gives the error.
INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.
But the destination folder is already there. And in the log file it says,
RESPONSE Status: 403 This request is not authorized to perform this operation.
For azcopy version 10.3.2:
1.Copy specific files, like only copy .pdf files: you should add --include-pattern "*.pdf" to your command. And also remember for the #Source variable, remove the wildcard *, so your #Source should be '"https://myblob.blob.core.windows.net/test/myfolder?SAS"'.
The completed command looks like this(please change it to meet your sql cmd):
azcopy cp "https://xx.blob.core.windows.net/test1/folder1?sas" "https://xx.blob.core.windows.net/test1/archive1?sas" --include-pattern "*.pdf" --recursive=true
2.For delete specific blobs, like only delete .pdf files, you should also add --include-pattern "*.pdf" to your azcopy rm command.
And also, there is no move command in azcopy, you should copy it first => then delete it. You can achieve this with the above 2 commands.
Related
I want to automate uploading certain files into my SQL Server every day, but each file has a different name.
I was researching and found a simple way was to schedule a bulk insert statement to run every day, but I don't know how to implement the file name change in the query. I'm not too familiar with using Windows command prompt but I'm open to using that as a solution.
The file name change is something like mmddyyyyfile with the mmddyyyy part changing to correspond with the day's date.
We use this technique in our system bulk loads when we have regular file extracts to import, similar to what you describe in your situation. If you have access to and are willing to use xp_cmdshell (which is sounds like you are) then doing something like this allows for dynamic filenames and you don't have to worry about what your date pattern is:
SET NOCOUNT ON;
DECLARE #cmdstr VARCHAR(1024) = 'dir c:\upload /B'; --set your own folder path here
DECLARE #FileName VARCHAR(1024);
DROP TABLE IF EXISTS #CmdOutput;
CREATE TABLE #CmdOutput (CmdOutput varchar(1024));
INSERT #CmdOutput EXEC master..xp_cmdshell #cmdstr;
DECLARE FILES CURSOR FAST_FORWARD FOR
SELECT CmdOutput
FROM #CmdOutput
WHERE CmdOutput IS NOT NULL;
OPEN FILES;
FETCH NEXT FROM FILES INTO #FileName;
WHILE ##FETCH_STATUS = 0
BEGIN
/*
Use dynamic SQL to do your bulk load here based on the value of #FileName
*/
FETCH NEXT FROM FILES INTO #FileName;
END;
CLOSE FILES;
DEALLOCATE FILES;
DROP TABLE #CmdOutput;
This will blindly take any file in the folder and include it in the cursor iterations. If the folder containing your .csv files will have something else that you don't then you can easily add filtering to the WHERE clause that defines the cursor to limit the files.
Finally, the obligatory warning about enabling and using xp_cmdshell on your SQL Server instance. I won't go into the details about that here (there is ample information that can be searched out), but suffice to say it is a security concern and needs to be used with the understanding of risks involved.
I've being searching for this on google but I haven't found any good explanation, so here's my issue.
I need to import product images, which are in a folder to SQL Server, I tried to use xp_cmdshell but without success.
My images are in C:\users\user.name\Images and the images have their names as the product id, just like [product_id].jpg and they're going to be inserted in a table with the product ID and the image binary as columns.
I just need to list the images on the folder, convert the images to binary and insert them in the table with the file name (as the product_id)
My questions are:
How do I list the images on the folder?
How do I access the folder with dots in their name (like user.name)
How do I convert the images to binary in order to store them in the database (if SQL Server doesn't do that automatically)
Thanks in advance
I figured I'd try an xp_cmdshell-based approach just for kicks. I came up with something that does appear to work for me, so I'd be curious to know what problems you ran into when you tried using xp_cmdshell. See the comments for an explanation of what's going on here.
-- I'm going to assume you already have a destination table like this one set up.
create table Images (fname nvarchar(max), data varbinary(max));
go
-- Set the directory whose images you want to load. The rest of this code assumes that #directory
-- has a terminating backslash.
declare #directory nvarchar(max) = N'D:\Images\';
-- Query the names of all .JPG files in the given directory. The dir command's /b switch omits all
-- data from the output save for the filenames. Note that directories can contain single-quotes, so
-- we need the REPLACE to avoid terminating the string literal too early.
declare #filenames table (fname varchar(max));
declare #shellCommand nvarchar(max) = N'exec xp_cmdshell ''dir ' + replace(#directory, '''', '''''') + '*.jpg /b''';
insert #filenames exec(#shellCommand);
-- Construct and execute a batch of SQL statements to load the filenames and the contents of the
-- corresponding files into the Images table. I found when I called dir /b via xp_cmdshell above, I
-- always got a null back in the final row, which is why I check for fname IS NOT NULL here.
declare #sql nvarchar(max) = '';
with EscapedNameCTE as (select fname = replace(#directory + fname, '''', '''''') from #filenames where fname is not null)
select
#sql = #sql + N'insert Images (fname, data) values (''' + E.fname + ''', (select X.* from openrowset(bulk ''' + E.fname + N''', single_blob) X)); '
from
EscapedNameCTE E;
exec(#sql);
I started with an empty Images table. Here's what I had after running the above:
Now I'm not claiming this is necessarily the best way to go about doing this; the link provided by #nscheaffer might be more appropriate, and I'll be reading it myself since I'm not familiar with SSIS. But perhaps this will help illustrate the kind of approach you were initially trying for.
What is the best way to find the path of an application, that could be installed anywhere on multiple types of PC/operating system.
I need to do this in SQL Server, the version could be 2008 all the way up to 2014.
I have the following that works well and returns what I need.
DECLARE #findpath varchar(500) = 'powershell.exe -noprofile (get-itemproperty -literalpath ''HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\App.exe'').''(default)'''
DECLARE #foundpath table(id int identity(1,1), [path] varchar(500))
INSERT #foundpath
EXEC xp_cmdshell #findpath
My problem is the application doesn't always exist under that registry location so I need an alternative way of finding it.
Presuming the application is installed to the same directory as the SQL database files, I have tried using the WHERE command but I can't always guarantee the application is going to be installed to the same location, and sometimes it may be installed to a different drive letter.
IF (SELECT left([path],16) FROM #foundpath WHERE id = 1) = 'Get-ItemProperty'
BEGIN
DECLARE #where varchar(500) = 'WHERE /R '+(
SELECT distinct vs.volume_mount_point
FROM sys.master_files AS f WITH (NOLOCK)
CROSS APPLY sys.dm_os_volume_stats(f.database_id, f.[file_id]) AS vs )+' '+'App.exe'
INSERT #foundpath
EXEC xp_cmdshell #where
END
Perhaps you can list the apps shown in the GUI of "Add or remove programs" using the command line. Maybe you can use wmic product get description (I ran it in PowerShell), which I found by looking around online. It took a while to run and doesn't include all my apps.
Perhaps the app does something consistently that you can rely on. For example, it might set an env variable (e.g., PATH) regardless of where the installation directory is.
As I am a beginner in SQL Server and my scripting is not very polished yet. I need suggestions on the below issue.
I receive files from a remote server to my machine (around 700/day) as follows :
ABCD.100.1601310200
ABCD.101.1601310210
ABCD.102.1601310215
Naming Convention:
Here the first part 'ABCD' remains the same, middle part is a sequence id which is in incremental order for every file. The last part is time stamp.
File structure
The file does not have any specific extension but can be opened with notepad/excel. Therefore can be called as flat file. Each files consist of 95 columns and 20000 rows fixed with some garbage value on top 4 and bottom 4 rows of column 1.
Now, I need to make a database in SQL server where I can import data from these flat files using a scheduler. Suggestion needed.
There are probably other ways of doing this, but this is one way:
Create a format file for your tables. You only need to create it once. Use this file in the import script in step 2.
Create an import script based on OPENROWSET(BULK '<file_name>', FORMATFILE='<format_file>'
Schedule the script from step 2 in SQL Server to run against the database you want the data imported in
Create the format file
This creates a format file to be used in the next step. The following script creates a format file in C:\Temp\imp.fmt based on an existing table (replace TEST_TT with the database you are importing to). This creates such a format file with a , as field seperator. If the files have tab as seperator, remove the -t, switch.
DECLARE #cmd VARCHAR(8000);
SET #cmd='BCP TEST_TT.dbo.[ABCD.100.1601310200] format nul -f "C:\Temp\imp.fmt" -c -t, -T -S ' + (SELECT ##SERVERNAME);
EXEC master..xp_cmdshell #cmd;
Before executing this you will to reconfigure SQL Server to allow the xp_cmdshell stored procedure. You only need to do this once.
EXEC sp_configure 'show advanced options', 1
GO
RECONFIGURE
GO
EXEC sp_configure 'xp_cmdshell', 1
GO
RECONFIGURE
GO
import script
This script assumes:
The files need to be imported to separate tables
The files are located in C:\Temp
The format file is C:\Temp\imp.fmt (generated in the previous step)
SET NOCOUNT ON;
DECLARE #store_path VARCHAR(256)='C:\Temp';
DECLARE #files TABLE(fn NVARCHAR(256));
DECLARE #list_cmd VARCHAR(256)='DIR ' + #store_path + '\ABCD.* /B';
INSERT INTO #files EXEC master..xp_cmdshell #list_cmd;
DECLARE #fullcmd NVARCHAR(MAX);
SET #fullcmd=(
SELECT
'IF OBJECT_ID('''+QUOTENAME(fn)+''',''U'') IS NOT NULL DROP TABLE '+QUOTENAME(fn)+';'+
'SELECT * INTO '+QUOTENAME(fn)+' '+
'FROM OPENROWSET(BULK '''+#store_path+'\'+fn+''',FORMATFILE=''C:\Temp\imp.fmt'') AS tt;'
FROM
#files
WHERE
fn IS NOT NULL
FOR XML PATH('')
);
EXEC sp_executesql #fullcmd;
I have a folder where files are going to be dropped for importing into my data warehouse.
\\server\share\loading_area
I have the following (inherited) code that uses xp_cmdshell shivers to call out to the command shell to run the DIR command and insert the resulting filenames into a table in SQL Server.
I would like to 'go native' and reproduce this functionality in SSIS.
Thanks in advance guys and girls. Here's the code
USE MyDatabase
GO
declare #CMD varchar(500)
declare #EXTRACT_PATH varchar(255)
set #EXTRACT_PATH = '\\server\share\folder\'
create table tmp_FILELIST([FILENUM] int identity(1,1), [FNAME] varchar(100), [FILE_STATUS] varchar(20) NULL CONSTRAINT [DF_FILELIST_FILE_STATUS] DEFAULT ('PENDING'))
set #CMD = 'dir ' + #EXTRACT_PATH + '*.* /b /on'
insert tmp_FILELIST([FNAME])
exec master..xp_cmdshell #CMD
--remove the DOS reply when the folder is empty
delete tmp_FILELIST where [FNAME] is null or [FNAME] = 'File Not Found'
--Remove my administrative and default/common, files not for importing, such as readme.txt
delete tmp_FILELIST where [FNAME] is null or [FNAME] = 'readme.txt'
Use the ForEach loop with the file enumerator.
Since you're only inserting file names in a table (i.e. not doing any processing on each file at the same time in SSIS), I suggest doing it all with .NET in a script task. This will also make it easy to add additional logic, such as filtering names etc. See the following items in System.Data.SqlClient:
SqlConnection
SqlCommand
SqlCommand.Parameters
SqlCommand.ExecuteNonQuery()