Batch file SQLCMD Restore Loop - Files Disappear - batch-file

So I've searched and found a lot of info in similar areas, but nothing that quite hits the issue I'm having.
I'm working in a test environment and need to restore from the same SQL database backup (.bak) to many named instances of SQL server. All the sql instances are pre-installed and running.
I played around with different ideas, but a batch file using sqlcmd seems to be the best suited for the job.
So I created a batch file (.bat) that asks for the starting and stopping instance numbers and then should restore from the backup to each SQL named instance incrementing the instance number along the way.
When it runs the sqlcmd appears to work fine. At the end it prints out
RESTORE DATABASE successfully processed X pages in Y seconds
Also the files (.mdf, .ndf, .ldf) are in the directory as expected and then it moves on to the next.
The problem is that when it moves on to the next, the files that were just restored disappear from the directory.
If anyone has any ideas it would certainly be appreciated.
Here's the batch...
ECHO OFF
ECHO Enter starting instance number for restore db
SET /P _IntStart=
ECHO Enter number of last instance for restore db
SET /P _IntStop=
SET /a _IntStop=_IntStop+1
:RestoreDb
If %_IntStart% GEQ %_IntStop% goto:EOF
ECHO Display Instance Number... IntStart = %_IntStart%
sqlcmd -e -s localhost\instance%_IntStart% -d master -U user -P password -Q "Use [master]; RESTORE DATABASE DBName1 FROM DISK = 'C:\DBName1.bak'WITH REPLACE, MOVE 'DBName1' TO 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE%_IntStart%\MSSQL\DATA\DBName1.mdf', MOVE 'DBName1_log' TO 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE%_IntStart%\MSSQL\DATA\DBName1_log.LDF', MOVE 'ftrow_DBName1Catalog' TO 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE%_IntStart%\MSSQL\DATA\DBName1_1.ndf';"
SET /a _IntStart=_IntStart+1
GOTO:RestoreDb
PAUSE
EXIT
=========================================
From SQL Mgmt. Studio I've also tried the below. It works if I comment out the loop and run it each time manually bumping the instance number. It will create separate copies of the db and files. The problem here is SQLCMD doesn't appear to like concatenation in Mgmt. Studio so I can't increment the instance number in the :CONNECT. It ends up trying to connect to localhost\instance$(SCintnum).
Declare #intnum int
Set #intnum = 1
Declare #intstr NVARCHAR(255)
Set #intstr = #intnum
Declare #PathName1 NVARCHAR(255)
Declare #PathName2 NVARCHAR(255)
Declare #PathName3 NVARCHAR(255)
Set #PathName1 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1.mdf'
Set #PathName2 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1_log.LDF'
Set #PathName3 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1_1.ndf'
While #intnum < 51
:SETVAR SCintnum 1
:CONNECT localhost\instance$(SCintnum) -U user -P password
Use [master];
RESTORE DATABASE DBName1 FROM DISK = 'C:\DBName1.bak'
WITH REPLACE,
MOVE 'DBName1' TO #PathName1,
MOVE 'DBName1_log' TO #PathName2,
MOVE 'ftrow_DBName1Catalog' TO #PathName3;
:SETVAR SCintnum $(SCintum)+1
Set #intnum = #intnum+1
Set #intstr = #intnum
Set #PathName1 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1.mdf'
Set #PathName2 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1_log.LDF'
Set #PathName3 = 'E:\Microsoft SQL Server\MSSQL10_50.INSTANCE' + #intstr + '\MSSQL\DATA\DBName1_1.ndf'
===================================================================
This is an example of what I ended up using in c#.....
///Set SQL Connection
SqlConnection myConnection = new SqlConnection("user id=sa;" +
"password="+ sapassword+";server="+servername+"\\instance"+currentinstancenum+";" +
"Trusted_Connection=yes;" +
"database=master; " +
"connection LifeTime=0; connection Timeout=30");
///Set SQL Command
string thesqlcommand = "USE [master]; RESTORE DATABASE " + dbname + " FROM DISK = '" + backuplocation + "' WITH REPLACE, MOVE '" + dbname + "' TO " + #PathName1 + ", MOVE '" + dbname + "_log' TO " + #PathName2 + ", MOVE 'ftrow_" + dbname + "Catalog' TO " + #PathName3 + ";";
SqlCommand myCommand = new SqlCommand(thesqlcommand, myConnection);
///Set SQL Command TimeOut, open connection, execute command, close command
myCommand.CommandTimeout = 180;
myCommand.Connection.Open();
myCommand.ExecuteNonQuery();
myConnection.Close();

It is doing what you asked...
MOVE 'DBName1'
and
MOVE 'DBName1_log'

Ended up creating a little utility in C# to do this. Wish I had started there as it was far simpler. I added an example to the bottom of the original post.

Related

Trouble executing SQLPS or PowerShell from T-SQL script using xp_cmdshell

I had originally created a set of scripts to extract the results from a system of dynamic queries to CSV using SQLCMD. However, due to large text fields in the datasets containing a large amount of formatting codes, the CSV files became unwieldly. I then switched to using SQLPS or PowerShell, which did a great job of formatting the output files. Unfortunately, I am unable to get my extract CSV files to save when running the T-SQL file manually from SSMS. I can paste the same output code into SQLPS and it works fine using PowerShell.exe or SQLPS.exe.
I checked file permissions on the extract folder, tried different options for SQLPS and PowerShell, and reviewed the execution in Process Monitor, but I cannot figure out why this will not work.
SET #Cmd = 'sqlps
$FromDate = "' + #FromDate + '";
$ToDate = "' + #ToDate + '";
$IncidentTypeName = "' + #IncidentTypeName + '";
$DateField = "' + #DateField + '";
$QueryType = "' + #QueryType + '";
$PersonTypeID = "' + #PersonTypeID + '";
$var = "FromDate=$FromDate", "ToDate=$ToDate", "IncidentTypeName=$IncidentTypeName", "DateField=$DateField", "QueryType=$QueryType", "PersonTypeID=$PersonTypeID";
invoke-sqlcmd -ServerInstance ' +#Server+ ' -Database ' +#Database+ ' -Username '+#Login+' -Password '+#Password + ' -QueryTimeout 0 ' +
' -InputFile "'+#SqlScriptFolder+#InputFileType+'" -Variable $var | export-csv -Delimiter "," -NoTypeInformation -Path "'+#ExtractFolder+#OutputFile+'.csv"'
print #Cmd
EXEC master..xp_cmdshell #Cmd
My requirements are to extract a standard CSV file, and unfortunately I am trying to leverage the large amount of T-SQL code I put together earlier in the project when working with SQLCMD. It will also need to be an automated process, which is why I was trying to leverage PowerShell or SQLPS. I am using SQL Server 2019 on a dev machine running Windows 10.
Any ideas on this one?
Well, after much research into my crazy process, which I should definitely rewrite to be a PowerShell script (and not use xp_cmdshell), I found the solution, which is to
escape the " (double-quotes) with a \ (backslash)
replace the carriage returns and newlines using SET #Cmd = REPLACE(REPLACE(#Cmd, NCHAR(13), N''), NCHAR(10), N' ');
from invoke-sqlcmd-doesnt-work-when-used-with-xp-cmdshell

User does not have permission to RESTORE database

I am using the below code to backup a Database on my local SqlExpress database.
procedure RestoreScoreDb(DBName,OldName,BackName : String);
var
cmd : WideString;
SqlBackupDir : String;
SqlDataDir : String;
begin
Try
ConnectionMaster.Connected := False;
ConnectionMaster.Close;
ConnectionMaster.ConnectionString := 'Provider=SQLNCLI11.1;Integrated Security="";Persist Security Info=False;User ID=SA;Password=Tccc1234;OLE DB Services=-2;Initial Catalog="master";Data Source=\SQLEXPRESS;Initial File Name="";Packet Size=4096;Auto Translate=True;Server SPN=""';
SqlBackupDir := ReadIniStr(IniCfg,'Dir','SqlBackup');
SqlDataDir := ReadIniStr(IniCfg,'Dir','SqlData');
cmd := 'RESTORE DATABASE '+DBName;
cmd := Cmd + ' FROM DISK = N'''+SqlBackupDir+'\'+BackName+'''';
cmd := Cmd + ' WITH FILE = 1';
cmd := cmd + ' , MOVE N'''+OldName+''' TO N'''+SqlDataDir+'\'+DBName+'.mdf''';
cmd := cmd + ' , MOVE N'''+OldName+'_log'' TO N'''+SqlDataDir+'\'+DBName+'.ldf''';
cmd := cmd + ' , NOUNLOAD, REPLACE, STATS = 10';
CmdRestore.CommandText := cmd;
CmdRestore.Connection := ConnectionMaster;
CmdRestore.Execute;
except
on E: Exception do
writeln(LogFile,'RestoreScoreDb = '+E.Message);
end;
end;
The Restore command is
cmd ='RESTORE DATABASE score_import FROM DISK =
N''C:\Program Files\Microsoft SQL Server\MSSQL11.SQLEXPRESS\MSSQL\Backup\score_import.bak''
WITH FILE = 1 , MOVE N''score_import'' TO
N''C:\Program Files\Microsoft SQL Server\MSSQL11.SQLEXPRESS\MSSQL\DATA\score_import.mdf'' ,
MOVE N''score_import_log'' TO N''C:\Program Files\Microsoft SQL Server\MSSQL11.SQLEXPRESS\MSSQL\DATA\score_import.ldf'' ,
NOUNLOAD, REPLACE, STATS = 10'
When I copy and paste into management studio it restore the table without trouble. but when I run this code I get an Error:
User does not have permission to RESTORE database "score_import"
SA has full permission. Any idea what I am missing? Delphi XE6, Windows 10 Sql 2012.
Thanks to Embarcadero support. They could reproduce my problem and resolved it by removing Integrated Security="" from the connection string. If you are interested in why, do a Google search for "ADO Integrated Security".

How to automate the process of restore db in SQL server?

In SQL server user have to follow below steps to restore database from backup file.
right click on the Databases container within object explorer.
from context menu select Restore database.
Specify To Database as either a new or existing database.
Specify Source for restore as from device.
Select Backup media as File.
Click the Add button and browse to the location of the BAK file.
Is there any script/command to restore the same without doing the above methods?
Just before you click on the last OK button on SSMS Backup/Restore Wizard ... you can Click on the script drop down button and pick script to New Query Window (The script drop down is Next to the Help button and above the Source/Database Name) and it will generate the Exact command that SSMS will execute against the DB. So you can use that command from the Query window to do the same thing programatically in TSQL. Indeed you can even put that inside a Stored PRocedure if you want to execute the same command all the time like so:
CREATE PROCEDURE dbo.dbBackup AS
BEGIN
RESTORE DATABASE [MyDataBaseName] FROM DISK = N'C:\SQLData\MyDB.bak'
WITH FILE = 1, NOUNLOAD, STATS = 5
END
And to take it one more level you could assign that SP to a hotkey as explained over here : https://www.mssqltips.com/sqlservertip/1287/assign-shortcuts-to-commands-in-sql-server-management-studio/
To access the screen below, open a query window and select from the menu Tools -> Options and then under the Environment node select Keyboard --> Query Shortcuts this will allow you to assign a stored procedure to execute for the Hotkey combination that is best for you.
One thing to be care full about though is to make sure that this does not replace an existing DB because you could accidentally overwrite a good DB if the restore command includes that option
"Automate" is a very broad term in your question, but if C# is an option, you can use the SQLCommand Class to execute the Restore, as below
sqlCmd.CommandText = "RESTORE DATABASE [" + restoreDbName + "] FROM DISK =
N'" + backupFileName + "' WITH FILE = 1, MOVE '" + logicalNameMdf + "' TO '"
+ restorePathMdf + "', MOVE '" + logicalNameLdf + "' TO '" + restorePathLdf + "', "
+ " NOUNLOAD, REPLACE";
sqlCmd.ExecuteNonQuery();

extraction data from file.mdf using pyodbc

I have an application for managing pharmacies (just an example ) this application is developed so that it relates to a single pharmacy . This application map on a Micrsoft DBMS MSSDE
each pharmacy generates database file ( pharmacy.mdf , pharmacy_log.mdf )
So after getting these files from different pharmacies, I wrote a script to automate data extraction from these all files
As the application is running, the script do the following tasks:
- Stops the server MSSQL server
- copy the two files from folder files recover and paste them in the path of the application
- Restart the server
- exctract the desired data
the connection with the database in question is using the python pyodbc Module
Once the extraction is run at a certain time pyodbc crash
i gotte this error:
cursor.execute ('select * from Pha.dbo.table_test ')
pyodbc.Error : ( '01000 ',' [ 01000 ] [ Microsoft] [ODBC SQL Server Driver] [ Shared Memory] ConnectionWrite ( ( send () ( 10054 ) ( SQLExecDirectW ) ')
i wrote this code for connection:
log = os.path.join(path,c,"SIC_log.ldf")
mdf = os.path.join(path,c,"SIC.mdf")
print(log)
print(mdf)
subprocess.call('net stop mssqlserver')
time.sleep(2)
os.system('copy "' + log + '" "' + MSSQL_DIR+'"')
os.system('copy "' + mdf + '" "' + MSSQL_DIR+'"')
time.sleep(2)
subprocess.call('net start mssqlserver')
time.sleep(2)
# Open a connection with the database
cnxn = pyodbc.connect('DSN=SIC_ODBC')
time.sleep(2)
extract_clo(cnxn, wb, ws)
cnxn.close()

Restoring differential backups from SQL Server

I have a strange problem. I have a database, called restorelogs.
An autobackup script do a full backup, and after that, in every 3 hours a differential backup.
Next day do a full again, and then the diffs again, and so on.
No i want to write a script, what is restore this files int the restorelogs database.
From php I am using these sql queries through sqlsrv:
USE master;
ALTER DATABASE restorelogs SET MULTI_USER WITH ROLLBACK IMMEDIATE;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307151435.bak' WITH REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307151745diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307152045diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307152345diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307160245diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307160545diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307160845diff.bak' WITH FILE= 1, REPLACE, NORECOVERY;
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307161145diff.bak' WITH FILE= 1, REPLACE, RECOVERY;
And now will comes the interesting part of the game.
If I copy these queries into SQL Server Management Studio, and run it, it works fine. There are the tables, no error messages, everything is all right.
BUT. When I try to do it from php, the database leave in restoring mode.
As I read through a lot of pages, I understood, the last query need to include the RECOVERY option to recover the database (I also try to not use the REPLACE, just at the first query).
How can it happens?
Please help me, what do i wrong.
UPDATE: Based on Adam comment: i exactly try to do that. I collect the backup files, and use the first (full), with NORECOVERY, REPLACE, and the last WITH RECOVERY.
This is what i try:
function pushToDatabase($files) {
$dbh = new Database();
$dbh->query("USE master");
pre($dbh->errorInfo(), 0);
$cnt = count($files);
$i = 0;
foreach ($files as $file) {
$withFile = '';
$option = '';
$moves = '';
if ($i == 0) {
//restoreFromDisk($file);
//truncateDatabase();
$option = " NORECOVERY, REPLACE";
} else {
$option = " RECOVERY";
}
if ($i > 0) {
$withFile = " FILE=1, ";
}
if ($i == 0 || $i == count($files) - 1) {
$sql = "RESTORE DATABASE " . DB_NAME . " FROM DISK = '" . $file . "' WITH " . $withFile . $option . ";";
echo $sql . "<br />";
$dbh->exec($sql);
pre($dbh->errorInfo(), 0);
}
$i++;
}
$sql = "RESTORE DATABASE " . DB_NAME . " WITH RECOVERY;";
$dbh->exec($sql);
pre($dbh->errorInfo(), 0);
echo $sql . "<br />";
die("NOW WE ARE DONE!");
}
The first statement is running without any problems:
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307151435.bak' WITH NORECOVERY, REPLACE;
But after this, when i want to restore the last diff file:
RESTORE DATABASE restorelogs FROM DISK = 'D:/Apache/htdocs/logreader/extracts/SRO_VT_LOG201307161145diff.bak' WITH FILE=1, RECOVERY;
i've get back an error message:
[Microsoft][SQL Server Native Client 10.0][SQL Server]A previous restore operation was interrupted and did not complete processing on file 'Log_DB_Data'. Either restore the backup set that was interrupted or restart the restore sequence.'
And the database state leaves in restoring mode.
I've also tried to play with the RESTORE FILELISTONLY, and moving the files into the right place. I could do that, but after that i can not recover the differential backup.
function restoreFromDisk($file) {
$sql = "RESTORE FILELISTONLY FROM DISK = '" . $file . "'";
$res = dbQuery($sql);
$option = ' ,REPLACE, RECOVERY';
while ($row = dbFetchAssoc($res)) {
if ($row["LogicalName"] == "Log_DB_Data") {
$move[] = " MOVE '" . $row["LogicalName"] . "' TO '" . MSSQL_DATA_PATH . "resetlogs.mdf'";
} else {
$move[] = " MOVE '" . $row["LogicalName"] . "' TO '" . MSSQL_DATA_PATH . "resetlogs.ldf'";
}
}
$moves = join(",", $move);
$sql = "RESTORE DATABASE " . DB_NAME . " FROM DISK = '" . $file . "' WITH " . $moves . $option . ";";
dbQuery($sql);
echo $sql . "<br />";
$sql = "RESTORE DATABASE " . DB_NAME . " WITH RECOVERY;";
dbQuery($sql);
echo $sql . "<br />";
}
You should be setting the database to single user, if there is the potential for users to be connected (and if it is okay to kill their sessions). Since you are restoring the database, I would think that it is okay. You will need to specify the replace option on the full backup restoration. Also you only need to restore the latest differential backup (using with recovery), as it will contain all the changes since the last full backup. It should also be noted that this restore will not include transactions that have occurred since the last differential backup.
I would think about using log backups every 3 hours. This will have less impact on your system and will allow the log to truncate (if you are in full recovery). In this restore scenario, you will need the full backup and all the log backups, with the last option specifying with recovery.
A different approach would be to write your SQL Script to a temp file, then use osql to execute it as a script. Of course this assumes that you're running on a machine that osql installed.
osql -E -S <server> -d master -i <sql file>

Resources