So I am trying to write a Powershell script that creates a backup of a databases, compresses the backup, and uploads it to an FTP site. Here is a part of my script
Sample Script/Code:
Write-Host "Backup of Database " $databaseName " is starting"
push-location
Invoke-Sqlcmd -Query "--SQL Script that backs up database--" -ServerInstance "$serverName"
pop-location
Write-Host "Backup of Database " + $databaseName " is complete"
#Create a Zipfile of the database
Write-Host "Compressing Database and creating Zip file...."
sz a -t7z "$zipfile" "$file"
Write-Host "Completed Compressing Database and creating Zip file!"
I am wanting to prevent any code after the "Invoke-Sqlcmd......." part from being executed until the SQL script backing up the database is complete because the compression line is failing to find the backup of the database because the backup takes a fairly long time to complete.
I am extremely new to using Powershell and didn't quite understand what a couple of the other possibly related questions I found were offering as a solution as I call my SQL statement a different way.
Possible Related Questions:
Get Powershell to wait for an SQL command to execute
Powershell run SQL job, delay, then run the next one
Are you sure your ...script that backs up the database isnt just throwing an error and the ps continuing?
This seems to indicate that it does in fact wait on that call:
Write-Host "starting"
push-location
Invoke-Sqlcmd -Query "waitfor delay '00:00:15';" -ServerInstance "$serverName"
pop-location
Write-Host "complete"
In any case, you should guard against the file existing, by either aborting if the file does not exist or polling until it does (i'm not 100% on when the .bak file is written to disk).
# abort
if(!(test-path $file)) {
}
# or, poll
while(!(test-path $file)) {
start-sleep -s 10;
}
Related
The end game is to create a database when building a docker container, and persist the data so that if the container is removed, I can start the container again and have my database with my persisted data.
I'm using microsoft/mssql-server-windows-developer with Windows containers with docker-compose.
The relevant part of my docker-compose file is (other services removed):
version: "3.9" services:
db:
build:
context: .
dockerfile: Database/Dockerfile
volumes:
- C:\temp:C:\temp
ports:
- 1433:1433
Basically, the db Dockerfile runs a powershell script (very similar to https://github.com/microsoft/mssql-docker/blob/master/windows/mssql-server-windows-developer/start.ps1). My powershell script starts MSSQLSERVER then runs sql files to create a database, run create table, create procs, etc scripts.
All of this works. docker-compose build then docker-compose up will create and run my database on localhost and everything is great. But, if I manipulate the data at all and remove the database then call docker-compose up again, my data is gone.
Everything I've read about persisting data includes using attach_db. I would like to do some sort of if exists, attach_db else create database.
The question (finally)... Why don't I have an mdf file after I create the database? Am I supposed to? I've messed with different ways to add volumes but my volume is always empty. It doesn't appear I'm creating an mdf file to add to my volume.
EDIT - Adding Dockerfile and ps script Dockerfile calls
Dockerfile:
FROM microsoft/mssql-server-windows-developer
ENV sa_password="nannynannybooboo" \
ACCEPT_EULA="Y" \
db1="db1" \
db2="db2"
EXPOSE 1433
RUN mkdir -p ./db1
RUN mkdir -p ./db2
COPY /Database/startsql.ps1 .
COPY /Database/db1/ ./db1
COPY /Database/db2/ ./db2
HEALTHCHECK CMD [ "sqlcmd", "-Q", "select 2" ]
RUN .\startsql -sa_password $env:sa_password -ACCEPT_EULA $env:ACCEPT_EULA -db_name $env:db2 -Verbose
RUN .\startsql -sa_password $env:sa_password -ACCEPT_EULA $env:ACCEPT_EULA -db_name $env:db1 -Verbose
startsql.ps1
# based off https://github.com/microsoft/mssql-docker/blob/master/windows/mssql-server-windows-developer/start.ps1
param(
[Parameter(Mandatory=$false)]
[string]$sa_password,
[Parameter(Mandatory=$false)]
[string]$ACCEPT_EULA,
[Parameter(Mandatory=$true)]
[string]$db_name
)
if($ACCEPT_EULA -ne "Y" -And $ACCEPT_EULA -ne "y")
{
Write-Verbose "ERROR: You must accept the End User License Agreement before this container can start."
Write-Verbose "Set the environment variable ACCEPT_EULA to 'Y' if you accept the agreement."
exit 1
}
# start the service
Write-Verbose "Starting SQL Server"
start-service MSSQLSERVER
if($sa_password -eq "_") {
if (Test-Path $env:sa_password_path) {
$sa_password = Get-Content -Raw $secretPath
}
else {
Write-Verbose "WARN: Using default SA password, secret file not found at: $secretPath"
}
}
Write-Verbose $sa_password
if($sa_password -ne "_")
{
Write-Verbose "Changing SA login credentials"
$sqlcmd = "ALTER LOGIN sa with password=" +"'" + $sa_password + "'" + ";ALTER LOGIN sa ENABLE;"
& sqlcmd -Q $sqlcmd
}
Write-Verbose "Started SQL Server"
Write-Verbose "Starting set up scripts..."
Write-Verbose $db_name
$exists = $true
$exists = #($sqlServer.Databases | % { $_.Name }) -contains $db_name
$creation = ".\"+$db_name+"\creation.sql"
$creation_rpt = ".\"+$db_name+"\creation.rpt"
$userdefined = ".\"+$db_name+"\userdefined.sql"
$userdefined_rpt = ".\"+$db_name+"\userdefined.rpt"
$presetup = ".\"+$db_name+"\pre.setup.sql"
$presetup_rpt = ".\"+$db_name+"\presetup.rpt"
$tables = ".\"+$db_name+"\tables.sql"
$tables_rpt = ".\"+$db_name+"\tables.rpt"
$procs = ".\"+$db_name+"\procs.sql"
$procs_rpt = ".\"+$db_name+"\procs.rpt"
$triggers = ".\"+$db_name+"\triggers.sql"
$triggers_rpt = ".\"+$db_name+"\triggers.rpt"
Write-Verbose $creation
Write-Verbose $exists
if ($exists -ne $true){
Write-Verbose "Starting creation script..."
Invoke-Sqlcmd -InputFile $creation | Out-File -FilePath $creation_rpt
Write-Verbose "Starting user defined script..."
Invoke-Sqlcmd -InputFile $userdefined | Out-File -FilePath $userdefined_rpt
Write-Verbose "Starting pre.setup script..."
Invoke-Sqlcmd -InputFile $presetup | Out-File -FilePath $presetup_rpt
Write-Verbose "Starting tables script..."
Invoke-Sqlcmd -InputFile $tables | Out-File -FilePath $tables_rpt
Write-Verbose "Starting triggers script..."
Invoke-Sqlcmd -InputFile $triggers | Out-File -FilePath $triggers_rpt
Write-Verbose "Starting procs script..."
Invoke-Sqlcmd -InputFile $procs | Out-File -FilePath $procs_rpt
}
Get-EventLog -LogName Application -Source "MSSQL*" -After (Get-Date).AddSeconds(-2) | Select-Object TimeGenerated, EntryType, Message
I can't share the sql files startsql calls, but 99% of the sql
is SSMS generate scripts from an existing DB that I am replicating. The 1% that isn't generated by SSMS is a command to link the two databases being created.
Volumes
You're spot on, volumes can (And should!) be used to persist your data.
Microsoft themselves have docs on how to persist data from containerised SQL servers, including the required commands:
https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-docker-container-configure?view=sql-server-ver15&pivots=cs1-bash#persist
However, this is for Linux, not Windows, so the paths will be different (Very likely the defaults for non-containerised work)
To find that location, you could probably use a query found below, or hop into the container while it is running (using docker exec) and navigate around:
https://www.netwrix.com/how_to_view_sql_server_database_file_location.html
When using volumes with docker-compose the spec can be found here, and is really simple to follow:
https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose
(Edit) Proof of Concept
I played around with the Windows container and managed to get the volumes working fine.
I ditched your Dockerfile, and just used the base container image, see below.
version: "3.9"
services:
db:
image: microsoft/mssql-server-windows-developer
volumes:
- .\db:C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA
ports:
- 1433:1433
environment:
SA_PASSWORD: "Abc12345678"
ACCEPT_EULA: "Y"
This works for me because I specified the MDF file location upon database creation:
/*
https://learn.microsoft.com/en-us/sql/relational-databases/databases/create-a-database?view=sql-server-ver15
*/
USE master ;
GO
CREATE DATABASE Sales
ON
( NAME = Sales_dat,
FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA\saledat.mdf',
SIZE = 10,
MAXSIZE = 50,
FILEGROWTH = 5 )
LOG ON
( NAME = Sales_log,
FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA\salelog.ldf',
SIZE = 5MB,
MAXSIZE = 25MB,
FILEGROWTH = 5MB ) ;
GO
EXEC sp_databases ;
You can see that the filepath in the container there correlates to the volume path in the docker-compose file. When I stopped the container, I could successfully see the mdf file in the .\db folder of my project. If you managed to locate the filepath from running your query, you can simply add that to the volume spec in the same fashion as above.When restarting the container, everything loaded fine, and the SP returned a valid list of all DB's.
Windows Containers
I knew they were regarded as a bad idea, but me oh my, did I not expect the base image to be 15GB.
This is ridiculously large, and depending on your use case, will present issues with the development, and deployment process, simply in terms of the time required to download the image.
If you can use Linux containers for your purposes, I highly recommend it as they are production ready, small, lightweight, and better supported. They can even be ran as the developer edition, and the Microsoft docs clearly state how to persist data from these containers
Linux: https://hub.docker.com/_/microsoft-mssql-server
Windows: https://hub.docker.com/r/microsoft/mssql-server-windows-developer/
Ex:
# Using Windows containers in Powershell
PS> docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
microsoft/mssql-server-windows-developer latest 19873f41b375 3 years ago 15.1GB
# Using Linux containers in WSL
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
mcr.microsoft.com/mssql/server 2019-latest 56beb1db7406 10 days ago 1.54GB
I am trying to delete remote (.csv) files in the FTP server older than 2 days files.
The files do not have their last modification time set correctly. I have to rely on a timestamp in their names.
Naming of the file is like Sales_201705010315.csv (date and time).
My current WinSCP script is:
option batch on
option confirm off
open login ftp credentials
cd /OUT
rm *<1D
exit
When I run the script, files are not deleting. Can someone please correct my scripting
This will indeed delete files "older than 1 day" (not 2 days):
rm *<1D
See file mask with time contraints.
But that syntax uses file modification time.
See also Delete files older than X days from FTP server with PowerShell or batch file.
If you need to select the files based on timestamp in their names, it's more complicated.
It's easy to delete files with a timestamp 2 days old:
rm Sales_%TIMESTAMP-2D#yyyymmdd%????.csv
This uses %TIMESTAMP% syntax with a relative time. The syntax will make the command resolve to (as of 2017-05-04):
rm Sales_20170502????.csv
But that won't delete files 3 and more days old. That's not a problem, if you run the script regularly every day. If you want to cater for 1 or few days of outages, you can delete files with timestamp 2, 3, 4... days old like:
rm Sales_%TIMESTAMP-2D#yyyymmdd%????.csv
rm Sales_%TIMESTAMP-3D#yyyymmdd%????.csv
rm Sales_%TIMESTAMP-4D#yyyymmdd%????.csv
...
If you really want to delete all files with timestamp 2 and more days old, you have to write the script in a more powerful language.
Example in PowerShell with use of WinSCP .NET assembly:
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions
# Set up session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Ftp
HostName = "ftp.examle.com"
UserName = "username"
Password = "password"
}
# Connect
Write-Host "Connecting..."
$session = New-Object WinSCP.Session
$session.Open($sessionOptions)
Write-Host "Listing files..."
$remotePath = "/OUT"
$files = $session.ListDirectory($remotePath).Files
$prefix = "Sales_"
$twoDaysBack = (Get-Date).AddDays(-2)
$timestamp = $twoDaysBack.ToString("yyyyMMdd")
foreach ($file in $files)
{
if (($file.Name.Length -gt ($prefix.Length + $timestamp.Length)) -and
($file.Name.SubString(0, $prefix.Length) -eq $prefix) -and
($file.Name.SubString($prefix.Length, $timestamp.Length) -le $timestamp))
{
$path = [WinSCP.RemotePath]::EscapeFileMask($file.FullName)
$session.RemoveFiles($path).Check()
Write-Host "Deleted $($file.Name)"
}
}
Write-Host "Done"
Be aware that the file on the FTP server will have the date/time the file was created on the FTP server, not the original file's date/time. So if your file was transferred by some automated task that is run overnight the FTP server date/time may be different. If the date/time on the FTP server isn't synced to a time server you'll run into the same problem. And you can experience the problem if the FTP transmitting machine and receiving machine are in different time zones.
I'm trying to write a script in Power Shell to connect to a remote server and execute a .bat file. Since I wanted to do this for more than one server I thought I could place the .bat file on another server which shares ist resources and call it from there, but that does not work. When I place the .bat file in my temp folder and execute it via the script it works. However trying to execute it from the shared resources does not. Why is that?
Note: I can manually start the process via the Task-Manager that starts the .bat file in the shared resources and it works.
script:
$CompName = "myotherserver"
$cred = get-credential
$process = get-wmiobject -query "SELECT * FROM Meta_Class WHERE __Class = 'Win32_Process'" -namespace "root\cimv2" -computername $CompName -credential $cred
$results = $process.Create( "cmd.exe `/c \\mysharedresc\temp\started.bat" )
started.bat:
echo started > c:\temp\started.txt
If you need more annotations or Information please ask for it.
Edit: Working means for me it creates the started.txt in C:\temp. Not working means it does not (but in this case there is no error given.)
I try to run this PowerShell script to get counter.
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[String]
$DatabaseName
)
$ErrorActionPreference = "Stop"
$VerbosePreference = "Continue"
cls
Write-Output "Collecting counters..."
Write-Output "Press Ctrl+C to exit."
$counters = #("\Processor(_Total)\% Processor Time", "\LogicalDisk(C:)\Disk Reads/sec", "\LogicalDisk(C:)\Disk Writes/sec",
"\LogicalDisk(C:)\Disk Read Bytes/sec", "\LogicalDisk(C:)\Disk Write Bytes/sec", "\SQLServer:Databases($DatabaseName)\Log Bytes Flushed/sec")
Get-Counter -Counter $counters -SampleInterval 1 -MaxSamples 3600 |
Export-Counter -FileFormat csv -Path "C:\sql-perfmon-log.csv" -Force
But I got error when ran the script.
Error on the screen.
I found out that error occur in "\SQLServer:Databases($DatabaseName)\Log Bytes Flushed/sec"
Can somebody prompt me How I can enter a specific instance probably with credentials?
I see you're trying to run a powershell script from a command prompt.
Make sure you're doing this on the machine your SQL databases are hosted on.
You need to either open powershell administratively on your computer, or if you already have an administrative command prompt open, type "powershell" into it.
You can either paste all the code into your current host after specifying a string for $DatabaseName or just dot-source . the script and pass $DatabaseName as a parameter into the script.
If you're not sure what any of that means, take a look at
https://technet.microsoft.com/en-us/library/dn425048.aspx for more information on getting started with using powershell.
Long time lurker, first time poster. I'm hoping somebody here can give me a hand trying to figure out an issue that's got me stumped. Essentially I am trying to write a Powershell script that will restore a multi-part SQL Server database to a remote server. I have messed around with the invoke-command and script-block features to a point where I understand how to use variables and parameters... However I am noticing some very interesting behaviour.
Here is my code right now:
cls
$strbaktest = "S:\Landing\testdb1.bak,S:\Landing\testdb2.bak"
echo "The backup files are $strbaktest"
write-host
$StrScrBlk = {param($StrBaktest); &Restore-SqlDatabase -ServerInstance "mydbserver" -Database testdb -BackupFile $strbaktest -ReplaceDatabase}
invoke-command -ComputerName mydbserver -ScriptBlock $StrScrBlk -argumentlist $strbaktest
This yields an error like this:
System.Data.SqlClient.SqlError: Cannot open backup device 'S:\Landing\testdb1.bak,S:\Landing\testdb2.bak'. Operating system error 123(The filename, directory name, or volume label syntax is incorrect.).
However, if I hardcode the same path into my script block, it works:
cls
$StrScrBlk = {Restore-SqlDatabase -ServerInstance "mydbserver" -Database testdb -BackupFile S:\Landing\testdb1.bak,S:\Landing\testdb2.bak -ReplaceDatabase}
invoke-command -ComputerName mydbserver -ScriptBlock $StrScrBlk
The other interesting thing I noticed was that if I only have a single file to restore, using the first method with a variable works:
cls
$strbaktest = "S:\Landing\testdb.bak"
echo "The backup files are $strbaktest"
write-host
$StrScrBlk = {param($StrBaktest); &Restore-SqlDatabase -ServerInstance "mydbserver" -Database testdb -BackupFile $strbaktest -ReplaceDatabase}
invoke-command -ComputerName mydbserver -ScriptBlock $StrScrBlk -argumentlist $strbaktest
Hopefully this makes sense to somebody. I've spent a couple hours trying a bunch of different ways to get this to work - however I can't seem to figure it out. Any help or advice is appreciated.
Thanks!
The comma is an operator or identifier that separates elements of an array. Compare:
PS C:\> 1,2
1
2
PS C:\> "1,2"
1,2
PS C:\>
Try this:
$strbaktest = "S:\Landing\testdb1.bak","S:\Landing\testdb2.bak"
It's working with "x","c" . The comma is not required, as it is waiting for an array.