MS SQL Server - Create schema dump without FILENAME property - sql-server

I would like to create an SQL Server schema dump from a local server that I can run from within an sql server docker container to create an empty database ready for integration tests.
I generate the scripts via SQL Server Management Studio. I right click the database > Tasks > Generate script and set Types of data to script to "Schema only".
I copy the sql files across as part of my docker-compose.yml and when I run the sql dump from within the docker container I get
Directory lookup for the file
"C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.mdf"
failed with the operating system error 2(The system cannot find the file specified.).
the part of the script that's causing the above is...
CREATE DATABASE [mydatabase]
CONTAINMENT = NONE
ON PRIMARY
( NAME = N'mydatabase', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase.mdf' , SIZE = 8192KB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB )
LOG ON
( NAME = N'mydatabase_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\mydatabase_log.ldf' , SIZE = 8192KB , MAXSIZE = 2048GB , FILEGROWTH = 65536KB )
WITH CATALOG_COLLATION = DATABASE_DEFAULT
GO
If I go through the same generation process from an azure hosted database the create code generated is...
CREATE DATABASE [mydatabase]
GO
and if I manually run the above from within the container, it creates as expected.
So what do I need to do to generate an SQL schema dump that doesn't include the FILENAME property of the create command?

Following #Jeroen Mostert's suggestion I generate the scripts without the CREATE DATABASE part, this is done by selecting "specific database objects" instead of "entire database and all database objects"
I then run a script I manually wrote that creates all the required databases, followed by running the scripts that create the tables and procedures against each database.
docker-compose.yml
version: "3.8"
services:
db:
image: mcr.microsoft.com/mssql/server:2019-latest
environment:
- ACCEPT_EULA=true
- SA_PASSWORD=localdevpassword#123
volumes:
- ./sql:/scripts/
ports:
- "1434:1433"
command: /bin/bash -c /scripts/init.sh
init.sh
#!/bin/bash
echo "Starting SQL server"
/opt/mssql/bin/sqlservr &
echo "Sleeping for 10 seconds"
sleep 10
echo "Done sleeping, run the SQL scripts"
cd /scripts/
echo "Creating databases using file databases.sql"
/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD -l 30 -e -i databases.sql
for FILENAME in *.sql; do
DATABASE="${FILENAME%.*}"
if [ "$DATABASE" != "databases" ]; then
echo "Running import for $DATABASE using file $FILENAME"
/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD -d $DATABASE -l 30 -e -i $FILENAME
fi
done
echo "SQL scripts run, sleep forever to keep container running"
sleep infinity

Related

Docker for Windows: SQL Server Database on persistent Volume with Windows-Container

On Docker for Windows and working with windows containers, I cannot get my persistent volume to work on the main database directory of the windows container. This would be C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\DATA
How can I get the benefits of persistent volumes for databases without having to mess with backups and restores into the mssql-server-container?
This may be because of the data directory having the master- and system-dbs stored inside this folder where I try to mount the persistent volume.
In SQL Server for linux containers this simply works, you can connect the persistent volume to /var/opt/mssql and have your database persistent.
I know I can recover a database from a backup into the container, but this has two major drawbacks: I have to have a big container size because I am working with a big database. So I extended the 20 GB limit of the container to 60 GB but... rebuilding the database each time from a backup is time consuming.
The second drawback is, if the mssql-dev container is killed, the database is lost, too. Any work on this database is then gone. This would be different if the database could reside on the persistent volume.
docker run -d -e sa_password=<Password> -e ACCEPT_EULA=Y -v "C:\mylocalfolder:C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\DATA" microsoft/mssql-server-windows-developer
The error message is 'failure in a Windows system call: the virtual computer or container was shutdown unexpectetly. (0xc0370106)
Workaround 1
connect persistent volume to another location like
c:\mydata to prevent the error message from above.
Then get the database connected to the server while not using the standard database folder.
Extract database .bak file, so there are mdf and log files
--Get the name of your DB
RESTORE FILELISTONLY
FROM DISK = 'c:\mydata'
GO
--do the extraction of the bak file to certain folder
RESTORE DATABASE mydatabase
FROM DISK = 'c:\mydata'
WITH REPLACE,
MOVE 'mydatabase' TO 'c:\mydata\extractedDb.mdf',
MOVE 'mydatabase_log' TO 'c:\mydata\extractedLog.ldf'
GO
With this done you should now have your database files ready on your persistent volume. Now attach the database to the server. This has to be done by creating a new db but this procedure only takes nanoseconds to complete!
CREATE DATABASE StackoverflowIsGreat
ON (FILENAME = 'c:\mydata\extractedDb.mdf'),
(FILENAME = 'c:\mydata\extractedLog.ldf')
FOR ATTACH;
Now the database is safe in a persistent volume. If the db-server container goes down or is rebuild, you simply run this last statement again (or even better implement it in your docker-compose or dockerfile):
CREATE DATABASE StackoverflowIsGreat
ON (FILENAME = 'c:\mydata\extractedDb.mdf'),
(FILENAME = 'c:\mydata\extractedLog.ldf')
FOR ATTACH;
Workaround 2
the -attach_dbs parameter seems to work the same way.
Docker run:
docker run -p 1433:1433 --name mssql-dev -e sa_password=<yourpassword> -e ACCEPT_EULA=Y -e attach_dbs="[{'dbName':'PowerSlide_SQLDB','dbFiles':['C:\\your\\path\\database.mdf','C:\\sqldata\\databaselog.ldf']}]" -v "d:\sqldata:C:\sqldata" microsoft/mssql-server-windows-developer
or if you prefer Docker-Compose, it is a little bit tricky. I had to omit the leading and closing ' outside of the brackets and replace the double quotation marks inside the brackets with ' to make it work.
version: '3.2'
services:
mssql-dev:
container_name: mssql-dev
image: 'microsoft/mssql-server-windows-developer'
volumes:
- "d:\\sqldata:C:\\sqldata"
ports:
- "1433:1433"
restart: always
environment:
- "ACCEPT_EULA=Y"
- "sa_password=yourpassword"
- attach_dbs=[{"dbName":"<yourDbName>","dbFiles":["C:\\<your>\\path\\database.mdf","C:\\your\\path\\databaselog.ldf"]}]
volumes:
mssql-dev-data:
It seems this question can be answered with workaround 1 and 2 from above.
Connect persistent volume to another location like c:\mydata to prevent the error message from above. Then get the database connected to the server while not using the standard database folder.
Extract database .bak file, so there are mdf and log files
--Get the name of your DB
RESTORE FILELISTONLY
FROM DISK = 'c:\mydata'
GO
--do the extraction of the bak file to certain folder
RESTORE DATABASE mydatabase
FROM DISK = 'c:\mydata'
WITH REPLACE,
MOVE 'mydatabase' TO 'c:\mydata\extractedDb.mdf',
MOVE 'mydatabase_log' TO 'c:\mydata\extractedLog.ldf'
GO
Attach the database to the server in one of the following three ways:
Docker run example:
docker run -p 1433:1433 --name mssql-dev -e sa_password=<yourpassword> -e ACCEPT_EULA=Y -e attach_dbs="[{'dbName':'PowerSlide_SQLDB','dbFiles':['C:\\your\\path\\database.mdf','C:\\sqldata\\databaselog.ldf']}]" -v "d:\sqldata:C:\sqldata" microsoft/mssql-server-windows-developer
if you prefer Docker-Compose, it is a little bit tricky. I had to omit the leading and closing ' outside of the brackets and replace the double quotation marks inside the brackets with ' to make it work. Example for docker-compose:
version: '3.2'
services:
mssql-dev:
container_name: mssql-dev
image: 'microsoft/mssql-server-windows-developer'
volumes:
- "d:\\sqldata:C:\\sqldata"
ports:
- "1433:1433"
restart: always
environment:
- "ACCEPT_EULA=Y"
- "sa_password=yourpassword"
- attach_dbs=[{"dbName":"<yourDbName>","dbFiles":["C:\\<your>\\path\\database.mdf","C:\\your\\path\\databaselog.ldf"]}]
volumes:
mssql-dev-data:
Or attach DB with SQL Command
CREATE DATABASE StackoverflowIsGreat
ON (FILENAME = 'c:\mydata\extractedDb.mdf'),
(FILENAME = 'c:\mydata\extractedLog.ldf')
FOR ATTACH;

T-SQL cannot create SQL Server database file on mounted docker volume

I'm trying to create a database in docker and have the .mdf and .ldf files be created on a mounted volume so they're not lost when the container shuts down. I followed Microsoft's instructions and added the volume to their example command like so:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<MY_PASSWORD>' \
-p 1433:1433 --name sql1 \
-v ~/mssql/data:/var/data \
-d mcr.microsoft.com/mssql/server:2017-latest
Here's the SQL used to create the database and the files:
CREATE DATABASE [MY_DB]
CONTAINMENT = NONE
ON PRIMARY
( NAME = N'MY_DB', FILENAME = N'/var/data/MY_DB.mdf' , SIZE = 204800KB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB )
LOG ON
( NAME = N'MY_DB_log', FILENAME = N'/var/data/MY_DB_log.ldf' , SIZE = 860160KB , MAXSIZE = 2048GB , FILEGROWTH = 65536KB )
I know this currently works if you're on a Windows machine running everything without docker because that's how we currently do it. However, when I try to execute the above SQL from inside the container (by connecting with a RDBMS), I get the error:
CREATE FILE encountered operating system error 2(The system cannot find the file specified.) while attempting to open or create the physical file '/var/data/SVT_MVP.mdf'
I tried execing into the container and running the SQL as root, but I get the same error. For funsies I even tried to chmod the mapped host directory to 0777 but no dice.
Another note is that everything works perfectly fine if there is no volume mounted. At this point I'm questioning if the volume part is worth it or even needed. I'm pretty new to docker.
For those curious, mounting volumes on macos has had issues since 2017. Instead you have to create a data volume (named volume):
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<MY_PASSWORD>' \
-p 1433:1433 --name sql1 \
-v sqlvolume:/var/opt/mssql/data \
-d mcr.microsoft.com/mssql/server:2017-latest
Thank you to Larnu who pointed me in the right direction.

When migrating database from Magento to localhost I get a timeout error

I have a Magento website with a 4.33 GB database. I need to manipulate some of the data so I exported the database, then zip it but when I try to import it inside phpmyadmin on localhost I get timeout error and only 1/3 of the tables get transferred. I have edited the my.ini as well as php.ini and made the execution time 6000 but the issue is not from there. I believe it should be something with Magento that is wrong.
You need to import your database using command line.
shell> mysqldump db_name > backup-file.sql
You can load the dump file back into the server like this:
UNIX
shell> mysql db_name < backup-file.sql
The same in Windows command prompt:
mysql -p -u [user] [database] < backup-file.sql
PowerShell
C:\> cmd.exe /c "mysql -u root -p db_name < backup-file.sql"
MySQL command line
mysql> use db_name;
mysql> source backup-file.sql;

How to restore a docker container with a SQL server database

I'm using MacOS Sierra with the latest version of the mssql docker file for linux.
I had built a database which grew to a size of ~69 GB. I started getting an error "Could not allocate a new page for database because of insufficient disk space in filegroup". I attempted to solve this problem by running this code:
USE [master]
GO
ALTER DATABASE [db]
MODIFY FILE ( NAME = N'db', FILEGROWTH = 512MB )
GO
ALTER DATABASE [db]
MODIFY FILE
(NAME = N'db_log', FILEGROWTH = 256MB )
GO
After doing this, I was no longer able to startup the the mssql container. I then manually replaced a backup copy of the container folder which in MacOs is called "com.docker.docker" and which contained the prior working version of the database.
After doing this, I stated getting the following error: "The extended event engine has been disabled by startup options. Features that depend on extended events may fail to start."
At this point I re-installed the docker container using the procedure mentioned in this post. the command I used was:
docker create -v /var/opt/mssql --name mssql microsoft/mssql-server-linux /bin/true
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Test#123' -p 1433:1433 --volumes-from mssql -d --name sql-server microsoft/mssql-server-linux
Although now I'm able to start the server with the new container, I would like restore the original SQL server database (~69 GB). I tried doing so by again manually copying the file named "Docker.qcow2" into the docker container folder. This is obviously not working.
How can I restore my database?

Restore SQL Server database to Linux Docker

I need to restore a large SQL Server database on a Linux Docker instance (https://hub.docker.com/r/microsoft/mssql-server-linux/)
I'm moving my .bak file to the docker and executing this command in mssql shell:
RESTORE DATABASE gIMM_Brag FROM DISK = '/var/opt/mssql/backup/BackupFull8H_gIMM.bak' WITH MOVE '[gIMM].Data' T'/var/opt/mssql/data/gIMM.mdf', MOVE '[gIMM].Log' TO '/var/opt/mssql/data/gIMM.ldf', MOVE 'TraceabilityData' TO '/var/opt/mssql/data/gIMM.TraceData.mdf', MOVE 'TraceabilityIndexes' TO '/var/opt/mssql/data/gIMM.TraceIndex.mdf', MOVE 'KpiData' TO '/var/opt/mssql/data/gIMM.KpiData.mdf', MOVE 'KpiIndexes' TO '/var/opt/mssql/data/gIMM.KpiIndex.mdf'
I'm mapping correctly every file that need to and I definitely have enough space on the docker instance but I'm getting this error:
Error: The backup or restore was aborted.
The same error occurs with a windows version of this docker actually... And as it's not supposed to be a Express version, the database size shouldn't be the issue here.
If anyone has more information about what is causing this error !
Thanks,
#TOUDIdel
You have to use the actual file system paths on linux rather than the virtual paths that are shown in the error.
RESTORE DATABASE Northwind FROM DISK='/var/opt/mssql/Northwind.bak' WITH MOVE 'Northwind' TO '/var/opt/mssql/data/NORTHWND.MDF', MOVE 'Northwind_log' TO '/var/opt/mssql/data/NORTHWND_log.ldf'
http://www.raditha.com/blog/archives/restoring-a-database-on-ms-sql-server-for-linux-docker/
You didn't mention it, but the thing that tricked me up was that I wasn't copying the BAK file to my Docker instance. In Terminal with docker and your mssql container running...
1) get container ID:
$docker inspect -f '{{.Id}}' <container_name>
2) copy BAK file to docker instance:
docker exec -i <container_id> bash -c 'cat > /var/opt/mssql/backup.bak' < '/source/path/backup.bak'
3) log into mssql:
mssql -u sa -p 'myPassword'
3) restore db: (you can replace this with your restore script, though this was sufficient for me)
RESTORE DATABASE [MyDatabase] FROM DISK = N'/var/opt/mssql/backup.bak' WITH FILE = 1, NOUNLOAD, REPLACE, STATS = 5
When I had this problem, it's because the restore command was taking long enough for mssql to time out (with a totally unhelpful error message). Specifying a long timeout when connecting allowed the restore to complete. eg
mssql -s localhost -p "<sa_password>" -t 36000000 -T 36000000
I am not sure it is worth mentioning, but neither of the answers alone worked when moving a .bak made in Windows server to the docker running in Linux version.
(Note that I am using the code from the two previous answers and thus any credit should go to the below-mentioned authors)
TabsNotSpaces' solution was good until step 3 where the restore crashed with path mismatch (C:/path_to_mssql_server could not be found).
Vinicius Krauspenhar's answer was then necessary to remap the MDF and LOG files to fully complete the backup.
Thus the solution that worked for me when importing a windows-server-made .bak file into the Linux docker instance was:
In Terminal with docker and your SQL Server container running...
1) get container ID:
$docker inspect -f '{{.Id}}' <container_name>
2) copy BAK file to docker instance:
docker exec -i <container_id> bash -c 'cat > /var/opt/mssql/backup.bak' < '/source/path/backup.bak'
3) log into mssql or in any DB software and
RESTORE DATABASE Northwind FROM DISK='/var/opt/mssql/Northwind.bak' WITH MOVE 'Northwind' TO '/var/opt/mssql/data/NORTHWND.MDF', MOVE 'Northwind_log' TO '/var/opt/mssql/data/NORTHWND_log.ldf'

Resources