I have a database running on SQL Server (13.01) on Windows. I like to deploy it to the Docker Container on Linux using SSDT.
I can perfectly connect to the server running on Docker and create/drop database manually and play with the data.
The problem is I can not publish it. I'm executing following script on Powershell
PS: SqlPackage.exe /Action:Publish /SourceFile:"d.dacpac" /TargetConnectionString:"server=containeraddress;database=thedatabase;user id=sa;password=thepassword;
and getting the following error.
Unable to connect to master or target server 'the database'. You must have a user with the same password in master or target server 'the database'. (Microsoft.Data.Tools.Schema.Sql)
I have the same user and same password on target and source servers.
Is there anybody has the same problem and know how to solve it?
I'll post this here as most of the answers refer to having an existing compiled dacpac file, which may not always be possible. I haven't seen similar ideas posted elsewhere to the solution I'm suggesting here.
Given your usage of docker and if you wish to compile your visual studio project inside the container, given certain combinations of the container base OS and image may not be possible to create a dacpac file with msbuild.
You can work around restoring the database using a series of unix based commands, taking note that the visual studio database project is usually just a series of SQL files, below I show an example of this, where I concat the SQL files into a single file and call sqlcmd to run the script;
FROM mcr.microsoft.com/mssql/server
WORKDIR /init
ENV ACCEPT_EULA=Y
ENV MSSQL_SA_PASSWORD=MyPassword
EXPOSE 1433:1433
RUN apt-get update && apt-get install dos2unix
COPY /solution_folder/database/Tables/*.sql /init/
WORKDIR /database
RUN echo "CREATE DATABASE [database_name];\nGO\nUSE [database_name];\n” >> /database/create.sql
RUN for f in /init/*.sql; do dos2unix $f; cat $f >> /database/create.sql; echo "\nGO\n" >> /database/create.sql; done
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P ‘MyPassword’ -i /database/create.sql && pkill sqlservr
The reason for "dos2unix" is that the SQL files created within visual studio have unique hidden cr/lf (and other characters) which the linux version of sqlcmd won't interpret successfully and will cause errors (which is kind of bizarre and this is exactly the kind of thing you'd want a cross platform database to be able to cope with)
Also, within the final run command you have to start-up the sql server service temporarily otherwise you'll also get errors; it's a little bit of work-around, and a bit fiddly and I'm not sure entirely that the microsoft sql server linux container is well designed enough for the simple task of restoring a database like this, the nuances are the differences between building and running a container and needing some sort of happy middle ground of both concepts for it to work.
Given here isn't a complete solution to restore, it only deals with Tables from the project file, although it should be trivial to expand to scalar functions and stored procedures.
Which version of SqlPackage.exe are you using? Only the most recent release candidate versions of SqlPackage.exe support SQL Server vNext CTP. The SqlPackage release candidate can be downloaded here: https://www.microsoft.com/en-us/download/details.aspx?id=54273
Related
I am unable to establish connection to my Oracle database from Azure Databricks although it works in ADF where I am able to query the table. But ADF takes time to filter the records so I am still trying to connect from Databricks.
I followed the steps from this Microsoft link, both manually and using init-script but error seems to persist.
When I looked into my cluster event log it says the init-script execution was successfully.
Error message when I tried to establish the connection:
DPI-1047: Cannot locate a 64-bit Oracle Client library: "/databricks/driver/oracle_ctl//lib/libclntsh.so: cannot open shared object file: No such file or directory".
When I executed the following command
dbutils.fs.ls("/databricks/driver/")
there was no such directory
This triggered me to post some questions here:
Does this mean the init-script did not perform its job?
Is /databricks/driver/oracle_ctl a hidden directory for dbutils.fs.ls?
Error message points to /databricks/driver/oracle_ctl//lib/libclntsh.so, when I manually inspected the downloaded oracle client, there is no such folder called lib although libclntsh.so exists in the main directory. Is there a problem that databricks is checking the wrong directory for the libclntsh.so?
Does this connections still works for others?
Syntax for connection: cx_Oracle.connect(user= user_name, password= password,dsn= IP+':'+Port+'/'+DB_name)
Above syntax works fine when connected from inside a on-premises machine.
Try installing the latest major release of cx_Oracle - which got renamed to python-oracledb, see the release announcement.
This version doesn't need Oracle Instant Client. The API is the same as cx_Oracle, although obviously the name is different.
If I understand the instructions, your init script would do something like:
/databricks/python/bin/pip install oracledb
Application code would be like:
import oracledb
connection = oracledb.connect(user='scott', password=mypw, dsn='yourdbhostname/yourdbservicename')
with connection.cursor() as cursor:
for row in cursor.execute('select city from locations'):
print(row)
Resources:
Home page: oracle.github.io/python-oracledb/
Quick start: Quick Start python-oracledb Installation
Documentation: python-oracle.readthedocs.io/en/latest/index.html
PyPI: pypi.org/project/oracledb/
Source: github.com/oracle/python-oracledb
Upgrading: Upgrading from cx_Oracle 8.3 to python-oracledb
Changed the path from "/databricks/driver/oracle_ctl/" to "/databricks/driver/oracle_ctl/instantclient" in the init-script and that error does not appear anymore.
Please use the following init script instead
dbutils.fs.put("dbfs:/databricks/<init-script-folder-name>/oracle_ctl.sh","""
#!/bin/bash
sudo apt-get install libaio1
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
mv /databricks/driver/oracle_ctl/instantclient* /databricks/driver/oracle_ctl/instantclient
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
""", True)
Notes:
The above init-script was advised by a databricks employee and can be found here.
As mentioned by Christopher Jones in one of the comments, cx_Oracle has been recently upgraded to oracledb with a thin and thick version.
You will get the above error if you don’t have Oracle instant client in your Cluster.
To resolve above error in azure databricks, please follow this code:
%sh
mkdir -p /opt/oracle
cd /opt/oracle
wget https://download.oracle.com/otn_software/nt/instantclient/19600/instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
unzip instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
set ORACLE_HOME=%ORABAS%\instantclient_19_3
set TNS_ADMIN=%ORACLE_HOME%
set PATH=%ORACLE_HOME%;%PATH%
To create init script, use the following code:
As per official doc,
dbutils.fs.put("dbfs:/databricks/<init-script-folder>/oracle_ctl.sh","""
#!/bin/bash
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
""", True)
To read data from oracle database in PySpark follow this article by Emrah Mete
For more information refer this official document:
https://docs.databricks.com/data/data-sources/oracle.html#oracle
I installed oracle db version 12c in my docker environment.
I used the following command:
docker run -d --name oracle -p 8080:8080 -p 1521:1521 quay.io/maksymbilenko/oracle-12c
I connected to the DB and everything went well but I wanted to enable unified audit.
In order to do that, at first you must shutdown the Database and in all the instructions that I found it says to use sqlplus as following:
sqlplus / as sysoper
SQL> shutdown immediate
SQL> exit
I connected successfully to the DB using the next command:
docker exec -it oracle "bash"
and then I ran the sqlplus command and I received "command not found"
[root#f30cc670f85f /]# sqlplus / as sysoper
bash: sqlplus: command not found
Am I doing it wrong?
What should I do in order to have sqlplus on my oracle DB?
I looked for it and didn't find anything that helped me.
I have mac if its relevant
I think that Docker image is just the database and enough of the OS to run the database. I don't think it includes client software such as SQL*Plus.
You need to have SQL*Plus installed on your Mac. If you haven't already, download the Oracle Instant Client for MacOS including the SQL*Plus extension. Or why not treat yourself and install the new-fangled sqlCL tool? It is easier to install and has all the SQL*Plus capabilities and a whole bunch more features. Find it here.
Whatever client you choose, once it's installed on your Mac you run it like any other app: when prompted for connection you give the string Maksym provides:
system/oracle#//localhost:1521/xe
If you need to connect as sys that would look like this:
sys/oracle#//localhost:1521/xe as sysdba
Sourcing the .bashrc should work to connect to sqlplus as sysdba.
docker-compose exec db bash -c "source /home/oracle/.bashrc; sqlplus sys/Oradoc_db1#ORCLCDB as sysdba;"
with this, you enter the image:
docker exec -it oracle /bin/bash
after that, you can use:
sqlplus sys as sysdba
When using the docker image store/oracle/database-enterprise:12.2.0.1-slim sqlplus and sqlldr tools are only available after the container has started.
You can't do the following in a Dockerfile:
RUN sqlplus sys/password AS SYSDBA #create_database.sql
The container images can be configured to run scripts after setup and on startup. Currently sh and sql extensions are supported.
In your Dockerfile, copy the SQL script into the startup directory:
COPY create_database.sql /opt/oracle/scripts/setup/01_create_database.sql
The database will be created on first startup of the container.
I don't have any experience with docker, but it looks for all the world like you are getting to a bash environment, so there we are on solid ground. The returned error ("bash: sqlplus: command not found") simply means that the executable (sqlplus) was not found in any directory listed in your PATH environment variable, as it exists within your shell environment. You actually need to set three variables: ORACLE_SID needs to be set to the value of your database name. ORACLE_HOME needs to be set to the value of the directory where your oracle binaries are installed. And PATH needs to have $ORACLE_HOME/bin added to it:
export PATH=$ORACLE_HOME/bin:$PATH
Obviously, since you are using the value of ORACLE_HOME in setting PATH, ORACLE_HOME needs to be set first.
For Windows OS:
Type docker ps in command line to show running containers and check container id.
Type docker exec -it container_id //bin/bash
Login via sqlplus command
Or the simplest way
docker exec -it container_id bash -c "source /home/oracle/.bashrc; sqlplus sys/Oradoc_db1#ORCLCDB as sysdba;"
More info is here: https://hub.docker.com/u/cgmmathaw/content/sub-90f0c051-b514-4b7b-a0fe-fc9d6f2172fa
I'm trying to change the default data directory for MSSQL Server RC1 2017 after installation and setup in linux (Ubuntu-16.10).
I used the following command to set the default data directory. Then restarted the mssql server.
sudo /opt/mssql/bin/mssql-conf set filelocation.defaultdatadir /mnt/var/opt/mssql/data/
systemctl restart mssql-server.service
After this I tried to create a simple database "test"
sqlcmd -s localhost -U sa -P "someStrongPassword" -Q "CREATE DATABASE test"
The error returned is as follows:
MODIFY FILE encountered operating system error 31(A device attached to
the system is not functioning.) while attempting to expand the
physical file '/mnt/var/opt/mssql/data/test.mdf'.
CREATE DATABASE
failed. Some file names listed could not be created. Check related
errors.
The error log indicates an OS error:
/mnt/var/opt/mssql/data/test.mdf: Operating system error 31(A device
attached to the system is not functioning.) encountered.
I cannot mount the data directory by any means. The permissions to "/mnt" directory are set to 777 too. Changing the default data directory to any other folder, works perfectly fine. Is this a known or recent bug with mssql server?
Yes, there is an issue with using remote storage through NFS and SMB that came up in CTP 2.1 and was not yet fixed in RC1. See release notes: https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-release-notes#a-idrc1-rc1-july-2017-a
The only workarounds are to use local storage or CTP 2.0. We are working on a fix. Release ETA is TBD ATM.
I installed Sqlserver on my Mac in a docker container, following the instructions from this article.
I run the container with Kitematic and managed to connect to the server using Navicat Essentials for SQl Server.
The server has four databases and I can create new ones, but, ideally, I would like to import an existing database as .bacpac.
The instructions from this answer have been of use to me in the past. Can I run something similar within the container? Or, more generally, is there a way to import a database in the container?
Hi all! We finally have a preview ready for sqlpackage that is built on dotnet core and is cross-platform! Below are the links to download from. They are evergreen links, i.e. each day a new build is uploaded. This way any checked in bug fix is available the next day. Included in the .zip file is the preview EULA.
linux
https://go.microsoft.com/fwlink/?linkid=873926
osx
https://go.microsoft.com/fwlink/?linkid=873927
windows
https://go.microsoft.com/fwlink/?linkid=873928
Release notes:
The /p:CommandTimeout parameter is hardcoded to 120
Build and deployment contributors are not supported
a. Need to move to .NET Core 2.1 where System.ComponentModel.Composition.dll is supported
b. Need to handle case-sensitive paths
SQL CLR UDT types are not supported.
a. This includes SQL Server Types SqlGeography, SqlGeometry, & SqlHierarchyId
Older .dacpac and .bacpac files that use Json serialization are not supported
Referenced .dacpacs (e.g. master.dacpac) may not resolve due to issues with case-sensitive file systems
For lack of a better method, please provide any feedback you have here on this GitHub issue.
Thanks for giving it a try and letting us know how it goes!
https://github.com/Microsoft/mssql-docker/issues/135#issuecomment-389245587
EDIT: I've made you a Docker image for this
https://hub.docker.com/r/samuelmarks/mssql-server-fts-sqlpackage-linux/
Example of setting up a container, creating a database, copying a .bacpac file over, and importing it into aforementioned database:
docker run -d -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 --name sqlfts0 samuelmarks/mssql-server-fts-sqlpackage-linux
docker exec -it sqlfts0 /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P '<YourStrong!Passw0rd>' -Q 'CREATE DATABASE MyDb0'
docker cp ~/Downloads/foo.bacpac sqlfts0:/opt/downloads/foo.bacpac
docker exec -it sqlfts0 dotnet /opt/sqlpackage/sqlpackage.dll /tsn:localhost /tu:SA /tp:'<YourStrong!Passw0rd>' /A:Import /tdn:MyDb0 /sf:foo.bacpac
It looks like Microsoft has implemented support of this on sqlpackage, with documentation!
You will have to add sqlpackage to your container.
You can download it here. (optionally, direct link to linux package here, hopefully doesn't change)
The following are instructions for running this from a windows machine -- obviously it's the bare minimum to get it working. Please change passwords, and probably put this in a docker-compose.yml for re-use.
I unzip the above package into a folder 'c:\sqlpackage' (my windows docker run doesn't allow relative paths), and then mount that into the container with the bacpac, like such:
docker run -d -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Asdf1234" -v c:\sqlpackage:/opt/sqlpackage -v c:\yourdb.bacpac:/tmp/yourdb.bacpac -p 1433:1433 --name mssql-server-example microsoft/mssql-server-linux:2017-latest
here is what a *nix user could run alternatively:
docker run -d -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Asdf1234' -v ./sqlpackage:/opt/sqlpackage -v ./yourdb.bacpac:/tmp/yourdb.bacpac -p 1433:1433 --name mssql-server-example microsoft/mssql-server-linux:2017-latest
and finally, attach to your container and run:
/opt/sqlpackage/sqlpackage /a:Import /tsn:. /tdn:targetdbname /tu:sa /tp:Asdf1234 /sf:/tmp/yourdb.bacpac
After this, you should be able to connect with SSMS to localhost, username and password as you provide them above, and see 'targetdbname'! These are mostly notes I wrote for myself but I'm sure others could use them too.
You can use free Azure Data Studio from Microsoft. Once you have it installed, install the extension "Admin Pack for SQL Server" from Microsoft. Then you can import bacpac files with ease.
This is not a supported feature with a LINUX implementation it seems.
See this link.
I want to install postgresql for use as the backend to a Windows application.
This seems to be no problem if postgresql is not already installed on the system.
If postgresql is already installed then unless the command line parameters contain the superpassword etc of the existing installation then the install fails.
As I will likely never know the superpassword or other account details of any pre-existing postgresql instances and the machine owners may not either it seems that this will frustrate any attempt to install postgresql in such a situation.
I believe it is possible to install completely independent instances of sql server but is this possible for postgresql?
BTW:
If the command line does contain the correct superpassword then the install just seems to overwrite the existing install and ignores parameters like --prefix etc .
I used init db to create a new database cluster before doing a second install but this new cluster was ignored?
In general you can have multiple independent instances of PostgreSQL. Strictly speaking it's database cluster with separate:
data directory
configuration (e.g. postgresql.conf, pg_hba.conf)
listening TCP/UDP port (default 5432+)
owner user and superuser role
locale and default encoding
log file
postmaster server process (on Windows postgres.exe)
Perfect well-done example is Debian with easy to use postgresql-common infrastructure (pg_ctlcluster, pg_lsclusters, pg_createcluster, pg_dropcluster, included SSL, log rotation and so on).
EDIT:
I found it's rather easy to install second, third, etc. instance of same versioned PostgreSQL under Windows with EnterpriseDB's installer, no need to use initdb and pg_ctl (assuming 64-bit installation, probably you need to use Program Files (x86) for 32-bit installation):
Open cmd with admin privileges (Run as Administrator)
Execute: cd "C:\Program Files\PostgreSQL\9.0\installer\server"
Create new database cluster (press Enter on every step): initcluster.vbs postgres postgres 12345 "C:\Program Files\PostgreSQL\9.0" "C:\Program Files\PostgreSQL\9.0\data2" 5433 DEFAULT
Register as Windows Service: startupcfg.vbs 9.0 postgres 12345 "C:\Program Files\PostgreSQL\9.0" "C:\Program Files\PostgreSQL\9.0\data2" postgresql-x64-9.0-2
Run newly created service postgresql-x64-9.0-2 using services.msc and you have second server
Change 12345 to your password specified during PostgreSQL installation. You don't have to use data2 directory, use whatever you like (but of course not existing data directory).
On Windows 7 I had success following these steps. You'll need the PsExec.exe utility available in the Sysinternals Suite. I assume here that the path to the Sysinternals Suite and the path to the bin folder of your existing PostgreSQL installation are in your PATH environment variable.
Open a cmd.exe window and enter the following command to open a prompt as the Network Service account.
psexec -i -u "nt authority\network service" cmd.exe
The Network Service account won't have access to your PATH, so cd 'C:\PostgreSQL\9.3\bin' and then enter the following command to initialize a data directory for your new instance. I've called mine "data2". It doesn't have to be in the postgres directory, but that's where the default data directory goes, so it's a reasonable choice.
initdb "C:\PostgreSQL\9.3\data2"
Edit C:\PostgreSQL\9.3\data2\postgresql.conf so that port = 5433 (the default instance uses 5432, and you shouldn't have two instances on the same port)
Leave the Network Service cmd prompt and in your standard prompt enter the following command to register the new service. Here I've named my new instance "pg_test"
pg_ctl register -N pg_test -U "nt authority\network service" -D "C:\PostgreSQL\9.3\data2"
Run the following command to start the service.
net start pg_test
The database owner role will be 'YOURMACHINENAME$'. If you want to change this to the standard 'postgres', you have to first create a new super user role that can rename the owner. From the command prompt, enter the following to create this super user.
createuser -s -r -l -i -P -h localhost -p 5433 -U YOURMACHINENAME$ mysuperuser
Finally, connect to the server with psql (psql -U mysuperuser -h localhost -p 5433 postgres) and enter the following commands to rename your database owner and add a password.
ALTER USER "YOURMACHINENAME$" RENAME TO postgres;
ALTER USER postgres WITH PASSWORD 'yourpassword';
Something like this should work (if not it's probably bug):
postgresql-9.0.4-1-windows_x64.exe ^
--mode unattended ^
--prefix c:\postgres\9.0-second ^
--servicename postgresql-x64-9.0-second ^
--serviceaccount postgres2 ^
--servicepassword <password> ^
--serverport 5433 ^
--superaccount postgres ^
--superpassword <password>
EDIT: after a couple of tests I believe it's not possible to create different Postgres instances of the same version using the One-click installer. Sorry.
OTOH you could always play with initdb and pg_ctl and use the existing installation to create a new instance. It would not be as easy as just starting the installer but it's doable.