OGR2OGR with MS SQL on Ubuntu 16.04 - sql-server

I need to access a Microsoft SQL Server with OGR2OGR from an Ubuntu Server 16.04. It is working on Microsoft, so the basic setup is fine.
My problem is that when I run my OGR2OGR command (after installing GDAL with sudo apt-get install gdal-bin):
ogr2ogr -overwrite -f MSSQLSpatial "MSSQL:server=tcp:<DATABASE_SERVER>,<PORT>;database=<DATABASE_NAME>;uid=<USER>;Pwd=<PASSWORD>;" "<IMPORT PATH FILE>" -nln "<TABLE NAME>" -progress
I get the error:
ERROR 1: Unable to initialize connection to the server for MSSQL:"DATABASE_SERVER";
[unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found
Try specifying the driver in the connection string from the list of available drivers:
I tried to install Microsoft ODBC Driver for SQL Server based on this instruction, but it cannot be installed when gdal is installed. The error is:
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
msodbcsql : Depends: unixodbc-utf16 (>= 2.3.1-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
I managed to install the Microsoft ODBC driver after a while but it meant removing GDAL. So I can either have GDAL or the Microsoft ODBC driver. Is there a way to solve this problem?
Thanks for your help!
UPDATE (2017.01.03): I found the following solution and I also commented on the according article on the MSDN Blog (so far without reply) here
Install UnixODBC if not already there
sudo apt-get install unixodbc unixodbc-dev
Download from somewhere: msodbcsql-11.0.2270.0.tar.gz (it could be better to use msodbcsql-13.x.x.x.tar.gz but I was not able to find it) and
tar xvfz msodbcsql-11.0.2270.0.tar.gz
cd msodbcsql-11.0.2270.0
ldd lib64/libmsodbcsql-11.0.so.2270.0
If there are missing dependencies install them, in my case it could be done like this: (everything except the apt-get install is to fix the naming of the file by creating a link:)
sudo apt-get install libssl1.0.0 libssl-dev
cd /lib/x86_64-linux-gnu
sudo ln -s libssl.so.1.0.0 libssl.so.10
sudo ln -s libcrypto.so.1.0.0 libcrypto.so.10
Installing and cleaning up:
sudo bash ./install.sh install --force --accept-license
rm -rf /tmp/msodbcubuntu
If the Driver is not found by the tool using it, (e.g. ogr2ogr or pyodbc) edit /etc/odbcinst.ini and create a connector for [SQL Server]
[SQL Server]
Description=Microsoft ODBC Driver 11 for SQL Server
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0
Threading=1
UsageCount=2

Shamelessly from : https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15
sudo su
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install msodbcsql17
Verify with
ogr2ogr --formats
Which should then list:
MSSQLSpatial -vector- (rw+): Microsoft SQL Server Spatial Database

Related

How to properly install pyodbc and drivers on M1 mac

I have a new M1 pro macbook and I'm trying to install pyodbc and relevant drivers on my machine.
So far I have installed unixodbc via homebrew and ODBC drivers according to the instructions found here: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15
I then pip installed pyodbc but when I try to run pyodbc.connect() within a test script, python just aborts with the following message, regardless which connection string I use:
[1] 2086 abort python test.py
After investigating a bit more I find that pyodbc doesn't have any drivers. When I run pyodbc.drivers() I get an empty list. So it seems like pyodbc cannot find the drivers I installed earlier.
Here is the output when I run odbcinst -j:
unixODBC 2.3.9
DRIVERS............: /opt/homebrew/etc/odbcinst.ini
SYSTEM DATA SOURCES: /opt/homebrew/etc/odbc.ini
FILE DATA SOURCES..: /opt/homebrew/etc/ODBCDataSources
USER DATA SOURCES..: /Users/kdot/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
and the odbcinst.ini file contains:
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/homebrew/lib/libmsodbcsql.17.dylib
UsageCount=3
I've also tried to explicitly give the driver file path to pyodbc in the connection string but then pyodbc.connect() just hangs instead of aborting right away.
I am running all of this within a virtual environment with the following versions:
python: 3.8.10
pyodbc: 4.0.32
Does anyone know how to make pyodbc see and use the relevant drivers?
First, install the ODBC driver as instructed by Microsoft here.
Further, Microsoft recommends creating symbolic links as follows:
sudo ln -s /usr/local/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /usr/local/etc/odbc.ini /etc/odbc.ini
However, the location of those files on M1 macs might be different, depending on the the default homebrew directory. In my case, the following paths worked:
sudo ln -s /opt/homebrew/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /opt/homebrew/etc/odbc.ini /etc/odbc.ini
This is what worked for me on a m2 macbook (from the official ms documentation).
Make sure you have homebrew installed.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql17 mssql-tools
sudo ln -s /opt/homebrew/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /opt/homebrew/etc/odbc.ini /etc/odbc.ini
The problem here is that "several critical CLI tools like nvm and brew do not have native versions built for the new M1 architecture." Source.
Luckily Apple Silicon comes with a translation layer named Rosetta 2, which is basically a CLI interface through which you have to install tools like brew in order for them to install correctly.
In this Medium post you can read how to use Rosetta 2. We got pyodbc working by installing Python 3.9, pip, wheel, openssl, unixodbc, freetds, the correct drivers and finally pyodbc (in this order) all with Rosetta 2.
Found the following link helpful to install pyodbc on MacBook M1 Chip.
We happen to get sql.h file missing error and if we try to install the wheel directly from the site it throws platform not supported error.
Follow the below document and then add the export commands into your zrc or bash file and you should be good to go.
https://whodeenie.medium.com/installing-pyodbc-and-unixodbc-for-apple-silicon-8e238ed7f216
Need more reading about the bug and how people tried to learn it.
https://github.com/mkleehammer/pyodbc/issues/846
I still had this issue with the default Homebrew & ODBC installations on macOS Ventura with a 2022 MacBook Pro.
The only thing that worked was to completely uninstall Homebrew and the ODBC drivers and then reinstall both with the prefix arch -x86_64.
$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ arch -x86_64 brew install msodbcsql18 mssql-tools18

Access external SQL server from inside of Docker

I am having a really hard time getting my docker container to access the MS SQL server. I have tried following this guide here, but to no avail.
Here is my Base Dockerfile:
FROM python:3
ADD ./odbcinst.ini /etc/odbcinst.ini
RUN apt-get update && apt-get install gcc
RUN apt-get install -y tdsodbc unixodbc-dev
RUN apt-get install unixodbc-bin -y
RUN apt-get clean -y
RUN apt-get update && apt-get install -y gcc unixodbc-dev mssql-python-pyodbc
RUN pip install pyodbc
RUN pip install plaster_pastedeploy pyramid pyramid_jinja2 pyramid_debugtoolbar waitress yagmail pyodbc
And here is the other docker file that extends it:
FROM pyodbc
COPY . .
RUN pip install -e companalysis/
CMD [ "pserve", "companalysis/development.ini" ]
EXPOSE 8081
and here is my sql connection string:
strconn = 'DRIVER=
{FreeTDS};SERVER=192.168.0.6;
DATABASE=xxxx;UID=xxxx;PWD=xxx'
and no matter what I do I get this error:
Error: ('01000', u"[01000] [unixODBC][Driver Manager]Can't open lib 'FreeTDS' : file not found (0) (SQLDriverConnect)")
I would love some assistance on this.
EDIT: I have gotten it to install with this in my dockerfile:
I got it to properly install the driver with this in my dockerfile:
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so" >> /etc/odbcinst.ini
RUN export PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1
RUN apt-get update && apt-get install -y unixodbc unixodbc-dev freetds-dev freetds-bin tdsodbc
I solved this issue by editing /etc/odbcinst.ini like so:
RUN echo "[FreeTDS]\n\
Description = FreeTDS unixODBC Driver\n\
Driver = /usr/lib/arm-linux-gnueabi/odbc/libtdsodbc.so\n\
Setup = /usr/lib/arm-linux-gnueabi/odbc/libtdsS.so" >> /etc/odbcinst.ini
Paths to libtdsodbc.so and libtdsS.so might be different depending on your architechture, as already pointed out in the linked questions.
If in doubt about the paths run the docker container, get into it (docker exec -ti <hash> bash) and search for the correct path.
Depending on your OS and architecture you could also use the Microsoft ODBC driver instead of the FreeTDS driver. For detailed instructions see the docs.

Error: apt-get install msodbcsql

When I execute the following in the terminal:
#sudo apt-get install msodbcsql
I get the following error:
Installation Failed, ODBC Driver 11 for SQL Server Detected!
I tried to reinstall following this tutorial:
https://learn.microsoft.com/en-us/sql/connect/odbc/linux/installing-the-microsoft-odbc-driver-for-sql-server-on-linux
but I still get the same error. Any ideas?
I'm using Debian with Ubuntu packages for mssql server. I had the same problem trying to reinstall mssql-tools, the message:
Installation Failed, ODBC Driver 13 for SQL Server Detected!
The solution:
In file /etc/odbcinst.ini delete the following lines:
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.8.0
UsageCount=1
Finally, I did apt -f install and problem solved.
To solve this i did a couple of things:
I removed the contents of /etc/odbcinst.ini as suggested by Sebastian
I stopped the mssql-server service sudo systemctl stop mssql-serverI don't know if this was necessary, but fiddling with tools related to it, it seemed the decent thing to do.
I did sudo apt remove mssql-tools
Then sudo apt remove msodbcsql The mssqlodbc part appended by whatever version number
Then sudo apt install mssql-toolsthis will install the newest version of mssql-tools for which msodbcsql is a requirement and thus it will itself fetch the required (newest i should think) version of that package. At this point, the problem has been fixed, The Sql service should still be started though
sudo systemctl start mssql-server
And thus everything was fine for me again.
Please notice that i did this in elementary OS 0.4 Loki.
I was trying to install ODBC 13 but had a very similar error
Installation Failed, ODBC Driver 13 for SQL Server Detected!
I had to:
sudo apt-get remove unixodbc mssql-tools odbcinst libodbc1
manually install the package apt had downloaded (I found it in /var/cache/apt/archives/msodbcsql_13.1.4.0-1_amd64.deb)
sudo dpkg -i msodbcsql_13.1.4.0-1_amd64.deb
ldd /opt/microsoft/msodbcsql/lib64/libmsodbcsql-*
then you can reinstall unixodbc, mssql-tools, libodbc1, etc.
I had to remove unixODBC before the installation could complete successfully.
MS were serious when they said "yum remove unixODBC #to avoid conflicts"
I had otherwise followed MS removal instructions for msodbc 11 prior to installing 13. (RedHat 7 intructions applied to Fedora 25 - DNF instead of YUM)
I discovered I needed to remove /usr/local/etc/odbcinst.ini too. With that config file in place I kept getting the "Installation Failed, ODBC Driver 11 for SQL Server Detected!" error.
Thus:
$ sudo bash
# apt-get remove unixodbc mssql-tools odbcinst libodbc1
# rm /usr/local/etc/odbcinst.ini
# apt-get -f install
# sudo dpkg -i msodbcsql_13.1.4.0-1_amd64.deb
Installation Failed, ODBC Driver 11 for SQL Server Detected!
If you've installed package with registration option (which is by default), remove the driver by:
odbcinst -u -d -n "ODBC Driver 13 for SQL Server"
Eventually locate your driver's ini file by: odbcinst -j and remove the driver section manually.
See: Installing the Microsoft ODBC Driver for SQL Server on Linux and macOS

Linux python3 - Can't open lib 'SQL Server'

I am trying to connect to an Microsoft Azure SQL server database.
This is how i am trying to connect:
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=%s' % (self.config.get("Sql", "DataSource")),
user= self.config.get("Sql", "UserId"),
password=self.config.get("Sql", "Password"),
database=self.config.get("Sql", "Catalog"))
I am getting an error while excuting this line. The error:
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found (0) (SQLDriverConnect)")
Can't figure why this is happening, Any idea?
replace DRIVER={SQL Server} with DRIVER={/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.0.so.1.1}
Update - December 2022
The current installation instructions for the ODBC driver are here
I also recommend you install the ODBC Driver and then try to use pyodbc. I am assuming you are on an Ubuntu 15.04+ machine.
To install the ODBC Driver follow the following instructions:
sudo su
wget https://gallery.technet.microsoft.com/ODBC-Driver-13-for-Ubuntu-b87369f0/file/154097/2/installodbc.sh
sh installodbc.sh
Once you do that, install pyodbc using pip and try the following script:
import pyodbc
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("SELECT ##version;")
row = cursor.fetchone()
while row:
print row
row = cursor.fetchone()
Let me know how that goes.
Cheers,
Meet
Download Dependencies depends on your platform,
(for other OS Download your Dependencies)
This example for Ubuntu:
# sudo su
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
#Download appropriate package for the OS version
#Choose only ONE of the following, corresponding to your OS version
#Ubuntu 14.04
# curl https://packages.microsoft.com/config/ubuntu/14.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
#Ubuntu 16.04
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# #Ubuntu 18.04
# curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
# #Ubuntu 18.10
# curl https://packages.microsoft.com/config/ubuntu/18.10/prod.list > /etc/apt/sources.list.d/mssql-release.list
# #Ubuntu 19.04
# curl https://packages.microsoft.com/config/ubuntu/19.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
exit
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install msodbcsql17
# optional: for bcp and sqlcmd
sudo ACCEPT_EULA=Y apt-get install mssql-tools
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
source ~/.bashrc
# optional: for unixODBC development headers
sudo apt-get install unixodbc-dev
and then change,
DRIVER={/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.0.so.1.1}
DRIVER={ODBC Driver 17 for SQL Server}
Check those links. It solved my problems which were kind of similar.
Installing FreeTDS
Connecting to SQL-Azure using FreeTDS
If you are using an offline REHL server, then follow the below method to setup connection to Microsoft SQL Server.
Download UNIXODBC & MSSQLTools packages—e.g., unixODBC-2.3.7-1.rh.x86_64.rpm/mssql-tools-17.9.1.1-1.x86_64.rpm—from https://packages.microsoft.com/rhel/, as per your REHL version.
Place downloaded files on the REHL server via winscp or any ssh client.
Install these two files in sequence given below:
yum localinstall unixODBC-2.3.7-1.rh.x86_64.rpm
yum localinstall mssql-tools-17.9.1.1-1.x86_64.rpm)
Go to the installation folder, and copy the path as shown in e.g.,
/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.9.so.1.1
Put this path in code:
driverpath = r"/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.9.so.1.1"
Your problem will get solved.
I see that this post has an accepted answer but it did not work for me running Python 3.8 on Ubuntu 20.04. I tried several things but I think that ultimately, it was a security issue having to do with the openSSL package. Here are the steps I recommend.
Check the openSSL version. Do this in the Linux console
openssl version
The problematic version was 1.1.1f on my system. If that is the version you have, update it with these commands
wget https://www.openssl.org/source/openssl-1.1.1p.tar.gz -O openssl-1.1.1p.tar.gz
tar -zxvf openssl-1.1.1p.tar.gz
cd openssl-1.1.1p
./config
make
sudo make install
sudo ldconfig
openssl version
Then you have to edit /etc/ssl/openssl.cnf . Put this in the beginning
openssl_conf = default_conf
And this at the end
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT:#SECLEVEL=0
Like I said, that worked for me. Hopefully it can help someone else too.
Hi I am using this snippet, which searches and return lastest available version of ODBC Driver, or raises an error:
def select_driver():
"""Find least version of: ODBC Driver for SQL Server."""
drv = sorted([drv for drv in pyodbc.drivers() if "ODBC Driver " in drv and " for SQL Server" in drv])
if len(drv) == 0:
raise Exception("No 'ODBC Driver XX for SQL Server' found.")
return drv[-1]
print(select_driver()) # ODBC Driver 17 for SQL Server

How to connect MSSQL in Laravel homestead?

I am trying to connect with a MSSQL database using laravel homestead. It always throws the exception:
PDOException in Connector.php line 55: could not find driver.
I've seen a lot of people talking about FreeTDS and Sybase drivers, but I can't seem to figure out what I need to enable laravel/php access to MSSQL databases from a Linux Web Server running the Laravel Homestead Vagrant box.
I did lots of R&D for resolving this issue. Finally found the solution:
Laravel vagrant is using php 7. For mssql support we have to enable the relevant sybase driver for linux :
To Enable MSSQL Support for PHP 7:
First, ssh into your box vagrant ssh from the Homestead folder.
Command : vagrant ssh
Install the Sybase package for enabling the support for PDO and Mssql.
Command : sudo apt-get install php7.0-sybase
then run php -m on the ssh to make sure pdo_dblib is enabled.
All Done!!!! Cheers
The following works for php72. You may try changing the version of php in the commands.
vagrant ssh
#set the default PHP version to 7.2
php72
sudo su
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
#Download appropriate package for the OS version
#Choose only ONE of the following, corresponding to your OS version
#Ubuntu 16.04
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
#Ubuntu 18.04
curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
#Ubuntu 20.04
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
exit
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install msodbcsql17
sudo ACCEPT_EULA=Y apt-get install mssql-tools
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
source ~/.bashrc
sudo apt-get install unixodbc-dev
sudo pecl config-set php_ini /etc/php/7.2/fpm/php.ini
sudo pecl install sqlsrv
sudo pecl install pdo_sqlsrv
sudo su
printf "; priority=20\nextension=sqlsrv.so\n" > /etc/php/7.2/mods-available/sqlsrv.ini
printf "; priority=30\nextension=pdo_sqlsrv.so\n" > /etc/php/7.2/mods-available/pdo_sqlsrv.ini
exit
sudo phpenmod -v 7.2 sqlsrv pdo_sqlsrv
sudo systemctl restart php7.2-fpm
# restart nginx, if needed:
sudo systemctl restart nginx.service
References:
https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#ubuntu17
https://learn.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac?view=sql-server-ver15#installing-the-drivers-on-ubuntu-1604-1804-and-2004
Laravel uses PDO extension to connect to database, you could check if the extension is enabled using
var_dump(class_exists('PDO'))
Also, if you could post the specific error, it would be great.
Thanks for this. Still works as of now with:
php7.2-sybase
using DB_HOST= IP_ADDRESS\NAMED_INSTANCE
Commenting out the PORT directive in both .env and database.php

Resources