How to properly install pyodbc and drivers on M1 mac - sql-server

I have a new M1 pro macbook and I'm trying to install pyodbc and relevant drivers on my machine.
So far I have installed unixodbc via homebrew and ODBC drivers according to the instructions found here: https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15
I then pip installed pyodbc but when I try to run pyodbc.connect() within a test script, python just aborts with the following message, regardless which connection string I use:
[1] 2086 abort python test.py
After investigating a bit more I find that pyodbc doesn't have any drivers. When I run pyodbc.drivers() I get an empty list. So it seems like pyodbc cannot find the drivers I installed earlier.
Here is the output when I run odbcinst -j:
unixODBC 2.3.9
DRIVERS............: /opt/homebrew/etc/odbcinst.ini
SYSTEM DATA SOURCES: /opt/homebrew/etc/odbc.ini
FILE DATA SOURCES..: /opt/homebrew/etc/ODBCDataSources
USER DATA SOURCES..: /Users/kdot/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8
and the odbcinst.ini file contains:
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/homebrew/lib/libmsodbcsql.17.dylib
UsageCount=3
I've also tried to explicitly give the driver file path to pyodbc in the connection string but then pyodbc.connect() just hangs instead of aborting right away.
I am running all of this within a virtual environment with the following versions:
python: 3.8.10
pyodbc: 4.0.32
Does anyone know how to make pyodbc see and use the relevant drivers?

First, install the ODBC driver as instructed by Microsoft here.
Further, Microsoft recommends creating symbolic links as follows:
sudo ln -s /usr/local/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /usr/local/etc/odbc.ini /etc/odbc.ini
However, the location of those files on M1 macs might be different, depending on the the default homebrew directory. In my case, the following paths worked:
sudo ln -s /opt/homebrew/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /opt/homebrew/etc/odbc.ini /etc/odbc.ini

This is what worked for me on a m2 macbook (from the official ms documentation).
Make sure you have homebrew installed.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql17 mssql-tools
sudo ln -s /opt/homebrew/etc/odbcinst.ini /etc/odbcinst.ini
sudo ln -s /opt/homebrew/etc/odbc.ini /etc/odbc.ini

The problem here is that "several critical CLI tools like nvm and brew do not have native versions built for the new M1 architecture." Source.
Luckily Apple Silicon comes with a translation layer named Rosetta 2, which is basically a CLI interface through which you have to install tools like brew in order for them to install correctly.
In this Medium post you can read how to use Rosetta 2. We got pyodbc working by installing Python 3.9, pip, wheel, openssl, unixodbc, freetds, the correct drivers and finally pyodbc (in this order) all with Rosetta 2.

Found the following link helpful to install pyodbc on MacBook M1 Chip.
We happen to get sql.h file missing error and if we try to install the wheel directly from the site it throws platform not supported error.
Follow the below document and then add the export commands into your zrc or bash file and you should be good to go.
https://whodeenie.medium.com/installing-pyodbc-and-unixodbc-for-apple-silicon-8e238ed7f216
Need more reading about the bug and how people tried to learn it.
https://github.com/mkleehammer/pyodbc/issues/846

I still had this issue with the default Homebrew & ODBC installations on macOS Ventura with a 2022 MacBook Pro.
The only thing that worked was to completely uninstall Homebrew and the ODBC drivers and then reinstall both with the prefix arch -x86_64.
$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ arch -x86_64 brew install msodbcsql18 mssql-tools18

Related

Version of driver installed with homebrew

I have installed via homebrew the unixODBC driver manager and the FreeTDS driver in order to create a connection to a SQL server.
Which is the brew command that can let me know which version of the driver has been installed?
You can obtain the version and some information about any package with brew info. In your case, you can run:
brew info unixodbc
and
brew info freetds

How to interact with mssql from python?

How to interact with mssql from python? I realy nead it to write my site.
I tried pyodbc
(
pip install pyodbc
)
but cmd says there is an error in setup.py.
Pleese, help me!
P.S. I am using windows and mssql.
[Updated Answer]
Follow these steps to install pyodbc and pymssql in 64-bit Windows with *Python 3.**
Update pip tool: python -m pip install pip -U
Update python setuptools : python -m pip install setuptools -U
Install pyodb : pip3 install pyodbc -U
Download and install MS Visual C++ 14.0 Build Tools. It's required for installing pymssql.
double click on downloaded file and select Workloads → Visual C++ build tools then in optional features tab select Windows 10 SDK(windows 8 user select Windows 8 SDK) and install.
Install pymssql: pip3 install pymssql -U
[Old Answer]
One can also try these unofficial wheel files, if above method doesn't work.
You can install the same using the unofficial pyodbc wheel for python 3.* from here, just download the appropriate wheel (*.whl) file form this link and install it using
pip install **downloaded_wheel_file_name**.whl
Note: Don't forget to change the downloaded_wheel_file_name with your downloaded wheel file's name.
You need to have Microsoft ODBC driver for sql in Windows to make
pip install pyodbc
work.
Try to follow this link and install required drivers.
Try this #Kostya2007: pip install --allow-external pyodbc --allow-unverified pyodbc pyodbc
or
pip install https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/pyodbc/pyodbc-3.0.7.zip
depending on your Python version
or
easy_install pyodbc-3.0.7.win32-py2.7.exe

Error: apt-get install msodbcsql

When I execute the following in the terminal:
#sudo apt-get install msodbcsql
I get the following error:
Installation Failed, ODBC Driver 11 for SQL Server Detected!
I tried to reinstall following this tutorial:
https://learn.microsoft.com/en-us/sql/connect/odbc/linux/installing-the-microsoft-odbc-driver-for-sql-server-on-linux
but I still get the same error. Any ideas?
I'm using Debian with Ubuntu packages for mssql server. I had the same problem trying to reinstall mssql-tools, the message:
Installation Failed, ODBC Driver 13 for SQL Server Detected!
The solution:
In file /etc/odbcinst.ini delete the following lines:
[ODBC Driver 13 for SQL Server]
Description=Microsoft ODBC Driver 13 for SQL Server
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.1.so.8.0
UsageCount=1
Finally, I did apt -f install and problem solved.
To solve this i did a couple of things:
I removed the contents of /etc/odbcinst.ini as suggested by Sebastian
I stopped the mssql-server service sudo systemctl stop mssql-serverI don't know if this was necessary, but fiddling with tools related to it, it seemed the decent thing to do.
I did sudo apt remove mssql-tools
Then sudo apt remove msodbcsql The mssqlodbc part appended by whatever version number
Then sudo apt install mssql-toolsthis will install the newest version of mssql-tools for which msodbcsql is a requirement and thus it will itself fetch the required (newest i should think) version of that package. At this point, the problem has been fixed, The Sql service should still be started though
sudo systemctl start mssql-server
And thus everything was fine for me again.
Please notice that i did this in elementary OS 0.4 Loki.
I was trying to install ODBC 13 but had a very similar error
Installation Failed, ODBC Driver 13 for SQL Server Detected!
I had to:
sudo apt-get remove unixodbc mssql-tools odbcinst libodbc1
manually install the package apt had downloaded (I found it in /var/cache/apt/archives/msodbcsql_13.1.4.0-1_amd64.deb)
sudo dpkg -i msodbcsql_13.1.4.0-1_amd64.deb
ldd /opt/microsoft/msodbcsql/lib64/libmsodbcsql-*
then you can reinstall unixodbc, mssql-tools, libodbc1, etc.
I had to remove unixODBC before the installation could complete successfully.
MS were serious when they said "yum remove unixODBC #to avoid conflicts"
I had otherwise followed MS removal instructions for msodbc 11 prior to installing 13. (RedHat 7 intructions applied to Fedora 25 - DNF instead of YUM)
I discovered I needed to remove /usr/local/etc/odbcinst.ini too. With that config file in place I kept getting the "Installation Failed, ODBC Driver 11 for SQL Server Detected!" error.
Thus:
$ sudo bash
# apt-get remove unixodbc mssql-tools odbcinst libodbc1
# rm /usr/local/etc/odbcinst.ini
# apt-get -f install
# sudo dpkg -i msodbcsql_13.1.4.0-1_amd64.deb
Installation Failed, ODBC Driver 11 for SQL Server Detected!
If you've installed package with registration option (which is by default), remove the driver by:
odbcinst -u -d -n "ODBC Driver 13 for SQL Server"
Eventually locate your driver's ini file by: odbcinst -j and remove the driver section manually.
See: Installing the Microsoft ODBC Driver for SQL Server on Linux and macOS

OGR2OGR with MS SQL on Ubuntu 16.04

I need to access a Microsoft SQL Server with OGR2OGR from an Ubuntu Server 16.04. It is working on Microsoft, so the basic setup is fine.
My problem is that when I run my OGR2OGR command (after installing GDAL with sudo apt-get install gdal-bin):
ogr2ogr -overwrite -f MSSQLSpatial "MSSQL:server=tcp:<DATABASE_SERVER>,<PORT>;database=<DATABASE_NAME>;uid=<USER>;Pwd=<PASSWORD>;" "<IMPORT PATH FILE>" -nln "<TABLE NAME>" -progress
I get the error:
ERROR 1: Unable to initialize connection to the server for MSSQL:"DATABASE_SERVER";
[unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found
Try specifying the driver in the connection string from the list of available drivers:
I tried to install Microsoft ODBC Driver for SQL Server based on this instruction, but it cannot be installed when gdal is installed. The error is:
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
msodbcsql : Depends: unixodbc-utf16 (>= 2.3.1-1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
I managed to install the Microsoft ODBC driver after a while but it meant removing GDAL. So I can either have GDAL or the Microsoft ODBC driver. Is there a way to solve this problem?
Thanks for your help!
UPDATE (2017.01.03): I found the following solution and I also commented on the according article on the MSDN Blog (so far without reply) here
Install UnixODBC if not already there
sudo apt-get install unixodbc unixodbc-dev
Download from somewhere: msodbcsql-11.0.2270.0.tar.gz (it could be better to use msodbcsql-13.x.x.x.tar.gz but I was not able to find it) and
tar xvfz msodbcsql-11.0.2270.0.tar.gz
cd msodbcsql-11.0.2270.0
ldd lib64/libmsodbcsql-11.0.so.2270.0
If there are missing dependencies install them, in my case it could be done like this: (everything except the apt-get install is to fix the naming of the file by creating a link:)
sudo apt-get install libssl1.0.0 libssl-dev
cd /lib/x86_64-linux-gnu
sudo ln -s libssl.so.1.0.0 libssl.so.10
sudo ln -s libcrypto.so.1.0.0 libcrypto.so.10
Installing and cleaning up:
sudo bash ./install.sh install --force --accept-license
rm -rf /tmp/msodbcubuntu
If the Driver is not found by the tool using it, (e.g. ogr2ogr or pyodbc) edit /etc/odbcinst.ini and create a connector for [SQL Server]
[SQL Server]
Description=Microsoft ODBC Driver 11 for SQL Server
Driver=/opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0
Threading=1
UsageCount=2
Shamelessly from : https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15
sudo su
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get install msodbcsql17
Verify with
ogr2ogr --formats
Which should then list:
MSSQLSpatial -vector- (rw+): Microsoft SQL Server Spatial Database

Connecting to SQL Server with ActiveRecord

Have you ever had to connect to SQL Server with ActiveRecord? Is this possible? Can anyone provide some starting points?
This what I used:
From here:
http://github.com/rails-sqlserver/2000-2005-adapter/tree/master
Installation
First, you will need Ruby DBI and Ruby ODBC. To my knowledge the ADO DBD for DBI is no longer supported. The installation below is not a comprehensive walk thru on how to get all the required moving parts like FreeTDS installed and/or configured. It will also assume gem installations of both the dependent libraries and the adapter itself.
It should be noted that this version of the adapter was developed using both the ancient 0.0.23 version of DBI up to the current stable release of 0.4.0. Because later versions of DBI will be changing many things, IT IS HIGHLY RECOMMENDED that you max your install to version 0.4.0 which the examples below show. For the time being we are not supporting DBI versions higher than 0.4.0. The good news is that if you were using a very old DBI with ADO, technically this adapter will still work for you, but be warned your path is getting old and may not be supported for long.
$ gem install dbi --version 0.4.0
$ gem install dbd-odbc --version 0.2.4
$ gem install rails-sqlserver-2000-2005-adapter -s http://gems.github.com
From here: http://lambie.org/2008/02/28/connecting-to-an-mssql-database-from-ruby-on-ubuntu/
Firstly, update your ~/.profile to include the following:
export ODBCINI=/etc/odbc.ini
export ODBCSYSINI=/etc
export FREETDSCONF=/etc/freetds/freetds.conf
Then reload your .profile, by logging out and in again.
Secondly, on Ubuntu 7.10 Server I needed to install some packages.
mlambie#ubuntu:~$ sudo aptitude install unixodbc unixodbc-dev freetds-dev sqsh tdsodbc
With FreeTDS installed I could configure it like this:
mlambie#ubuntu:/etc/freetds$ cat freetds.conf
[ACUMENSERVER]
host = 192.168.0.10
port = 1433
tds version = 7.0
The important thing here is ACUMENSERVER, which is the DSN that I’ll use when connecting to the database. The host, and port are self-explanatory, and it’s worth noting that I had to use 7.0 specifically as the tds version.
Testing FreeTDS is not too hard:
mlambie#ubuntu:~$ sqsh -S ACUMENSERVER -U username -P password
sqsh: Symbol `_XmStrings' has different size in shared object, consider re-linking
sqsh-2.1 Copyright (C) 1995-2001 Scott C. Gray
This is free software with ABSOLUTELY NO WARRANTY
For more information type '\warranty'
1> use acumen
2> go
1> select top 1 firstname, lastname from tblClients
2> go
[record returned]
(1 row affected)
1> quit
Next up it’s necessary to configure ODBC:
mlambie#ubuntu:/etc$ cat odbcinst.ini
[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
Driver = /usr/lib/odbc/libtdsodbc.so
Setup = /usr/lib/odbc/libtdsS.so
CPTimeout =
CPReuse =
FileUsage = 1
mlambie#ubuntu:/etc$ cat odbc.ini
[ACUMENSERVER]
Driver = FreeTDS
Description = ODBC connection via FreeTDS
Trace = No
Servername = ACUMENSERVER
Database = ACUMEN
I then tested the connection with isql:
mlambie#ubuntu:~$ isql -v ACUMENSERVER username password
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL> use ACUMEN
[][unixODBC][FreeTDS][SQL Server]Changed database context to 'Acumen'.
[ISQL]INFO: SQLExecute returned SQL_SUCCESS_WITH_INFO
SQLRowCount returns -1
SQL> select top 1 firstname from tblClients;
[record returned]
SQLRowCount returns 1
1 rows fetched
SQL> quit
OK, so we’ve got ODBC using FreeTDS to connect to a remote MSSQL server. All that’s left is to add Ruby into the mix.
mlambie#ubuntu:~$ sudo aptitude install libdbd-odbc-ruby
The last thing to test is that Ruby can use DBI and ODBC to hit the actual database, and that’s easy to test:
mlambie#ubuntu:~$ irb
irb(main):001:0> require "dbi"
=> true
irb(main):002:0> dbh = DBI.connect('dbi:ODBC:ACUMENSERVER', 'username', 'password')
=> #<DBI::DatabaseHandle:0xb7ac57f8 #handle=#<DBI::DBD::ODBC::Database:0xb7ac5744
#handle=#<odbc::database:0xb7ac576c>, #attr={}>, #trace_output=#</odbc::database:0xb7ac576c><io:0xb7cbff54>,
#trace_mode=2>
irb(main):003:0> quit
And a more complete test (only with SQL SELECT, mind you):
#!/usr/bin/env ruby
require 'dbi'
db = DBI.connect('dbi:ODBC:ACUMENSERVER', 'username', 'password')
select = db.prepare('SELECT TOP 10 firstname FROM tblClients')
select.execute
while rec = select.fetch do
puts rec.to_s
end
db.disconnect
</io:0xb7cbff54>
From here (to fix the odbc lib being in the wrong place):
http://ubuntuforums.org/showthread.php?t=433435&page=2
libtdsodbc.so
with freeTDS (freetds-dev, tdsodbc), you can either edit the path in the odbcinst.ini file for the [FreeTDS] driver section OR cp the /usr/lib/odbc/libtdsodbc.so into /usr/lib/libtdsodbc.so.
either way works when accessing mssql from the prompt
isql -v $dsn $user $passwd
i found this to be useful
http://www.unixodbc.org/doc/FreeTDS.html#Configuration
And then in the database.yml file:
development:
adapter: sqlserver
mode: odbc
dsn: dsn_name
username: my_username
password: my_password
These are the steps i've compiled for Centos 5.3. It took me a lot of trial and error to get this working. It's from a completely clean Centos installation.
Install EPEL:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
Install ruby, rubygems, freetds, unixODBC, development tools:
yum install gcc
yum install freetds
yum install ruby-devel
yum install unixODBC-devel
yum install ruby rubygems
Install rails:
gem install rails
Install DB related gems:
gem install dbd-odbc
gem install rails-sqlserver-2000-2005-adapter -s http://gems.github.com
Download, build and install ruby-odbc:
wget http://www.ch-werner.de/rubyodbc/ruby-odbc-0.9997.tar.gz
tar zxvf ruby-odbc-0.9997.tar.gz
cd ruby-odbc-0.9997
ruby extconf.rb
make
make install
Final gem list:
# gem list
*** LOCAL GEMS ***
actionmailer (2.3.2)
actionpack (2.3.2)
activerecord (2.3.2)
activeresource (2.3.2)
activesupport (2.3.2)
dbd-odbc (0.2.4)
dbi (0.4.1)
deprecated (2.0.1)
rails (2.3.2)
rails-sqlserver-2000-2005-adapter (2.2.17)
rake (0.8.7)
You can use various tools like isql to test your ODBC connection. Brian's post covers this in nice detail so I won't repeat. In rails you need a database.yml that looks something like this:
development:
adapter: sqlserver
mode: odbc
dsn: dsnName
username: username
password: password
On Ubuntu, I use FreeTDS and the activerecord-sqlserver-adapter gem.
You can install FreeTDS through apt:
sudo apt-get install freetds
And add this to your Gemfile:
gem 'activerecord-sqlserver-adapter'
I had to change the configured FreeTDS version number in /etc/freetds/freetds.conf to 8.0 in order to get things working correctly.
[global]
tds version = 8.0
activerecord-sqlserver-adapter on GitHub
FreeTDS project

Resources