I am unable to establish connection to my Oracle database from Azure Databricks although it works in ADF where I am able to query the table. But ADF takes time to filter the records so I am still trying to connect from Databricks.
I followed the steps from this Microsoft link, both manually and using init-script but error seems to persist.
When I looked into my cluster event log it says the init-script execution was successfully.
Error message when I tried to establish the connection:
DPI-1047: Cannot locate a 64-bit Oracle Client library: "/databricks/driver/oracle_ctl//lib/libclntsh.so: cannot open shared object file: No such file or directory".
When I executed the following command
dbutils.fs.ls("/databricks/driver/")
there was no such directory
This triggered me to post some questions here:
Does this mean the init-script did not perform its job?
Is /databricks/driver/oracle_ctl a hidden directory for dbutils.fs.ls?
Error message points to /databricks/driver/oracle_ctl//lib/libclntsh.so, when I manually inspected the downloaded oracle client, there is no such folder called lib although libclntsh.so exists in the main directory. Is there a problem that databricks is checking the wrong directory for the libclntsh.so?
Does this connections still works for others?
Syntax for connection: cx_Oracle.connect(user= user_name, password= password,dsn= IP+':'+Port+'/'+DB_name)
Above syntax works fine when connected from inside a on-premises machine.
Try installing the latest major release of cx_Oracle - which got renamed to python-oracledb, see the release announcement.
This version doesn't need Oracle Instant Client. The API is the same as cx_Oracle, although obviously the name is different.
If I understand the instructions, your init script would do something like:
/databricks/python/bin/pip install oracledb
Application code would be like:
import oracledb
connection = oracledb.connect(user='scott', password=mypw, dsn='yourdbhostname/yourdbservicename')
with connection.cursor() as cursor:
for row in cursor.execute('select city from locations'):
print(row)
Resources:
Home page: oracle.github.io/python-oracledb/
Quick start: Quick Start python-oracledb Installation
Documentation: python-oracle.readthedocs.io/en/latest/index.html
PyPI: pypi.org/project/oracledb/
Source: github.com/oracle/python-oracledb
Upgrading: Upgrading from cx_Oracle 8.3 to python-oracledb
Changed the path from "/databricks/driver/oracle_ctl/" to "/databricks/driver/oracle_ctl/instantclient" in the init-script and that error does not appear anymore.
Please use the following init script instead
dbutils.fs.put("dbfs:/databricks/<init-script-folder-name>/oracle_ctl.sh","""
#!/bin/bash
sudo apt-get install libaio1
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
mv /databricks/driver/oracle_ctl/instantclient* /databricks/driver/oracle_ctl/instantclient
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
""", True)
Notes:
The above init-script was advised by a databricks employee and can be found here.
As mentioned by Christopher Jones in one of the comments, cx_Oracle has been recently upgraded to oracledb with a thin and thick version.
You will get the above error if you don’t have Oracle instant client in your Cluster.
To resolve above error in azure databricks, please follow this code:
%sh
mkdir -p /opt/oracle
cd /opt/oracle
wget https://download.oracle.com/otn_software/nt/instantclient/19600/instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
unzip instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
set ORACLE_HOME=%ORABAS%\instantclient_19_3
set TNS_ADMIN=%ORACLE_HOME%
set PATH=%ORACLE_HOME%;%PATH%
To create init script, use the following code:
As per official doc,
dbutils.fs.put("dbfs:/databricks/<init-script-folder>/oracle_ctl.sh","""
#!/bin/bash
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
""", True)
To read data from oracle database in PySpark follow this article by Emrah Mete
For more information refer this official document:
https://docs.databricks.com/data/data-sources/oracle.html#oracle
adb server version (39) doesn't match this client (40); killing...
could not read ok from ADB Server
* failed to start daemon
error: cannot connect to daemon
adb: error: failed to get feature set: protocol fault (couldn't read status): Connection reset by peer
* daemon not running; starting now at tcp:5037
* daemon started successfully
- waiting for device -
error: protocol fault (couldn't read status): Connection reset by peer
I have the same issue when running adb devices command.
adb devices
List of devices attached
adb server version (36) doesn't match this client (40); killing...
* daemon started successfully
2322dc3d device
I ran command which adbwhich gave me the location of the adb that the previous command was using, in my case output was:
which adb
/usr/bin/adb
Then i ran whereis adb command which gave me the location of adb's:
whereis adb
adb: /usr/bin/adb /home/arefin/Android/Sdk/platform-tools/adb
/usr/share/man/man1/adb.1.gz
I get rid of this problem by moving the adb from /usr/bin directory with mv command: (meaning this is of no use thus instead of deleting just put in desktop),
/usr/bin$ `sudo mv adb /home/arefin/Desktop/`
After this i executed this command adb kill-server then ran adb devices. I found everything is fine this time.
My Android development related path in in $HOME/.profile file is, as below:
export ANDROID_HOME=/home/arefin/Android/Sdk
export PATH=$PATH:$ANDROID_HOME/platform-tools
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/emulator
export PATH=$PATH:$JAVA_HOME/bin
When encountered this error in Linux,I deleted the adb file in my
/usr/bin/
and replaced with one which located in path (usually in home path)
~/Android/Sdk/platform-tools/
and it worked.
As pointed out in previous answers, this is due to having multiple adb versions installed in your system. Probably one from the android-sdk-platform-tools installed via apt and other from Android Sdk itself (which is usually newer version).
The simplest way to solve this in Ubuntu (or Ubuntu-based OS's) is to add this line at the end of ~/.bashrc file:
alias adb='~/Android/Sdk/platform-tools/adb'
#Change path according to your Android Sdk installation directory
And then reopen terminal for changes to reflect.
This avoids the need to mess with system files as well.
Probably you have two versions of adb in two different locations in your system. The running server version is 39 and the adb client version is 40. You can resolve this by keeping only one adb in your PC.
You can figure it out as shown below-
In Linux below command gives you adb location if adb binary is added to path.
which adb
if adb is not added to path and still you are getting this error then search for adb using below command.
locate adb
if your adb client is other than above location, you may add newest version to above path and delete the old adb version.
If you are using windows, you figure it out the two adb locations and keep only at single place.
I know this is a little old, but if you have this problem and you have Vysor installed, then you can resolve it by looking in the Vysor application folder for the adb.exe (and Adb*.dll files) and replacing them from your Android SDK folder. I'm on a Windows machine and the relevant paths for me were:
Vysor
C:\Users\\AppData\Roaming\Vysor\crx\gidgenkbbabolejbgbpnhbimgjbffefm\app-2.1.7.crx-unpacked\native\win32*
(I'm guessing the path will vary according to the version, etc, but this will hopefully help figure it out.)
APK
C:\Users\\AppData\Local\Android\Sdk\platform-tools
On my machine, the relevant files were:
adb.exe,
AdbWinApi.dll,
AdbWinUsbApi.dll
After this, you will need to kill the existing ADB task as described in the other answers to this post.
Solved by
Open terminal in Android studio
adb kill-server
sudo cp ~/Android/Sdk/platform-tools/adb /usr/bin/adb
sudo chmod +x /usr/bin/adb
adb start-server
Run
tasklist | findstr adb
then kill the duplicate process
TASKKILL /PID "PID_NUMBER" /f
Encountered this problem myself, none of the solutions online worked for me.
did this by chance and it solved my problem!
I'm using Android Studio on a Macbook Pro.
first I tried to install adb from brew:
brew cask install android-platform-tools
then I uninstalled it:
brew cask uninstall android-platform-tools
After the last line adb didn't work in terminal but did work in Android Studio immediately! =D
posted so it may help someone else - good luck!
For ubuntu (more precise - kubuntu) i had the similar problem.
The problem was in android-studio, to fix it you need to do these steps:
Go to SDK-Manager
In SDK-Manager click the SDK Tools menu.
Uncheck "Android SDK Command-line Tools (latest)"
I had the same issue connecting to my SFTP Server app and could solve it as follows:
The error was caused (in my case, Ubuntu 18.04.5 LTS) by conflicting adb installations. One Installation coming with AndroidStudio (IntelliJ) located in .../Sdk/platform-tools and one coming with the package manager installation apt install located in the standard directory for executables /usr/bin. The installation location for AndroidStudio (IntelliJ) can be found in the AndroidStudio (IntelliJ) Menu: Tools → Android → SDK Manager: Android SDK Location. And the installation location related to the package manager can be found invoking the command which adb.
And the solution to this problem is to uninstall one of them.
Depending on your use case, if you’re most of the time working with AndroidStudio (IntelliJ) (and this version is up to date), then remove the (outdated) version installed with the package manager as follows.
To uninstall first find out which packages relates to /usr/bin/adb with the following command: dpkg -S /usr/bin/adb.
And then call apt autoremove adb to uninstall the adb package and all its dependencies.
And last but not least in order to still being able to call adb from the command line update the PATH variable in “/etc/profile”, if it should be accessible for all users, or ~/.bash_profile or ~/.profile (whichever exists) if it should only be accessible for the current user and append the path “.../Sdk/platform-tools” (that you looked up above) at the end. And then reboot or log-out & log-in for the new path to get applied.
In ubuntu
delete the adb
-> usr/bin
-> (delete command) sudo rm -rf /usr/bin/adb
Android/Sdk/platform-tools/adb
-> copy this folderpath.
paste into
->usr/bin
->(paste command) sudo ln -s /home/yourfile/Android/Sdk/platform-tools/adb /usr/bin
restarting your system, would also work for you.
I have faced the following error and what worked in my case was just to restart the system.
could not read ok from ADB Server
* failed to start daemon
error: cannot connect to daemon code here
adb.exe: failed to check server version: cannot connect to daemon
I had a similar error. Two different adb versions were conflicting with each other which caused it to keep restarting.
Run this command in terminal => where adb.exe
This will show you where the different adb files are. You can look at the file property date to see which is the latest one. Then copy the latest one and use it to overwrite/replace the older one(s). This will make both adb versions be the same and prevent a conflict.
This generally happens when there are two adb paths are available. Uninstall one will help
--> npm uninstall adb
This worked in my case.
None of the other solutions worked for me - Windows/WSL2-Ubuntu
The error is misleading but for me it was cos the adb port 5037 was in use.
Solution: find and kill the process using port 5037
Windows: netstat -aon | findstr 5037 and Stop-Process -Id <id-from-netstat>
WSL will attempt to connect to Windows port 5037
In my case wsl Ubuntu has different adb --version than Windows, I don't have to keep my Android Studio latest(since this might not your choice and do not guarantee same version as apt) or try with $PATH(wsl run Windows exe is not make sense).
The solution is simple, I download both Linux and Windows SDK Platform-Tools from official site which guarantee same version, then invoke relevant adb on each platform.
In my case I'd installed "AirDroid" on my windows machine and it runs "AirDroid_adb.exe", I had to kill that to get things to work. I have reported that they are using an old version of adb and they will hopefully fix it.
A Working Simple Answer for Windows:
make sure you have the sideload file (whatever.bin) in a sub directory of your adb executable.
make sure your adb executable folder, and sub directories are in PATH (look up DOS commands for PATH in Windows)
open a command prompt in the same directory as the sideload file
enter your command for example:
adb sideload mynewrom-5.4.3.2.1.bin
(Your device needs to be in recovery - sideload state before you send)
i was installing postgresql on ubuntu using linuxbrew:
brew install postgresql
it seems to work fine but after that because i was installing PostgreSQL for the first time i tried creating a database:
initdb /usr/local/var/postgres -E utf8
but it returned as:
initdb: command not found
i tried running the command with sudo but that doesn't helped
run locate initdb it should give you the list to chose. smth like:
MacBook-Air:~ vao$ locate initdb
/usr/local/Cellar/postgresql/9.5.3/bin/initdb
/usr/local/Cellar/postgresql/9.5.3/share/doc/postgresql/html/app-initdb.html
/usr/local/Cellar/postgresql/9.5.3/share/man/man1/initdb.1
/usr/local/Cellar/postgresql/9.6.1/bin/initdb
/usr/local/Cellar/postgresql/9.6.1/share/doc/postgresql/html/app-initdb.html
/usr/local/Cellar/postgresql/9.6.1/share/man/man1/initdb.1
/usr/local/bin/initdb
/usr/local/share/man/man1/initdb.1
So in my case I want to run
/usr/local/Cellar/postgresql/9.6.1/bin/initdb
If you don't have mlocate installed, either install it or use
sudo find / -name initdb
There's a good answer to a similar question on SuperUser.
In short:
Postgres groups databases into "clusters", each of which is a named collection of databases sharing a configuration and data location, and running on a single server instance with its own TCP port.
If you only want a single instance of Postgres, the installation includes a cluster named "main", so you don't need to run initdb to create one.
If you do need multiple clusters, then the Postgres packages for Debian and Ubuntu provide a different command pg_createcluster to be used instead of initdb, with the latter not included in PATH so as to discourage end users from using it directly.
And if you're just trying to create a database, not a database cluster, use the createdb command instead.
I had the same problem and found the answer here.
Ubuntu path is
/usr/lib/postgresql/9.6/bin/initdb
Edit: Sorry, Ahmed asked about linuxbrew, I'm talking about Ubuntu.
I Hope this answer helps somebody.
I had a similar issue caused by the brew install postgresql not properly linking postgres. The solve for me was to run:
brew link --overwrite postgresql
you can add the PATH to run from any location
sudo nano ~/.profile
inside nano go to the end and add the following
# set PATH so it includes user's private bin if it exists
if [ -d "/usr/lib/postgresql/14/bin/" ] ; then
PATH="/usr/lib/postgresql/14/bin/:$PATH"
fi
and configure the alternative
sudo update-alternatives --install /usr/bin/initdb initdb /usr/lib/postgresql/14/bin/initdb 1
I have a postgreSQL 9.6 installation on Ubuntu 16.04, provided by the EnterpriseDB. I've been connecting to the local server and using it for a number of months. I figured I should backup the server at this point, so I tried using the pg_dump command as follows:
cd to postgres bin at /opt/PostgreSQL/9.6/bin
Run pg_dump via sudo pg_dumpall > /sata_ssd2/pgdump/0419.bkp
This gives an error: Error: You must install at least one postgresql-client-<version> package. I tried installing via: sudo apt-get install postgresql-client-9.6 but that package does not seem to exist.
I also tried to run pg_dump and pg_dumpall via pgAdmin4. It first made me fill in the binary paths for EDB Advanced Server Binary Path and PostgreSQL Binary Path; I put in opt/PostgreSQL/9.6/bin/, which is where the pg_dump executable is. When I tried to run the backups, I got the error: 'ascii' codec can't decode byte 0xe1 in position 1: ordinal not in range(128). I found an answer to this elsewhere that recommended changing from client_encoding = SQLascii to client_encoding = utf8 in postgresql.conf, but this did not help.
Does anyone have any advice on where to go from here?
In a unix/linux system the current directory . is not on the PATH and not searched for binaries.
If you want to run the command pg_basebackup in /opt/PostgreSQL/9.6/bin/, you run /opt/PostgreSQL/9.6/bin/pg_basebackup.
If you did want to run the command pg_basebackup that you knew was in the current directory you could write ./pg_basebackup instead. But just using the full path rather than cding unnecessarily is better.
I have a database running on SQL Server (13.01) on Windows. I like to deploy it to the Docker Container on Linux using SSDT.
I can perfectly connect to the server running on Docker and create/drop database manually and play with the data.
The problem is I can not publish it. I'm executing following script on Powershell
PS: SqlPackage.exe /Action:Publish /SourceFile:"d.dacpac" /TargetConnectionString:"server=containeraddress;database=thedatabase;user id=sa;password=thepassword;
and getting the following error.
Unable to connect to master or target server 'the database'. You must have a user with the same password in master or target server 'the database'. (Microsoft.Data.Tools.Schema.Sql)
I have the same user and same password on target and source servers.
Is there anybody has the same problem and know how to solve it?
I'll post this here as most of the answers refer to having an existing compiled dacpac file, which may not always be possible. I haven't seen similar ideas posted elsewhere to the solution I'm suggesting here.
Given your usage of docker and if you wish to compile your visual studio project inside the container, given certain combinations of the container base OS and image may not be possible to create a dacpac file with msbuild.
You can work around restoring the database using a series of unix based commands, taking note that the visual studio database project is usually just a series of SQL files, below I show an example of this, where I concat the SQL files into a single file and call sqlcmd to run the script;
FROM mcr.microsoft.com/mssql/server
WORKDIR /init
ENV ACCEPT_EULA=Y
ENV MSSQL_SA_PASSWORD=MyPassword
EXPOSE 1433:1433
RUN apt-get update && apt-get install dos2unix
COPY /solution_folder/database/Tables/*.sql /init/
WORKDIR /database
RUN echo "CREATE DATABASE [database_name];\nGO\nUSE [database_name];\n” >> /database/create.sql
RUN for f in /init/*.sql; do dos2unix $f; cat $f >> /database/create.sql; echo "\nGO\n" >> /database/create.sql; done
RUN ( /opt/mssql/bin/sqlservr --accept-eula & ) | grep -q "Service Broker manager has started" && /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P ‘MyPassword’ -i /database/create.sql && pkill sqlservr
The reason for "dos2unix" is that the SQL files created within visual studio have unique hidden cr/lf (and other characters) which the linux version of sqlcmd won't interpret successfully and will cause errors (which is kind of bizarre and this is exactly the kind of thing you'd want a cross platform database to be able to cope with)
Also, within the final run command you have to start-up the sql server service temporarily otherwise you'll also get errors; it's a little bit of work-around, and a bit fiddly and I'm not sure entirely that the microsoft sql server linux container is well designed enough for the simple task of restoring a database like this, the nuances are the differences between building and running a container and needing some sort of happy middle ground of both concepts for it to work.
Given here isn't a complete solution to restore, it only deals with Tables from the project file, although it should be trivial to expand to scalar functions and stored procedures.
Which version of SqlPackage.exe are you using? Only the most recent release candidate versions of SqlPackage.exe support SQL Server vNext CTP. The SqlPackage release candidate can be downloaded here: https://www.microsoft.com/en-us/download/details.aspx?id=54273