I am unable to establish connection to my Oracle database from Azure Databricks although it works in ADF where I am able to query the table. But ADF takes time to filter the records so I am still trying to connect from Databricks.
I followed the steps from this Microsoft link, both manually and using init-script but error seems to persist.
When I looked into my cluster event log it says the init-script execution was successfully.
Error message when I tried to establish the connection:
DPI-1047: Cannot locate a 64-bit Oracle Client library: "/databricks/driver/oracle_ctl//lib/libclntsh.so: cannot open shared object file: No such file or directory".
When I executed the following command
dbutils.fs.ls("/databricks/driver/")
there was no such directory
This triggered me to post some questions here:
Does this mean the init-script did not perform its job?
Is /databricks/driver/oracle_ctl a hidden directory for dbutils.fs.ls?
Error message points to /databricks/driver/oracle_ctl//lib/libclntsh.so, when I manually inspected the downloaded oracle client, there is no such folder called lib although libclntsh.so exists in the main directory. Is there a problem that databricks is checking the wrong directory for the libclntsh.so?
Does this connections still works for others?
Syntax for connection: cx_Oracle.connect(user= user_name, password= password,dsn= IP+':'+Port+'/'+DB_name)
Above syntax works fine when connected from inside a on-premises machine.
Try installing the latest major release of cx_Oracle - which got renamed to python-oracledb, see the release announcement.
This version doesn't need Oracle Instant Client. The API is the same as cx_Oracle, although obviously the name is different.
If I understand the instructions, your init script would do something like:
/databricks/python/bin/pip install oracledb
Application code would be like:
import oracledb
connection = oracledb.connect(user='scott', password=mypw, dsn='yourdbhostname/yourdbservicename')
with connection.cursor() as cursor:
for row in cursor.execute('select city from locations'):
print(row)
Resources:
Home page: oracle.github.io/python-oracledb/
Quick start: Quick Start python-oracledb Installation
Documentation: python-oracle.readthedocs.io/en/latest/index.html
PyPI: pypi.org/project/oracledb/
Source: github.com/oracle/python-oracledb
Upgrading: Upgrading from cx_Oracle 8.3 to python-oracledb
Changed the path from "/databricks/driver/oracle_ctl/" to "/databricks/driver/oracle_ctl/instantclient" in the init-script and that error does not appear anymore.
Please use the following init script instead
dbutils.fs.put("dbfs:/databricks/<init-script-folder-name>/oracle_ctl.sh","""
#!/bin/bash
sudo apt-get install libaio1
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
mv /databricks/driver/oracle_ctl/instantclient* /databricks/driver/oracle_ctl/instantclient
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/instantclient/"' >> /databricks/spark/conf/spark-env.sh
""", True)
Notes:
The above init-script was advised by a databricks employee and can be found here.
As mentioned by Christopher Jones in one of the comments, cx_Oracle has been recently upgraded to oracledb with a thin and thick version.
You will get the above error if you don’t have Oracle instant client in your Cluster.
To resolve above error in azure databricks, please follow this code:
%sh
mkdir -p /opt/oracle
cd /opt/oracle
wget https://download.oracle.com/otn_software/nt/instantclient/19600/instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
unzip instantclient-basic-windows.x64-19.6.0.0.0dbru.zip
set ORACLE_HOME=%ORABAS%\instantclient_19_3
set TNS_ADMIN=%ORACLE_HOME%
set PATH=%ORACLE_HOME%;%PATH%
To create init script, use the following code:
As per official doc,
dbutils.fs.put("dbfs:/databricks/<init-script-folder>/oracle_ctl.sh","""
#!/bin/bash
wget --quiet -O /tmp/instantclient-basiclite-linuxx64.zip https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip
unzip /tmp/instantclient-basiclite-linuxx64.zip -d /databricks/driver/oracle_ctl/
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
""", True)
To read data from oracle database in PySpark follow this article by Emrah Mete
For more information refer this official document:
https://docs.databricks.com/data/data-sources/oracle.html#oracle
The Visual Code remote SSH extension allows editing any remote file in the embedded terminal just using code <filename> which is super useful.
However when opening a file with sudo e.g. sudo code /etc/fstabs it returns
sudo: code: command not found
Is there any way to allow the use of code to any user?
I don't believe this is possible at the time of writing as it would require the underlying node which is running VSCode server to be running with sudo, which it is not.
I have opened a GitHub Issue here.
I'm setting up a new remote host and every time i initiate it i get the following error output: Any feedback or direction on how to resolve this issue?
Pseudo-terminal will not be allocated because stdin is not a terminal.
Linux Destiny 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
mesg: ttyname failed: Inappropriate ioctl for device
bash: cannot set terminal process group (3202): Inappropriate ioctl for device
bash: no job control in this shell
mesg: ttyname failed: Inappropriate ioctl for device
Installing...
Downloading with wget
WARNING: tar exited with non-0 exit code
Found running server...
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
*
cat: /root/.vscode-remote/.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.log: No such
file or directory
Server did not start successfully. Full server log:
cat: /root/.vscode-remote/.X.log51ec4692-
4da4-4ec0-b613-5a3563034cf1====
: No such file or directory
"install" terminal command done
Received install output: : No such file or directory
Failed to parse remote port from server output: : No such file or directory
If the server fails to shut down properly, sometimes it leaves dangling lockfiles. This can cause startup to fail and produce the "Failed to parse remote port from server output" error message. In this case the solution is to simply to delete the lockfiles:
.vscode-server/bin/[:xdigit:]*/vscode-remote-lock.*
Fixed the issue. It appears I had 2 other server agents running incorrectly. I killed both server agents using kill (PID) and removed ".vscode_remote" directory from user home directory. Then i reinitialized remote-ssh from vscode. Successfully connected!
On remote machine you do not have a tar installed. It's in log output
Installing... Downloading with wget
WARNING: tar exited with non-0 exit code
so under a root run:
apt-get install tar
or with sudo, if you have a user with sudoers configured:
sudo apt-get install tar
I also got the same issue and my workaround was to provide proper rights to the home or user folder, so vscode can create a remote folder and do the required installation on it.
Example :
sudo chmod -R 777 home/
In this case, I have provided all rights to my home folder and It worked like a charm for all the users.
I ssh'd onto the remote server (linux) and then deleted both directories as follows:
$ rm -r .vscode-server.backup2022-04-03T16:20:18-05:00
$ rm -r .vscode-server
In case someone else encounters the same issue - I had an instance where remote target had no space left on device. After extending root volume of target machine, connection worked fine.
I had the same issue because vscode was looking for my .vscode-server directory in the wrong location (it was in a custom location due to restrictions on where files can be saved). This can be fixed by using How to change vscode-server directory. Specifically add:
"remote.SSH.lockfilesInTmp": true,
"remote.SSH.serverInstallPath":{
"hostname":"/path/to/.vscode-server/.."
}
To your settings.json
In my case, it wasn't working because of server asking for new password when starting a session. What I did was to open a new default terminal (not VSC terminal but your OS default terminal like ZSH, CMD, and so on). And I used the ssh command to login. I logged in successfully and changed the password. Then I tried connecting with the new password and it worked because the server didn't asked for password change now.
Command:
ssh username#IP
Enter password and you'll get asked to change the password. Change the password and try connecting again with new password using SSH VSC extension.
If yout authorize by ssh-key - also check the value of User parameter in VsCode ssh config. User must have matching key in ~/.ssh/authorized_keys on remote host.
#Sachin's answer directed me in the right direction, VSCode needs permissions in order to create some files, but instead of giving 777 permissions to your home folder (which can be dangerous) you can just chown the user that wants to log in (the user for me was ubuntu):
sudo chown -R ubuntu /home
I also got the same issue and my workaround was to provide proper rights to the home or user folder, so vscode can create a remote folder and do the required installation on it.
Step 1: Add port to your config file :
Host hostname
Port 22
User username
Step 2 : Go to File->Prefrences ->Open settings.json fle
Search for lockfilesInTmp
and check the box next to that
bash: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.com': No error
https://github.com/atom/atom/issues/8984
Same problem but I worked around it by changing the git remote url to
include my git username and password.
List your remote URL using:
git remote -v Update the remote URL to include username and password
just before github.com:
git remote set-url
https://:#github.com//.git
Not sure why this was necessary, I thought I'd be prompted for my
credentials if git didn't know them.
More details, hopefully relevant:
OS Windows 10 Git version is 2.6.0 from http://git-scm.com/ Recently
done a Windows re-install
good video https://www.youtube.com/watch?v=SKXkC4SqtRk
I'm trying to install postgres on a sun solaris sparc instance in my home directory.
Everything is fine except when I try to start postgres server, I get the following error
FATAL: "/home/reic/var/lib/pgsql/data" is not a valid data directory
DETAIL: File "/home/reic/var/lib/pgsql/data/PG_VERSION" is missing.
I used the command pg_ctl -l logfile start to start the server.
I have followed all the necessary steps for installation on sun solaris.
Any idea as to why is this happening ?
Solution:
All the necessary files are there in the PGDATA directory and I ran it as the same user which is trying to start the server. The problem is with the placement of env variables in the .bashrc instead of .bash_profile file. Apparently the customisations are not propagated to sub shells when the env variables are put in .bashrc.
My bad for not realising this! Thank you all....