I want to connect Azure MS SQL Database with Azure Databricks via python spark. I could do this with pushdown_query if I run Select * from.... But I need to run ALTER DATABASE to scale up/down.
I must change this part
spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties)
otherwise I get this error Incorrect syntax near the keyword 'ALTER'.
Anyone can help me. Much appreciated.
jdbcHostname = "xxx.database.windows.net"
jdbcDatabase = "abc"
jdbcPort = 1433
jdbcUrl = "jdbc:sqlserver://{0}:{1};database={2}".format(jdbcHostname, jdbcPort, jdbcDatabase)
connectionProperties = {
"user" : "..............",
"password" : "............",
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}
pushdown_query = "(ALTER DATABASE [DBNAME] MODIFY (SERVICE_OBJECTIVE = 'S0')) dual_down"
df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, properties=connectionProperties)
display(df)
Related
I have been able to read/write from Databricks into SQL Server table using JDBC driver. However this time I have to execute a command before I write to a SQL Server.
I need to execute this command on SQL server: SET IDENTITY_INSERT <sqlserver_table_name> ON
How to do this from Databricks ? Any help/pointers are appreciated. Thanks.
You can't do this with the JDBC Spark Connector (or the SQL Server Spark Connector), but it's trivial when using JDBC directly in Scala or Java. When using JDBC directly you have explicit control of the session, and you can issue multiple batches in the same session, or multiple statements in the same batch. EG
%scala
import java.util.Properties
import java.sql.DriverManager
val jdbcUsername = dbutils.secrets.get(scope = "kv", key = "sqluser")
val jdbcPassword = dbutils.secrets.get(scope = "kv", key = "sqlpassword")
val driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
// Create the JDBC URL without passing in the user and password parameters.
val jdbcUrl = s"jdbc:sqlserver://xxxxxx.database.windows.net:1433; . . ."
val connection = DriverManager.getConnection(jdbcUrl, jdbcUsername, jdbcPassword)
val stmt = connection.createStatement()
val sql = """
SET IDENTITY_INSERT <sqlserver_table_name> ON
"""
stmt.execute(sql)
//run additional batches here with IDENTITY_INSERT ON
connection.close()
And you can always use the Spark Connector to load a staging table, then use JDBC to run a stored procedure or ad-hoc SQL batch to load the staging data into the target table.
Am trying to restore the database from python 3.7 in Windows using below script.
Drop database functions correctly as expected.
The restore database doesn't work as expected, database always shows "Restoring...." and never completes.
Database files are there in the specified path, but database is not usable.
How to fix this?
import pyodbc
try:
pyconn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=MY-LAPTOP\\SQLEXPRESS;DATABASE=master;UID=sa;PWD=sa123')
cursor = pyconn.cursor()
pyconn.autocommit = True
sql = "IF EXISTS (SELECT 0 FROM sys.databases WHERE name = 'data_test') BEGIN DROP DATABASE data_test END"
pyconn.cursor().execute(sql)
sql = """RESTORE DATABASE data_test FROM DISK='G:\\dbbak\\feb-20-2020\\data_test_backup_2020_02_20_210010_3644975.bak' WITH RECOVERY,
MOVE N'Omnibus_Data' TO N'd:\\db\\data_test.mdf',
MOVE N'Omnibus_Log' TO N'd:\\db\\data_test_1.ldf';"""
print(sql)
pyconn.cursor().execute(sql)
while pyconn.cursor().nextset():
pass
pyconn.cursor().close()
except Exception as e:
print(str(e))
You're not using a single cursor, and so your program is exiting before the restore is complete, aborting it in the middle.
Should be something like:
conn = pyodbc.connect(' . . .')
conn.autocommit = True
cursor = conn.cursor()
cursor.execute(sql)
while cursor.nextset():
pass
cursor.close()
0
After hours I found solution. It must be performed no MASTER, other sessions must be terminated, DB must be set to OFFLINE, then RESTORE and then set to ONLINE again.
def backup_and_restore():
server = 'localhost,1433'
database = 'myDB'
username = 'SA'
password = 'password'
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE=MASTER;UID='+username+';PWD='+ password)
cnxn.autocommit = True
def execute(cmd):
cursor = cnxn.cursor()
cursor.execute(cmd)
while cursor.nextset():
pass
cursor.close()
execute("BACKUP DATABASE [myDB] TO DISK = N'/usr/src/app/myDB.bak'")
execute("ALTER DATABASE [myDB] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;")
execute("ALTER DATABASE [myDB] SET OFFLINE;")
execute("RESTORE DATABASE [myDB] FROM DISK = N'/usr/src/app/myDB.bak' WITH REPLACE")
execute("ALTER DATABASE [myDB] SET ONLINE;")
execute("ALTER DATABASE [myDB] SET MULTI_USER;")
I am trying to connect to local MS SQL Express Edition. I am using canopy for Python editing.
Code:
import pymssql
conn = pymssql.connect(server='******\SQLEXPRESS',user = 'MEA\*****',password='*****',database='BSEG')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Table')
print(cursor.fetchone())
conn.close()
Error::
pymssql.pyx in pymssql.connect (pymssql.c:10734)()
_mssql.pyx in _mssql.connect (_mssql.c:21821)()
_mssql.pyx in _mssql.MSSQLConnection.init (_mssql.c:5917)()
ValueError: too many values to unpack (expected 2)
user = 'MEA\*****',password='*****'
MEA\***** seems to be Windows login, in this case you shouldn't pass in any password, your user name is enough, but you also should use Integrated security or Trusted parameter in your connection string
It should be smth like this:
server='******\SQLEXPRESS',Trusted_Connection=yes,database='BSEG'
I am trying to connect to a redshift server and run some sql commands. Here is the code that I have written:
Class.forName("org.postgresql.Driver")
val url: String = s"jdbc:postgres://${user}:${password}#${host}:${port}/${database}"
val connection: Connection = DriverManager.getConnection(url, user, password)
val statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
val setSearchPathQuery: String = s"set search_path to '${schema}';"
statement.execute(setSearchPathQuery)
But I am getting the following error:
java.sql.SQLException: No suitable driver found for jdbc:postgres://user:password#host:port/database
But when I am using play framework's default database library with the same configuration, then I am able to connect to database successfully. Below is the configuration for the default database:
db.default.driver=org.postgresql.Driver
db.default.url="postgres://username:password#hostname:port/database"
db.default.host="hostname"
db.default.port="port"
db.default.dbname = "database"
db.default.user = "username"
db.default.password = "password"
The problem was with the url. The correct format for the url is:
jdbc:postgresql://hostname:port/database
I'm trying to connect to Microsoft SQL Server using Google Apps Script. I'm using SQL Server 2008 R2 and I am using one of the suggested scripts that's supposed to read data and put it into a Spreadsheet:
https://developers.google.com/apps-script/jdbc#reading_from_a_database
The error message is:
Failed to establish a database connection. Check connection string, username, and password
Username and password are OK, the user is DBowner. The port is also correct, I tried to connect to the server via Telnet using:
o IP-address 1433
and it works.
Here's the code:
function foo() {
var conn = Jdbc.getConnection("jdbc:sqlserver://IP-adress:1433/DBName","user","password");
var stmt = conn.createStatement();
stmt.setMaxRows(100);
var start = new Date();
var rs = stmt.executeQuery("select * from person");
var doc = SpreadsheetApp.getActiveSpreadsheet();
var cell = doc.getRange('a1');
var row = 0;
while (rs.next()) {
for (var col = 0; col < rs.getMetaData().getColumnCount(); col++) {
cell.offset(row, col).setValue(rs.getString(col + 1));
}
row++;
}
rs.close();
stmt.close();
conn.close();
var end = new Date();
Logger.log("time took: " + (end.getTime() - start.getTime()));
}
Do you have any idea of what can be wrong? Do I have to make some configuration on my Server? Or in the database? The instructions mentioned above says to ensure that Google's Ip-addresses can reach the database. But instead of listing all of Google's IP-addresses I granted access to all on that port. I also enabled TCP/IP Protocol in SQL Server Configuration Manager. And I granted "Remote connections" to the Server in MSSMS. Any other idea, please?
Well, I found the answer here:
Google Apps Scripts/JDBC/MySQL
Obviously, the connection string has to look like this:
var conn = Jdbc.getConnection("jdbc:sqlserver://IP-address:1433;" + "databaseName=DBName;user=username;password=password;");
I don't understand why connection string differs from the one in Google Documentation but this one works for me...
The Issue is in the Connection string.
It should be like so
address = '%YOUR SQL HOSTNAME%';
user = '%YOUR USE%';
userPwd = '%YOUR PW%';
dbUrl = 'jdbc:sqlserver://' + address + ':1433;databaseName=' + queryDb;
var conn = Jdbc.getConnection(dbUrl, user, userPwd);
I have an entire tool on GitHub where you are able to connect to MySQL and SQL Servers. I you will be able to use it to help you. I will be constantly updating with more features overtime! You can find it here.
GOOGLE SPREADSHEET JDBC CONNECTOR TOOL