Using Scala to Write a DF to a SQL Server Table - sql-server

Question

I'm trying to use the code below.
import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._
// Aquire a DataFrame collection (val collection)
val config = Config(Map(
"url" -> "mysqlserver.database.windows.net",
"databaseName" -> "MyDatabase",
"dbTable" -> "dbo.Clients"
"user" -> "username",
"password" -> "*********"
))
import org.apache.spark.sql.SaveMode
collection.write.mode(SaveMode.Append).sqlDB(config)
The script is from this link.
https://github.com/Azure/azure-sqldb-spark
I'm running this in a databricks environment. I'm getting these errors:
command-836397363127942:5: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.connect._
^
command-836397363127942:4: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.config.Config
^
command-836397363127942:7: error: not found: value Config
val bulkCopyConfig = Config(Map(
^
command-836397363127942:18: error: value sqlDB is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row]
df.write.mode(SaveMode.Append).sqlDB(bulkCopyConfig)
I'm guessing that some kind of library is not installed correctly. I Googled for an answer, but didn't find anything useful. Any idea how to make this work? Thanks.

Answer 1

If you are getting the sqldb error means all other support libraries already imported to your notebook and only the latest JAR with dependencies are missing.
As per the repro, I got the same error message as shown above:
After bit of research, I had found that you will experience this error due to missing JAR with dependencies.
To resolve this issue, you need to download the JAR file from here: https://search.maven.org/artifact/com.microsoft.azure/azure-sqldb-spark/1.0.2/jar
After downloaded the jar file, upload the JAR library into the cluster and install it.
Note: After installing both the libraries, make sure to restart the cluster.
Now, you will be able to run the command successfully.

Answer 2

I think you have missed the library
If your using Maven Build Add the following library in pom.xml
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-sqldb-spark</artifactId>
<version>1.0.2</version>
</dependency>
If your using SBT Build Add the following library in build.sbt
libraryDependencies += "com.microsoft.azure" % "azure-sqldb-spark" % "1.0.2"

Answer 3

Have you imported and installed the library in DataBricks?
I found it easiest to import the library using Maven. See this answer: How to install a library on a databricks cluster using some command in the notebook?
Note: You need to install the Library on your cluster and then restart the cluster before you can use it.