SparkSQL MS SQL Server , Get message "No suitable driver" After compiled - sql-server

Question

build.sbt
name := "BigData"
version := "0.1"
scalaVersion := "2.12.7"
libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.3.5"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
// https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
libraryDependencies += "com.microsoft.sqlserver" % "mssql-jdbc" % "6.1.0.jre8" % Test
SparkMSSQL.scala
import org.apache.spark.sql.SparkSession
object SparkMSSQL {
def main(args: Array[String]): Unit = {
val spark = SparkSession.
builder.master("local[*]")
.appName("Simple Application")
.getOrCreate()
val url = "jdbc:sqlserver://localhost;databaseName=scalatest;integratedSecurity=true";
// Define database table to load into DataFrame
val jdbcDbTable = "dbo.user_profiles"
val df = spark
.read
.format("jdbc")
.option("url", url)
.option("dbtable", "dbo.user_profiles")
.load()
df.printSchema()
}
}
Error after compile
Exception in thread "main" java.sql.SQLException: No suitable driver
at java.sql.DriverManager.getDriver(DriverManager.java:315)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:105)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:105)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:35)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
Please advice , what is wrong with my code?

Answer 1

First, you have your jdbc driver in the test scope, so the jar is probably not loaded at runtime. But also, spark needs driver class information to create JDBC connection, so try adding the following option to the DF initializer:
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")