Apache-spark: Equal column data structure, different outcome in UDF function - arrays

Question

I have two array of columns
arrayColumns1: org.apache.spark.sql.Column = array("col1","col2")
arrayColumns2: org.apache.spark.sql.Column = array("col1","col2")
Both seems equals, but they came from different sources.
arrayColumns1 is from a conversion of Array("col1","col2") to an array, using this function:
def asLitArray[T](xs: Seq[T]) = array(xs map lit: _*)
arrayColumns2 is from writing the textual Array.
Now when I try to use arrayColumns1 as input to an UDF function:
.withColumn("udfFunction",udfFunction(arrayColumns))
where
val udfFunction= udf(
{ xs : Seq[Double] =>
DO_SOMETHING
(output)
}
)
It thorows me this error:
org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(array(col1,col2))' due to data type mismatch: argument 1 requires array<double> type, however, 'array('col1','col2')' is of array<string> type.;;
But when I use arrayColumns2 it works fine. What did I do wrong?
I'm using Spark 2.1 over scala 2.11

Answer 1

It does not make much sense to pass an array of literals to an UDF, because what you want to pass is the name of the columns, not literal values. Your second case fails because you are creating columns of type string (lit("col1") is a literal column with a content of "col1", it does not reference the column col1)
I would to it like this:
def asColArray(xs: Seq[String]) = array((xs.map(x => col(x))): _*)
val arrayColumns = asColArray(Array("col1","col2"))
df.withColumn("udfFunction",udfFunction(arrayColumns))
If you really want to use literal values, you would need to do something like this:
val arrayColumns = asLitArray(Array(1.0,2.0))
But this would give you a constant output of your udf