Aggregating arrays element wise - arrays

Question

Pretty new to spark/scala. I am wondering if there is an easy way to aggregate an Array[Double] in a column-wise fashion. Here is an example:
c1 c2 c3
-------------------------
1 1 [1.0, 1.0, 3.4]
1 2 [1.0, 0,0, 4.3]
2 1 [0.0, 0.0, 0.0]
2 3 [1.2, 1.1, 1.1]
Then, upon aggregation, I would end with a table that looks like:
c1 c3prime
-------------
1 [2.0, 1.0, 7.7]
2 [1.2, 1.1, 1.1]
Looking at UDAF now, but was wondering if I need to code at all?
Thanks for your consideration.

Answer 1

Assuming the array values of c3 are of the same size, you can sum the column element-wise by means of a UDF like below:
val df = Seq(
(1, 1, Seq(1.0, 1.0, 3.4)),
(1, 2, Seq(1.0, 0.0, 4.3)),
(2, 1, Seq(0.0, 0.0, 0.0)),
(2, 3, Seq(1.2, 1.1, 1.1))
).toDF("c1", "c2", "c3")
def elementSum = udf(
(a: Seq[Seq[Double]]) => {
val zeroSeq = Seq.fill[Double](a(0).size)(0.0)
a.foldLeft(zeroSeq)(
(a, x) => (a zip x).map{ case (u, v) => u + v }
)
}
)
val df2 = df.groupBy("c1").agg(
elementSum(collect_list("c3")).as("c3prime")
)
df2.show(truncate=false)
// +---+-----------------------------+
// |c1 |c3prime |
// +---+-----------------------------+
// |1 |[2.0, 1.0, 7.699999999999999]|
// |2 |[1.2, 1.1, 1.1] |
// +---+-----------------------------+

Answer 2

Here's one without UDF. It utilizes Spark's Window functions. Not sure how efficient it is, since it involves multiple groupBys
df.show
// +---+---+---------------+
// | c1| c2| c3|
// +---+---+---------------+
// | 1| 1|[1.0, 1.0, 3.4]|
// | 1| 2|[1.0, 0.0, 4.3]|
// | 2| 1|[0.0, 0.0, 0.0]|
// | 2| 2|[1.2, 1.1, 1.1]|
// +---+---+---------------+
import org.apache.spark.sql.expressions.Window
val window = Window.partitionBy($"c1", $"c2").orderBy($"c1", $"c2")
df.withColumn("c3", explode($"c3") )
.withColumn("rn", row_number() over window)
.groupBy($"c1", $"rn").agg(sum($"c3").as("c3") )
.orderBy($"c1", $"rn")
.groupBy($"c1")
.agg(collect_list($"c3").as("c3prime") ).show
// +---+--------------------+
// | c1| c3prime|
// +---+--------------------+
// | 1|[2.0, 1.0, 7.6999...|
// | 2| [1.2, 1.1, 1.1]|
// +---+--------------------+

Answer 3

You can combine some inbuilt functions such as groupBy, agg, sum, array, alias(as) etc. to get the desired final dataframe.
import org.apache.spark.sql.functions._
df.groupBy("c1")
.agg(sum($"c3"(0)).as("c3_1"), sum($"c3"(1)).as("c3_2"), sum($"c3"(2)).as("c3_3"))
.select($"c1", array("c3_1","c3_2","c3_3").as("c3prime"))
I hope the answer is helpful.