how to read file and split in scala - arrays

Question

If I had a file(like csv, txt...).
I wish get two array such as
Array(Array(1.0,2.0),Array(4.0,5.0),Array(7.0, 8.0),Array(10.0,11.0),Array(13.0,14.0))
and
Array(3.0, 6.0, 9.0, 12.0, 15.0)
What's the ideal way to do this in scala?

Answer 1

val rdd = sc.textFile("1.csv").map(_.split(',').map(_.trim().toDouble))
rdd.map(_.take(2)).collect()
res0: Array[Array[Double]] = Array(Array(1.0, 2.0), Array(4.0, 5.0), Array(7.0, 8.0), Array(10.0, 11.0), Array(13.0, 14.0))
rdd.map(_(2)).collect()
res2: Array[Double] = Array(3.0, 6.0, 9.0, 12.0, 15.0)

Answer 2

You can get both arrays in one go, so that you don't need to traverse the data twice:
val (first, second) = {
io.Source.fromFile(name).getLines
.map(_.split(",").map(_.toDouble))
.foldRight(Seq.empty[Array[Double]] -> Seq.empty[Double]) {
case (Array(x, y, z), (as, bs)) => (Array(x, y) +: as, z +: bs)
}
}
Now, you end up with two lists rather that arrays. Of that matters to you, first.toArray and second.toArray will do the conversion for you.

Answer 3

Similar to #Vitaliy Kotlyarenko's answer, but without using 3rd parties like Spark (Spark is great if your data is large, but an overkill otherwise):
val lines: Iterator[String] = scala.io.Source.fromFile("txt.csv").getLines()
val matrix: Array[Array[Double]] = lines.map(_.split(",").map(_.trim.toDouble)).toArray
val twoFirstColumns: Array[Array[Double]] = matrix.map(_.take(2))
val thirdColumn: Array[Double] = matrix.map(_(2))