I create zeroed Arrays in Scala with
(0 until Nrows).map (_ => 0).toArray but is there anything faster ? map is slow.
I have the same question but with 1 instead of O, i.e. I also want to accelerate (0 until Nrows).map (_ => 1).toArray
Zero is the default value for an array of Ints, so just do this:
val array = new Array[Int](NRows)
If you want all those values to be 1s then use .fill() (with thanks to #gourlaysama):
val array = Array.fill(NRows)(1)
However, looking at how this works internally, it involves the creation of a few objects that you don't need. I suspect the following (uglier) approach may be quicker if speed is your main concern:
val array = new Array[Int](NRows)
for (i <- 0 until array.length) { array(i) = 1 }
For multidimensional arrays consider Array.ofDim, for instance,
scala> val a = Array.ofDim[Int](3,3)
a: Array[Array[Int]] = Array(Array(0, 0, 0), Array(0, 0, 0), Array(0, 0, 0))
Likewise,
scala> val a = Array.ofDim[Int](3)
a: Array[Int] = Array(0, 0, 0)
In the context here,
val a = Array.ofDim[Int](NRows)
For setting (possibly nonzero) initial values, consider Array.tabulate, for instance,
scala> Array.tabulate(3,3)( (x,y) => 1)
res5: Array[Array[Int]] = Array(Array(1, 1, 1), Array(1, 1, 1), Array(1, 1, 1))
scala> Array.tabulate(3)( x => 1)
res18: Array[Int] = Array(1, 1, 1)
Related
object Solution extends App {
val arr1 = Array(
Array(1,2,3),
Array(4,5,6)
)
var arr2 = Array.ofDim[Int](2,3)
Array.copy(arr1,0,arr2,0,arr1.length)
arr1(0)(1) = 23
println(arr1.map(_.mkString(",")).mkString("\n"))
println()
println(arr2.map(_.mkString(",")).mkString("\n"))
}
1,23,3
4,5,6
1,23,3
4,5,6
what is wrong, why is the 23 appearing in both arrays
Because Array in Scala, or if to be more precise in JVM, because of Scala interop with Java - is a mutable structure, and you performing shallow copy and not a deep copy. Meaning - you copying the upper structure (or top array in your case) and not entire structure recursively, like all downstream array.
Solution might look like:
val source = Array(
Array(1, 2, 3),
Array(4, 5, 6)
)
val target = Array.ofDim[Int](2, 3)
source.zipWithIndex.foreach { case (row, index) =>
Array.copy(source(index), 0, target(index), 0, source.length)
}
target(0)(1) = 23
println(source.map(_.mkString(",")).mkString("\n"))
println()
println(target.map(_.mkString(",")).mkString("\n"))
which will print out result:
1,2,3
4,5,6
1,23,0
4,5,0
Scatie example: https://scastie.scala-lang.org/lrrHyGqZRxKk7mZ6CbLoiA
UPDATE
As correctly stated #Luis Miguel Mejía Suárez in the comment section - zipWithIndex expensive operation. More optimal solution would be
(0 until source.length).foreach { index =>
Array.copy(source(index), 0, target(index), 0, source.length)
}
Array.copy uses System.arrayCopy which modifies both the arrays. In the doc:
Copy one array to another. Equivalent to Java's System.arraycopy(src, srcPos, dest, destPos, length), except that this also works for polymorphic and boxed arrays.
Note that the passed-in dest array will be modified by this call.
You can try a simple map with identity:
scala> val arr1 = Array(Array(1,2,3),Array(4,5,6))
arr1: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6))
scala> val arr3 = arr1.map(_.map(identity))
arr3: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6))
scala> arr1(0)(1) = 23
scala> arr1
res16: Array[Array[Int]] = Array(Array(1, 23, 3), Array(4, 5, 6))
scala> arr3
res17: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6))
Hi I've tried to insert element to rdd array[String] using scala in spark.
Here is example.
val data = RDD[Array[String]] = Array(Array(1,2,3), Array(1,2,3,4), Array(1,2)).
I want to make length 4 of all arrays in this data.
If the length of array is less than 4, I want to fill the NULL value in the array.
here is my code that I tried to solve.
val newData = data.map(x =>
if(x.length < 4){
for(i <- x.length until 4){
x.union("NULL")
}
}
else{
x
}
)
But The result is Array[Any] = Array((), Array(1, 2, 3, 4), ()).
So I tried another ways. I used yield on for loop.
val newData = data.map(x =>
if(x.length < 4){
for(i <- x.length until 4)yield{
x.union("NULL")
}
}
else{
x
}
)
The result is Array[Object] = Array(Vector(Array(1, 2, 3, N, U, L, L)), Array(1, 2, 3, 4), Vector(Array(1, 2, N, U, L, L), Array(1, 2, N, U, L, L)))
these are not what I want. I want to return like this
RDD[Array[String]] = Array(Array(1,2,3,NULL), Array(1,2,3,4), Array(1,2,NULL,NULL)).
What should I do?
Is there a method to solve it?
union is a functional operation, it doesn't change the array x. You don't need to do this with a loop, though, and any loop implementations will probably be slower -- it's much better to create one new collection with all the NULL values instead of mutating something every time you add a null. Here's a lambda function that should work for you:
def fillNull(x: Array[Int], desiredLength: Int): Array[String] = {
x.map(_.toString) ++ Array.fill(desiredLength - x.length)("NULL")
}
val newData = data.map(fillNull(_, 4))
I solved your use case with the following code:
val initialRDD = sparkContext.parallelize(Array(Array[AnyVal](1, 2, 3), Array[AnyVal](1, 2, 3, 4), Array[AnyVal](1, 2, 3)))
val transformedRDD = initialRDD.map(array =>
if (array.length < 4) {
val transformedArray = Array.fill[AnyVal](4)("NULL")
Array.copy(array, 0, transformedArray, 0, array.length)
transformedArray
} else {
array
}
)
val result = transformedRDD.collect()
If I have an array of array (similar to a matrix) in Scala, what's the efficient way to sum up each column of the matrix? For example, if my array of array is like below:
val arr = Array(Array(1, 100, ...), Array(2, 200, ...), Array(3, 300, ...))
and I want to sum up each column (e.g., sum up the first element of all sub-arrays, sum up the second element of all sub-arrays, etc.) and get a new array like below:
newArr = Array(6, 600, ...)
How can I do this efficiently in Spark Scala?
There is a suitable .transpose method on List that can help here, although I can't say what its efficiency is like:
arr.toList.transpose.map(_.sum)
(then call .toArray if you specifically need the result as an array).
Using breeze Vector:
scala> val arr = Array(Array(1, 100), Array(2, 200), Array(3, 300))
arr: Array[Array[Int]] = Array(Array(1, 100), Array(2, 200), Array(3, 300))
scala> arr.map(breeze.linalg.Vector(_)).reduce(_ + _)
res0: breeze.linalg.Vector[Int] = DenseVector(6, 600)
If your input is sparse you may consider using breeze.linalg.SparseVector.
In practice a linear algebra vector library as mentioned by #zero323 will often be the better choice.
If you can't use a vector library, I suggest writing a function col2sum that can sum two columns -- even if they are not the same length -- and then use Array.reduce to extend this operation to N columns. Using reduce is valid because we know that sums are not dependent on order of operations (i.e. 1+2+3 == 3+2+1 == 3+1+2 == 6) :
def col2sum(x:Array[Int],y:Array[Int]):Array[Int] = {
x.zipAll(y,0,0).map(pair=>pair._1+pair._2)
}
def colsum(a:Array[Array[Int]]):Array[Int] = {
a.reduce(col2sum)
}
val z = Array(Array(1, 2, 3, 4, 5), Array(2, 4, 6, 8, 10), Array(1, 9));
colsum(z)
--> Array[Int] = Array(4, 15, 9, 12, 15)
scala> val arr = Array(Array(1, 100), Array(2, 200), Array(3, 300 ))
arr: Array[Array[Int]] = Array(Array(1, 100), Array(2, 200), Array(3, 300))
scala> arr.flatten.zipWithIndex.groupBy(c => (c._2 + 1) % 2)
.map(a => a._1 -> a._2.foldLeft(0)((sum, i) => sum + i._1))
res40: scala.collection.immutable.Map[Int,Int] = Map(2 -> 600, 1 -> 6, 0 -> 15)
flatten array and zipWithIndex to get index and groupBy to map new array as column array, foldLeft to sum the column array.
Consider a 2 dimensional Array, for instance
scala> val a = Array.tabulate(2,3){_+_}
a: Array[Array[Int]] = Array(Array(0, 1, 2), Array(1, 2, 3))
How to define a function
def getCol(ith: Int, a: Array[Array[Int]]): Array[Int]
that delivers
val col2 = getCol(2, a)
col2: Array[Int] = Array(1,2)
A simple and inefficient approach includes
def getCol(ith: Int, a: Array[Int]): Array[Int] = {
val t = a.transpose
t(ith)
}
Thus to ask also for more efficient ways.
def getCol(n: Int, a: Array[Array[Int]]) = a.map{_(n - 1)}
Note that you have to use n - 1 for Nth element.
I have an array of array of integers. Like:
val t1 = Array(Array(1, 2, 3), Array(2), Array(4, 5, 6), Array(5, 6))
I want to remove the arrays that are subsets of another array. So, the result should be:
Array(Array(1, 2, 3), Array(4, 5, 6))
Ideally, these should be Sets, but in the context of my program, they are arrays, and I don't want to convert them to sets due to performance reasons.
I solved it this way in Scala, but I would like to know if there is a more elegant (and/or more efficient) way to do this:
def removeSubsets[T: ClassManifest](clusters: Array[Array[T]]) = {
val sortedClusters = clusters.sortBy(-1 * _.length)
sortedClusters.foldLeft(Array[Array[T]]()){ (acc, ele) =>
val isASubset = acc.exists(arr => (ele diff arr).length == 0)
if (isASubset) acc else acc :+ ele
}
}