Array of Arrays as Iterable of Iterables - arrays

Let's consider the following definition to add up all elements in a nested Iterable structure of Ints:
def add(xss : Iterable[Iterable[Int]]) : Int = xss.map(_.sum).sum
However, evaluating following expression yields a type error:
scala> add(Array(Array(1,2,3)))
<console>:9: error: type mismatch;
found : Array[Array[Int]]
required: Iterable[Iterable[Int]]
add(Array(Array(1,2,3)))
^
The function works as expected with other Iterables (like Lists). How can I avoid that error? What's the rationale for it? Guess that is something to do with Arrays being native from Java, but don't know the specifics details in this case.
Thanks

It doesn't work because Scala would need to use 2 implicit conversion in a row to go from Array[Array[Int]] to Iterable[Iterable[Int]], which it (purposefully) doesn't do.
You could specify the outer Array's type :
scala> add(Array[Iterable[Int]](Array(1,2,3)))
res4: Int = 6
or transform its elements to Iterable[Int] (thus bypassing an implicit conversion) :
scala> add(Array(Array(1,2,3)).map(_.toIterable))
res5: Int = 6
The problem comes from the fact that Scala's Array[T] is just a representation for Java's T[]. What makes Array[T] behave like an usual Scala collection is an implicit conversion in Predef.
From Array's documentation :
Two implicit conversions exist in scala.Predef that are frequently
applied to arrays: a conversion to mutable.ArrayOps
and a conversion to mutable.WrappedArray (a subtype of scala.collection.Seq). Both types make available many of the standard
operations found in the Scala collections API. The conversion to
ArrayOps is temporary, as all operations defined on ArrayOps return an
Array, while the conversion to WrappedArray is permanent as all
operations return a WrappedArray.
The conversion to ArrayOps takes priority over the conversion to
WrappedArray.

Your intuition is correct. See the Array type signature:
final class Array[T] extends java.io.Serializable with java.lang.Cloneable
contrast that with Seq type signature:
trait Seq[+A] extends Iterable[A] with collection.Seq[A] ...
As you can see Array does not relate to the Iterable[A] trait.
You can fix this by calling toIterable on the instance:
scala> add(Array(Array(1,2,3).toIterable).toIterable)
res1: Int = 6

Related

How to define an extention method for a non-generic array?

I cannot figure out what is the correct syntax to define an extension method on a non-generic array. This question deals with generic arrays, but trying a similar construct for a byte array doesn't work.
I have tried a lot of options:
type Byte ``[]`` with
type byte ``[]`` with
type Byte array with
type byte array with
type array<Byte> with
type array<byte> with
type []<Byte> with
type []<byte> with
as well as all of these constructs wrapped in double backticks or parenthesis, but nothing works. I downloaded the language specification, but it only has a generic array example.
This is likely a quirk with Optional Type Extensions, which can get pretty funky when generics are involved. I'd use an extension method like this instead:
open System.Runtime.CompilerServices
[<Extension>]
type ByteArrayExtensions =
[<Extension>]
static member inline Sum(xs: byte[]) = Array.sum xs
let f() =
let xs = [| byte(1); byte(2); byte(3) |]
xs.Sum()

Does ccall really convert arguments passed by pointer?

Considering a dynamic library with this native function that returns the sum of all even (32-bit unsigned) numbers in an array:
uint32_t sum_of_even(const uint32_t *numbers, size_t length);
The implementation of the function above was written in Rust as below, and packaged into a C dynamic library.
use libc::size_t;
use std::slice;
#[no_mangle]
pub extern "C" fn sum_of_even(n: *const u32, len: size_t) -> u32 {
let numbers = unsafe {
assert!(!n.is_null());
slice::from_raw_parts(n, len as usize)
};
numbers
.iter()
.filter(|&v| v % 2 == 0)
.sum()
}
I wrote the following Julia (v1.0.1) wrapper function:
lib = Libdl.dlopen(libname)
sumofeven_sym = Libdl.dlsym(lib, :sum_of_even)
sumofeven(a) = ccall(
sumofeven_sym,
UInt32,
(Ptr{UInt32}, Csize_t),
a, length(a)
)
The documentation states multiple times that arguments in ccall are converted to become compatible with the C function prototype (emphasis mine):
Each argvalue to the ccall will be converted to the corresponding argtype, by automatic insertion of calls to unsafe_convert(argtype, cconvert(argtype, argvalue)). (See also the documentation for unsafe_convert and cconvert for further details.) In most cases, this simply results in a call to convert(argtype, argvalue).
And moreover, that when passing an Array{T} by Ptr{U} to a C function, the call is invalidated if the two types T and U are different, since no reinterpret cast is added (section Bits Types):
When an array is passed to C as a Ptr{T} argument, it is not reinterpret-cast: Julia requires that the element type of the array matches T, and the address of the first element is passed.
Therefore, if an Array contains data in the wrong format, it will have to be explicitly converted using a call such as trunc(Int32, a).
However, this is seemingly not the case. If I deliberately pass an array with another type element:
println(sumofeven(Float32[1, 2, 3, 4, 5, 6]))
The program calls the C function with the array passed directly, without converting the values nor complaining about the different element types, resulting in either senseless output or a segmentation fault.
If I redefine the function to accept a Ref{UInt32} instead of a Ptr{UInt32}, I am prevented from calling it with the array of floats:
ERROR: LoadError: MethodError: Cannot `convert` an object of type Array{Float32,1} to an object of type UInt32
Closest candidates are:
convert(::Type{T<:Number}, !Matched::T<:Number) where T<:Number at number.jl:6
convert(::Type{T<:Number}, !Matched::Number) where T<:Number at number.jl:7
convert(::Type{T<:Integer}, !Matched::Ptr) where T<:Integer at pointer.jl:23
...
However, Ref was not designed for arrays.
Making the example work with Ptr{UInt32} requires me to either specify Array{UInt32} as the type of input a (static enforcement), or convert the array first for a more flexible function.
sumofeven(a:: Array{UInt32}) = ccall( # ← either this
sumofeven_sym,
UInt32,
(Ptr{UInt32}, Csize_t),
convert(Array{UInt32}, a), # ← or this
length(a))
With that, I still feel that there is a gap in my reasoning. What is the documentation really suggesting when it says that an array passed to C as a Ptr{T} is not reinterpret-cast? Why is Julia letting me pass an array of different element types without any explicit conversion?
This turned out to be either a bug in the core library or a very misguided documentation, depending on the perspective (issue #29850). The behavior of the function unsafe_convert changed from version 0.4 to 0.5, in a way that makes it more flexible than what is currently suggested.
According to this commit, unsafe_convert changed from this:
unsafe_convert(::Type{Ptr{Void}}, a::Array) = ccall(:jl_array_ptr, Ptr{Void}, (Any,), a)
To this:
unsafe_convert{S,T}(::Type{Ptr{S}}, a::AbstractArray{T}) = convert(Ptr{S}, unsafe_convert(Ptr{T}, a))
For arrays, this relaxed implementation will enable a transformation from an array of T to a pointer of another type S. In practice, unsafe_convert(cconvert(array)) will reinterpret-cast the array's base pointer, as in the C++ nomenclature. We are left with a dangerously reinterpreted array across the FFI boundary.
The key takeaway is that one needs to take extra care when passing arrays to C functions, as the element type of an array in a C-call function parameter is not statically enforced. Use type signatures and/or explicit conversions where applicable.

Scala generic arrays instantiating

I have a function that makes Arrays of specific type:
def mkArray[A:ClassTag:Ordering](size:Int):Array[A] = Array.ofDim[A](size)
And I want to make array arr of type Int or String depending on String str like so:
var arr = if(str=="i"){mkArray[Int](size)}else{mkArray[String](size)}
and now I try to add values to the array like so:
arr(n) = num.toInt // num is String like "123"
But it says:
- type mismatch; found : Int required: _366 where type _366 >: Int
with String
How can I get around this, making arr of type Array[Int] or Array[String] depending on string str?
Any help is appreciated,
Thanks!
Scala is a statically typed language, and in your case the type of arr is Array[_ >: Int with String]. Hence, if you give it an Int, you get a type error.
Depending on how you're using the array further in your code, I'd recommend you take a look at Either[1], since it may be helpful in keeping an Array[Either], and processing it using pattern matching differently for when it contains an Int vs String.
[1] http://danielwestheide.com/blog/2013/01/02/the-neophytes-guide-to-scala-part-7-the-either-type.html

implicit conversion of an array of arrays

I have a case class called Point defined as follows:
case class Point(x: Double, y: Double)
and a functions that takes an array of points:
def f(coords: Array[Point]) ...
I want to be able to implictly pass an array of double arrays to my function. To achieve this I have defined the following two implicit functions:
implicit def arrayToPoint(a: Array[Double]) = new Point(a(0), a(1))
implicit def arraysToPoints(a: Array[Array[Double]]) = a map(p => Point(p(0), p(1)))
My questions is is there any way I can achieve this with just a single implict conversion function to simplify matters?
And as a related question what would be the best approach if I wanted to be able to pass an Array of Ints instead of Doubles?
Regards
Des
Your method arraysToPoints is superfluous. You could use a view bound on the array argument for your method f and add the conversion to the companion object of Point, like so:
object Point {
implicit def arrayToPoint[A](a: Array[A])(implicit view: A => Double): Point =
Point(a(0), a(1))
}
case class Point(x: Double, y: Double)
def f[P](coords: Array[P])(implicit view: P => Point): Unit = coords.foreach { p =>
println(p: Point)
}
f(Array(Point(1, 2), Point(2, 3)))
f(Array(Array(1.0, 2.0), Array(3.0, 4.0)))
f(Array(Array(1, 2), Array(3, 4)))
In order to allow both arrays of Int and Double to be covered, I have used a second view bound on the arrayToPoint method. Otherwise you would need two separate conversion methods for Array[Int] and Array[Double].
You can read this definition of f as, "take an array of elements of a type P which can be viewed as type Point". One spot where the compiler looks for such views is the companion object of the target type, thus object Point. This is a good place for implicit methods.
The second possibility would be to use the magnet pattern. Instead of converting point by point with a view in f, you would create a single wrapper object at once. This is a bit prettier and for large arrays minimises the penalty on direct Array[Double] arguments (because you instantiate the wrapper once, but then do not need to call the view function any more). arrayToPoint is used whenever the array element type A once again can be viewed as a Double. This is true for Double itself of course, but also for Int which can be seen as a Double though what Scala calls numeric widening (e.g., you can say val x: Double = 33 and the integer 33 is implicitly "widened" to a double).
object Points {
implicit def direct(a: Array[Point]): Points =
new Points {
val peer = a
}
implicit def indirect[A](a: Array[Array[A]])(implicit view: A => Double): Points =
new Points {
lazy val peer = a.map { c => Point(c(0), c(1)) }
}
}
trait Points {
def peer: Array[Point]
}
def f(coords: Points): Unit = coords.peer.foreach(println)
This looks in the companion object for an implicit method from the argument type to the special magnet type Points. I use lazy val for the non-direct arrays so that we might save the actual conversion action if the peer method is not called.

Why are Arrays invariant, but Lists covariant?

E.g. why does
val list:List[Any] = List[Int](1,2,3)
work, but
val arr:Array[Any] = Array[Int](1,2,3)
fails (because arrays are invariant). What is the desired effect behind this design decision?
Because it would break type-safety otherwise.
If not, you would be able to do something like this:
val arr:Array[Int] = Array[Int](1,2,3)
val arr2:Array[Any] = arr
arr2(0) = 2.54
and the compiler can't catch it.
On the other hand, lists are immutable, so you can't add something that is not Int
This is because lists are immutable and arrays are mutable.
The normal answer to give is that mutability combined with covariance would break type safety. For collections, this can be taken as a fundamental truth. But the theory actually applies to any generic type, not just collections like List and Array, and we don't have to try and reason about mutability at all.
The real answer has to do with the way function types interact with subtyping. The short story is if a type parameter is used as a return type, it is covariant. On the other hand, if a type parameter is used as an argument type, it is contravariant. If it is used both as a return type and as an argument type, it is invariant.
Let's look at the documentation for Array[T]. The two obvious methods to look at are for the ones for lookup and update:
def apply(i: Int): T
def update(i: Int, x: T): Unit
In the first method T is a return type, while in the second T is an argument type. The rules of variance dictate that T must therefore be invariant.
We can compare the documentation for List[A] to see why it is covariant. Confusingly, we would find these methods, which are analogous to the methods for Array[T]:
def apply(n: Int): A
def ::(x: A): List[A]
Since A is used as both a return type and as an argument type, we would expect A to be invariant just like T is for Array[T]. However, unlike with Array[T], the documentation is lying to us about the type of ::. The lie is good enough for most calls to this method, but isn't good enough to decide the variance of A. If we expand the documentation for this method and click on "Full Signature", we are shown the truth:
def ::[B >: A](x: B): List[B]
So A does not actually appear as an argument type. Instead, B (which can be any supertype of A) is the argument type. This does not place any restriction on A, so it really can be covariant. Any method on List[A] which has A as an argument type is a similar lie (we can tell because these methods are marked as [use case]).
The difference is that Lists are immutable while Arrays are mutable.
To understand why mutability determines variance, consider making a mutable version of List - let's call it MutableList. We'll also make use of some example types: a base class Animal and 2 subclasses named Cat and Dog.
trait Animal {
def makeSound: String
}
class Cat extends Animal {
def makeSound = "meow"
def jump = // ...
}
class Dog extends Animal {
def makeSound = "bark"
}
Notice that Cat has one more method (jump) than Dog.
Then, define a function that accepts a mutable list of animals and modifies the list:
def mindlessFunc(xs: MutableList[Animal]) = {
xs += new Dog()
}
Now, horrible things will happen if you pass a list of cats into the function:
val cats = MutableList[Cat](cat1, cat2)
val horror = mindlessFunc(cats)
If we were using a careless programming language, this will be ignored during compilation. Nevertheless, our world will not collapse if we only access the list of cats using the following code:
cats.foreach(c => c.makeSound)
But if we do this:
cats.foreach(c => c.jump)
A runtime error will occur. With Scala, writing such code is prevented, because the compiler will complain.

Resources