Related
I am desperately trying to split strings within an xarray.Dataarray.
What should happen to every element of the array is e.g.
"aaabbbccc" --> [aaa, bbb, ccc]
Fortunately, such a function already exists in the textwrap library, but applying it to my Dataarray is a different story:
xds = riox.open_rasterio(fp_output_tmp_mlsieved, chunks = "auto")
<xarray.DataArray (band: 1, y: 2, x: 2)>
dask.array<transpose, shape=(1, 2, 2), dtype=<U18, chunksize=(1, 2, 2), chunktype=numpy.ndarray>
Coordinates:
* band (band) int64 1
* x (x) float64 3.077e+06 3.077e+06 ... 3.077e+06 3.077e+06
* y (y) float64 1.865e+06 1.865e+06 ... 1.865e+06 1.865e+06
spatial_ref int64 0
Loaded it looks like this:
array([[['000000000000000000', '000000000000000000'],
['000000000000000000', '000000000000000000']]], dtype='<U18')
I think a solution is to apply it with xr.apply_ufunc(). I have managed to do that with a simpler numpy function before, but with wrap() all I get is a bunch of errors. I think the main issue is that it is not a vectorized numpy function and second that I can´t get the dimensions to work out. My latest try looks like that:
def decompressor(s, l):
return np.array(wrap(s.item(), l))
def ufunc_decompressor(s, l):
return xr.apply_ufunc(
decompressor,
s, l,
output_dtypes=[np.dtype(f"U{l}")],
input_core_dims=[["band"],[]],
output_core_dims=[["band"]],
exclude_dims=set(("band",)),
dask="parallelized",
vectorize=True
)
xds_split = ufunc_decompressor(xds, 3).load()
What I get is a cryptic error:
File "/home/.../miniconda3/envs/postproc/lib/python3.10/site-packages/dask/array/gufunc.py", line 489, in <genexpr>
core_output_shape = tuple(core_shapes[d] for d in ocd)
KeyError: 'dim0'
I have a simple class I always implement when working with a new Language, MergeSort. So I am looking at my implementations of it with type Int and it looks great. Then I wanted to genericize it. I started with a simple implementation of T, but i noticed that needed to relfect the ClassTag. How do i assign the reflected ClassTag + extending?
class MergeSort[T: scala.reflect.ClassTag] {
var array: Array[T] = Array[T]()
var length: Int = 0
var tempArray: Array[T] = new Array(length)
def sort(data: Array[T]): Unit = {
array = data;
length = data.length;
tempArray = new Array[T](length)
//sort(0, length - 1)
}
}
Now this looks nice! It works, but when I i do the sort and rest of the functionality, I need to be able to compare 2 items of type T. The "Java" way was to just make sure the Object has the compareTo method. So i was thinking: [T extends Comparable]
but in scala, I am doing assignment for T with ClassTag, and
class MergeSort[T: scala.reflect.ClassTag extends Comparable] {} for example. It will error saying:
']' expected, but 'extends' found.
I was thinking this would sorta be the way to do things, but i am not sure whats going on here.
The endstate is to implement the merge portion of the class:
def merge(lower: Int, center: Int, upper: Int){
// ...
// loop
// if (tempArr(i) <= tempArr(j)) {} // OLD WAY, since First attempt was with Int.
// if (tempArr(i).compareTo(tempArr(j)) < 0) {} // Modified way with Comparable
}
Is this the scala way of implementing? I was noticing that people were mentioning Ordering, but i thought Comparable made sense.
The Scala way of implementing merge sort is using List and vals and the Ordering trait. The advantage of Ordering (the Java Comparator) is that Scala gives you implicit orderings for all standard library types by default.
def msort[T: Ordering](xs: List[T]): List[T] = {
#tailrec
def merge(xs: List[T], ys: List[T], acc: List[T] = Nil): List[T] =
(xs, ys) match {
case (Nil, _) => acc.reverse ++ ys
case (_, Nil) => acc.reverse ++ xs
case (x :: xs1, y :: ys1) =>
if (implicitly[Ordering[T]].lt(x, y))
merge(xs1, ys, x :: acc)
else
merge(xs, ys1, y :: acc)
}
xs match {
case Nil | _ :: Nil => xs
case _ =>
val (xs1, xs2) = xs splitAt (xs.length / 2)
merge(msort(xs1), msort(xs2))
}
}
msort(List(4, 23, 1, 2, 5, 76, 3, 142, 4321, 213, 42323))
// List(1, 2, 3, 4, 5, 23, 76, 142, 213, 4321, 42323)
msort(List("John", "Chris", "Helen", "Danny", "Michelle"))
// List(Chris, Danny, Helen, John, Michelle)
Another advantage over Ordered is that Scala provides implicit conversions from Ordered[A] => Ordering[A], which means your custom types that mix in Ordered will work with msort without the need to define implicit orderings.
Finally, the last advantage over Ordered is when using numeric types: Int, Double, etc. do not mix in Ordered, so you will not be able to sort elements of these types with Ordered, this is why most use Ordering instead.
I'm well aware this variant is not in-memory, but it does not require ClassTag at all to implement.
First, I am new to Scala, So apologies if the following question is too simple.
I have written the following code to the find the values of the keys that I supply in an array from the map.
def stringToCountMap(inputArray: Array[String], inputMap:Map[String,Int]) : Array[Int] = {
return inputArray.map(x => inputMap.get(x))
}
I got the following error,
type mismatch;
found : Array[Option[Int]]
required: Array[Int]
return inputArray.map(x => inputMap.get(x))
Question:
1) Can anyone explain what is Option[Int]?
2) What is my mistake here ?
Thanks in advance.
Option is Scala's option type (also called a nullable type). It represents a case where the value may not exist.
Consider a map that doesn't contain a requested key. How would you handle a request for the key? One option is to result in an error, such as by throwing an exception. Another is to return a special value that indicates there is no value. Map.get does the latter, using Option as the special type and None as the value. This means the return type of Map.get isn't the value type of the map (Int), but Option applied to the value type (Option[Int]).
To correct the type declaration, change the return type:
def stringToCountMap(inputArray: Array[String], inputMap:Map[String,Int]) : Array[Option[Int]] = {
inputArray.map(x => inputMap.get(x))
}
You can leave out the return type of stringToCountMap and let type inference handle it:
def stringToCountMap(inputArray: Array[String], inputMap:Map[String,Int]) = {
inputArray.map(x => inputMap.get(x))
}
As a consequence, missing keys from the input map get carried through:
scala> stringToCountMap(Array("a", "def"), Map("a" -> 1, "bc" -> 2))
res0: Array[Option[Int]] = Array(Some(1), None)
Option[T] is a wrapper around a value of type T. Its purpose is to prevent NullPointerException, that you might know from Java. A value of type Option[T] might either be None, which, as the name implies is an object that represents nothing, or it might be Some(x: T), which represents an existing value.
inputMap(x) returns an Option[Int], since you have no guarantee that the x key exists in inputMap. If it does, it returns Some(value: Int), else it returns None.
Calling stringToCountMap(Array("a", "b", "c"), Map("a" -> 1, "c" -> 2)) results in Array(Some(1), None, Some(2))
If you want an Array[Int] instead, you might do something like inputArray.map(x => inputMap.getOrElse(x, 0)).get. The getOrElse method has two parameters, where the first one is the key, and the second one is the default value. inputArray.map(x => inputMap.get(x).getOrElse(0)) has the same effect, since calling getOrElse(value) on an Option either unwraps the Some object, or returns the default value.
Now, stringToCountMap(Array("a", "b", "c"), Map("a" -> 1, "c" -> 2)) results in Array(1, 0, 2).
You might also want to omit the keys missing in the input array. In that case, you might do inputArray.flatMap(x => inputMap.get(x)). flatMap is a function similar to map, but it returns strictly, as the name implies, flat collections. For example, calling flatMap(x => x) on an Array[Array[Int]] would return an Array of all the values in the 2D array in a single row.
Here, Option is a collection, as well. If it is of type Some, it contains a single value, if it is None, it is an empty collection. Thus, in the resulting array you would only have the values of the keys present in the map, and the keys not present in the map are skipped.
Now, stringToCountMap(Array("a", "b", "c"), Map("a" -> 1, "c" -> 2)) results in Array(1, 2).
First, there is no need to use 'return' in scala to return any value.
def stringToCountMap(inputArray: Array[String], inputMap:Map[String,Int]) = {
inputArray.map(x => inputMap.get(x))
}
When you get the value of key from Map, It returns result in Option.
For example:
scala> val map = Map(1-> "a",2 -> "b")
map: scala.collection.immutable.Map[Int,String] = Map(1 -> a, 2 -> b)
scala> map.get(1)
res0: Option[String] = Some(a)
scala> map.get(3)
res1: Option[String] = None
When you try to get the value of key, which does not exist. In java, you have encountered with NullPointerException. So when there is no value, it returns None to avoid exception.
For more info refer
In your method, you have given return type as Array[Int] but function returns Array[Option[Int]] that's why it throws compilation error.
I searched for a while and I was not able to find a clear and satisfying answer to this question.
Suppose I have this little function to compute the mean value of an Array:
def meanofArray(s:Array[Double]) : Double = s.sum/s.length
the problem is that is Array has to be Double.
If for example I have this Array:
val x = Array( 1 , 2 , 3 , 4 )
and make the call :
println(meanofArray(x))
I get the error found : Array[Int]
required: Array[Double]
How can I implement meanofArray so as to be able to accept any type of Int , Long , Float and Double with no errors ?
My thought was Generic types :
def meanofArray(s:Array[T]) : Double = s.sum/s.length
but I could not get it working.
Any other idea is welcomed !
The issue you are running into demonstrates the beauty of type abstraction.
Based solely on the signature of the function you provided
def meanofArray(s:Array[T]) : T
we know quite a bit about what this function can and can not do, as long as it does not do shady things like reflection of course. It can not do things like sort the Array, find the sum, etc. Based on some unfortunate characteristics of Scala, there are some things which we can not be sure it does not do, like use equals or toString for some T.
You need to declare some additional information about the type T in the function signature.
def meanofArray[T](s: Array[T])(implicit ev: Numeric[T]): Double
Here we are declaring that T must have a definition of Numeric[T] defined and in scope at call sites. Numeric[T] provides the definitions of functions required in order to do the calculations you need. We are now declaring that this function may do more things like sort the array or find the sum.
And the implementation you are looking for...
def meanofArray[T](s: Array[T])(implicit ev: Numeric[T]): Double =
s.view.map(ev.toDouble).sum / s.length
EDIT: Map s to view of Array[Double] to decrease likelihood of overflow. Taken from #Eastsun's answer. Taking Numeric[T] implicitly is strongly preferred over implicit view from T => Double in my opinion though.
Here is what you want(The parameter's type of mean is Seq[A] so that you can use it as mean(List(1, 2, 3)) or mean("ABCDEFG")):
scala> def mean[A](arr: Seq[A])(implicit view: A => Double): Double =
(arr map view sum) / arr.size
mean: [A](arr: Seq[A])(implicit view: A => Double)Double
scala> mean(Array(1, 2, 3, 4))
res0: Double = 2.5
Compare with #drstevens 's solution:
scala> def meanofArray[T](s: Array[T])(implicit ev: Numeric[T]): Double =
| ev.toDouble(s.sum) / s.length
meanofArray: [T](s: Array[T])(implicit ev: scala.math.Numeric[T])Double
scala> meanofArray(Array[Byte](111, 111, 111, 111))
res3: Double = -17.0 //Wrong!!!
scala> mean((Array[Byte](111, 111, 111, 111)))
res4: Double = 111.0 //Right !!!
This is the cleanest solution i could come up for your question.
def mean[A <% Double : Numeric](seq:Seq[A]) = seq.sum/seq.length
Here, Seq.sum needs an evidence of A conforms to be a Numeric type. (: in the type parameter)
And that division to work, the result of seq.sum should be viewable as a Double. Altogether this results a result mean of Double type
According to Scaladoc, there is no method named map in Array class, but there is an implicit function implicit def intArrayOps (xs: Array[Int]): ArrayOps[Int] defined in scala.Predef. So you can apply map on Array(1,2,3,4) if you like. But What I am confused about is that the map result is of type Array[Int], not ArrayOps[Int]. Here is my test:
scala> val array = Array(1,2,3,4)
array: Array[Int] = Array(1, 2, 3, 4)
scala> array.map(x => x)
res18: Array[Int] = Array(1, 2, 3, 4)
scala> res18.isInstanceOf[Array[Int]]
res19: Boolean = true
scala> res18.isInstanceOf[scala.collection.mutable.ArrayOps[Int]]
warning: there wre 1 unchecked warnings; re-run with -unchecked for details
res20: Boolean = false
It indeed returns an array, as intended and as is convenient, there is no reason you would need an ArrayOps, it is intended only to provide extra methods to arrays. The doc is wrong.
The routine is actually not implemented in ArrayOps. As most collection methods, it is inherited from TraversableLike. And you see two map methods in the doc:
def map [B] (f: (T) ⇒ B): ArrayOps[B]
def map [B, That] (f: (T) ⇒ B)(implicit bf: CanBuildFrom[Array[T], B, That]): That
Only the second one exists (inherited from TraversableLike). It is intended to allow implementation of map in just one place (traversable like) while allways giving the best possible behavior. For instance, a String is a of Seq[Char], if you map with a function from character to character, you get a String, but if you map from collection to say Int, the result cannot be a String and it will just be a Seq. This is explained in much detail in the paper fighting the bit rot with types.
However, this makes for a very complex signature, which does not reflect the simplicity of using the method, and makes very poor documentation most of the time (you normally would have to chase to which CanBuildFrom in implicit scope would work). This was discussed in this most famous scala question of stack overflow. So the tool scaladoc was extended so that a simpler entry, corresponding to intended usage, may appear. If you look at the source of GenTraversableLike, where the routine is introduced, you will see the following in the scaladoc for map (and a similar one in many methods)
#usecase def map[B](f: A => B): $Coll[B]
Subtypes add in their doc #define Coll <className>, and map (among others) appears with the simplified signature, marked [Use case]. In the source of ArrayOps, there is a #define Coll ArrayOps where it should be Array.
You can use the REPL with the -Xprint:typer option to see what's going on. Here is the output of the map method, reformatted for easier reading:
$ scala -Xprint:typer
scala> Array(1,2,3,4).map(x => x)
[[syntax trees at end of typer]]// Scala source: <console>
// some lines deleted
private[this] val res0: Array[Int] =
scala.this.Predef.intArrayOps(scala.Array.apply(1, 2, 3, 4))
.map[Int, Array[Int]]
(( (x: Int) => x ))
(scala.this.Array.canBuildFrom[Int](reflect.this.Manifest.Int));
So simplifying for package names here is what happens:
intArrayOps(Array(1,2,3,4)) // converts to ArrayOps
.map[Int, Array[Int]] // calls map with parameter lists below
((x:Int) => x) // pass identity function as fisrt param
(Array.canBuildFrom[Int]// pass builder for Array[Int] as second param
(Manifest.Int)) // pass class manifest for Int
So there is indeed a conversion to ArrayOps (first line). It returns ArrayOps[Int].
The ArrayOps.map[Int, Array[Int]] method is then called on it. Then as didierd explain, the original signature for map - not the simplified signature - indicates that the return type inferred will be Array[Int]