Related
Let's say i have an array with 10 elments:
arr = [1,2,3,4,5,6,7,8,9,10]
Then I want to define a function that takes this arr as parameter to perform
a calculation, let's say for this example the calculation is the difference of means, for example:
If N=2 (That means group the elements of arr in groups of size 2 sequentially):
results=[]
result_1 = 1+2/2 - 3+4/2
result_2 = 3+4/2 - 5+6/2
result_3 = 5+6/2 - 7+8/2
result_4 = 7+8/2 - 9+10/2
The output would be:
results = [-2,-2,-2,-2]
If N=3 (That means group the elements of arr in groups of size 3 sequentially):
results=[]
result_1 = 1+2+3/3 - 4+5+6/3
result_2 = 4+5+6/3 - 7+8+9/3
The output would be:
results = [-3,-3]
I want to do this defining two functions:
Function 1 - Creates the arrays that will be used as input for 2nd function:
Parameters: array, N
returns: k groups of arrays -> seems to be ((length(arr)/N) - 1)
Function 2 - Will be the fucntion that gets the arrays (2 by 2) and perfoms the calculations, in this case, difference of means.
Parameters: array1,array2....arr..arr..
returns: list of the results
Important Note
My idea is to apply these fucntions to a stream of data and the calculation will be the PSI (Population Stability Index)
So, if my stream has 10k samples and I set the first function to N=1000, then the output to the second function will be 1k samples + next 1k samples.
The process will be repetead till the end of the datastream
I was trying to do this in python (I already have the PSI code ready) but now I decided to use Julia for it, but I am pretty new to Julia. So, if anyone can give me some light here will be very helpfull.
In Julia if you have a big Vector and you want to calculate some statistics on groups of 3 elements you could do:
julia> a = collect(1:15); #creates a Vector [1,2,...,15]
julia> mean.(eachcol(reshape(a,3,length(a)÷3)))
5-element Vector{Float64}:
2.0
5.0
8.0
11.0
14.0
Note that both reshape and eachcol are non-allocating so no data gets copied in the process.
If the length of a is not divisible by 3 you could truncate it before reshaping - to avoid allocation use view for that:
julia> a = collect(1:16);
julia> mean.(eachcol(reshape(view(a,1:(length(a)÷3)*3),3,length(a)÷3)))
5-element Vector{Float64}:
2.0
5.0
8.0
11.0
14.0
Depending on what you actually want to do you might also want to take a look at OnlineStats.jl https://github.com/joshday/OnlineStats.jl
Well, I use JavaScript instead Of Python, But it would be same thing in python...
You need a chunks function, that take array and chunk_size (N), lets say chunks([1,2,3,4], 2) -> [[1,2], [3,4]]
we have a sum method that add all element in array, sum([1,2]) -> 3;
Both JavaScript and python support corouting, that you can use for lazy evaluation, And its called Generator function, this type of function can pause its execution, and can resume on demand! This is useful for calculate stream of data.
let arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// Javascript Doesn't support `chunks` method yet, So we need to create one...
Array.prototype.chunks = function* (N) {
let chunk = [];
for (let value of this) {
chunk.push(value)
if (chunk.length == N) {
yield chunk;
chunk = []
}
}
}
Array.prototype.sum = function () {
return this.reduce((a, b) => a + b)
}
function* fnName(arr, N) {
let chunks = arr.chunks(N);
let a = chunks.next().value.sum();
for (let b of chunks) {
yield (a / N) - ((a = b.sum()) / N)
}
}
console.log([...fnName(arr, 2)])
console.log([...fnName(arr, 3)])
I have a method that returns some numpy array like below one
def numpy_array():
......
......
return true, test_1, test_2, test_3
I have another method that calculates the subtraction between true and rest of the arrays like the below one
def subtraction():
true, test_1, test_2, test_3 = numpy_array()
sub_1 = np.subtract(true, test_1)
sub_2 = np.subtract(true, test_2)
......
......
return sub_1, sub_2
The problem is, I could have a lot of arrays in the def numpy_array() method. I want to write the def subtraction() method in a dynamic way. So that I do not need to subtract arrays manually (np.subtract(true, test_1), np.subtract(true, test_2), and so on).
Could you tell me how can I do this?
numpy_array or any python function with multiple arguments will return a tuple.
You can then iterate over the tuple to get the differences you want:
def subtraction():
matrices = numpy_array()
# a tuple of some number of numpy arrays
first = matrices[0]
# the first array, 'true' in question
result = [first - matrices[i] for i in range(1,len(matrices))]
# the difference between the first and subsequent arrays
return tuple(result)
In Kotlin we can do:
val arr = intArrayOf(1,2,3)
if (2 in arr)
println("in list")
But if I want to check if 2 or 3 are in arr, what is the most idiomatic way to do it other than:
if (2 in arr || 3 in arr)
println("in list")
I'd use any() extension method:
arrayOf(1, 2, 3).any { it == 2 || it == 3 }
This way, you traverse the array only once and you don't create a set instance just to check whether it's empty or not (like in one of other answers to this question).
This is the shortest and most idiomatic way I can think of using any and in:
val values = setOf(2, 3)
val array = intArrayOf(1, 2, 3)
array.any { it in values }
Of course you can use a functional reference for the in operator as well:
array.any(values::contains)
I use setOf for the first collection because order does not matter.
Edit: I switched values and array, because of alex.dorokhow's answer. The order doesn't matter for the check to work but for performance.
The OP wanted the most idiomatic way of solving this. If you are after a more efficient way, go for aga's answer.
You can use intersect method, it takes iterable as paramter and returns set containing only items that are both in your collection and in iterable you provided. Then on that set you just need to do size check.
Here is sample:
val array1 = arrayOf(1, 2, 3, 4, 5, 6)
val array2 = arrayOf(2, 5)
// true if array1 contains any of the items from array2
if(array1.intersect(array2.asIterable()).isNotEmpty()) {
println("in list")
}
Combining #aga and #willi-mentzel solutions for better efficiency and dynamic set of checked values:
val numbers = setOf(2, 3)
arrayOf(1, 2, 3).any(numbers::contains)
Is this case the array will be iterated completely only once (at most, in the worst case).
This is more efficient than (suggested by #WilliMentzel):
numbers.any(arrayOf(1, 2, 3)::contains) // don't do that!
Where the array will be iterated set.count times in the worst case.
Note that Set.contains has O(1) complexity, but IntArray::contains has O(N).
Of course, this optimization makes sense only if the set or array are big enough.
Another handy way is actually not Kotlin's but with use of Java Collections.
It is also good to know it.
Collections.disjoint(Collection<?> c1, Collection<?> c2)
Returns {#code true} if the two specified collections have no elements
in common.
#Test
fun disjointCollections() {
val list = listOf(1, 2, 3)
assertTrue(Collections.disjoint(list, listOf(7, 8)))
assertFalse(Collections.disjoint(list, listOf(1)))
}
For me, the best way to solve this is to define a containsAny(elements:) extension function on the Array and/or Collection class. All of the places where you need to check whether an Array or Collection contains any of the elements of another then remains nice and neat, as follows:
arrayOf(1, 2, 3).containsAny(2, 3)
The implementation detail is tucked away inside of that function and can be changed in a single place in future should you wish to change the implementation.
Here's an example implementation of an extension function defined on the Collection class, the details of which are inspired by other answers in this thread:
/**
* Returns true if the receiving collection contains any of the specified elements.
*
* #param elements the elements to look for in the receiving collection.
* #return true if any element in [elements] is found in the receiving collection.
*/
fun <T> Collection<T>.containsAny(vararg elements: T): Boolean {
return containsAny(elements.toSet())
}
/**
* Returns true if the receiving collection contains any of the elements in the specified collection.
*
* #param elements the elements to look for in the receiving collection.
* #return true if any element in [elements] is found in the receiving collection.
*/
fun <T> Collection<T>.containsAny(elements: Collection<T>): Boolean {
val set = if (elements is Set) elements else elements.toSet()
return any(set::contains)
}
Likewise, here's an extension function defined on the Array class:
/**
* Returns true if the receiving array contains any of the specified elements.
*
* #param elements the elements to look for in the receiving array.
* #return true if any element in [elements] is found in the receiving array.
*/
fun <T> Array<T>.containsAny(vararg elements: T): Boolean {
return any(elements.toSet()::contains)
}
I think it's most readable to write the statement the other way around:
val arr = intArrayOf(1,2,3)
val match = setOf(2, 3).any(arr::contains)
It might be even possible to use ranges in certain scenarios:
val match = (2..3).any(arr::contains)
In the end, your solution looks pretty good to me already. Although not using fancy library functionality.
You can make an extension function to check for more values:
infix fun <T> Iterable<T>.containsAny(values: Iterable<T>): Boolean =
intersect(values).isNotEmpty()
Have a look at this blog post: https://www.codevscolor.com/kotlin-check-array-contains-one-multiple-values
Sample code
fun main() {
val givenArray = intArrayOf(1, 2, 3, 4, 5, 7, 9, 11)
println(givenArray.any{ it == 5 || it == 12})
println(givenArray.any{ it == 5 || it == 11})
println(givenArray.any{ it == 15 || it == 21})
}
If you use Apache Commons in your project, you can use CollectionUtils.containsAny.
we can use any() funtion and a Set
val arr = intArrayOf(1,2,3)
if(arr.any{ it in setOf(2,3) })
println("in list")
I want to randomly sample from a Scala list or array (not an RDD), the sample size can be much longer than the length of the list or array, how can I do this efficiently? Because the sample size can be very big and the sampling (on different lists/arrays) needs to be done a large number of times.
I know for a Spark RDD we can use takeSample() to do it, is there an equivalent for Scala list/array?
Thank you very much.
An easy-to-understand version would look like this:
import scala.util.Random
Random.shuffle(list).take(n)
Random.shuffle(array.toList).take(n)
// Seeded version
val r = new Random(seed)
r.shuffle(...)
For arrays:
import scala.util.Random
import scala.reflect.ClassTag
def takeSample[T:ClassTag](a:Array[T],n:Int,seed:Long) = {
val rnd = new Random(seed)
Array.fill(n)(a(rnd.nextInt(a.size)))
}
Make a random number generator (rnd) based on your seed. Then, fill an array with random numbers from 0 until the size of your array.
The last step is applying each random value to the indexing operator of your input array. Using it in the REPL could look as follows:
scala> val myArray = Array(1,3,5,7,8,9,10)
myArray: Array[Int] = Array(1, 3, 5, 7, 8, 9, 10)
scala> takeSample(myArray,20,System.currentTimeMillis)
res0: scala.collection.mutable.ArraySeq[Int] = ArraySeq(7, 8, 7, 3, 8, 3, 9, 1, 7, 10, 7, 10,
1, 1, 3, 1, 7, 1, 3, 7)
For lists, I would simply convert the list to Array and use the same function. I doubt you can get much more efficient for lists anyway.
It is important to note, that the same function using lists would take O(n^2) time, whereas converting the list to arrays first will take O(n) time
If you want to sample without replacement -- zip with randoms, sort O(n*log(n), discard randoms, take
import scala.util.Random
val l = Seq("a", "b", "c", "d", "e")
val ran = l.map(x => (Random.nextFloat(), x))
.sortBy(_._1)
.map(_._2)
.take(3)
Using a for comprehension, for a given array xs as follows,
for (i <- 1 to sampleSize; r = (Math.random * xs.size).toInt) yield a(r)
Note the random generator here produces values within the unit interval, which are scaled to range over the size of the array, and converted to Int for indexing over the array.
Note For pure functional random generator consider for instance the State Monad approach from Functional Programming in Scala, discussed here.
Note Consider also NICTA, another pure functional random value generator, it's use illustrated for instance here.
Using classical recursion.
import scala.util.Random
def takeSample[T](a: List[T], n: Int): List[T] = {
n match {
case n: Int if n <= 0 => List.empty[T]
case n: Int => a(Random.nextInt(a.size)) :: takeSample(a, n - 1)
}
}
package your.pkg
import your.pkg.SeqHelpers.SampleOps
import scala.collection.generic.CanBuildFrom
import scala.collection.mutable
import scala.language.{higherKinds, implicitConversions}
import scala.util.Random
trait SeqHelpers {
implicit def withSampleOps[E, CC[_] <: Seq[_]](cc: CC[E]): SampleOps[E, CC] = SampleOps(cc)
}
object SeqHelpers extends SeqHelpers {
case class SampleOps[E, CC[_] <: Seq[_]](cc: CC[_]) {
private def recurse(n: Int, builder: mutable.Builder[E, CC[E]]): CC[E] = n match {
case 0 => builder.result
case _ =>
val element = cc(Random.nextInt(cc.size)).asInstanceOf[E]
recurse(n - 1, builder += element)
}
def sample(n: Int)(implicit cbf: CanBuildFrom[CC[_], E, CC[E]]): CC[E] = {
require(n >= 0, "Cannot take less than 0 samples")
recurse(n, cbf.apply)
}
}
}
Either:
Mixin SeqHelpers, for example, with a Scalatest spec
Include import your.pkg.SeqHelpers._
Then the following should work:
Seq(1 to 100: _*) sample 10 foreach { println }
Edits to remove the cast are welcome.
Also if there is a way to create an empty instance of the collection for the accumulator, without knowing the concrete type ahead of time, please comment. That said, the builder is probably more efficient.
Did not test for performance, but the following code is a simple and elegant way to do the sampling and I believe can help many that come here just to get a sampling code. Just change the "range" according to the size of your end sample. If pseude-randomness is not enough for your need, you can use take(1) in the inner list and increase the range.
Random.shuffle((1 to 100).toList.flatMap(x => (Random.shuffle(yourList))))
I want to test and see if everything in an array passes my check, here is how I am presently doing it and my fantasy code which crashes the compiler.
Current:
def mylist = [1,2,3,4]
def presumeTrue = true
mylist.each{
if(it<0)presumeTrue=false
}
if(presumeTrue)println "Everything was greater than 0!!"
Fantasy:
def mylist = [1,2,3,4]
if(mylist*>0)println "Everything was greater than 0, but this sugar doesn't work."
Is there a correct way to apply an if test to everything in a list one one line?
Use the every method:
myList.every { it > 0 }
The operator you were trying to use is "spread dot", which is *. (not *). You'd need to use the method name (compareTo), which takes an argument. But map isn't what you're trying to do.
You're not trying to apply the method to all of mylist's members, you're trying to aggregate the result of applying the method to all the members, more like:
mylist.inject(true) { acc, n -> acc && n > 0 }
This works for me...
def mylist = [1,2,3,4]
if(!mylist.find {it < 1}) println "Everything was greater than 0, and this sugar DID work."