Multi-threaded array initialization in Kotlin - arrays

Say, I have the following functions that allow me to create matrices:
inline fun <reified T> Matrix2D(w: Int, h: Int, init: (Int, Int) -> T) =
Array(w){ x -> Array(h){ y -> init(x, y) } }
inline fun <reified T> Matrix2D(w: Int, h: Int, value: T) =
Array(w){ Array(h){ value } }
// For example:
val m = Matrix2D(400, 400) { x, y ->
Color(x.toDouble() / 400.0, y.toDouble() / 400.0, 0.0)
}
I also don't want to allow for nullability, cause it's going to be a pain to deal with element access later.
How would I initialize such a matrix using multiple threads/coroutines, where each one initializes its own tile of the matrix, without allowing for nullability of the matrix cells?

Yes, you can create a Matrix2D in parallel, for example:
val threads = Matrix2D(10, 3){x, y-> Thread.currentThread() }.flatten().distinct()
// v--- size > 1
println(threads.size)
inline fun <reified T> Matrix2D(w: Int, h: Int, crossinline init: (Int, Int) -> T)=
// v--- create array in parallel
generatorOf(w){x -> Array(h) {y -> init(x, y) } }.map{ it() }.toTypedArray()
// get the final result `Matrix2D` ---^
// v--- create array generator
inline fun <reified T> generatorOf(size: Int, crossinline init:(Int) -> T) =
//v--- you can use `ForkJoinPool#commonPool`, but it will run task in work thread
ForkJoinPool(20).let {
Array(size) { i -> it.submit(Callable { init(i) }).let { { it.get() } } }
// return the lambda generator ()->T ---^
}
You also can write an extension function toArray to remove the middle map operation & promote the performance, for example:
inline fun <reified T> Matrix2D(w: Int, h: Int, crossinline init: (Int, Int) -> T)=
// v--- init array elements in parallel
generatorOf(w) { x -> Array(h) { y -> init(x, y) } }.toArray{it()}
// transform to the `Matrix2D` directly ---^
// v--- convert Array<T> to Array<R>
inline fun <T,reified R> Array<T>.toArray(transform:(T)->R):Array<R>{
return Array(size){i->transform(get(i))}
}
Why did the Matrix2D start threads in outer array initialization, you can see sum two double[][] with parallel stream as further.

Related

array of closures in Rust

The following Rust code tries to store a closure of zero arguments in an array and call the function.
fn main() {
println!("The answer is: {}", solution_fns[0]());
}
const solution_fns: [fn() -> isize] =
[|| (1..=999).filter(|e| divides(3, e) || divides(5, e)).sum()];
fn divides(d: usize, n: usize) -> bool {
n % d == 0
}
Link to Rust playground. Unfortunately it does not compile:
error[E0277]: the size for values of type [fn() -> isize] cannot be
known at compilation time --> src/main.rs:5:21 | 5 | const
solution_fns: [fn() -> isize] = |
^^^^^^^^^^^^^^^ doesn't have a size known at compile-time | =
help: the trait Sized is not implemented for [fn() -> isize]
I understand that you cannot constuct an array (or Vec) of things of which the size is not known at compile-time. However I understand the type [fn() -> isize] to be array of function pointers, and I don't see why a function pointer should not have a known size. Sticking the closure in a Box does not seem to help:
const solution_fns: [Box<fn() -> isize>] = [Box::new(|| {
(1..=999).filter(|e| divides(3, e) || divides(5, e)).sum()
})];
How then can I store an array of closures?
The problem is not in the fn pointer, but that your array doesnt have a size. It is simply solved by adding the expected size to the array declaration:
const solution_fns: [fn() -> usize; 1] =
[|| (1..=999usize).filter(|&e| divides(3, e) || divides(5, e)).sum()];
Playground
If you use a slice, then you don't have to specify the size:
fn main() {
println!("The answer is: {}", SOLVERS[0]());
}
const SOLVERS : &[fn() -> usize] =
&[|| (1..=999).filter(|&e| divides(3, e) || divides(5, e)).sum()];
fn divides(d: usize, n: usize) -> bool {
n % d == 0
}
Link to the playground.

Function accepting ndarray and float at the same time

I would like to write a function computing the refractive index of some material as a function of wavelength. I use the ndarray crate for arrays, similar to numpy in Python.
At the moment the function is implemented the following way:
fn calc_refractive_index(l: &Array1<f64>) -> Array1<f64> {
fn refractive_index(&self, l: &Array1<f64>) -> Array1<f64> {
let V =
self.A[0] + self.A[1] / (1. + self.A[2] * (self.A[3] * l / self.pitch).mapv(f64::exp));
let W =
self.B[0] + self.B[1] / (1. + self.B[2] * (self.B[3] * l / self.pitch).mapv(f64::exp));
(self.material.refractive_index(&l).powi(2)
- 3. * l.powi(2) / 4. / (consts::PI * self.pitch).powi(2) * (V.powi(2) - W.powi(2)))
.mapv(f64::sqrt)
}
}
With the following trait implemented, to make potentiation shorter:
trait Squared<T> {
fn powi(&self, e: i32) -> T;
}
impl Squared<Array1<f64>> for Array1<f64> {
fn powi(&self, e: i32) -> Array1<f64> {
self.mapv(|a| a.powi(e))
}
}
However, I may also want to compute the refractive index at only one specific wavelength, so I would like to also accept float64 values. What is the best way to implement this, without implementing two separate functions?
Edit:
The function is using ndarray's syntax for squaring: l.mapv(|x| x.powi(2)), which is unfortunately different than for float64
Edit 2:
As asked I included the function body.
It is definitely possible, although it might take more work than simply adding another function. You will need to first define two helper traits that specify the mapv and powi (much like Squared) operations for both f64 and Array1<f64>:
trait PowI {
fn powi(&self, e: i32) -> Self;
}
impl PowI for f64 {
fn powi(&self, e: i32) -> Self {
f64::powi(*self, e)
}
}
impl PowI for Array1<f64> {
fn powi(&self, e: i32) -> Self {
self.mapv(|a| a.powi(e))
}
}
trait MapV {
fn mapv(&self, f: impl FnMut(f64) -> f64) -> Self;
}
impl MapV for f64 {
fn mapv(&self, mut f: impl FnMut(f64) -> f64) -> Self {
f(*self)
}
}
impl MapV for Array1<f64> {
fn mapv(&self, f: impl FnMut(f64) -> f64) -> Self {
Array1::mapv(self, f)
}
}
You can then make your refractive index calculation function be generic over a type T that implements both these traits (i.e. T: PowI + MapV).
Note that you will also need additional bounds on T that specify that it can be added, divided and multiplied with Ts and f64s (which I assume is the type of self.pitch and the elements in the self.A and self.B arrays). You can do this by requiring that T, &'a T or even f64 implement the appropriate Add, Mul and Div traits. For example, your implementation may require that f64: Mul<&'a T, Output=T> and T: Div<f64, Output=T> in addition to many other bounds. The compiler errors will help you figure out exactly which ones you need.

Initialize Array of non-optionals without using constructor

I am using an Array of non-optional values, and I want them to stay non-optional, but I can't use Array's default constructor because of problems described here.
Furthermore, the .also{} trick described in the linked won't work for me, because mine is not an array of some primitive type with its own special WhateverArray class.
Is there some Kotlin trick by which I can initialize my a below? Or must I resort to building some list and then converting it?
// please assume Stream<MyNonprimitiveType> magically gives me
// size() -> Int and
// next() -> MyNonprimitiveType
val stream : Stream<MyNonprimitiveType> = Stream<MyNonprimitiveType>()
val size : Int = stream.size()
val a : Array<MyNonprimitiveType> = ??? // use stream.next()
Here's a complete example doing what you want, without using a temporary list:
class Stream<T>(private val list: List<T>) {
val size = list.size;
private val it = list.iterator()
fun next(): T {
return it.next()
}
}
inline fun <reified T: Any> Stream<T>.toArray(): Array<T> {
val tmp: Array<T?> = arrayOfNulls(size)
for (i in 0 until size) {
tmp[i] = next()
}
return tmp as Array<T>
}
fun main() {
val stream : Stream<String> = Stream(listOf("a", "b"))
val a: Array<String> = stream.toArray()
println(Arrays.toString(a))
}

How to check a generic type inside a Kotlin function?

I am using Kotlin to parse JSON. For example, I have this representation of a country: {"code":"US", "name":"United States of America"}. To produce a Country object from such a JSONObject, I have this function:
val produceCountry = fun (js: JSONObject) =
Country(js.getString("code"), js.getString("name"))
I can easily parse an array of Country with this function. Besides arrays of Country, however, I also have arrays of Cat, Car, Cart, CordlessPhone, etc. Each has their own produce* function transforming a JSONObject to a Kotlin object of that type. To generalize array parsing, I have this function:
fun <T> produceSetOf(array: JSONArray, element: (JSONObject) -> T): Set<T> {
val set = mutableSetOf<T>()
for (i in 0 until array.length())
set.add(element(array.getJSONObject(i)))
return set
}
So I can call produceSetOf(jsonArray, produceCountry) on encountering an array whose elements are of type Country. This works well on arrays of Cat, Car, Cart, CordlessPhone too.
Problem arises when I see an array of strings. Instead of array.getJSONObject(i), I have to use array.getString(i). In effect, I am thinking of introducing another parameterized type to the function above and have it make the call differently:
fun <S,T> produceSetOf(array: JSONArray, element: (S) -> T): Set<T> {
val set = mutableSetOf<T>()
for (i in 0 until array.length()) {
when (S) {
is String ->
set.add(element(array.getString(i)))
is JSONObject ->
set.add(element(array.getJSONObject(i)))
}
}
return set
}
Of course, Kotlin does not allow me to do that. Any suggestion how I could do that while maintaining the generality of produceSetOf() and without introducing another layer of abstraction (e.g. an element iterator, or a function transforming an index into String/JSONObject)?
Thank you.
Here is one possible solution using reified type parameters.
inline fun <reified S, T> produceSetOf(array: JsonArray, element: (S) -> T): Set<T> {
val set = mutableSetOf<T>()
for (i in 0 until array.size()) {
when (S::class) {
String::class -> set.add(element(array[i].string as S))
JsonObject::class -> set.add(element(array[i].obj as S))
}
}
return set
}
val stringArray = listOf("1", "2").toJsonArray()
val stringSet = produceSetOf<String, Int>(stringArray) { it.toInt() }
println(stringSet) // prints [1, 2]
val objArray = listOf(jsonObject("key" to "value"), jsonObject("key" to "other")).toJsonArray()
val objSet = produceSetOf<JsonObject, String>(objArray) { it["key"].string }
println(objSet) // print [value, other]
I used gson for the Json objects, since I didn't know where yours were from.
A possible shorter solution:
inline fun <reified S, T> produceSetOf(array: JsonArray, element: (S) -> T): Set<T> = array.map {
when (S::class) {
String::class -> element(it.string as S)
JsonObject::class -> element(it.obj as S)
else -> throw UnsupportedOperationException("${S::class.simpleName} is not supported")
}
}.toSet()

Eigen unsupported Tensor to Eigen matrix

I get a Eigen::Tensor<std::complex, 2> after some operations on a Tensor with more dimensions. Is there an easy way to create a Eigen::MatrixXcf from this Tensor-object or do I have to copy the values manually?
Original answer. See the update further down.
I am currently using Eigen version 3.3.4, and there is no easy built-in function to cast between Eigen::Tensor types and the more familiar Matrix or Array types. It would have been handy to have something like the .matrix() or .array() methods.
There is also no easy way to add such methods via the plugin mechanism, because the Tensor module doesn't seem to support plugins. If anybody knows how please comment.
In the meantime it is possible to make make fairly efficient workarounds using the Map functions. The following works in C++14, for casting Tensors to and from Matrices
#include <unsupported/Eigen/CXX11/Tensor>
#include <iostream>
template<typename T>
using MatrixType = Eigen::Matrix<T,Eigen::Dynamic, Eigen::Dynamic>;
template<typename Scalar,int rank, typename sizeType>
auto Tensor_to_Matrix(const Eigen::Tensor<Scalar,rank> &tensor,const sizeType rows,const sizeType cols)
{
return Eigen::Map<const MatrixType<Scalar>> (tensor.data(), rows,cols);
}
template<typename Scalar, typename... Dims>
auto Matrix_to_Tensor(const MatrixType<Scalar> &matrix, Dims... dims)
{
constexpr int rank = sizeof... (Dims);
return Eigen::TensorMap<Eigen::Tensor<const Scalar, rank>>(matrix.data(), {dims...});
}
int main () {
Eigen::Tensor<double,4> my_rank4 (2,2,2,2);
my_rank4.setRandom();
Eigen::MatrixXd mymatrix = Tensor_to_Matrix(my_rank4, 4,4);
Eigen::Tensor<double,3> my_rank3 = Matrix_to_Tensor(mymatrix, 2,2,4);
std::cout << my_rank3 << std::endl;
return 0;
}
This works with complex types just as well.
Unfortunately these functions only take tensors, not tensor operations. For instance, this doesn't work:
Eigen::MatrixXd mymatrix = Tensor_to_Matrix(my_rank4.shuffle(Eigen::array<long,4>{1,0,3,2}), 4,4);
Update August 2021:
A memory leak was fixed, see the code below. In short, one must remember to call .cleanup() member function on the Eigen::TensorEvaluator object to deallocate internal temporary buffers.
Update October 2021:
Since I submitted the answer above back in 2018, I have learned how to map and cast between Eigen::Tensor and Eigen::Matrix (and expressions thereof) with a minimal amount of temporaries.
I have used the helper functions below since Eigen 3.3.7 using C++17
#include <unsupported/Eigen/CXX11/Tensor>
#include <iostream>
template<typename T>
using MatrixType = Eigen::Matrix<T,Eigen::Dynamic, Eigen::Dynamic>;
template<typename T>
using VectorType = Eigen::Matrix<T,Eigen::Dynamic, 1>;
/*
*
* Convert Eigen::Tensor --> Eigen::Matrix
*
*/
template<typename Derived, typename Device = Eigen::DefaultDevice>
class selfCleaningEvaluator {
private:
using Evaluator = Eigen::TensorEvaluator<const Eigen::TensorForcedEvalOp<const Derived>, Device>;
Evaluator m_eval;
public:
selfCleaningEvaluator(const Evaluator &eval) : m_eval(eval) {}
selfCleaningEvaluator(const Eigen::TensorBase<Derived, Eigen::ReadOnlyAccessors> &expr, const Device &device = Device())
: m_eval(Evaluator(expr.eval(), device)) {
m_eval.evalSubExprsIfNeeded(nullptr);
}
~selfCleaningEvaluator() {
// This whole point of this object is to call cleanup automatically on destruct.
// If there are pending operations to evaluate, m_eval will allocate a buffer to hold a result,
// which needs to be deallocated.
m_eval.cleanup();
}
constexpr auto rank() {
using DimType = typename decltype(m_eval)::Dimensions::Base;
return DimType{}.size(); // Because Derived::Dimensions is sometimes wrong
}
const Evaluator *operator->() const { return &m_eval; }
Evaluator *operator->() { return &m_eval; }
constexpr auto map() {
// We inspect m_eval to get the type, rank and dimensions, because it has the resulting tensor,
// whereas Derived is the tensor type _before_ whatever operation is pending (if any).
using DimType = typename decltype(m_eval)::Dimensions::Base;
constexpr auto rank = DimType{}.size();
using Scalar = typename Eigen::internal::remove_const<typename decltype(m_eval)::Scalar>::type;
return Eigen::TensorMap<Eigen::Tensor<Scalar, rank>>(m_eval.data(), m_eval.dimensions());
}
};
// Evaluates expressions if needed
template<typename T, typename Device = Eigen::DefaultDevice>
auto asEval(const Eigen::TensorBase<T, Eigen::ReadOnlyAccessors> &expr, // An Eigen::TensorBase object (Tensor, TensorMap, TensorExpr... )
const Device &device = Device()) { // Override to evaluate on another device, e.g. thread pool or gpu.
return selfCleaningEvaluator(expr, device);
}
// Converts any Eigen::Tensor (or expression) to an Eigen::Matrix with shape rows/cols
template<typename T, typename sizeType, typename Device = Eigen::DefaultDevice>
auto MatrixCast(const Eigen::TensorBase<T, Eigen::ReadOnlyAccessors> &expr, const sizeType rows, const sizeType cols, const Device &device = Device()) {
auto tensor = asEval(expr, device);
auto tensorMap = tensor.map();
using Scalar = typename decltype(tensorMap)::Scalar;
return static_cast<MatrixType<Scalar>>(Eigen::Map<const MatrixType<Scalar>>(tensorMap.data(), rows, cols));
}
// Converts any Eigen::Tensor (or expression) to an Eigen::Matrix with shape rows/cols
template<typename T, typename Device = Eigen::DefaultDevice>
auto VectorCast(const Eigen::TensorBase<T, Eigen::ReadOnlyAccessors> &expr, const Device &device = Device()) {
auto tensor = asEval(expr, device);
auto tensorMap = tensor.map();
auto size = Eigen::internal::array_prod(tensorMap.dimensions());
using Scalar = typename decltype(tensorMap)::Scalar;
return static_cast<VectorType<Scalar>>(Eigen::Map<const VectorType<Scalar>>(tensorMap.data(), size));
}
// View an existing Eigen::Tensor as an Eigen::Map<Eigen::Matrix>
template<typename Scalar, auto rank, typename sizeType>
auto MatrixMap(const Eigen::Tensor<Scalar, rank> &tensor, const sizeType rows, const sizeType cols) {
return Eigen::Map<const MatrixType<Scalar>>(tensor.data(), rows, cols);
}
// View an existing Eigen::Tensor of rank 2 as an Eigen::Map<Eigen::Matrix>
// Rows/Cols are determined from the matrix
template<typename Scalar>
auto MatrixMap(const Eigen::Tensor<Scalar, 2> &tensor) {
return Eigen::Map<const MatrixType<Scalar>>(tensor.data(), tensor.dimension(0), tensor.dimension(1));
}
// View an existing Eigen::Tensor of rank 1 as an Eigen::Map<Eigen::Vector>
// Rows is the same as the size of the tensor.
template<typename Scalar, auto rank>
auto VectorMap(const Eigen::Tensor<Scalar, rank> &tensor) {
return Eigen::Map<const VectorType<Scalar>>(tensor.data(), tensor.size());
}
/*
*
* Convert Eigen::Matrix --> Eigen::Tensor
*
*/
// Converts an Eigen::Matrix (or expression) to Eigen::Tensor
// with dimensions specified in std::array
template<typename Derived, typename T, auto rank>
Eigen::Tensor<typename Derived::Scalar, rank>
TensorCast(const Eigen::EigenBase<Derived> &matrix, const std::array<T, rank> &dims) {
return Eigen::TensorMap<const Eigen::Tensor<const typename Derived::Scalar, rank>>
(matrix.derived().eval().data(), dims);
}
// Converts an Eigen::Matrix (or expression) to Eigen::Tensor
// with dimensions specified in Eigen::DSizes
template<typename Derived, typename T, auto rank>
Eigen::Tensor<typename Derived::Scalar, rank>
TensorCast(const Eigen::EigenBase<Derived> &matrix, const Eigen::DSizes<T, rank> &dims) {
return Eigen::TensorMap<const Eigen::Tensor<const typename Derived::Scalar, rank>>
(matrix.derived().eval().data(), dims);
}
// Converts an Eigen::Matrix (or expression) to Eigen::Tensor
// with dimensions as variadic arguments
template<typename Derived, typename... Dims>
auto TensorCast(const Eigen::EigenBase<Derived> &matrix, const Dims... dims) {
static_assert(sizeof...(Dims) > 0, "TensorCast: sizeof... (Dims) must be larger than 0");
return TensorCast(matrix, std::array<Eigen::Index, sizeof...(Dims)>{dims...});
}
// Converts an Eigen::Matrix (or expression) to Eigen::Tensor
// with dimensions directly as arguments in a variadic template
template<typename Derived>
auto TensorCast(const Eigen::EigenBase<Derived> &matrix) {
if constexpr(Derived::ColsAtCompileTime == 1 or Derived::RowsAtCompileTime == 1) {
return TensorCast(matrix, matrix.size());
} else {
return TensorCast(matrix, matrix.rows(), matrix.cols());
}
}
// View an existing Eigen::Matrix as Eigen::TensorMap
// with dimensions specified in std::array
template<typename Derived, auto rank>
auto TensorMap(const Eigen::PlainObjectBase<Derived> &matrix, const std::array<long, rank> &dims) {
return Eigen::TensorMap<const Eigen::Tensor<const typename Derived::Scalar, rank>>(matrix.derived().data(), dims);
}
// View an existing Eigen::Matrix as Eigen::TensorMap
// with dimensions as variadic arguments
template<typename Derived, typename... Dims>
auto TensorMap(const Eigen::PlainObjectBase<Derived> &matrix, const Dims... dims) {
return TensorMap(matrix, std::array<long, static_cast<int>(sizeof...(Dims))>{dims...});
}
// View an existing Eigen::Matrix as Eigen::TensorMap
// with dimensions determined automatically from the given matrix
template<typename Derived>
auto TensorMap(const Eigen::PlainObjectBase<Derived> &matrix) {
if constexpr(Derived::ColsAtCompileTime == 1 or Derived::RowsAtCompileTime == 1) {
return TensorMap(matrix, matrix.size());
} else {
return TensorMap(matrix, matrix.rows(), matrix.cols());
}
}
int main () {
Eigen::Tensor<double,4> my_rank4 (2,2,2,2);
my_rank4.setRandom();
Eigen::MatrixXd mymatrix = MatrixCast(my_rank4, 4,4); // Cast Eigen::Tensor --> Eigen::Matrix
Eigen::Tensor<double,3> my_rank3 = TensorCast(mymatrix, 2,2,4); // Cast Eigen::Matrix --> Eigen::Tensor
std::cout << my_rank3 << std::endl;
return 0;
}
Try on compiler-explorer

Resources