I want to Create a global matrix
G=+---+---+
| A | B |
+---+---+
| C | D |
+---+---+
Is that possible to create a reference to each blocks? So that I can consider each block as a matrix individually?
Yes, there is the Ref class for that:
MatrixXd G(100,100); // global matrix
// reference to sub-blocks:
Ref<MatrixXd> A = G.topLeftCorner(50,50);
Ref<MatrixXd> B = G.topRightCorner(50,50);
Ref<MatrixXd> C = G.bottomLeftCorner(50,50);
Ref<MatrixXd> D = G.bottomRightCorner(50,50);
// Accessing/modifiying the submatrices:
A.setOnes();
B.setRandom();
C.setIdentity();
D = A+B+C;
If G is const, you can make Ref<const MatrixXd> to the sub-matrices (these are read-only, of course).
Related
I'm a beginner in spark and I'm dealing with a large dataset (over 1.5 Million rows and 2 columns). I have to evaluate the Cosine Similarity of the field "features" beetween each row. The main problem is this iteration beetween the rows and finding an efficient and rapid method. I will have to use this method with another dataset of 42.5 Million rows and it would be a big computational problem if I won't find the most efficient way of doing it.
| post_id | features |
| -------- | -------- |
| Bkeur23 |[cat,dog,person] |
| Ksur312kd |[wine,snow,police] |
| BkGrtTeu3 |[] |
| Fwd2kd |[person,snow,cat] |
I've created an algorithm that evaluates this cosine similarity beetween each element of the i-th and j-th row but i've tried using lists or creating a spark DF / RDD for each result and merging them using the" union" function.
The function I've used to evaluate the cosineSimilarity is the following. It takes 2 lists in input ( the lists of the i-th and j-th rows) and returns the maximum value of the cosine similarity between each couple of elements in the lists. But this is not the problem.
def cosineSim(lista1,lista2,embed):
#embed = hub.KerasLayer(os.getcwd())
eps=sys.float_info.epsilon
if((lista1 is not None) and (lista2 is not None)):
if((len(lista1)>0) and (len(lista2)>0)):
risultati={}
for a in lista1:
tem = a
x = tf.constant([tem])
embeddings = embed(x)
x = np.asarray(embeddings)
x1 = x[0].tolist()
for b in lista2:
tem = b
x = tf.constant([tem])
embeddings = embed(x)
x = np.asarray(embeddings)
x2 = x[0].tolist()
sum = 0
suma1 = 0
sumb1 = 0
for i,j in zip(x1, x2):
suma1 += i * i
sumb1 += j*j
sum += i*j
cosine_sim = sum / ((sqrt(suma1))*(sqrt(sumb1))+eps)
risultati[a+'-'+b]=cosine_sim
cosine_sim=0
risultati=max(risultati.values())
return risultati
The function I'm using to iterate over the rows is the following one:
def iterazione(df,numero,embed):
a=1
k=1
emp_RDD = spark.sparkContext.emptyRDD()
columns1= StructType([StructField('Source', StringType(), False),
StructField('Destination', StringType(), False),
StructField('CosinSim',FloatType(),False)])
first_df = spark.createDataFrame(data=emp_RDD,
schema=columns1)
for i in df:
for j in islice(df, a, None):
r=cosineSim(i[1],j[1],embed)
if(r>0.45):
z=spark.createDataFrame(data=[(i[0],j[0],r)],schema=columns1)
first_df=first_df.union(z)
k=k+1
if(k==numero):
k=a+1
a=a+1
return first_df
The output I desire is something like this:
| Source | Dest | CosinSim |
| -------- | ---- | ------ |
| Bkeur23 | Ksur312kd | 0.93 |
| Bkeur23 | Fwd2kd | 0.673 |
| Ksur312kd | Fwd2kd | 0.76 |
But there is a problem in my "iterazione" function.
I ask you to help me finding the best way to iterate all over this rows. I was thinking also about copying the column "features" as "features2" and applying my function using WithColumn but I don't know how to do it and if it will work. I want to know if there's some method to do it directly in a spark dataframe, avoiding the creation of other datasets and merging them later, or if you know some method more rapid and efficient. Thank you!
This question already has answers here:
How to swap the elements of an array, slice, or Vec?
(1 answer)
How to get mutable references to two array elements at the same time?
(8 answers)
Temporarily move out of borrowed content
(3 answers)
Closed 2 years ago.
I would like to write a function fn f<A>(xs: &mut [A; 9]) that reorders an array in-place from:
[a, b, c,
d, e, f,
g, h, i]
to:
[g, d, a,
h, e, b,
i, f, c]
I can't reassign the array due to moving elements out of array:
fn f1<A>(xs: &mut [A; 9]) {
*xs = [xs[6], xs[3], xs[0], xs[7], xs[4], xs[1], xs[8], xs[5], xs[2]];
}
error[E0508]: cannot move out of type `[A; 9]`, a non-copy array
--> src/lib.rs:2:12
|
2 | *xs = [xs[6], xs[3], xs[0], xs[7], xs[4], xs[1], xs[8], xs[5], xs[2]];
| ^^^^^
| |
| cannot move out of here
| move occurs because `xs[_]` has type `A`, which does not implement the `Copy` trait
I cannot do multiple mutable borrows:
fn f2<A>(xs: &mut [A; 9]) {
std::mem::swap(&mut xs[0], &mut xs[6]);
}
error[E0499]: cannot borrow `xs[_]` as mutable more than once at a time
--> src/lib.rs:2:32
|
2 | std::mem::swap(&mut xs[0], &mut xs[6]);
| -------------- ---------- ^^^^^^^^^^ second mutable borrow occurs here
| | |
| | first mutable borrow occurs here
| first borrow later used by call
There is no built-in function that does this transformation for me.
How to implement this?
I'm a Matlab newbie and I would like to assign NaN values to an array of size(j, k, l) wherever the dimension is j < k or j < l. How do I do this most efficiently?
You can use bsxfun to do it efficiently:
J = (1:size(A,1)).';
K = 1:size(A,2);
L = reshape(1:size(A,3),1,1,[]);
A(bsxfun(#or,bsxfun(#lt,J,K),bsxfun(#lt,J,L))) = NaN;
In MATLAB r2016b or Octave you can simply write:
J = (1:size(A,1)).';
K = 1:size(A,2);
L = reshape(1:size(A,3),1,1,[]);
A(J<K|J<L)=NaN;
Result of a test on a matrix A = rand(500,400,300):
________________________________
| METHOD | MEMORY | SPEED |
|==========|==========|==========|
| MESHGRID | 1547 MB | 1.24 Secs|
|----------|----------|----------|
| BSXFUN | 57 MB | 0.18 Secs|
|__________|__________|__________|
Use fancy vecotization:
% this may be memory expensive for big matrices:
[j,k,l]=meshgrid(1:size(A,1),1:size(A,2),1:size(A,3));
% Tada!
A(j<k | k<l)=NaN;
If you do not have enough RAM (or do not want to use it for this), then the best option is just loopy:
for jj=1:size(A,1)
for k=1:size(A,2)
for l=1:size(A,3)
if (jj<k | k<l)
A(jj,k,l)=NaN;
end
end
end
end
This will likely be slower, but doesn't need any extra memory.
I have a matrix (c) of 10x10 (M = 10) elements in which I divide the matrix by rows to be executed by 5 different processes (slaves = 5) with each process corresponding to 2 rows of that matrix.
offset = 0;
rows = (M / slaves);
MPI_Send(&c[offset][0], rows*M, MPI_DOUBLE, id_slave,0,MPI_COMM_WORLD);
offset= offset+rows;
Now I want to divide the matrix but by columns. I did the test as follows by changing array indices but not working:
MPI_Send(&c[0][offset], rows*M, MPI_DOUBLE, id_slave,0,MPI_COMM_WORLD);
Do you know how to do it? Thank you.
You are using the wrong datatype. As noted by Jonathan Dursi, you need to create a strided datatype that tells MPI how to access the memory in such a way that it matches the data layout of a column or a set of consecutive columns.
In your case, instead of
MPI_Send(&c[0][offset], rows*M, MPI_DOUBLE, id_slave, 0, MPI_COMM_WORLD);
you have to do:
MPI_Datatype dt_columns;
MPI_Type_vector(M, rows, M, MPI_DOUBLE, &dt_columns);
MPI_Type_commit(&dt_columns);
MPI_Send(&c[0][offset], 1, dt_columns, id_slave, 0, MPI_COMM_WORLD);
MPI_Type_vector(M, rows, M, MPI_DOUBLE, &dt_columns) creates a new MPI datatype that consists of M blocks of rows elements of MPI_DOUBLE each with the heads of the consecutive blocks M elements apart (stride M). Something like this:
|<------------ stride = M ------------->|
|<---- rows --->| |
+---+---+---+---+---+---+---+---+---+---+--
| x | x | x | x | | | | | | | ^
+---+---+---+---+---+---+---+---+---+---+ |
| x | x | x | x | | | | | | | |
+---+---+---+---+---+---+---+---+---+---+
. . . . . . . . . . . M blocks
+---+---+---+---+---+---+---+---+---+---+
| x | x | x | x | | | | | | | |
+---+---+---+---+---+---+---+---+---+---+ |
| x | x | x | x | | | | | | | v
+---+---+---+---+---+---+---+---+---+---+--
>> ------ C stores such arrays row-wise ------ >>
If you set rows equal to 1, then you create a type that corresponds to a single column. This type cannot be used to send multiple columns though, e.g., two columns, as MPI will look for the second one there, where the first one ends, which is at the bottom of the matrix. You have to tell MPI to pretend that a column is just one element wide, i.e. resize the datatype. This can be done using MPI_Type_create_resized:
MPI_Datatype dt_temp, dt_column;
MPI_Type_vector(M, 1, M, MPI_DOUBLE, &dt_temp);
MPI_Type_create_resized(dt_temp, 0, sizeof(double), &dt_column);
MPI_Type_commit(&dt_column);
You can use this type to send as many columns as you like:
// Send one column
MPI_Send(&c[0][offset], 1, dt_column, id_slave, 0, MPI_COMM_WORLD);
// Send five columns
MPI_Send(&c[0][offset], 5, dt_column, id_slave, 0, MPI_COMM_WORLD);
You can also use dt_column in MPI_Scatter[v] and/or MPI_Gather[v] to scatter and/or gather entire columns.
The problem with your code is the following:
your c array is continuous in memory, and in C it stored row-major order, and the dividing it by row like you do will just add constant offset from the beginning.
and the way you are going to divide it by columns just gives you wrong offset.
You can imagine it for 3x3 matrix and 3 slave processes:
a[3][3] = {{a00 a01 a02},
{a10 a11 a12},
{a20 a21 a22}}
which is actually in memory looks like:
A = {a00,a01,a02,a10,a11,a12,a20,a21,a22}
For example we want to send data to CPU with id = 1. In this case a[1][0] will point you to the forth element of A and the a[0][1] will point you to the second element of A. And the in both cases you just send the rows*M elements from the specific point in A.
In first case it will be:
a10,a11,a12
And in second case:
a01,a02,a10
One of the way to solve things you want is to transpose your matrix and the send it.
And also it is much natural to use MPI_Scatter than MPI_Send for this problem,
something like it explained here: scatter
I have given an array of integers of length up to 10^5 & I want to do following operation on array.
1-> Update value of array at any position i . (1 <= i <= n)
2-> Get products of number at indexes 0, X, 2X, 3X, 4X.... (J * X <= n)
Number of operation will be up to 10^5.
Is there any log n approach to answer query and update values.
(Original thought is to use Segment Tree but I think that it is not needed...)
Let N = 10^5, A:= original array of size N
We use 0-based notation when we saying indexing below
Make a new array B of integers which of length up to M = NlgN :
First integer is equal to A[0];
Next N integers is of index 1,2,3...N of A; I call it group 1
Next N/2 integers is of index 2,4,6....; I call it group 2
Next N/3 integers 3,6,9.... I call it group 3
Here is an example of visualized B:
B = [A[0] | A[1], A[2], A[3], A[4] | A[2], A[4] | A[3] | A[4]]
I think the original thoughts can be used without even using Segment Tree..
(It is overkill when you think for operation 2, we always will query specific range on B instead of any range, i.e. we do not need that much flexibility and complexity to maintain the data structure)
You can create the new array B described above, also create another array C of length M, C[i] := products of Group i
For operation 1 simply use O(# factors of i) to see which Group(s) you need to update, and update the values in both B and C (i.e. C[x]/old B[y] *new B[y])
For operation 2 just output corresponding C[i]
Not sure if I was wrong but this should be even faster and should pass the judge, if the original idea is correct but got TLE
As OP has added a new condition: for operation 2, we need to multiply A[0] as well, so we can special handle it. Here is my thought:
Just declare a new variable z = A[0], for operation 1, if it is updating index 0, update this variable; for operation 2, query using the same method above, and multiply by z afterwards.
I have updated my answer so now I simply use the first element of B to represent A[0]
Example
A = {1,4,6,2,8,7}
B = {1 | 4,6,2,8,7 | 6,8 | 2 | 8 | 7 } // O(N lg N)
C = {1 | 2688 | 48 | 2 | 8 | 7 } // O (Nlg N)
factorization for all possible index X (X is the index, so <= N) // O(N*sqrt(N))
opeartion 1:
update A[4] to 5: factors = 1,2,4 // Number of factors of index, ~ O(sqrt(N))
which means update Group 1,2,4 i.e. the corresponding elements in B & C
to locate the corresponding elements in B & C maybe a bit tricky,
but that should not increase the complexity
B = {1 | 4,6,2,5,7 | 6,5 | 2 | 5 | 7 } // O(sqrt(N))
C = {1 | 2688 | 48/8*5 | 2 | 8/8*5 | 7 } // O(sqrt(N))
update A[0] to 2:
B = {2 | 4,6,2,5,7 | 6,5 | 2 | 5 | 7 } // O(1)
C = {2 | 2688/8*5 | 48/8*5 | 2 | 8/8*5 | 7 } // O(1)
// Now A is actually {2,4,6,2,5,7}
operation 2:
X = 3
C[3] * C[0] = 2*2 = 4 // O(1)
X = 2
C[2] * C[0] = 30*2 = 60 // O(1)