I'm working on a project where I have a number of arrays of the same size in the 1st, 2nd and 3rd dimension, although the sizes may vary in the 4th dimension.
I would like to group these arrays by constructing a pointer which concatenates these arrays.
To make this less abstract, let's say I have 2 arrays:
A (size: N1 x N2 x N3 x N4a)
B (size: N1 x N2 x N3 x N4b)
in previous versions of the project these arrays where copied to an array C of size N1 x N2 x N3 x (N4a + N4b) which would then be passed to a subroutine to perform ffts on this array.
I would like to avoid this copying operation and construct a pointer p which would contain the same data as the array C in the previous version but without the explicit copying and additional memory allocation.
Is this possible in Fortran?
No. A pointer cannot point across two otherwise independent objects like that at the same time.
Depending on your situation, what might be workable is to start with an array that is of dimension (N1,N2,N3,N4a+N4b) and then make A and B associated (pointer, storage or argument) to the relevant parts of that initial big array in some way.
REAL, TARGET :: c(N1,N2,N3,N4a+N4b)
REAL, POINTER :: a(:,:,:,:)
REAL, POINTER :: b(:,:,:,:)
a => c(:,:,:,:n4a)
b => c(:,:,:,n4a+1:)
! Go forth and do things with a and b.
! Then later do things with c.
In the dark times, before Fortran had dynamic memory allocation of any sort, this sort of one-array-to-rule-them-all that then got divvied out was in common usage.
Related
Suppose I have n sorted integer arrays (a_1, ..., a_n, there may be duplicated elements in a single array), and T is a threshold value between 0 and 1. I would like to find all pairs of arrays the similarity of which is larger than T. The similarity of array a_j w.r.t. array a_i is defined as follows:
sim(i, j) = intersection(i, j) / length(i)
where intersection(i, j) returns the number of elements shared in a_i and a_j, and length(i) returns the length of array a_i.
I can enumerate all pairs of arrays and compute the similarity value, but this takes too much time for a large n (say n=10^5). Is there any data structure, pruning strategy, or other techniques that can reduce the time cost of this procedure? I'm using Java so the technique should be easily applicable in Java.
There are (n^2 - n)/2 pairs of arrays. If n=10^5, then you have to compute the similarity of 5 billion pairs of arrays. That's going to take some time.
One potential optimization is to shortcut your evaluation of two arrays if it becomes clear that you won't reach T. For example, if T is 0.5, you've examined more than half of the array and haven't found any intersections, then it's clear that that pair of arrays won't meet the threshold. I don't expect this optimization to gain you much.
It might be possible to make some inferences based on prior results. That is, if sim(1,2) = X and sim(1,3) < T, there's probably a value of X (likely would have to be very high) at which you can say definitively that sim(2,3) < T.
I have the same problem described and solved in this thread, where julia loops were considerably slowed down due to the repeated access to memory using a not optimal variable type. In my case however I have tensors (with dimensions higher than two) stored as multidimensional Array which need to be summed over and over in a for loop, something like:
function randomTensor(M,N)
T = fill( zeros( M, M, M, M) , N)
for i in 1 : N
Ti = rand( M, M, M, M)
T[i] += Ti
end
return T
end
I'm new to Julia (in fact I'm using it just for a specific project for which I need some julia modules), so I have only a general understanding of variable types and so forth, and till now I've not been able to define my tensors if not as general arrays. I would like a solution as the one proposed in the aforementioned link, where they use Matrix{Float64}[] to define variables, but apparently that works only for 1 or 2-dimensional arrays. Cannot use Tuples either, as I need to sum over the values. Any tip for different implementations?
Thank you all in advance.
Edit: the sample code I posted is just an example featuring the same structure of my code, I'm not interested in generating and summing random numbers : )
In other words, my question is: is there a better variable type than Array{Float, N} to do operations (e.g. tensor times scalar...) or that have "better access" to memory?
It's quite unclear what the question is. Your function is equivalent to:
randomTensor(M,N) = [rand(M, M, M, M) for i in 1:N]
Are you saying that this is too slow? The majority of the time should be in creating the random numbers and allocating the memory for the output.
Depending on the size of M, you might notice some advantage from using .+= instead of +=. The reason is that x += y is exactly equivalent to x = x + y, including allocating a new array to hold the result. In contrast, x .+= y is in-place. You can see that for yourself in the following way:
julia> a = [1,2,3]
3-element Vector{Int64}:
1
2
3
julia> pointer(a)
Ptr{Int64} #0x00007fa007d2c980
julia> a += a
3-element Vector{Int64}:
2
4
6
julia> pointer(a)
Ptr{Int64} #0x00007fa02a06eac0 # different pointer means new memory was allocated
julia> a .+= a
3-element Vector{Int64}:
4
8
12
julia> pointer(a)
Ptr{Int64} #0x00007fa02a06eac0 # same pointer means memory re-use
In Matlab I have an array v of length m, a matrix of order n and a function F that takes as an input a single matrix and outputs a number. Starting from v I would like to apply the function to the whole array of matrices whose i-th element consists of a matrix M_i whose entries are obtained by multiplicating all the entries of M by v_i. The output would be itself an array of length n.
As far as I can see there are two ways of achieving this:
Looping on all i=1:n, computing F on all the M_is and store all the corresponding values in an array
Defining a 3-array structure that contains all the matrices M_i and correspondingly extending the function F as to act on 3-arrays instead of matrices. However this entails overloading some matrix operators and functions (transpose, exponential, logarithm, square root, inverse etc...) as to formally handle a 3-array.
I have done the simpler option 1. It takes a long time to execute. Number 2 promises to be faster- However, I am not sure if this is the case, and I am not familiar with overloading operators on Matlab. In particular: how to extend a matrix operator to a 3-array in such a way that it performs the related function on all of its entries.
A for loop is probably no slower than vectorising this, especially for larger problems where memory starts to limit speed. Nevertheless, here are two ways of doing it:
M=rand(3,3,5) % I'm using a 3x3x5 matrix
v=1:5
F=#sum % This is the function
M2=bsxfun(#times,M,permute(v.',[2 3 1])) % Multiply the M(:,:,i) matrix by v(i)
R=arrayfun(#(t) F(M2(:,:,t)),(1:size(M,3)).','UniformOutput',false) % applies the function F to the resulting matrices
cell2mat(R) % Convert from cell array to matrix, since my F function returns row vectors
R2=zeros(size(M,3),size(M,1)); % Initialise R2
for t=1:size(M,3)
R2(t,:)=F(M(:,:,t)*v(t)); % Apply F to M(:,:,i)*v(i)
end
R2
You should do some testing to see which will be more efficient for your actual problem. The vectorised version should be faster for small problems, but use more memory, whereas the for loop will be slower for small problems but use less memory, and so could be faster on larger problems.
How can one combine bunch of large allocatable vectors in an array in Fortran? I'd like to avoid copying and reshaping using reshape as the arrays are large. The effect I want to achieve is just like Fortran's equivalence, to illustrate:
program test_equiv
integer x(10), y(10), z(10), xyz(10,3)
equivalence (x, xyz(1,1))
equivalence (y, xyz(1,2))
equivalence (z, xyz(1,3))
x = 1
y = 2
z = 3
! and we can use just normal array syntax
print *, xyz(3,:)
end program
This will, however, not work with allocatable arrays. If it's about accessing vectors of matrix it can be easily implemented via pointers. But how can one combine vectors in a two dimensional array? Till now I came only to array of pointers which has problems:
program test_arofpntrs
implicit none
integer :: i
integer, allocatable, target :: xs(:), ys(:), zs(:)
type t_p_xs
integer, pointer :: p_xs(:)
end type t_p_xs
type(t_p_xs), allocatable :: coords(:)
allocate(coords(3), xs(10), ys(10), zs(10))
xs = 1
ys = 2
zs = 3
coords(1) % p_xs => xs
coords(2) % p_xs => ys
coords(3) % p_xs => zs
print *, coords(1) % p_xs(:)
! this fails:
!print *, coords(:) % p_xs(1)
end program
This is ugly and one cannot access xs(i), ys(i), zs(i). Is it possible to do what I want without copies?
This will not be possible, if you start with separate 1D arrays. The allocatable arrays may be anywhere in the memory. Although Fortran arrays don't have to be contiguous, there has to be some system of strides.
! this fails:
!print *, coords(:) % p_xs(1)
is prohibited by the Fortran standard, because one can not calculate the address of the next element simply. The 1D arrays are even not guaranteed to have the same length.
Also reshape as such does not have to be inefficient, it may just syntactically help with indexing, but don't touch the data at all.
Pointers are a great tool and may help here. You would have to use your 1D arrays bit differently. For example allocate a long 1D array and have 1D pointers for parts of it and a 2D pointer as a whole, or better the other way round:
real,allocatable,target :: xyz(:,:)
real,pointer :: x(:),y(:),z(:)
allocate(xyz(1:10,1:3))
x => xyz(:,1)
y => xyz(:,2)
z => xyz(:,3)
Even the other order of indexes is possible, i.e., xyz(3,10); x => xyz(1,:).
Also you can do
long1D(1:size(xyz)) => xyz
but note it is a Fortran 2008 feature in this direction (2003 otherwise).
In GSL a real n * m matrix M is represented internally as an array of size n*m. To access the (i,j) element of M, internally GSL has to access the (i-1) * n + j - 1 location of the array, which involves integer multiplications and additions.
In Numerical Recipes for C, they recommend the alternative method of declaring an array of n pointers, each pointing to an array of m numbers. Then to access the (i,j) element, one puts M[i-1][j-1]. They claim that this is more efficient because it avoids the integer multiplication. The downside is that one has to initialize each pointer separately.
I am wondering, what are the advantages/disadvantages of each approach?
In C:
#define n 2
#define m 3
int M[n*m];
is the same as
int M[n][m];
in C matrices are said to be stored in row-major order
http://en.wikipedia.org/wiki/Row-major_order
In C,
M[1][2]
is the same as
*(M + 1*m + 2) // if M is define as M[n][m]
You could define M as an array of n pointers, but you still have to put the data somewhere and the best place is probably a 2D array. I would suggest:
int M[n][m];
int* Mrows[n] = {M[0], M[1]};
You can then do a direct offset into rows to get to the row you want. Then:
Mrows[1][2]
is the same as
*((*(Mrows + 1)) + 2)
Its more work for the programmer and probably only worth it if you want to go really fast. In that case you may want to look into more optimizations such as specific machine instructions. Also, depending on your algorithm, you may be able to just use + operations (like if you are iterating over the matrix)