While compiling the lapack system from the f2cl library, SBCL drops into the low-level debugger with this error message about heap exhaustion:
Heap exhausted during garbage collection: 0 bytes available, 64 requested.
Gen Boxed Unboxed LgBox LgUnbox Pin Alloc Waste Trig WP GCs Mem-age
0 0 0 0 0 0 0 0 26843545 0 0 0.0000
1 26205 12796 0 0 9 1277771072 213696 767547401 39001 1 1.3696
2 14599 27405 117 18 88 1114241264 266569488 2000000 42139 0 0.8257
3 0 0 0 0 0 0 0 2000000 0 0 0.0000
4 0 0 0 0 0 0 0 2000000 0 0 0.0000
5 0 0 0 0 0 0 0 2000000 0 0 0.0000
6 449 220 64 47 0 24788848 770192 2000000 780 0 0.0000
7 0 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes allocated = 2416801184
Dynamic-space-size bytes = 2684354560
GC control variables:
*GC-INHIBIT* = true
*GC-PENDING* = true
*STOP-FOR-GC-PENDING* = false
fatal error encountered in SBCL pid 31717(tid 0x7fbb53033280):
Heap exhausted, game over.
I found postings on the maxima mailing list that recommend increasing the heap size with the --dyanamic-space-size option for sbcl, so I tried that. Even when I gave it as much memory as my RAM (~7.44 GB) it still exhausted the heap, albeit after a much longer time. I'm not quite sure where to go from here. Any ideas?
System Specs: Arch Linux and SBCL 1.4.16.
High memory consumption during compilation may happen because you are asking a high amount of debugging information. I tried compiling lapack, which failed too. It happens that in my ~/.sbclrc, I have:
(sb-ext:restrict-compiler-policy 'debug 3)
(sb-ext:restrict-compiler-policy 'safety 3)
In a fresh SBCL, the compiler policy is as follows:
* (describe-compiler-policy)
Basic qualities:
COMPILATION-SPEED = 1
DEBUG = 3
SAFETY = 3
SPACE = 1
SPEED = 1
INHIBIT-WARNINGS = 1
Dependent qualities:
SB-C::CHECK-CONSTANT-MODIFICATION = 1 -> 3 (yes)
SB-C::TYPE-CHECK = 1 -> 3 (full)
SB-C::CHECK-TAG-EXISTENCE = 1 -> 3 (yes)
SB-C::LET-CONVERSION = 1 -> 0 (off)
SB-C:ALIEN-FUNCALL-SAVES-FP-AND-PC = 1 -> 3 (yes)
SB-C:VERIFY-ARG-COUNT = 1 -> 3 (yes)
SB-C::INSERT-DEBUG-CATCH = 1 -> 3 (yes)
SB-C::RECOGNIZE-SELF-CALLS = 1 -> 0 (no)
SB-C::FLOAT-ACCURACY = 1 -> 3 (full)
SB-C:INSERT-STEP-CONDITIONS = 1 -> 3 (full)
SB-C::COMPUTE-DEBUG-FUN = 1 -> 3 (yes)
SB-C::EVAL-STORE-SOURCE-FORM = 1 -> 3 (yes)
SB-C::PRESERVE-SINGLE-USE-DEBUG-VARIABLES = 1 -> 3 (yes)
SB-C::INSERT-ARRAY-BOUNDS-CHECKS = 1 -> 3 (yes)
SB-C::STORE-XREF-DATA = 1 -> 3 (yes)
SB-C:STORE-COVERAGE-DATA = 1 -> 0 (no)
SB-C::INSTRUMENT-CONSING = 1 -> 1 (no)
SB-C::STORE-CLOSURE-DEBUG-POINTER = 1 -> 0 (no)
SB-KERNEL:ALLOW-NON-RETURNING-TAIL-CALL = 1 -> 0 (no)
In particular, the policy is restricted as follows:
* (restrict-compiler-policy)
((SAFETY . 3) (DEBUG . 3))
Those are the minimal amount of debugging and safety required, which is thus at least 3 for both.
When I remove the debug restriction, with the following line:
* (restrict-compiler-policy 'debug)
((SAFETY . 3))
... then compiling the lapack system works, without changing the dynamic space size. Adding debugging information is fine when developing, and I'd encourage anyone to keep them at all time because you never know when you'll need to debug something, but in that case it is simply better to turn them off.
Just a quick note to add: that's a very old version of SBCL. If you can, you may want to consider upgrading to the latest.
Related
I basically want to use the function ismember, but for a range. For example, I want to know what data points in array1 are within n distance to array2, for each element in array2.
I have the following:
array1 = [1,2,3,4,5]
array2 = [2,2,3,10,20,40,50]
I want to know what values in array2 are <= 2 away from array1:
indices(1,:) (where array1(1) = 1) = [1 1 1 0 0 0 0]
indices(2,:) (where array1(2) = 2) = [1 1 1 0 0 0 0]
indices(3,:) (where array1(3) = 3) = [1 1 1 0 0 0 0]
indices(4,:) (where array1(4) = 4) = [1 1 1 0 0 0 0]
indices(5,:) (where array1(5) = 5) = [0 0 1 0 0 0 0]
Drawbacks:
My array1 is 496736 elements, my array2 is 9268 elements, so aI would rather not use a loop.
Looping is a valid option here. Intialise an array output to the size of array1 X array2, then loop over all elements in array1 an subtract array2 from that, then check whether the absolute value is less than or equal to 2:
array1 = [1,2,3,4,5];
array2 = [2,2,3,10,20,40,50];
output = zeros(numel(array1), numel(array2),'logical');
for ii = 1:numel(array1)
output(ii,:) = abs(array1(ii)-array2)<=2;
end
output =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
i.e. loops are not the problem.
Thanks to Rahnema1's suggestion, you can initialise output directly as a logical matrix:
output = zeros(numel(array1),numel(array2),'logical');
whose size is just 4.3GB.
On timings: Hans' code runs in a matter of seconds for array1 = 5*rand(496736,1); array2 = 25*rand(9286,1);, the looped solution takes about 15 times longer. Both solutions are equal to one another. obcahrdon's ismembertol solution is somewhere in between on my machine.
On RAM usage:
Both implicit expansion, as per Hans' answer, as well as the loop suggested in mine work with just 4.3GB RAM on your expanded problem size (496736*9286)
pdist2 as per Luis' answer and bsxfun as per Hans' on the other hand try to create an intermediate double matrix of 34GB (which doesn't even fit in my RAM, so I cannot compare timings).
obchardon's ismembertol solution outputs a different form of the solution, and takes ~5.04GB (highly dependent on the amount of matches found, the more, the larger this number will be).
In general this leads me to the conclusion that implicit expansion should be your option of choice, but if you have R2016a or earlier, ismembertol or a loop is the way to go.
Using implicit expansion, introduced in MATLAB R2016b, you can simply write:
abs(array1.' - array2) <= 2
ans =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
For earlier MATLAB versions, you can get this using the bsxfun function:
abs(bsxfun(#minus, array1.', array2)) <= 2
ans =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
Hope that helps!
P.S. On the "MATLAB is slow for loops" myth, please have a look at that blog post for example.
EDIT: Please read Adriaan's answer on the RAM consumption using this and/or his approach!
If you have the Statistics Toolbox, you can use pdist2 to compute the distance matrix:
result = pdist2(array1(:), array2(:)) <= 2;
As noted by #Adriaan, this is not efficient for large input sizes. In that case a better approach is a loop with preallocation of the logical matrix output, as in his answer.
You can also use ismembertol with some specific option:
A = [1,2,3,4,5];
B = [2,2,3,10,20,40,5000];
tol = 2;
[~,ind] = ismembertol([A-tol;A+tol].',[B-tol;B+tol].',tol, 'ByRows', true, ...
'OutputAllIndices', true, 'DataScale', [1,Inf])
It will create a 5x1 cell array containing the corresponding linear indice
ind{1} = [1,2,3]
ind{2} = [1,2,3]
...
ind{5} = [3]
In this case using linear indices instead of logical indices will greatly reduce the memory consumption.
I have an array A of 1s and 0s and want to see if the larger array of bits B contains those bits in that exact order?
Example: A= [0 1 1 0 0 0 0 1]
B= [0 1 0 0 1 1 0 0 0 0 1 0 1 0 1]
would be true as A is contained in B
Most solutions I have found only determine if a value IS contained in another matrix, this is no good here as it is already certain that both matrices will be 1s and 0s
Thanks
One (albeit unusual) option, since you're dealing with integer values, is to convert A and B to character arrays and use the contains function:
isWithin = contains(char(B), char(A));
There are some obtuse vectorized ways to to do this, but by far the easiest, and likely just as efficient, is to use a loop with a sliding window,
A = [0 1 1 0 0 0 0 1];
B = [0 1 0 0 1 1 0 0 0 0 1 0 1 0 1];
vec = 0:(numel(A)-1);
for idx = 1:(numel(B)-numel(A)-1)
if all(A==B(idx+vec))
fprintf('A is contained in B\n');
break; % exit the loop as soon as 1 match is found
end
end
Or if you want to know the location(s) in B (of potentially multiple matches) then,
A = [0 1 1 0 0 0 0 1];
B = [0 1 0 0 1 1 0 0 0 0 1 0 1 0 1];
C = false(1,numel(B)-numel(A)-1);
vec = 0:(numel(A)-1);
for idx = 1:numel(C)
C(idx) = all(A==B(idx+vec));
end
if any(C)
fprintf('A is contained in B\n');
end
In this case
>> C
C =
1×6 logical array
0 0 0 1 0 0
You can use the cross-correlation between two signals for this, as a measure of local similarity.
For achieving good results, you need to shift A and B so that you don't have the value 0 any more. Then compute the correlation between the two of them with conv (keeping in mind that the convolution is the cross-correlation with one signal flipped), and normalize with the energy of A so that you get a perfect match whenever you get the value 1:
conv(B-0.5, flip(A)-0.5, 'valid')/sum((A-0.5).^2)
In the normalization term, flipping is removed as it does not change the value.
It gives:
[0 -0.5 0.25 1 0 0 -0.25 0]
4th element is 1, so starting from index equal to 4 you get a perfect match.
So I have this matrix A, which is made of 1 and zeros, I have about 10 to 14 white spots of many pixels, but I want only 1 white pixel/centers coordinate for every cluster of white, how do I calculate how many cluster there are and their centers.
Try to imagine the matrix A as the night sky with white starts in black sky and how to I count the stars and the stars centers, plus the star are made of cluster of white pixels.
also the clusters are not all exactly the same size.
Here is some code using bwlabel and/or regioprops, which are used to identify connected components in a matrix and a buch of other properties, respectively. I think it suits your problem quite well; however you might want to adapt my code a bit as its more of a starting point.
clear
clc
%// Create dummy matrix.
BW = logical ([ 1 1 1 0 1 1 1 0
1 1 1 0 1 1 1 0
1 1 1 0 1 1 1 0
0 0 0 0 0 0 0 0
0 0 0 0 0 1 1 0
1 1 1 1 0 1 1 0
1 1 1 1 0 1 1 0
1 1 1 1 0 0 0 0]);
%// Identify clusters.
L = bwlabel(BW,4)
Matrix L looks like this:
L =
1 1 1 0 3 3 3 0
1 1 1 0 3 3 3 0
1 1 1 0 3 3 3 0
0 0 0 0 0 0 0 0
0 0 0 0 0 4 4 0
2 2 2 2 0 4 4 0
2 2 2 2 0 4 4 0
2 2 2 2 0 0 0 0
Here you have many ways to locate the center of the clusters. The first one uses the output of bwlabel to find each cluster and calculate the coordinates in a loop. It works and its didactic but it's a bit long and not so efficient. The 2nd method, as mentioned by #nkjt, uses regionprops which does exactly what you want using the 'Centroid' property. So here are the 2 methods:
Method 1: a bit complicated
So bwlabel identified 4 clusters, which makes sense. Now we need to identify the center of each of those clusters. My method could probably be simplified; but I'm a bit out of time so fell free to modify it as you see fit.
%// Get number of clusters
NumClusters = numel(unique(L)) -1;
Centers = zeros(NumClusters,2);
CenterLinIdices = zeros(NumClusters,1);
for k = 1:NumClusters
%// Find indices for elements forming each cluster.
[r, c] = find(L==k);
%// Sort the elements to know hot many rows and columns the cluster is spanning.
[~,y] = sort(r);
c = c(y);
r = r(y);
NumRow = numel(unique(r));
NumCol = numel(unique(c));
%// Calculate the approximate center of the cluster.
CenterCoord = [r(1)+floor(NumRow/2) c(1)+floor(NumCol/2)];
%// Actually this array is not used here but you might want to keep it for future reference.
Centers(k,:) = [CenterCoord(1) CenterCoord(2)];
%// Convert the subscripts indices to linear indices for easy reference.
CenterLinIdices(k) = sub2ind(size(BW),CenterCoord(1),CenterCoord(2));
end
%// Create output matrix full of 0s, except at the center of the clusters.
BW2 = false(size(BW));
BW2(CenterLinIdices) = 1
BW2 =
0 0 0 0 0 0 0 0
0 1 0 0 0 1 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 0
0 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0
Method 2 Using regionprops and the 'Centroid' property.
Once you have matrix L, apply regionprops and concatenate the output to get an array containing the coordinates directly. Much simpler!
%// Create dummy matrix.
BW = logical ([ 1 1 1 0 1 1 1 0
1 1 1 0 1 1 1 0
1 1 1 0 1 1 1 0
0 0 0 0 0 0 0 0
0 0 0 0 0 1 1 0
1 1 1 1 0 1 1 0
1 1 1 1 0 1 1 0
1 1 1 1 0 0 0 0]);
%// Identify clusters.
L = bwlabel(BW,4)
s = regionprops(L,'Centroid');
CentroidCoord = vertcat(s.Centroid)
which gives this:
CentroidCoord =
2.0000 2.0000
2.5000 7.0000
6.0000 2.0000
6.5000 6.0000
Which is much simpler and gives the same output once you use floor.
Hope that helps!
This question already has answers here:
Find all combinations of a given set of numbers
(9 answers)
Closed 8 years ago.
I hope all you are doing great.
I have an interesting question, which has stuck me. Its about generating combinations in a precise order.
For example i have 4 variables(can be vary) and these 4 variables has some limit to increase for example in this case 2. so i want to generate 2d matrix in a order as:
0 0 0 0
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
1 1 0 0
1 0 1 0
1 0 0 1
0 1 1 0
0 1 0 1
0 0 1 1
1 1 1 0
0 1 1 1
1 1 1 1
2 0 0 0
0 2 0 0
0 0 2 0
0 0 0 2
2 1 0 0
2 0 1 0
......
......
and so on.
the number of variables ( in this case 4 ) can be varied and also the maximum limit ( in this case 4) can be varied.
Even i have also find all possible combinations but i am not able to arrange them in this sequence.
it would be great if somebody gives an answer.
cheers!
I'm going to assume you've got n variables, each of which is allowed to range from 0 to b-1. What you want is just counting n-digit numbers in base b. For example, if n = 2 and b = 3, then the sequence you want to produce is
00
01
02
10
11
12
20
21
22
To implement this, write a loop something like the following: (warning: untested code)
def inc(v, b):
for i in range(len(v)):
v[i] = v[i] + 1
if v[i] < b:
break
v[i] = 0
def is_zero(v):
for i in range(len(v)):
if v[i] != 0:
return False
return True
v = [0, 0, 0]
b = 3
while True:
print(v)
inc(v, b)
if is_zero(v):
break
If you look carefully at how this works, you should see how to generalize it if your variables have different upper bounds.
I have a training set that has input and outputs in this way:
Input:
0.832 64.643
0.818 78.843
1.776 45.049
0.597 88.302
1.412 63.458
1.468 49.535
1.985 33.387
2.073 30.279
1.431 55.231
1.116 68.521
1.617 44.362
2.159 66.512
Output:
0 0 1
0 0 1
0 1 0
0 0 1
0 0 1
1 0 0
0 0 1
1 0 0
1 0 0
0 0 1
0 0 1
0 1 0
1 0 0
1 0 0
0 1 0
0 1 0
I need to implement one linear layer neural network that can represent the data set best in MATLAB. What would be the algorithm to do it in MATLAB?
The target output is "1 for a particular class that the corresponding input belongs to and "0 for the remaining 2 outputs.
Consider this example of training a feed-forward ANN of one hidden layer (with 3 nodes).
Since your data seems to have more output points than input, I'm using a demo dataset, but the idea is the same:
%# load sample data
laod simpleclass_dataset
input = simpleclassInputs; %# 2x1000, 2-dimensional points
output = simpleclassTargets; %# 4x1000, 4 classes
%# split data into training/testing sets
trainInd = 1:500;
testInd = 501:1000;
%# create ANN and initialize network weights
net = newpr(input, output, 3);
net = init(net);
net.trainParam.epochs = 25; %# max number of iterations
%# learn net weights from training data
net = train(net, input(:,trainInd), output(:,trainInd));
%# predict output of net on testing data
pred = sim(net, input(:,testInd));
%# classification confusion matrix
[err,cm] = confusion(output(:,testInd), pred);
The output is:
err =
0.075075
cm =
81 0 0 0
0 82 0 0
9 0 52 16
0 0 0 93
Obviously you will need access to the Neural Network Toolbox.