how to generate an image in the format of the mnist database? - arrays

I need to make a handwritten image to be tested with a neural network in Matlab. When I see the data contained in the training images from the MNIST I see that it is an array of different gray scales like:
Columns 343 through 351
0 -0.0240 0.4002 0.6555 0.0235 -0.0062 0 0 0
Columns 352 through 360
0 0 0 0 0 0 0 0 0
Columns 361 through 369
0 0 0 -0.0079 0.1266 0.3272 -0.0233 0.0005
corresponding to a 20x20 image, unrolled into a 1*400 dimensional array.
I have downloaded an image in jpeg format and did the following:
im=imread('image.jpg');
gi=rgb2gray(im);
gi=gi(:);
gi=gi';
that generates me an array gi that says <1*400 uint8>, the last part of uint8 does not appear in the MNIST samples when I put it in Matlab. When I check it up my array it appear the following values:
Columns 289 through 306
58 105 128 133 142 131 76 21 1 0 3 0 2 4 17 12 7 0
Columns 307 through 324
1 15 42 75 97 105 98 73 31 4 1 0 0 0 0 2 4 3
Columns 325 through 342
0 0 1 4 21 37 55 59 46 26 9 0 0 0 0 0 0 0
Columns 343 through 360
1 1 0 0 0 1 7 14 21 21 14 5 0 0 0 0 0 0
Columns 361 through 378
0 0 0 0 0 0 0 0 0 1 2 1 0 0 0 2 0 0
when I visualize them all is fine, but when I want to run my program the following message appears:
??? Error using ==> mtimes
MTIMES is not fully supported for integer classes. At least one input must be scalar.
Error in ==> predict at 15
h1 = sigmoid([ones(m, 1) X] * Theta1');
Error in ==> ex4 at 241
pred = predict(Theta1, Theta2, gi);
situation that does not occur when I test my program even with one random sample ofc the MNIST data; any help?

You could try something like this:
imfile = 'image.jpg';
im = double(rgb2gray(imread(imfile))); % double and convert to grayscale
im = imresize(im,[20,20]); % change to 20 by 20 dimension
im = im(:); % unroll matrix to vector
im = im./max(im);
Note the MNIST dataset is intended to be a good dataset to require minimal preprocessing and the images were actually originally black and white (bilevel) whereas you are using color image. Also they do normalisation and other preprocessing to make nice 28 by 28 image dataset, my brief snippet of code above is unlikely to be anywhere near as good as MNIST dataset and is just intended to attempt to fix your error.
Your specific error is likely because you don't use double().
You may also get further errors because your code needs right dimensions, which can be achieved using imresize.
More information on MNIST dataset here:
http://yann.lecun.com/exdb/mnist/

Related

Matlab: Creating a binned RGB histogram [duplicate]

This question already has an answer here:
Content-Based Image Retrieval and Precision-Recall graphs using Color Histograms in MATLAB
(1 answer)
Closed 7 years ago.
I want to implement the following Matlab function:
function hist = binnedRgbHist(im, numChannelBins)
Given an image im and a number between 1 and 256 numChannelBins, it should create a histogram sized (numChannelBins)^3.
For example, if numChannelBins is 2, it should produce the following 8-sized histogram:
Number of pixels with R < 128, G < 128, B < 128
Number of pixels with R < 128, G < 128, B >= 128
Number of pixels with R < 128, G >= 128, B < 128
Number of pixels with R < 128, G >= 128, B >= 128
Number of pixels with R > 128, G < 128, B < 128
Number of pixels with R > 128, G < 128, B >= 128
Number of pixels with R > 128, G >= 128, B < 128
Number of pixels with R > 128, G >= 128, B >= 128
It is like creating a cube where each axis represents one of (R,G and B), where each axis is divided into 2 bins => Finally there are 8 bins in the cube.
My questions:
It there a built-in function for it?
If not, how is it better to implement it in manners of runtinme using the GPU? Should I better iterate over the pixels once and create the histogram manually, or should I better iterate over the bins and each time count the number of pixels which satisfy the bin's conditions?
accumarray is very suited for this. Let
im: input image;
N: number of bins per color component.
Then
result = accumarray(reshape(permute(ceil(im/255*N), [3 1 2]), 3, []).', 1, [N N N]);
How it works
ceil(im/255*N) quantizes each color vaue to 1, 2, ..., N.
reshape(permute(..., [3 1 2]), 3, []).' transforms the quantized image into a three-column matrix where each row is a pixel and each column is a (quantized) color component.
accumarray(..., 1, [N N N]) considers each row of that matrix as 3D index, and counts how many times each index appears, giving filling indices that don't appear with a 0.
Example 1
Data:
>> N = 2;
>> im = randi(256,4,5,3)
im(:,:,1) =
113 152 157 65 229
138 71 215 39 41
13 108 230 160 153
142 128 125 220 214
im(:,:,2) =
208 215 182 27 230
205 161 8 95 180
225 53 73 129 31
103 97 160 83 255
im(:,:,3) =
242 29 185 89 55
202 225 156 174 96
160 197 35 87 113
244 176 146 85 120
Result:
result(:,:,1) =
1 1
3 4
result(:,:,2) =
2 4
3 2
It can be checked for example that there is only 1 pixel with all R,G,B less than 128.
Example 2
Data:
>> im = repmat(150,20,30,3);
>> N = 4;
Result:
result(:,:,1) =
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
result(:,:,2) =
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
result(:,:,3) =
0 0 0 0
0 0 0 0
0 0 600 0
0 0 0 0
result(:,:,4) =
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
In this case all pixels belong to the same 3D-bin:
I see #Luis Mendo gave a great one-liner solution as I was writing this. In case it provides some deeper intuition, my solution makes use of histcounts and accumarray:
im = randi([1 255],[10,5,3]); %// A random 10-by-5 "image"
numChannelBins = 2;
[~,~,binR]=histcounts(im(:,:,1),[1 ceil((1:numChannelBins)*(255/numChannelBins))]);
[~,~,binG]=histcounts(im(:,:,2),[1 ceil((1:numChannelBins)*(255/numChannelBins))]);
[~,~,binB]=histcounts(im(:,:,3),[1 ceil((1:numChannelBins)*(255/numChannelBins))]);
hist=accumarray([binR(:) binG(:) binB(:)],1,[numChannelBins,numChannelBins,numChannelBins])
Explanation:
the three calls to histcounts bin the red, green, blue pixels separately -- the third output [~,~,binX] of histcounts gives the bin index for each pixel
accumarray accumulates all the unique index triplets

Replacing specific elements in a table with a specific element from a range in APLX

I'm learning a spread of programming languages in a class, and we're working on an APLX project at the moment. A restriction we have to work around is we cannot use If, For, While, etc. No loops or conditionals. I have to be able to take a plane of numbers, ranging 0-7, and replace each number 2 or greater into the depth of that number, and, ideally, change the 1's to 0's. For example:
0100230 => 0000560
I have no idea how I'm supposed to do the replacement with depth aspect, though the change from ones to zeros is quite simple. I'm able to produce the set of integers in a table and I understand how to replace specific values, but only with other specific values, not values that would have to be determined during the function. The depth should be the row depth, rather than the multi-dimensional depth.
For the record this is not the whole of the program, the program itself is a poker dealing and scoring program. This is a specific aspect of the scoring methodology that my professor recommended I use.
TOTALS„SCORE PHAND;TYPECOUNT;DEPTH;ISCOUNT;TEMPS;REPLACE
:If (½½PHAND) = 0
PHAND„DEAL PHAND
:EndIf
TYPECOUNT„CHARS°.¹PHAND
DEPTH„2Þ(½TYPECOUNT)
REPLACE „ 2 3 4 5 6 7
ISCOUNT „ +/ TYPECOUNT
ISCOUNT „ ³ISCOUNT
((1=,ISCOUNT)/,ISCOUNT)„0
©((2=,ISCOUNT)/,ISCOUNT)„1
©TEMPS „ ISCOUNT
Œ„ISCOUNT
Œ„PHAND
You may have missed the first lessons of your prof and it might help to look at at again to learn about vectors and how easy you can work with them - once you unlearned the ideas of other programming languages ;-)
Assume you have a vector A with numbers from 1 to 7:
A←⍳7
A
1 2 3 4 5 6 7
Now, if you wanted to search for values > 3, you'd do:
A>3
0 0 0 1 1 1 1
The result is a vector, too, and you can easily combine the two in lots of operations:
multiplication to only keep values > 0 and replace others with 0:
A×A>3
0 0 0 4 5 6 7
or add 500 to values >3
A+500×A>3
1 2 3 504 505 506 507
or, find the indices of values > 3:
(A>3)×⍳⍴A
0 0 0 4 5 6 7
Now, looking at your q again, the word 'depth' has a specific meaning in APL and I guess you meant something different. Do I understand correctly that you want to replace values > 2 with the ' indices' of these values?
Well, with what I've shown before, this is easy:
A←0 1 0 0 2 3 0
(A≥2)×⍳⍴A
0 0 0 0 5 6 0
edit: looking at multi-dimensional arrays:
let's look into this example:
A←(⍳5)∘.×⍳10
A
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 21 24 27 30
4 8 12 16 20 24 28 32 36 40
5 10 15 20 25 30 35 40 45 50
Now, which numbers are > 20 and < 30?
z←(A>20)∧A<30
z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 0
0 0 0 0 0 1 1 0 0 0
0 0 0 0 1 0 0 0 0 0
Then, you can multiply the values with that boolean result to filter out only the ones satisfying the condition:
A×z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 21 24 27 0
0 0 0 0 0 24 28 0 0 0
0 0 0 0 25 0 0 0 0 0
Or, perhaps you're interested in the column-index of the values?
z×[2]⍳¯1↑⍴z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 7 8 9 0
0 0 0 0 0 6 7 0 0 0
0 0 0 0 5 0 0 0 0 0
NB: this statement might not work in all APL-dialects. Here's another way to formulate this:
z×((1↑⍴z)⍴0)∘.+⍳¯1↑⍴z
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 7 8 9 0
0 0 0 0 0 6 7 0 0 0
0 0 0 0 5 0 0 0 0 0
I hope this gives you some ideas to play with. In general, using booleans to manipulate arrays in mathematical operations is an extremely powerful idea in APL which will take you loooooong ways ;-)
Also, if you'd like to see more of the same, have a look at the FinnAPL Idioms - some useful shorties grown over the years ;-)
edit re. "maintaining untouched values":
going back to example array A:
A←(⍳5)∘.×⍳10
A
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 21 24 27 30
4 8 12 16 20 24 28 32 36 40
5 10 15 20 25 30 35 40 45 50
Replacing values between 20 and 30 with the power 2 of these values, keeping all others unchanged:
touch←(A>20)∧A<30
(touch×A*2)+A×~touch
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 441 576 729 30
4 8 12 16 20 576 784 32 36 40
5 10 15 20 625 30 35 40 45 50
I hope you get the idea...
Or better: ask a new q, as otherwise this would truly take epic dimensions, whereas the idea of stackoverflow is more like "one issue - one question"...

How to populate binary matrices with all the combinations?

I want to have 2^n matrices with all the combinations of 0 and 1 in them. For example, for n=6 (n=#rows x #columns) array{1}=[0 0 0; 0 0 0],array{2}=[0 0 0; 0 0 1]... array{64}=[1 1 1;1 1 1]. I am using MATLAB and I came across with combn.m (M = COMBN(V,N) returns all combinations of N elements of the elements in vector V. M has the size (length(V).^N)-by-N.), dec2bin() but I can't get it quite right. Another idea of mine was to create a large matrix and then split it into 2^n matrices. For instance,for n=6( 2 x 3), i did this M=combn([0 1],3) which gives me:
M =
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
Then, use this M to create a larger matrix like this M2=combn(M,2), but this produces the wrong results. However, if i concatenate M row like this:
M=combn([000;010;100;001;110;011;101;111],2)' I get something closer to what I expect i.e
M =
Columns 1 through 21
0 0 0 0 0 0 0 0 10 10 10 10 10 10 10 10 100 100 100 100 100
0 10 100 1 110 11 101 111 0 10 100 1 110 11 101 111 0 10 100 1 110
Columns 22 through 42
100 100 100 1 1 1 1 1 1 1 1 110 110 110 110 110 110 110 110 11 11
11 101 111 0 10 100 1 110 11 101 111 0 10 100 1 110 11 101 111 0 10
Columns 43 through 63
11 11 11 11 11 11 101 101 101 101 101 101 101 101 111 111 111 111 111 111 111
100 1 110 11 101 111 0 10 100 1 110 11 101 111 0 10 100 1 110 11 101
Column 64
111
111
where I can get each column and convert it separately into 64 matrices.So, for example column 1 would be converted from [0;0] to [0 0 0;0 0 0] etc. However, i believe it is a much easier problem which it can be solved in less time, elegantly.
Using dec2bin:
r = 2; %// nunber of rows
c = 3; %// number of columns
M = dec2bin(0:2^(r*c)-1)-'0'; %// Or: M = de2bi(0:2^(r*c)-1);
M = reshape(M.',r,c,[]);
M is a 3D-array of size r x c x 2^(r*c), such that M(:,:,1) is the first matrix, M(:,:,2) is the second etc.
How it works:
dec2bin gives a binary string representation of a number. So dec2bin(0:2^(r*c)-1) gives all numbers from 0 to 2^(r*c)-1 expressed in binary, each in one row. The -'0' part just turns the string into a numeric vector of 0 and 1 values. Then reshape puts each of those rows into a r x c form, to make up each of the the desired matrices.

Incorrect output from Fortran loop

I have written a routine to give the column index position for the furthers right cell containing a 1, marking the right edge of a polygon in a mask array.
But when I print the index arrays (I realize the row position array is redundant), all rows are reporting a position (other than 0), which I know shouldn't be the case. I can't seem to find where the following could be incorrect.
Example mask array:
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1 0 0 0 0 0 0
0 0 0 0 0 0 1 1 1 0 0 0 0 0
0 0 0 0 0 1 1 1 1 1 1 0 0 0
0 0 0 1 1 1 1 1 1 1 1 0 0 0
0 0 0 1 1 1 1 1 1 1 1 0 0 0
0 0 0 0 0 0 1 1 1 1 0 0 0 0
0 0 0 0 0 0 0 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
Desired output:
[0,10,10,11,11,11,9,8,0]
Is this a Fortran thing, or is my logic just off (
Routine:
subroutine get_right_idx(mask_array, idx_x, idx_y)
integer, parameter :: x = 169 ! num columns
integer, parameter :: y = 124 ! num rows
integer i ! iterator for rows
integer j ! iterator for columns
integer row_x ! x position for furthest south cell in row
integer row_y ! y position for furthest south cell in row
integer :: idx_x(y) ! index positions for lowest lat in model grid - x
integer :: idx_y(y) ! index positions for lowest lat in model grid - y
real mask_array(y,x) ! mask array of zeros, containing polygon of ones
do j=1,y
row_x = 0
row_y = 0
do i=1,x
if (mask_array(j,i).eq.1) then
row_x = i
row_y = j
endif
enddo
idx_x(j)=row_x
idx_y(j)=row_y
enddo
endsubroutine get_right_idx
Actual mask (zoomed out in Open Office):
Below is the mask that I am trying to evaluate. My resulting array has a non-zero value in for all elements, where there should be zero elements at the start and end of the array, no matter which direction it is evaluated from.
Output:
125 104 104 104 104 104 104 114 114 114 114 103 103 103 108 108 103 103 103 103 97 97 97 107 107 107 107 107 107 107 107 107 97 101 101 101 101 101 111 111 111 111 111 111 101 101 100 105 105 105 105 105 105 100 115 115 104 104 104 104 104 104 104 104 104 104 98 98 98 98 108 108 108 108 108 108 108 108 98 102 102 102 102 102 112 112 112 112 112 112 101 101 101 106 106 106 101 101 101 95 95 105 105 105 105 105 105 105 105 105 105 99 99 99 99 99 109 109 109 109 109 109 109 99 99
I made a quick template of your code, but with my suggestion. Since you want to find the largest index j that has a 1 in it, you can simply test this by starting your j-loop with the maximum (in the example, 14) and working downwards but exiting the inner do-loop if mask_array==1.
You could generalize this by replacing 14 and 9 with the maximum values for those dimensions.
program get_right
implicit none
integer, dimension(9,14) :: mask_array
integer :: i,j,mask_out(9),j_tmp
mask_array(1,:)=[0,0,0,0,0,0,0,0,0,0,0,0,0,0]
mask_array(2,:)=[0,0,0,0,0,0,0,1,0,0,0,0,0,0]
mask_array(3,:)=[0,0,0,0,0,0,1,1,1,0,0,0,0,0]
mask_array(4,:)=[0,0,0,0,0,1,1,1,1,1,1,0,0,0]
mask_array(5,:)=[0,0,0,1,1,1,1,1,1,1,1,0,0,0]
mask_array(6,:)=[0,0,0,1,1,1,1,1,1,1,1,0,0,0]
mask_array(7,:)=[0,0,0,0,0,0,1,1,1,1,0,0,0,0]
mask_array(8,:)=[0,0,0,0,0,0,0,0,0,1,0,0,0,0]
mask_array(9,:)=[0,0,0,0,0,0,0,0,0,0,0,0,0,0]
mask_out=0
do i=1,9
j_tmp=0
print '(14(i0,2x))',mask_array(i,:)
do j=14,1,-1
if(mask_array(i,j)==1) then
j_tmp=j
exit
endif
enddo
mask_out(i)=j
enddo
print *,""
print '(9(i0,2x))',mask_out
end program get_right
I got 0,8,9,11,11,11,10,10,0 as the answer (which is what you got but backwards).
Or, if typing makes you tired you could evaluate this
maxval(mask_array*spread([(ix,ix=lbound(mask_array,dim=2),&
ubound(mask_array,dim=2))],dim=1, ncopies=size(mask_array,dim=1)),dim=2)
As you can see this makes a temporary array (which may be undesirable if your masks are large) using spread and an implied-do loop, each element of this temporary contains its own column index number. Then it multiplies the temporary with the mask_array, performing element-wise multiplication not matrix multiplication. Finally take the maxval of each row of the result. This returns the vector in the same order as Kyle's code does.
I've edited the code to use lbound and ubound rather than 1 and size in case you want to use the code on arrays with lower bounds other than 1.
Bob's yer uncle, but don't ask whether this is faster than Kyle's code. If you are interested in execution speed test and measure.
Incidentally, since this returns just one vector and doesn't modify its arguments or have any other side effects, I'd package it as a function rather than as a subroutine.

Perl Nested Loop: Arrays - Calculating Minimum Distance

temp.bgf
ATOM 218 CB ASN 1 34 -7.84400 -9.19900 -5.03100 C_3 4 0 -0.18000 0 0
ATOM 221 CG ASN 1 34 -7.37700 -7.83400 -4.55200 C_R 3 0 0.55000 0 0
ATOM 226 C ASN 1 34 -9.18200 -10.62100 -6.58300 C_R 3 0 0.51000 0 0
ATOM 393 CB THR 2 69 -3.33000 -7.97700 -7.72000 C_3 4 0 0.14000 0 0
ATOM 397 CG2 THR 2 69 -4.75300 -8.54400 -7.67200 C_3 4 0 -0.27000 0 0
ATOM 401 C THR 2 69 -2.58000 -9.55700 -5.85500 C_R 3 0 0.51000 0 0
ATOM 417 CB THR 2 71 1.99100 -9.86800 -2.77000 C_3 4 0 0.14000 0 0
ATOM 421 CG2 THR 2 71 2.86300 -10.15400 -1.55700 C_3 4 0 -0.27000 0 0
ATOM 425 C THR 2 71 -0.19100 -10.14200 -1.62900 C_R 3 0 0.51000 0 0
ATOM 492 CB CYS 2 77 -5.17100 -14.77100 4.04000 C_3 4 0 -0.11000 0 0
ATOM 495 SG CYS 2 77 -6.29600 -14.88500 2.59500 S_3 2 2 -0.23000 0 0
ATOM 497 C CYS 2 77 -4.65100 -13.75800 6.12000 C_R 3 0 0.51000 0 0
ATOM 2071 CB SER 7 316 -3.87300 -2.15900 1.02300 C_3 4 0 0.05000 0 0
ATOM 2076 C SER 7 316 -4.79700 -1.16500 -1.10800 C_R 3 0 0.51000 0 0
target.bgf
ATOM 575 CB ASP 2 72 -2.80100 -7.45000 -2.09400 C_3 4 0 -0.28000 0 0
ATOM 578 CG ASP 2 72 -3.74900 -6.45900 -1.31600 C_R 3 0 0.62000 0 0
ATOM 581 C ASP 2 72 -3.19300 -9.62400 -0.87900 C_R 3 0 0.51000 0 0
I got two files of data. The first file contains data for the residues I want to calculate the distance to. The second file contains the coordinates for the target residue.
I want to calculate the minimum distance between the two quantities (i.e. ASP and the residues in the temp.bgf). I haven't been able to come up with an optimal way to store the x,y,z values and compare the distance in temp.bgf.
There have been questions as to how the calculation should be done. Here is the idea I have
#asp_atoms
#asn_atoms
$asnmin, aspmin
foreach $ap (#asp_atoms)
{
foreach $an (#asn_atoms)
{
dist = dist($v..$g...);
if($dist < $min)
{
$min = $dist;
}
}
}
I hope that clarifies questions as to how to implement the code. However, the problem I am having is how to store the values in the array and traverse through the file.
Also, to clarify how exactly(i.e. what numbers will be used for distance, here is an example of what I want to do).
For the ASP CB atoms with the following coordinates: -2.80100 -7.45000 -2.09400
I want to calculate the distance between the ASN CB, ASN CG, ASN C atoms. The minimum is the value printed out. Unfortunately, I don't have an exact value as to what that minimum would be, but I have to print out values less than 5 units of distance. Then, the ASP CG atoms distance would be calculated to all the ASN atoms to see the min. So I am trying to find the min distance here.
You can solve this by simply splitting each row from your file on white spaces, then storing the results in arrays of arrays and then slicing out only the parameters you need in loops (in this case x,y,z). This is not a complete answer to your problem but it should give you an idea of how this can be accomplished.
open (my $temp,"<","temp.bgf");
open (my $target,"<","target.bgf");
my #temps = create_ar($temp);
my #targets = create_ar($target);
sub create_ar {
my $filehan = shift;
my #array;
foreach (<$filehan>) {
push #array,[split(/\s+/,$_)];
}
return #array;
}
foreach my $ap (#targets) {
my ($target_X,$target_Y,$target_Z) = #{$ap}[6,7,8];
foreach my $an (#temps) {
my ($temp_X,$temp_Y,$temp_Z) = #{$an}[6,7,8];
...
}
}

Resources