Radial basis network character recognition - arrays
I want to develop a simple character recognition program by implementing a given neural network kind; a simple command line-type is enough.
The radial basis function neural network was assigned to me and I already studied the weight training, input-to-hidden-to-output procedures but I am still doubtful of in implementing it. My references are (1) and (2).
A simple one-dimensional array of a 10 by 10 binary object (that represents a character) is the input. For example, the array below
input = array(
0,0,0,1,1,1,1,0,0,0,
0,0,1,0,0,0,0,1,0,0,
0,1,0,0,0,0,0,0,1,0,
1,0,0,0,0,0,0,0,0,1,
1,1,1,1,1,1,1,1,1,1,
1,0,0,0,0,0,0,0,0,1,
1,0,0,0,0,0,0,0,0,1,
1,0,0,0,0,0,0,0,0,1,
1,0,0,0,0,0,0,0,0,1,
1,0,0,0,0,0,0,0,0,1 )
is the representation of the character "A":
0 0 0 1 1 1 1 0 0 0
0 0 1 0 0 0 0 1 0 0
0 1 0 0 0 0 0 0 1 0
1 0 0 0 0 0 0 0 0 1
1 1 1 1 1 1 1 1 1 1
1 0 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0 1
I plan to take the total weight of the input and compare it to the training set as in the saved 1-D arrays of the other characters of the alphabet and the one with the closest is the prediction.
The problem is I tend to understand algorithms better if presented in a CLRS-manner or similar type as opposed to mathematical formula. I find it hard to understand the explanations in those two papers (which I find the easiest to read among others here in the Google search).
Can someone point me to a friendly algorithm for a RBNFF that takes in an array and produces an output of weights? If not, a paper that explains this in Layman's manner would be appreciated.
Training
For what I could find there is no "one right way" to train them.
The simplest training I could find was by a composition of two algorithms
(Clustering) Taking the left part (input weights and RBFs) of the network and doing unsupervised clustering. There is a few things you can try out hard/soft and the number of clusters/RBFs.
Each cluster is a representation of a single RBF with the weights connecting to it.
How you go from having clusters to get rbf and rbf weights depends on what clustering you are using. (I can extend this if it's unclear)
(Neural Network) The solving the left out part of the original RBFNN from the last step by using the output from the clustering as input to an ordinary single layer neural network.
Probably easier to find these more primitive algorithms easily explained
EDIT
found some "pseudo"-code with explanations that might explain it all better (written in C#)
http://msdn.microsoft.com/en-us/magazine/dn532201.aspx
(Supposedly) working python code
https://github.com/andrewdyates/Radial-Basis-Function-Neural-Network
Related
which clustering technique i should use?
i have a data matrix given as below.. it is the user access matrix..each row represents users and each column represents page category visited by that user. 0 8 1 0 0 8 0 0 0 0 0 0 0 11 2 2 0 1 0 7 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 6 1 0 0 0 2 6 0 0 0 0 1 0 0 0 0 0 5 3 2 0 2 0 0 0 0 0 1 0 0 0 1 0 0 2 3 0 1 0 1 0 0 0 0 0 1 0 3 0 0 0 9 0 1 1 0 0 5 0 0 0 1 2 0 0 0 0 0 5 1 4 0 0 0 1 0 0 2 0 0 0 9 0 0 0 5 5 0 2 0 1 0 0 0 0 1 1 0 0 0 0 0 1 2 0 0 2 3 3 0 0 1 1 0 0 0 4 0 0 0 1 0 1 0 2 0 0 1 0 0 0 0 2 0 0 0 5 4 0 0 1 0 0 0 0 0 1 0 0 2 0 0 0 0 0 0 2 0 0 2 12 1 0 0 0 2 0 0 0 0 6 1 0 0 0 0 58 15 7 0 1 0 0 0 0 0 0 1 0 2 0 0 1 1 0 0 0 2 0 0 0 0 0 0 I need to apply biclustering technique on it. This biclustering technique will first generates user clusters and then generates page clusters.after that it combine both user and page clusters to generate biclusters. Now i am confused about which clustering technique i should use for this purpose. the best clustering will generate coherent biclusters from this matrix.
Here is a summary of several clustering algorithms that can help to answer the question "which clustering technique i should use?" There is no objectively "correct" clustering algorithm Ref Clustering algorithms can be categorized based on their "cluster model". An algorithm designed for a particular kind of model will generally fail on a different kind of model. For eg, k-means cannot find non-convex clusters, it can find only circular shaped clusters. Therefore, understanding these "cluster models" becomes the key to understanding how to choose among the various clustering algorithms / methods. Typical cluster models include: [1] Connectivity models: Builds models based on distance connectivity. Eg hierarchical clustering. Used when we need different partitioning based on tree cut height. R function: hclust in stats package. [2] Centroid models: Builds models by representing each cluster by a single mean vector. Used when we need crisp partitioning (as opposed to fuzzy clustering described later). R function: kmeans in stats package. [3] Distribution models: Builds models based on statistical distributions such as multivariate normal distributions used by the expectation-maximization algorithm. Used when cluster shapes can be arbitrary unlike k-means which assumes circular clusters. R function: emcluster in the emcluster package. [4] Density models: Builds models based on clusters as connected dense regions in the data space. Eg DBSCAN and OPTICS. Used when cluster shapes can be arbitrary unlike k-means which assumes circular clusters.. R function dbscan in package dbscan. [5] Subspace models: Builds models based on both cluster members and relevant attributes. Eg biclustering (also known as co-clustering or two-mode-clustering). Used when simultaneous row and column clustering is needed. R function biclust in biclust package. [6] Group models: Builds models based on the grouping information. Eg collaborative filtering (recommender algorithm). R function Recommender in recommenderlab package. [7] Graph-based models: Builds models based on clique. Community structure detection algorithms try to find dense subgraphs in directed or undirected graphs. Eg R function cluster_walktrap in igraph package. [8] Kohonen Self-Organizing Feature Map: Builds models based on neural network. R function som in the kohonen package. [9] Spectral Clustering: Builds models based on non-convex cluster structure, or when a measure of the center is not a suitable description of the complete cluster. R function specc in the kernlab package. [10] subspace clustering : For high-dimensional data, distance functions could be problematic. cluster models include the relevant attributes for the cluster. Eg, hddc function in the R package HDclassif. [11] Sequence clustering: Group sequences that are related. rBlast package. [12] Affinity propagation: Builds models based on message passing between data points. It does not require the number of clusters to be determined before running the algorithm. It is better for certain computer vision and computational biology tasks, e.g. clustering of pictures of human faces and identifying regulated transcripts, than k-means, Ref Rpackage APCluster. [13] Stream clustering: Builds models based on data that arrive continuously such as telephone records, financial transactions etc. Eg R package BIRCH [https://cran.r-project.org/src/contrib/Archive/birch/] [14] Document clustering (or text clustering): Builds models based on SVD. It has used in topic extraction. Eg Carrot [http://search.carrot2.org] is an open source search results clustering engine which can cluster documents into thematic categories. [15] Latent class model: It relates a set of observed multivariate variables to a set of latent variables. LCA may be used in collaborative filtering. R function Recommender in recommenderlab package has collaborative filtering functionality. [16] Biclustering: Used to simultaneously cluster rows and columns of two-mode data. Eg R function biclust in package biclust. [17] Soft clustering (fuzzy clustering): Each object belongs to each cluster to a certain degree. Eg R Fclust function in the fclust package.
You cannot tell which clustering algorithms is best by just looking at the matrix. You must try different algorithms (maybe k-means, bayes, nearest-neighbor or whatever your library has). Make a cross validation (clustering is just a type of categorization where you categorize users to cluster centers) and evaluate the results. You could even print it to a chart. Then make a decision. No decision will be perfect, you will always have errors. And the result depends of what you expect. Maybe a result with more errors will have better results in your personal view. Have you tried any algorithm yet?
find largest rectangle not (necessary) aligned with image boundary in binary matrix
I am using this solution to find rectangles aligned with the image border in a binary matrix. Suppose now I want to find a rectangle that is not aligned with the image border, and I don't know its orientation; what would be the fastest way to find it? For the sake of the example, let's look for a rectangle containing only 1's. For example: 1 1 1 1 0 0 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 0 0 1 0 0 1 1 0 0 0 0 1 1 1 1 1 0 1 0 0 1 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 Then the algorithm described in the solution I described above would only find a rectangle of size 6 (3x2). I would like to find a bigger rectangle that is tilted; we can clearly see a rectanble of at least size 10 or more... I am working in C/C++ but an algorithm description in any language or pseudo-code would help me a lot. Some more details: there can be more than one rectangle in the image: I need the biggest only the rectangle is not a beautiful rectangle in the image (I adapted my example above a little bit) I work on large images (1280x1024) so I'm looking for the fastest solution (a brute-force O(n³) algorithm will be very slow) (optional) if the solution can be parallellized, that is a plus (then I can boost it more using GPU, SIMD, ...)
I only have a partial answer for this question, and only a few thoughts on complexity or speed for what I propose. Brute Force The first idea that I see is to use the fact that your problem is discrete to implement a rotation around the center of the image and repeat the algorithm you already use in order to find the axis aligned solution. This has the downside of checking a whole lot of candidate rotations. However, this check can be done in parallel since they are indepedant of one another. This is still probably very slow, although implementing it (shouldn't be too hard) and would provide a more definite answer to the question speed once parallelized. Note that your work-space being a discrete matrix, there is only a finite number of rotation to browse through. Other Approach The second solution I see is: To cut down your base matrix so as to separate the connected components [1] (corresponding to the value set you're interested in). For each one of those smaller matrices -- note that they may be overlapping depending on the distribution -- find the minimum oriented bounding box for the value set you're interested in. Still for each one of those, rotate your matrix so that the minimum oriented bounding box is now axis-aligned. Launch the algorithm you already have to find the maximum axis-aligned rectangle containing only values from your value set. The solution found by this algorithm would be the largest rectangle obtained from all the connected components. This second solution would probably give you an approximation of the soluiton, but I believe it might prove to be worth trying. For reference The only solutions that I have found for the problem of the maximum/largest empty rectangle are axis-aligned. I have seen many unanswered questions corresponding to the oriented version of this problem on 2D continuous space. EDIT: [1] Since what we want is to separate the connected component, if there is a degree of overlap, you should do as in the following example: 0 1 0 0 0 1 0 1 0 0 0 1 should be divided into: 0 0 0 0 0 0 0 1 0 0 0 1 and 0 1 0 0 0 1 0 0 0 0 0 0 Note that I kept the original dimensions of the matrix. I did that because I'm guessing from your post it has some importance and that a rectangle expanding further away from the boundaries would not be found as a solution (i.e. that we can't just assume there are zero values beyond the border). EDIT #2: The choice of whether or not to keep the matrix dimensions is debatable since it will not directly influence the algorithm. However, it is worth noting that if the matrices corresponding to connected components do not overlap on non-zero values, you may choose to store those matrices "in-place". You also need to consider the fact that if you wish to return as output the coordinates of the rectangle, creating a matrix with different dimensions for each connected component, this will force you to store the coordinates of your newly created matrix in the original one (actually, one point, say for instance the up-left one, should be enough).
Efficient way to "fill" a binary matrix to create smooth, connected zones
I have a large matrix of 1's and 0's, and am looking for a way to "fill" up areas that are locally dense with 1's. I first did this task for an array, and counted the number of 1's within a certain radius of the element in questions. If the radius was 5, for example, and my threshold was 4, then a point that had 4 elements marked "1" within 5 elements to the left or right would be changed to a 1. Basically I would like to generalized this to a two - dimensional array and have a resulting matrix that has "smooth" and "connected" regions of 1's and no "patchy" spots. As an example, the matrix 1 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 1 1 1 0 0 would ideally be changed to 1 0 0 1 1 0 0 0 0 1 1 1 0 0 0 1 1 1 1 0 0 0 0 1 1 1 0 0 or something similar
For binary images, the morphologial operations that are implemented in MATLAB are perfect for manipulating the shape and size of connected regions. Specifically, the process of image closing is designed to fill holes in connected regions. In MATLAB, the function is imclose, which takes the image and a structuring element, similar to a filter kernel, for how neighboring pixels effect the filling of holes and gaps. A simple invocation of imclose is,
IM2 = imclose(IM,strel(ones(3)));
Larger gaps can be filled by increasing the area of the influence of of neighboring pixes, via larger structuring elements. For example, we an use a disk of radius 10 pixels:
IM2 = imclose(IM,strel('disk',10));
While, imclose supports grayscale and binary (0 and 1) images, the function bwmorph is designed for operation on binary images only but provides a generic interface to all of the morphological operations and various neat combinations of operations (e.g. 'bothat', 'tophat', etc.). The syntax for closing is simplified with bwmorph:
BW2 = bwmorph(BW,'close');
Here the structuring element is the standard ones(3).
A simple filter such as the following might do the trick: h = [ 0 1 0 1 0 1 0 1 0]; img2=(imfilter(img,h)>2) | img; For instance: img = 1 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 1 1 1 0 0 img2 = 1 0 0 1 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 0 0 0 0 1 1 1 0 0 You can try different filters to modify the output img2. This uses the image processing toolbox. If you don't have that, you may want to look up equivalent routines from the matlab exchange.
MPI: How to concatenate sub-arrays in multiple processors into a larger single array
I am using MPI in C. I was able to distribute parts of an array to different processors. And the different processors did all the manipulation I wanted. Now I am at the point where I wanted to combine all the sub-arrays that are in all the processors into one big array. For example if the different processors had sub-arrrays as follows: Processor 1: 0 1 1 0 0 0 1 0 Processor 2: 0 0 1 0 1 1 0 1 Processor 3: 1 1 0 0 1 1 1 1 ... I want to be able to combine, or "concatenate", all the sub-arrays together. For example I would want the large array to be: 0 1 1 0 0 0 1 0 0 0 1 0 1 1 0 1 1 1 0 0 1 1 1 1 ... I was trying to use MPI_Reduce but I couldn't find an operation to do what I wanted to do. Is there another MPI method I could use to achieve what I am looking for?
You are looking for MPI_Gather: Each process (root process included) sends the contents of its send buffer to the root process. The root process receives the messages and stores them in rank order. For documentation and examples, see here and here. The section 5.5 in the MPI 2.2 Standard also has examples.
Neural network using MATLAB
I have a training set that has input and outputs in this way: Input: 0.832 64.643 0.818 78.843 1.776 45.049 0.597 88.302 1.412 63.458 1.468 49.535 1.985 33.387 2.073 30.279 1.431 55.231 1.116 68.521 1.617 44.362 2.159 66.512 Output: 0 0 1 0 0 1 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 1 0 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 0 1 0 0 0 1 0 0 1 0 I need to implement one linear layer neural network that can represent the data set best in MATLAB. What would be the algorithm to do it in MATLAB? The target output is "1 for a particular class that the corresponding input belongs to and "0 for the remaining 2 outputs.
Consider this example of training a feed-forward ANN of one hidden layer (with 3 nodes). Since your data seems to have more output points than input, I'm using a demo dataset, but the idea is the same: %# load sample data laod simpleclass_dataset input = simpleclassInputs; %# 2x1000, 2-dimensional points output = simpleclassTargets; %# 4x1000, 4 classes %# split data into training/testing sets trainInd = 1:500; testInd = 501:1000; %# create ANN and initialize network weights net = newpr(input, output, 3); net = init(net); net.trainParam.epochs = 25; %# max number of iterations %# learn net weights from training data net = train(net, input(:,trainInd), output(:,trainInd)); %# predict output of net on testing data pred = sim(net, input(:,testInd)); %# classification confusion matrix [err,cm] = confusion(output(:,testInd), pred); The output is: err = 0.075075 cm = 81 0 0 0 0 82 0 0 9 0 52 16 0 0 0 93 Obviously you will need access to the Neural Network Toolbox.