I am currently learning Tensorflow, and as part of this I'm building a neural network. As an input, it'll take a 42-length array and a 7-length array, and as output it'll put out a 7-length array or a single digit, it doesn't matter. I want to have 69 hidden layers.
Somehow, I need to train my Tensorflow model on a bunch of groups of 42-and-7-length arrays, but I'm not sure how to group them.
I currently have a large array like this:
myLargeArray = numpy.ndarray([[[42-length-array],[7-length-array]],[[42-length-array],[7-length-array]],[[42-length-array],[7-length-array]],[[42-length-array],[7-length-array]],[[42-length-array],[7-length-array]](and so on)]
How can I pass in each one of the grouped arrays into my Tensorflow model? I can't quite understand how because all of the input data in the Tensorflow tutorials is processed from CSVs.
Looks like you want to feed different arrays to different tensors. In this case, there's no need to combine the two into a single myLargeArray.
Here's an example MNIST classifier that also recognizes a digit. It uses x of length 784 and y of length 10, but the idea is the same.
# None is for batch processing
x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10]) # one-hot encoded
[...]
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
sess.run([optimizer, cost], feed_dict={x: batch_xs, y: batch_ys})
Arrays batch_xs and batch_ys have different size and both fed to the session. The result of the model is probability distribution of each digit, i.e., also of size 10.
Related
Let's say I have vectors
Real x[5]={1,2,3,4,5};
and
Real y[5]={0,0,0,0,0};
I want to assign values to elements of y given some condition on x, say for example, for every element in x greater than or equal to 3, corresponding y should be set to 1, for every x<3: y=sin(x)
In matlab, I would have written as follows:
y(x>=3)=1;
y(x<3)=sin(x(x<3));
so
x=[1 2 3 4 5]
Results in
y=[sin(1) sin(2) 1 1 1]
Can something similar be done in Modelica, and if so, how?
Unfortunately there is no array filtering in Modelica, so x>=3 does not work and there is nothing available that is comparable to that.
But still there are some ways get the resulting vectors, depending if you want an equation or an algorithm. In a model using equations you could e.g. use an array constructor with one iterator:
model DemoModel
Real x[5]={1,2,3,4,5};
Real y[:]= {if i >= 3 then 1 else sin(i) for i in x};
end DemoModel;
With this method all elements are assigned at once. In models its not possible to update only the values which match a criteria (or at least I don't know how to do so), since:
Vector sizes must be known during translation
Every element of a vector needs an equation
With a filter the number of equations would vary dynamically, depending on the values of x and the condition.
Inside functions you have more possibilities (you can have flexible array sizes), but when you use the output of the function in a model you are facing the same limitations again.
I'm having trouble iterating through two arrays using the haversine Ruby gem.
I'll post the full gist here but give you an idea of what I'm trying to accomplish.
https://gist.github.com/ornerymoose/c4a45540304706e78191894dfe6b2539
We have a fiber network. Just think about it as lines on a map. I have all the lat/long pairs that make up this network, roughly 24,000 rows.
We have potential customers that are in the area of the fiber network. We want to see what customers are within 1,000 feet of our network. For our potential customers, we have roughly 3,000 rows.
The Haversine formula takes four arguments: lat1, long1, lat2, long2. The first two will represent the customer, the second two will represent the fiber network.
For a hardcoded customer location, I can do this and it returns the correct value in feet. Verified via Google Earth.
File.open('distance-output.csv', 'w') do |csv_object|
lat.zip(long).each do |lat_locs,long_locs|
csv_object << Haversine.distance(28.59569, -81.21464, lat_locs.pop.to_f, long_locs.pop.to_f).to_feet
csv_object << "\n"
end
end
Now, how would I implement the same type of .zip setup for the customer lat/long data?
If I wrap lat.zip(long).each do |lat_locs,long_locs| in the customer .zip, it returns values but incorrect (like, really incorrect). If I wrap the customer zip block in lat.zip(long).each do |lat_locs,long_locs|, the returned values are also incorrect.
Any input on this is greatly appreciated.
I suspect your problem is that you're popping your coordinates out of the one-element arrays. That permanently modifies each array. When you do it in a loop, the second time through the loop the arrays will be empty.
The quick fix would be to use .last (which just returns the last element of the array rather than removing it) rather than .pop. Since there is only one column in each CSV, .first and .last are equivalent.
However, I suggest rewriting the part of your code that you didn't include in the question. The loops over the arrays read from your CSVs don't do anything useful at all; they just copy the arrays to new arrays.
Instead, strip off the one-element arrays, zip those, and then loop over the result:
fib_coords = fib_lat.map(&:last).zip(fib_long.map(&:last))
cust_coords = cust_lat.map(&:last).zip(cust_long.map(&:last))
fib_coords.each do |fib_lat, fib_long|
cust_coords.each do |cust_lat, cust_long|
# ...
end
end
Many of the plotting functions I use to visualize data (plot_surface, plot_wireframe, mayavi's contour3, etc.) take as arguments 2-D arrays X, Y, Z, and some scalar value of the function.
I usually have information from a file in the format
x y z data
0 0 1 45
...
Which is on a regular grid. I have way too many values to be able to hold the output from meshgrid in my memory, but I can hold the full dataset as either an Nx4 or four Nx1 arrays.
Is there a way to make a view, or restructure the existing gridded data to a format that is compatible with these functions?
I realize that I can use griddata and interpolate to lower the resolution, and that is my current approach.
UPDATE:
The specific task I'm working on uses mayavi's contour3 function to generate an isosurface plot, but the approach of multidimensionalizing the arrays should be general.
You can greatly reduce memory use by passing copy=False to meshgrid. This creates views into the original arrays, see the docs
Im using HDF5 to store massive sparse arrays in Coordinate format (basically, an M x 3 array which stores the value, x index and y index for each non-zero element).
This is great for processing the whole dataset in an iterative manner, but I am struggling with random lookups based on index values.
E.g, given a 100x100 matrix, I might store then non sparse elements like so:
[[1,2,3,4,5], // Data values
[13, 14, 55, 67, 80], // X-indices
[45, 12, 43, 55, 12]] // Y-indices
I then wish to get all the data values between 10<x<32 and 10<y<32, for example. With the current format, all I can do is iterate through the x and y index arrays looking for matching indices. This is very very slow, with multiple reads from disk (my real data typically has as size of 200000x200000 with perhaps 10000000 non-sparse elements).
Is there a better way to store large (larger than RAM) sparse matrices and support rapid index-based lookups?
I'm using HDF5, but happy to be pointed in other directions
First, let's suppose that, as your example hints but you don't state conclusively, you store the elements in order sorted by x first and by y second.
One easy technique for more rapid lookup would be to store an x-index-index, a vector of tuples (following your example this might be [(10,1),(20,null),(30,null),(40,null),(50,3),...]) pointing to locations in the x-index vector at which runs of elements start. If this index-index fits comfortably in RAM you could get away with reading it from disk only once at the start of your computation.
Of course, this only supports rapid location of x indices, and then a scan for the y. If you need to support rapid location of both you're into the realm of spatial indexing, and HDF5 might not be the best on-disk storage you could choose.
One thought that does occur, though, would be to define a z-order curve across your array and to store the elements in your HDF5 file in that order. To supplement that you'd want to define a z-index which would identify the location of the start of the elements in each 'tile' of the array. This all begins to get a bit hairy, I suggest you look at the Wikipedia article on z-order curves and do some head scratching.
Finally, in case it's not crystal clear, I've looked at this only from the point of view of reading values out of the file. All the suggestions I've made make creating and updating the file more difficult.
Finally, finally, you're not the first person to think about effective and efficient indexing for sparse arrays and your favourite search engine will throw up some useful resources for your study.
I totally understand the purpose of arrays, yet I do not feel I have "mastered" them. Does anyone have some really good problems or readings involving arrays. I program in PHP and C++ so if there are examples with those languages that would be preferable but is not necessary.
Draw everything out on graph paper.
Memory is just little boxes, it's a lot clearer to see on paper with a few arrows than in some complex markup language
Define 'mastered'.
Does anyone have some really good problems or readings involving arrays.
Try array based
Linked-list implementation.
Implement stacks, queues, etc and then use stack(s) to emulate a queue etc
Heaps
A lot of people seem to struggle with the concept of arrays at first, particularly arrays of >2 dimensions.
It's a little too abstract. However, getting past that initial block just requires a concrete exploration of the mechanics. So, here's an example I've used a lot that shows the basic mechanics in a nerd-friendly way:
Concrete Example (in psuedocode)
Let's say you're building a role playing game and you want to keep track of your character's stats. You could use an array of integers like this:
Stats(0) could be strength
Stats(1) could be dexterity
Stats(2) could be intelligence
...And so on.
Now let's add a level of complexity. Maybe we want to introduce a potion of strength that increases strength by 5 for 10 turns. We could represent the stat side of that by making this into a 2-dimensional array:
Stats(0, 0) - this is my current strength.
Stats(0, 1) - this is my normal strength.
Stats(0, 2) - this is the number of turns until the strength potion wears off.
Stats(1, 0) - this is my current dexterity.
...And so on. You get the idea.
So I've added a 2nd dimension to hold details about the 1st dimension. What if I wanted our Stats array to handle statistics for more than one character? I could represent that by making this into a 3-dimensional array:
Stats(0, 0, 0) - character 0's current strength.
Stats(1, 1, 0) - character 1's current dexterity.
It would be even better to create some constants or enums to eliminate magic numbers from the code:
Const Strength = 0
Const Dexterity = 1
Const Intelligence = 2
Const CurrentValue = 0
Const NormalValue = 1
Const PotionTurns = 2
Then I could do:
Stats(1, Dexterity, NormalValue) = 5 'For character 1, set the normalvalue of dex to 5.
A few more thoughts about arrays... At least in the .Net world where I live, most of us don't have to use them in our day-to-day lives too often because they're slowly being relegated to underpinnings for more elaborate data structures like collections.
In fact, if I were to implement character stats, realistically I would not use arrays.
However, it's still important to get your head around them because they are rocket-fast and there are definitely cases where they're invaluable.
Try manually (no built in methods) sorting with arrays (bubblesort is a good one to get yourself going)
An array is a contiguous block of memory devoted to N items of the same type, where N is a fixed number indicating the number of items. In order to expand an array (as they are fixed in size), you can use the C function realloc(..). In C++, you can manually re-allocate an array by creating a new, larger array and copying the contents of the old array into the new one (an expensive operation).
Modern languages have options that supplant arrays (so as to overcome the inherent limitations of arrays). In C++, you can use the Standard Template Library (STL) for this purpose; the STL "vector" data type is a typical replacement for standard arrays in C++. Platform-dependent APIs like MFC have built-in constructs like the CArrayList for a richer array usage experience. ;-)
I'm going to try to explain the best i can arrays in their fundamental form.
Ok, so an array is basically a way to store data. For instance if you want a list of shopping items, we would use a one dimensional array:
[0] - "Bread"
[1] - "Milk"
[2] - "Eggs"
[3] - "Butter"
.
.
.
[n] - "Candy"
Each index 0,1,2,3,...n holds a specific data. The data as you can see are represented as Strings. Now i can't use something like :
[n+1] = 1000
because this will put an integer as index n+1, the compiler will tell you that it's no good and you need to fix that issue.
Moving on to matrices or 2-dimensional arrays. Take a piece of squared paper like the ones you use for math and draw a Cartesian system and some dots. Put the coordinates on different piece of paper and next to them put a 1. For example:
[0,0] = 1
[0,1] = 1
[2,3] = 1
What this means is that at index [0,0],[0,1],[2,3] i have 1. A representation would be like so:
Cartesian System in form of a matrices :
1) 2) 3)
1) 1 1 0
2) 0 0 1
3) 0 0 0
I used just simple arrays to illustrate what are they, for example if you want to go 3D the data structure for that would be a array of matrices aka a list which holds each Cartesian location at a specific height.
If we had 10 as height and the same dots as above it would be something like:
[10,0,0] - 1
[10,0,1] - 1
[10,2,3] - 1
If you want a way to master: grab a simple list of problems and try to implement them using arrays of any kind. There is no quick way to do it, just have patience and practice.
Alright, so a one-dimensional array is just a grouping of variables. Useful if you've got a lot of something, and you like to save time and space. Functionally,
element1:=5;
element2:=6;
element3:=7;
...
is the same as saying
element[1]:=5;
element[2]:=6;
element[3]:=7;
...
except now the computer knows what you're talking about and you can write something like:
for i:=1 to n do
element[i]:=element[i]+1;
Moving on, a two dimensional array is somewhat more complicated, but can be thought of as an array of arrays. So we could have something like this:
type
arrayA=array[1..50] of integer;
arrayB=array[1..50] of arrayA;
arrayB=array[1..50,1..50] of integer; //an equivalent declaration in Pascal to the above two
More specifically, a two-dimensional array is a table. So for example, if arrayA contains the grades of a student in 50 classes, then arrayB represents the grades of a bunch of students. So, arrayB[3,5] would be the grade of the third student on class number 3.
It's easy to expand the same logic to add another dimension to the array:
arrayC=array[1..50] of arrayB;
We could say that C represents a school, so arrayC[2,4,6] is the grade of the second schools fourth student in class 6.
Now, what are arrays used for? Storing groups of similar information, or information which'll need to be processed in bulk. In practice, though, you'll mostly be using a one-, at most two-, dimensional array. If you can imagine your data as a table, you'll almost definitely want an two-dimensional array, for example. How else would you represent a chess board, if you had to?
Three-dimensional arrays tend to be used less, but what if you had to save some sort of state for each of your table elements? Something like:
Table=array[1..50, 1..50, 0..1] of integer;
Then the third value is 1 if some condition is true for a given element, and 0 otherwise. Of course, a trivial example, but it's another way of understand three-dimensional arrays more easily.