I'm having an issue with my numpy array, which is of size (29000,200,1024)(7Go). Its the features of the images of my dataset.
Once loaded, my function receives the indexes to build the current batch as a tensor.
Unfortunately, using :
tf.gather(array, indices)
freezes. Though printing for example array[0] work instantly.
I tried to transform my numpy array with convert_to_tensor so I can use directly array_tensor(indice) but again, convert_to_tensor leads to a memory limit error.
Any work around ?
Thank you very much
Passing numpy array directly into tf op construction API converts it to tf.constant op which contains data in the op definition, so you are inlining the whole thing into GraphDef, subject to 2GB GraphDef limit.
To avoid this, create var=tf.Variable(my_placeholder) and initialize this variable by running var.initializer, feed_dict={my_placeholder: np_array}. This puts numpy array data directly into variable store.
Related
so I have a final project for a class where I need to make a video game in LabView. The issue I'm having at the moment is that I can't figure out the 'right' way to put 'yourShip.png' into the 2d array of 2d pictures at [0,0]. Every tutorial I can find basically has exactly what I have down below in the screenshot, and it makes sense to me. However, running the program quickly shows that it does nothing.
To describe the code, I have a path constant that leads to the picture, which feeds to a draw flattened pixelmap function. Up to this point I know the code works, since creating a test indicator reveals as such. However, next I try to use the replace array subset function to replace the (default blank) 2d picture at [0,0] with yourShip.png. 'screen' is a 5x5 2d array of 2d pictures. The local variable of the same name being outputted to is indeed the very same array.
My main guess with why my code doesn't work is because of the way I'm taking screen as the input variable and then outputting to it via a local variable. However, if this is wrong, I'm confused with how I should do it. All I want to do is 'spawn' the image at the correct index.
The replace array subset works quite literally, i.e. it can only replace existing elements.
If there is no element at the specified index because the array is smaller, the function will do just nothing.
I guess your array is empty, so, initialize your screen array first to a size of at least 1x1.
I'm having this issue and I'm not sure why is happening, and why it does not happen to all variables.
I wrote an optimization algorithm for several parameters.
example I have two multidimensional numpy arrays
"parameters" is a 3 dimensional array and
"lambda" a 2 dimensional array
I will summarize the code since is a few hundred lines long
parametersOpt,lambdaOpt=myOptModule(parameters,lambda)
def myOptModule(parameters,lambda):
**Optimization code here**
parameters=parameters-delta_parameters
lambda=lambda-delta_lambda
return parameters, lambda
what happens is that code is executed and for some reason the variable "parameters" changes and becomes the same as "parametersOpt" it changes automatically even tough the operations occur inside a module, and this is not the case to variable "lambda", lambda and lambdaOpt are not the same as it should be expected. Any ideas? I tries changing the names of variables inside and outside the module but that didn't work
When using the MATLAB jsonencode function it seems very difficult to convert size 1 arrays into the correct JSON format i.e. [value]. For example if I do:
jsonencode(struct('words', [string('hello'), string('bye')]))
Then this produces:
{"words":["hello","bye"]}
which is correct. If however I do:
jsonencode(struct('words', [string('hello')]))
Then it produces:
{"words":"hello"}
losing the square brackets, which it needs because it is in general an array. The same happens when using a cell rather than an array, although using a cell does work if it's not inside a struct.
Any idea how I can work around this issue?
It seems this can be solved by using a cell rather than an array and then not creating the struct inline. Like
s.words = {'hello'};
jsonencode(s)
Output:
{"words":["hello"]}
I presume when created inline the cell functionality of matlab is actually trying to make multiple structs rather than multiple strings. Note that this still won't work with arrays as matlab treats a size one array as a scalar.
you might ask "why don't you just use C?",
but I would like to try constructing an array
with specific element size using python 3, is that possible?
I know bytearray() but it's limited to one byte per element, is there a more flexible function?
also, is there any equivalent statement in python 3 to "sizeof(int)" in C?
I am not talking about sys.getsizeof(int) since it gives me the bytesize of the whole int class
x = [None]*10
This will initialize a list with size 10, but you can also take a look at arrays from the numpy module. This will also have lots of other optimization in it too.
I don't have much experience with your memory problem, but maybe this code will help? http://code.activestate.com/recipes/546530/
I know that for object, we can forEach the collection and update the object as we like but for immutable objects like Strings, how can we update the array with new object without converting it into an array again.
For e.g, I have an array of string. I want to iterate through each string and trim them. I would otherwise have to do something like this:
Arrays.stream(str).map(c -> c.trim()).collect(Collectors.toList())
In the end, I will get a List rather then String[] that I initially gave. Its a whole lot of processing. Is there any way I can do something similar to:
for(int i = 0; i < str.length; i++) {
str[i] = str[i].trim();
}
using java streams?
Streams are not intended for manipulating other data structures, especially not for updating their source. But the Java API consists of more than the Stream API.
As Alexis C. has shown in a comment, you could use Arrays.setAll(arr, i -> arr[i].trim());
There’s even a parallelSetAll that you could use when you have a really large array.
However, it might be easier to use just Arrays.asList(arr).replaceAll(String::trim);.
Keep in mind that the wrapper returned by Arrays.asList allows modifications of the wrapped array through the List interface. Only adding and removing is not supported.
Use toArray :
str = Arrays.stream(str).map(c -> c.trim()).toArray(String[]::new);
The disadvantage here (over your original Java 7 loop) is that a new array is created to store the result.
To update the original array, you can re-write your loop with Streams, though I'm not sure what's the point :
IntStream.range (0, str.length).forEach (i -> {str[i] = str[i].trim();});
It's not that much processing as you might think, the array has a known size and the spliterator from it will be SIZED, thus the resulting collection size will be known before processing and the space for it can be allocated ahead of time, without having to re-size the collection.
It's also always interesting that in the absence of actual tests we almost always assume that this is slow or memory hungry.
of course if you want an array as the result there is a method for that :
.toArray(String[]::new);