Is there an equivalent to numpy.array(someArray, dtype=numpy.uint16) by just using the array module in Python 3? I'm trying to build the equivalent of the buffer of a Javascript Uint16Array object: Uint16Array(someArray).buffer).
Here's what I have so far:
import numpy as np
someArray = []
someArray.append(0)
someArray.append(216)
someArray.append(162)
someArray.append(52)
print(bytearray(np.array(someArray, dtype=np.uint16)))
Output: bytearray(b'\x00\x00\xd8\x00\xa2\x004\x00')
But, if I try the following:
import array as arrayModule
someArray = arrayModule.array("I", [])
someArray.append(0)
someArray.append(216)
someArray.append(162)
someArray.append(52)
print(bytearray(someArray.tobytes()))
Output: bytearray(b'\x00\x00\x00\x00\xd8\x00\x00\x00\xa2\x00\x00\x004\x00\x00\x00')
Using the numpy module works but I'd rather find a native way to accomplish the goal as this is the only place that I use numpy... seems inefficient to import a large module just to use it once.
You want to use "H" (unsigned short) instead of "I" (unsigned int). In C, int can be 2 or 4 bytes depending on architecture and its usually 4. You could check someArray.itemsize to verify on your machine.
Related
I'm trying to call np.random.choice, without replacement, row-by-row in a 2-D numpy array. I'm using Cython to get a speed boost. The code is only running a factor of 3 faster than a pure-python implementation, which is not a great result. The bottleneck is the numpy function call itself. When I comment it out, and just supply a static result of, say [3, 2, 1, 0] to each row, I get a factor of 1000 speedup (of course then it's not doing much of anything :)
My question: is there something I'm doing wrong in calling the numpy function that's causing it to go super slow? In theory it's C talking to C, so it should be fast. I looked at the compiled code, and the call to the numpy function looks complex, with statements like __Pyx_GOTREF and __Pyx_PyObject_GetAttrStr that lead me to believe it's using pure python in the process (bad!!).
My code:
# tag: numpy
import numpy as np
# compile-time info for numpy
cimport numpy as np
np.import_array()
# array dtypes
W_DTYPE = np.float
C_DTYPE = np.int
cdef int NUM_SELECTIONS = 4 # FIXME should be function kwarg
#compile-time dtypes
ctypedef np.float_t W_DTYPE_t
ctypedef np.int_t C_DTYPE_t
def allocate_choices(np.ndarray[W_DTYPE_t, ndim=2] round_weights,
np.ndarray[C_DTYPE_t, ndim=1] choice_labels):
"""
For ea. row in `round_weights` select NUM_SELECTIONS=4 items among
corresponding `choice_labels`, without replacement, with corresponding
probabilities in `round_weights`.
Args:
round_weights (np.ndarray): 2-d array of weights, w/
size [n_rounds, n_choices]
choice_labels (np.ndarray): 1-d array of choice labels,
w/ size [n_choices]; choices must be *INTEGERS*
Returns:
choices (np.ndarray): selected items per round, w/ size
[n_rounds, NUM_SELECTIONS]
"""
assert round_weights.dtype == W_DTYPE
assert choice_labels.dtype == C_DTYPE
assert round_weights.shape[1] == choice_labels.shape[0]
# initialize final choices array
cdef int n_rows = round_weights.shape[0]
cdef np.ndarray[C_DTYPE_t, ndim=2] choices = np.zeros([n_rows, NUM_SELECTIONS],
dtype=C_DTYPE)
# Allocate choices, per round
cdef int i, j
cdef bint replace = False
for i in range(n_rows):
choices[i] = np.random.choice(choice_labels,
NUM_SELECTIONS,
replace,
round_weights[i])
return choices
Update on this, after chatting with some folks and examining the compiled code: #DavidW's comment above put it well:
" In theory it's C talking to C, so it should be fast" - no. Not true.
The main bit of Numpy that cimport numpy gives direct access to is
just faster indexing of arrays. Numpy functions are called using the
normal Python mechanism. They may ultimately be implemented in C, but
that doesn't give a shortcut from Cython's point-of-view.
So this issue here is, calling this Numpy function requires translating the inputs back into python objects, passing them in, and then letting numpy do its thing. I don't think this is the case for all Numpy functions (from timing experiments, some of them I call work quite fast), but plenty are not "Cythonized".
I am practising my Haskell skills by attempting to build a command line version of Tetris. For the game board I am using UArray as I can freeze and thaw it, and that allows me to see if the current Tetris piece has collided with currently placed pieces without searching through the whole board (which is what I would need to do if I used lists). I have run into the issue that I am not sure how to convert this array to Text or String to output it to the console.
For now I am working with just one row of the board, which I initialise with the following function:
gameBoardWidth = 10 :: Int
initBoard :: UArray Int Char
initBoard = runSTUArray $ do
let lastCol = gameBoardWidth - 1
row <- newArray (0,lastCol) ' '
return row
Now I am not sure how to get the Char out of the array for printing. From the standard Array interface elems looks like what I need, but that does not appear to work on UArrays.
*Main Console Lib Paths_haskell_tetris Data.Array> elems initBoard
• Couldn't match expected type ‘Array i0 e’
with actual type ‘Data.Array.Base.UArray Int Char’
The other idea I had was to try and use the readArray function in a do block, but I am not sure how to concatenate the result of each string in a functional language
Just in case my issue is to do with the packages I have imported, these are my imports:
import Data.Array.Unboxed
import Data.Array.ST
import Control.Monad
import Control.Monad.ST
Your test uses Data.Array.elems which only works on Array.
You need to call instead the class method Data.Array.IArray.elems which works on any immutable array, including UArray.
Here is a snippet of code I have to convert a numpy array to c_float ctype array so I can pass it to some functions in C language:
arr = my_numpy_array
arr = arr/255.
arr = arr.flatten()
new_arr = (c_float*len(arr))()
new_arr[:] = arr
but since the last line is actually a for loop and we all know how notorious python is when it comes to for loops for a medium size image array it takes about 0.2 seconds!! so this one line is right now the bottle neck of my whole pipeline. I want to know if there is any faster way of doing it.
Update
Please note "to pass to a function in C" in the question. To be more specific I want to put a numpy array in IMAGE data structure and pass it to rgbgr_image function. You can find both here
The OP's answer makes 4 copies of the my_numpu_array, at least 3 of which should be unnecessary. Here's a version that avoids them:
# random array for demonstration
my_numpy_array = np.random.randint(0, 255, (10, 10))
# copy my_numpy_array to a float32 array
arr = my_numpy_array.astype(np.float32)
# divide in place
arr /= 255
# reshape should return a view, not a copy, unlike flatten
ctypes_arr = np.ctypeslib.as_ctypes(arr.reshape(-1))
In some circumstances, reshape will return a copy, but since arr is guaranteed to own it's own data, it should return a view here.
So I managed to do it in this weird way using numpy:
arr = my_numpu_array
arr = arr/255.
arr = arr.flatten()
arr_float32 = np.copy(arr).astype(np.float32)
new_arr = np.ctypeslib.as_ctypes(arr_float32)
In my case it works 10 times faster.
[Edit]: I don't know why it doesn't work without np.copy or with reshape(-1). So it would be awesome if anyone can explain.
I'm trying to convert the Long to array byte. This code block is working but this solution is a Java solution. I'm looking for a good solution in Scala. How can I convert the Long to array byte in Scala way?
val arrayByteFromLong: Array[Byte] = ByteBuffer.allocate(8).putLong(myLong).array()
You can leverage scala.math.BigInt:
import scala.math.BigInt
val arrayByteFromLong: Array[Byte] = BigInt(myLong).toByteArray
If you want to also pad the array to 8 Bytes you can do (quick-and-dirty not so efficient version):
arrayByteFromLong.reverse.padTo(8,0).reverse
For unpacking complex binary strings with mixed doubles and integers using Ruby's String.unpack I need to determine offsets within the binary string. Commonly, doubles are 8 bytes and integers are 4 bytes, but in order to make my code machine-independent, I would like to query these sizes from within my Ruby code.
What is the easiest way to determine the size of integers and doubles from within Ruby, i.e., request the response of a request to C's sizeof( type ) method?
I have found a workable solution, by using Array#pack, the inverse method of String#unpack. By packing a string of 1 integer or 1 double it is possible to determine the size of integers and doubles from within ruby itself which gives the same results as using sizeof(int) or sizeof(double) (because of the implementation of Array#pack in C):
[1.to_i].pack("i").size # 4
[1.to_f].pack("d").size # 8
This is so obvious that i am probably missing the point:
puts 1.size #=> 4
puts (256**10 - 1).size #=> 12
I'm not familiar with the Ruby language or its build/distribution system. However, the existence of Rake suggests that it's possible to create custom files on package installation - for example a configuration file which exports the result of the following commands as Ruby constants:
echo __SIZEOF_INT__ | gcc -E -P -
echo __SIZEOF_DOUBLE__ | gcc -E -P -
My current solution is to have a small C program, sizeofdouble.c:
#include <stdio.h>
int main() {
printf( "%lu", sizeof(double) );
}
which, when compiled to a sizeofdouble executable can be called from ruby with
sizeofdouble = `./sizeofdouble`.to_i
but that's an ugly hack, rather than a clean solution to this problem.
I know I'm late to the party here, but ruby's Fiddle library exposes the sizes of all the data types you might want as constants. Here are the results on my system (64-bit MacOS 10.13):
require 'fiddle'
Fiddle::SIZEOF_SHORT # => 2
Fiddle::SIZEOF_INT # => 4
Fiddle::SIZEOF_DOUBLE # => 8
Fiddle::SIZEOF_SIZE_T # => 8
Fiddle::SIZEOF_SSIZE_T # => 8
See the ruby docs here