A faster way to merge a list of 1D-arrays? - arrays

Question

I have a function distance that take a natural number as an input and return a 1-D array of length 199. My goal is to merge all the arrays distance(0), ..., distance(499). My code to do so is as follows:
import numpy as np
np.random.seed(42)
n = 200
d = 500
sample = np.random.uniform(size = [n, d])
def distance(i):
value = list(sample[i, 0:3])
temp = value - sample[(i + 1):n, 0:3]
return np.sqrt(np.sum(temp**2, axis = 1))
temp = [distance(i) for i in range(n - 1)]
result = [j for i in temp for j in i]
Because I work with large d, I want to optimize as good as possible. I would like to ask for a faster way to merge such arrays.
Thank you so much!

Answer 1

If you are just trying to compute the pairwise distance:
from scipy.spatial.distance import cdist
dist = cdist(sample[:,:3], sample[:,:3])
Of course you get back a symmetric array with all pairwise distances. To get your result, you can do:
result = dist[np.triu_indices(n,k=1)]
Regarding the broadcasting comment, cdist will do something similar to this:
dist = np.sum((sample[None,:,:3]-sample[:,None,:3])**2, axis=-1)**0.5
For reference, below is the run time for each:
%%timeit -n 100
temp = [distance(i) for i in range(n - 1)]
result = [j for i in temp for j in i]
6.41 ms ± 197 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit -n 100
temp = [distance(i) for i in range(n - 1)]
result = np.hstack(temp)
4.86 ms ± 295 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit -n 100
temp = [distance(i) for i in range(n - 1)]
result = np.concatenate(temp)
4.28 ms ± 175 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit -n 100
dist = np.sum((sample[None,:,:3]-sample[:,None,:3])**2, axis=-1)**0.5
result = dist[np.triu_indices(n,k=1)]
1.47 ms ± 61 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit -n 100
dist = cdist(sample[:,:3], sample[:,:3])
result = dist[np.triu_indices(n,k=1)]
415 µs ± 26.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)