Python list comprehension for Numpy

Question

I'm looking for list-comprehension method or similar in Numpy to eliminate use of a for-loop eg. index_values is a Python dictionary list of lists (each list containing a different number of index values) and s is a numpy vector:

for i in range(33):
    s[index_values[i]] += 4.1

Is there a method available that allows eliminating the for-loop?

I don't think there's a more general way than using python's list comprehensions... but if you're doing something specific, like a specific mathematical operation, there's probably a way. What are you trying to do? — Emil Stenström, Commented Jan 1, 2012 at 22:04

senderle · Accepted Answer · 2012-01-02 18:32:42Z

I don't fully understand what kind of object index_values is. But if it were an ndarray, or could be converted to an ndarray, you could just do this:

>>> s = numpy.arange(20)
>>> index_values = (numpy.random.random((3, 3)) * 20).astype('i')
>>> s[index_values] = 4
>>> s
array([ 0,  1,  4,  4,  4,  5,  6,  4,  8,  4,  4, 11, 12, 
       13,  4, 15,  4,  4,  4, 19])

Edit: But it seems that won't work in this case. On the basis of your edits and comments, here's a method I think might work for you. A random list of lists with varying lengths...

>>> index_values = [list(range(x, x + random.randrange(1, 5)))
...                 for x in [random.randrange(0,50) for y in range(33)]]

...isn't hard to convert into an array:

>>> index_value_array = numpy.fromiter(itertools.chain(*index_values), 
                                       dtype='i')

If you know the length of the array, specify the count for better performance:

>>> index_value_array = numpy.fromiter(itertools.chain(*index_values), 
                                       dtype='i', count=83)

Since your edit indicates that you want histogram-like behavior, simple indexing won't do, as pointed out by Robert Kern. So use numpy.histogram:

>>> hist = numpy.histogram(index_value_array, bins=range(0, 51))

histogram is really constructed for floating point histograms. This means that bins has to be a bit larger than expected because the last value is included in the last bin, and so 48 and 49 would be in the same bin if we used the more intuitive range(0, 50). The result is a tuple with an array of n counts and an array of n + 1 bin borders:

>>> hist
(array([2, 2, 1, 2, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 5, 5, 5, 3, 3, 
        3, 3, 3, 2, 1, 0, 2, 3, 3, 1, 0, 2, 3, 2, 2, 2, 3, 2, 1, 1, 2, 2, 
        2, 0, 0, 0, 1, 0]), 
 array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
        17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
        34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]))

Now we can scale the counts up by a factor of 4.1 and perform vector addition:

>>> s = numpy.arange(50, dtype='f')
>>> hist[0] * 4.1 + s
array([  8.2,   9.2,   6.1,  11.2,   8.1,   5. ,   6. ,   7. ,  12.1,
        13.1,  14.1,  15.1,  16.1,  13. ,  18.1,  19.1,  20.1,  37.5,
        38.5,  39.5,  32.3,  33.3,  34.3,  35.3,  36.3,  33.2,  30.1,
        27. ,  36.2,  41.3,  42.3,  35.1,  32. ,  41.2,  46.3,  43.2,
        44.2,  45.2,  50.3,  47.2,  44.1,  45.1,  50.2,  51.2,  52.2,
        45. ,  46. ,  47. ,  52.1,  49. ])

I have no idea if this suits your purposes, but it seems like a good approach, and should probably happen at near c speed since it uses only numpy and itertools.

This is the right answer for numpy arrays. The only thing to watch out for is extending this to augmented assignment. When there are repetitions in index_values, the augmented assignment won't happen repeatedly as it would in the full for loop (for reasons that are complicated to go into). So you can't use this kind of indexing for doing ad hoc histograms, as many people try to do. — Robert Kern, Commented Jan 1, 2012 at 22:38
index_values is a Python dictionary list of lists eg. [[3, 6, 7], [5, 7, 11, 25, 99], [8, 45]]. index_values cannot be an ndarray - sorry! — Henry Thornton, Commented Jan 1, 2012 at 22:48
@dbv, I think what's confusing me is "dictionary list of lists." I don't know what a "dictionary list" is. Do you mean simply a dictionary of lists? If so, why are you using ints to index it? Why not just use a list of lists? A list of lists can be passed to numpy.array to produce an ndarray of the corresponding shape like so: numpy.array([[1, 2, 3], [1, 2, 3]]). If you must use a dictionary, you can at least gain some speed with a comprehension like so: numpy.array([d[i] for i in range(2)]). — senderle, Commented Jan 2, 2012 at 1:13
@dev using an ndarray to index s is almost definitely the way to go, let us know why you think you cannot turn index_values into an ndarray and maybe we can come up with a fix. Also there is a relatively simple workaround to the issue that Robert Kern raised, let me know if it's relevant in your case and I'll post the code. — Bi Rico, Commented Jan 2, 2012 at 6:44
guys, i'm going to delete this question later today and re-formulate as a new question. thank-you and hopefully see you on the other side. — Henry Thornton, Commented Jan 2, 2012 at 12:58

Ricardo Cárdenes · Accepted Answer · 2012-01-01 22:31:55Z

1

What about:

s[reduce(lambda x,y: x+y, [index_values[x] for x in range(33)], [])] = 4.1

edited Jan 1, 2012 at 22:31

answered Jan 1, 2012 at 22:25

Ricardo Cárdenes

9,1721 gold badge23 silver badges35 bronze badges

see comment under Tadeck. Thanks!
– Henry Thornton
Commented Jan 1, 2012 at 22:58
Under Tadeck? :? Can't see any Tadek at the moment :) (people keeps removing their answers and it's difficult to keep track!)
– Ricardo Cárdenes
Commented Jan 1, 2012 at 23:05
Yeah, just saw that @Tadeck's answer has been removed. I've edited the original question above to show that the Numpy vector s is updated in-place.
– Henry Thornton
Commented Jan 1, 2012 at 23:22
Yeah, of course. My comment on @Tadeck's answer was on the use he was making of comprehensions and filter, not on the final result. Of course I understand you intend to modify parts of s, not to create a new array :)
– Ricardo Cárdenes
Commented Jan 1, 2012 at 23:24
The reduce function is accumulating the values in index_values[x]. Say index_value[x] = [4, 7, 9, 11, 13], then we want s[[4, 7, 9, 11, 13]] = s[[4, 7, 9, 11, 13]] + 4.1 ie. s[4] += 4.1, s[7] += 4.1, etc. for x in range(33).
– Henry Thornton
Commented Jan 2, 2012 at 0:22

| Show 1 more comment

Collectives™ on Stack Overflow

Python list comprehension for Numpy

2 Answers 2

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Linked

Related