14

Take this code:

import itertools as it
import numpy as np
data = ['a','b','c','d']
dw = np.array([1, 3], dtype=np.int64)
print(list(it.islice(data,dw[0],dw[1],1)))

On Python 2.7 it prints ['b', 'c',] as expected.

On Python 3.6 it throws an exception:

ValueError: Stop argument for islice() must be None or an integer: 0 <= x <= sys.maxsize.

The same goes for np.int32, and other methods of the itertools package throw similar errors, e.g. when you use permutations you get TypeError: Expected int as r.

I couldn't find much on this apart from this numpy issue and related ones, but that one was closed 3 years ago implying it was solved.

And basic things like indexing with numpy ints data[dw[0]] or boolean comparisons like dw[0] == 1 work just fine.

Am I missing something? Could this be a Python 3 bug?

Improve this question
3
  • the simplest fix would be to use type casting: int(dw[0]), int(dw[1]) Commented Jun 1, 2017 at 10:15
  • Even where it works, py2.7, the use an array instead of a list will slower. Commented Jun 1, 2017 at 10:24
  • 2
    Exactly the sort of thing that makes me hate Python. Commented Jun 1, 2017 at 20:15

4 Answers 4

Reset to default
14

a numpy.int64 is apparently not a subclass of int

a, b = dw[0], dw[1]

type(a)

numpy.int64

isinstance(a, int)

False

Numpy documentation

The documentation mentions this explicitly

Warning

The int_ type does not inherit from the int built-in under Python 3, because type int is no longer a fixed-width integer type.

Solution

print(list(it.islice(data, int(dw[0]) , int(dw[1]), 1)))

or numpy slicing

data[dw[0]:dw[1]:1]
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

So why does itertools.islice require ints while regular list slicing is fine without them (e.g. data[dw[0]])?
@Chris_Rands: Regular list slicing and indexing supports anything that implements the __index__ method. NumPy integer types implement that method.
@MarkDickinson Thank you; is there a reason for that design? Why not let islice support anything with the __index__ method too? Although perhaps there is no real need for this given how easy it is to cast to int...
@Chris_Rands: I think that's a question for bugs.python.org.
@MarkDickinson Ok, a bug issue has been raised (not by me) bugs.python.org/issue30537
|
6

I'm not sure if it's a bug in Python 3 or not, but it looks like the behaviour has changed since 2.7. As the numpy issue you linked described, under py27, either numpy.int32 or numpy.int64 would appear to be a subclass of int (depending on whether you use a 32- or 64-bit build of Python); under py3, the types are no longer related (numpy has fixed-width numeric types, python's int is variable-width).

The implementation of itertools.islice requires its arguments to be objects of type PyLong (which is the Python API name for the Python int type). Specifically, it calls PyLong_AsSize_t, which converts a Python object into a C size_t value. This method seems to require that its argument is actually a Python int object, since it calls PyLong_Check. I think this method is broadly equivalent to Python's isinstance(obj, int), which explains the difference in behaviour between py2 and py3 here.

Normal list indexing uses another more tolerant method to coerce arguments into positive integer values, called PyNumber_AsSsize_t. This checks if its argument is an int, and, if not, falls back to trying to call its argument's __index__ method; as @MarkDickinson points out, numpy's numeric types implement this method, so everything works fine. Perhaps this would be a more intuitive thing for itertools.islice to do.

Improve this answer

2 Comments

Given that things like indexing or boolean comparisons work just fine shouldn't this considered to be a bug? Apparently many more basic functionalities are implemented in a way that works.
@Khris I'm personally inclined to view this as a shortcoming of islice, yes. It would seem more sensible to have islice consume any argument which is in some way "numeric" (including, for example, the base type np.integer).
3

This looks like a case for the __index__ magic method (which numpy's integers already implement). I suggest raising an issue on the tracker, requesting this as an enhancement - that islice accept any object that implements __index__.

Improve this answer

Comments

1

If you want to keep a islice/slice-like object, use np.s_:

slice = np.s_[dw[0]: dw[1]: 1]
data[slice]

['b', 'c']

since np.s_ is a numpy object, it doesn't mind the numpy integers.

Improve this answer

5 Comments

Regular slicing was never an issue; list(it.islice(data,np.s_[dw[0]])) still fails
The point is to skip creating an iterable object altogether and use a numpy object with equivalent function. The reason why itertools fails is covered in other answers.
Thus the introductory phrase "If you want to keep a slice-like object". i.e. if you're going to be using that slice on a bunch of different lists
But then why not just use slice() like a = slice(dw[0],dw[1]); data[a]?
. . . Because I had a completely unrelated error pop when I tried slice() and I assumed it was the same integer type mismatch. Whoops. np.s_[] still looks nicer, I guess!

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.