373

This is how I am accustomed to filter, map, and reduce working in Python 2:

>>> def f(x):
        return x % 2 != 0 and x % 3 != 0
>>> filter(f, range(2, 25))
[5, 7, 11, 13, 17, 19, 23]

>>> def cube(x):
        return x*x*x
>>> map(cube, range(1, 11))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]

>>> def add(x,y):
        return x+y
>>> reduce(add, range(1, 11))
55

However, all of these seem to break in Python 3:

>>> filter(f, range(2, 25))
<filter object at 0x0000000002C14908>

>>> map(cube, range(1, 11))
<map object at 0x0000000002C82B70>

>>> reduce(add, range(1, 11))
Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    reduce(add, range(1, 11))
NameError: name 'reduce' is not defined

Why are the results different? How can I get Python 3 code to work like the Python 2 code did?

See also: What is the problem with reduce()? for specific motivation for the change to put reduce into a standard library module rather than leaving it as a builtin.

See Getting a map() to return a list in Python 3.x for more specific answers about map.

Improve this question
1
  • 1
    In short, list is not the only datatype. If you want a list, say you want a list. But in most cases, you want something else anyway. Commented Dec 18, 2018 at 8:02

8 Answers 8

Reset to default
389

You can read about the changes in What's New In Python 3.0. You should read it thoroughly when you move from 2.x to 3.x since a lot has been changed.

The whole answer here are quotes from the documentation.

Views And Iterators Instead Of Lists

Some well-known APIs no longer return lists:

  • [...]
  • map() and filter() return iterators. If you really need a list, a quick fix is e.g. list(map(...)), but a better fix is often to use a list comprehension (especially when the original code uses lambda), or rewriting the code so it doesn’t need a list at all. Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).
  • [...]

Builtins

  • [...]
  • Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
  • [...]
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Adding list(map(...) everywhere .. how in the world is that helping readability.. python can't seem to handle progressive / streaming application of functional combinators. Other languages I can chain a dozen operations against a collection in a row and it's readable. Here? what do you want - a dozen way nested in ??
If you're working in an imperative context, then a for-loop is probably the more readable option. But there are good reasons to prefer a functional context--and breaking from that to go back to procedural can be pretty darn ugly.
@javadba Are you sure in a "streaming application" you need to add the list call at all? I thought the meaning of "streaming" is "no list is created at all; process each element of the input fully before moving on to the next".
I still cant grasp how a readability argument leads to such a change. If it was for performance reasons I might understand...
A "quick fix" (read: hack) is to use list(map...) but notice the "better fix" is to use a list comprehension instead - like [Foo(x) for x in mylist]. This doesn't lead to adding list() everywhere and longer term may be better. (@javadba FYI)
|
104

The functionality of map and filter was intentionally changed to return iterators, and reduce was removed from being a built-in and placed in functools.reduce.

So, for filter and map, you can wrap them with list() to see the results like you did before.

>>> def f(x): return x % 2 != 0 and x % 3 != 0
...
>>> list(filter(f, range(2, 25)))
[5, 7, 11, 13, 17, 19, 23]
>>> def cube(x): return x*x*x
...
>>> list(map(cube, range(1, 11)))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>> import functools
>>> def add(x,y): return x+y
...
>>> functools.reduce(add, range(1, 11))
55
>>>

The recommendation now is that you replace your usage of map and filter with generators expressions or list comprehensions. Example:

>>> def f(x): return x % 2 != 0 and x % 3 != 0
...
>>> [i for i in range(2, 25) if f(i)]
[5, 7, 11, 13, 17, 19, 23]
>>> def cube(x): return x*x*x
...
>>> [cube(i) for i in range(1, 11)]
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>>

They say that for loops are 99 percent of the time easier to read than reduce, but I'd just stick with functools.reduce.

Edit: The 99 percent figure is pulled directly from the What’s New In Python 3.0 page authored by Guido van Rossum.

Improve this answer

6 Comments

You do not need to create extra functions in list comprehensions. Just use [i*i*i for i in range(1,11)]
You are absolutely correct. I kept the function in the list comprehension examples to keep it looking similar to the filter/map examples.
i**3 is also equivalent of i*i*i
@Breezer actually i**3 will call i.__pow__(3) and i*i*i i.__mul__(i).__mul__(i) (or something like that). With ints it doesn't matter but with numpy numbers/custom classes it might even produce different results.
I have noticed that whenever we hear that "Guido made decision X" that pain is a likely outcome. This is a great example: list(list(list(.. ))) to do what was already verbose in python.
|
12

As an addendum to the other answers, this sounds like a fine use-case for a context manager that will re-map the names of these functions to ones which return a list and introduce reduce in the global namespace.

A quick implementation might look like this:

from contextlib import contextmanager    

@contextmanager
def noiters(*funcs):
    if not funcs: 
        funcs = [map, filter, zip] # etc
    from functools import reduce
    globals()[reduce.__name__] = reduce
    for func in funcs:
        globals()[func.__name__] = lambda *ar, func = func, **kwar: list(func(*ar, **kwar))
    try:
        yield
    finally:
        del globals()[reduce.__name__]
        for func in funcs: globals()[func.__name__] = func

With a usage that looks like this:

with noiters(map):
    from operator import add
    print(reduce(add, range(1, 20)))
    print(map(int, ['1', '2']))

Which prints:

190
[1, 2]

Just my 2 cents :-)

Improve this answer

1 Comment

python as a language is a mess - but it has v good to excellent libraries: numpy, pandas, statsmodels and friends.. I had been buliding convenience libraries like you show here to reduce the pain of the native language - but have lost the energy and try not to stray far from a data.frame / datatable, or xarray. But kudos for trying..
10

Since the reduce method has been removed from the built in function from Python3, don't forget to import the functools in your code. Please look at the code snippet below.

import functools
my_list = [10,15,20,25,35]
sum_numbers = functools.reduce(lambda x ,y : x+y , my_list)
print(sum_numbers)
Improve this answer

Comments

6

One of the advantages of map, filter and reduce is how legible they become when you "chain" them together to do something complex. However, the built-in syntax isn't legible and is all "backwards". So, I suggest using the PyFunctional package (https://pypi.org/project/PyFunctional/). Here's a comparison of the two:

flight_destinations_dict = {'NY': {'London', 'Rome'}, 'Berlin': {'NY'}}

PyFunctional version

Very legible syntax. You can say:

"I have a sequence of flight destinations. Out of which I want to get the dict key if city is in the dict values. Finally, filter out the empty lists I created in the process."

from functional import seq  # PyFunctional package to allow easier syntax

def find_return_flights_PYFUNCTIONAL_SYNTAX(city, flight_destinations_dict):
    return seq(flight_destinations_dict.items()) \
        .map(lambda x: x[0] if city in x[1] else []) \
        .filter(lambda x: x != []) \

Default Python version

It's all backwards. You need to say:

"OK, so, there's a list. I want to filter empty lists out of it. Why? Because I first got the dict key if the city was in the dict values. Oh, the list I'm doing this to is flight_destinations_dict."

def find_return_flights_DEFAULT_SYNTAX(city, flight_destinations_dict):
    return list(
        filter(lambda x: x != [],
               map(lambda x: x[0] if city in x[1] else [], flight_destinations_dict.items())
               )
    )
Improve this answer

3 Comments

Would it also work to try something like: python def find_return_flights(city): return [key for key, val in flight_destinations_dict.items() if city in val]
It probably would, but that wouldn't be strictly speaking functional programming.
Awesome! Coming from JS this is just what I needed!
2 
from functools import reduce

def f(x):
    return x % 2 != 0 and x % 3 != 0

print(*filter(f, range(2, 25)))
#[5, 7, 11, 13, 17, 19, 23]

def cube(x):
    return x**3
print(*map(cube, range(1, 11)))
#[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]

def add(x,y):
    return x+y

reduce(add, range(1, 11))
#55

It works as is. To get the output of map use * or list

Improve this answer

1 Comment

I like the *-notation, but note that it does not produce the output you indicate. It doesn't produce lists, rather, it unpacks the filter- and map objects.
0

Here are examples of Filter, map and reduce functions.

numbers = [10,11,12,22,34,43,54,34,67,87,88,98,99,87,44,66]

filter:

oddNumbers = list(filter(lambda x: x%2 != 0, numbers))

print(oddNumbers)

map:

multiplyOf2 = list(map(lambda x: x*2, numbers))

print(multiplyOf2)

The reduce function, since it is not commonly used, was removed from the built-in functions in Python 3. It is still available in the functools module, so you can do:

from functools import reduce

sumOfNumbers = reduce(lambda x,y: x+y, numbers)

print(sumOfNumbers)
Improve this answer

Comments

0

functools.reduce is also useful for drilling deep into nested-keys data structures:

from functools import reduce
from operator import getitem

# ...

megayaml = defaultdict(lambda: {})

for datum in someInput:
    deepK = datum['Key'].split('/')
    value = b64decode(datum['Value'])
     
    start = megayaml[f"{some}-{complicated}-{toplevel}-{key}.irrelevant"]

    parents, leaf = deepK[:-1], deepK[-1]
    bottom = reduce(getitem, parents, start)
    bottom[leaf] = value

The operator.getitem that I imported is, of course, morally equivalent to lambda drill, step: drill[step]. In combination with collections.defaultdict, this snippet allows me to "unflatten" a tree, reconstructing it from a tabular dump.

Yes Guido, it's easy to say "reduce can always be rewritten as a for-loop" — except when you're already several loops deep, and it really isn't. Then reduce(getitem, …) is much simpler.

As for the laziness of map & filter results, there's a trivial idiom:

data = filter(…)
data = map(…, data)
data = list(data) # force lazy list
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.