Python - Check for list duplicates

Question

I have a list in python like this

my_list = [{'1':'A','2':'B'}]

And now I want to append some more JSON to my_list but first I want to check if the JSON I am adding already exists, how could I go about this?

So if I was going to append {'2':'B','1':'A'}, I would that not to be added?

How could I do this?

Thanks

Answer 1

You can check if two dictionaries are identical with ==

In [2]: {'1':'A','2':'B'}=={'2':'B','1':'A'}
Out[2]: True

Therefore to check if JSON exists in my_list you can simply do

if JSON in my_list:
    #blahblah

Update:

To use set with your data, you can define your own subclass and implement __hash__() method. You can start from here:

class MyJSON(dict):
    def __hash__(self):
        return hash(json.dumps(self,sort_keys=True))

Example:

a=MyJSON({'1':'A','2':'B'})
b=MyJSON({'1':'A','2':'C'})
c=MyJSON({'2':'B','1':'A'}) ## should be equal to a
print a==c # should be True
my_set=set()
my_set.add(a)
my_set.add(b)
my_set.add(c)
for item in my_set:
    print item,
## output is {'1': 'A', '2': 'C'} {'1': 'A', '2': 'B'}

Answer 2

The canonical data structure to use for avoiding duplicates is the set. As you've mentioned, you can't do this because you'll be adding dicts to the set, which are unhashable.

The usual fix for this is either to define a custom dict-like object that is hashable, or to freeze your dict into something hashable and add that to the set. We'll do the latter.

my_list = [{1:2, 3:4}, {3:4, 1:2}]
result = set()

for json_data in my_list:
    result.add(frozenset(json_data.items()))

print(result)
# {frozenset({(1, 2), (3, 4)})}