Remove Python Console Duplicates from Lists
by Dinesh[ Edit ] 2012-08-27 12:56:35
Remove Python Console Duplicates from Lists
If you want to remove duplicates from a list, just put every element into a dict as a key (e.g. with None as value) and check dict.keys(). I found this optimized version in the WWW:
from operator import setitem
def distinct(l):
d = {}
map(setitem, (d,)*len(l), l, [])
return d.keys()
This makes use of the fact that map fills up shorter lists (in this case the empty one) with None. Newer Python versions allow for an even more concise formulation of the above:*
def distinct(l):
return dict.fromkeys(l).keys() # works with python 2.3+
def distinct24(l):
return list(set(l)) # needs python 2.4 or later
These are clearly concise enough to be used in-place. Note that all variants so far have two limitations:
The elements are returned in arbitrary order.
Lists with unhashable elements (e.g. sub-lists) may not be processed.
