Collections

Unite

merge(*colls)

Merges several collections of same type into one: dicts, sets, lists, tuples, iterators or strings. For dicts values of later dicts override values of former ones with same keys.

Can be used in variety of ways, but merging dicts is probably most common:

def utility(**options):
    defaults = {...}
    options = merge(defaults, options)
    ...

If you merge sequences and don’t need to preserve collection type, then use concat() or lconcat() instead.

join(colls)

Joins collections of same type into one. Same as merge(), but accepts iterable of collections.

Use cat() and lcat() for non-type preserving sequence join.

Transform and select

All functions in this section support Extended function semantics.

walk(f, coll)

Returns a collection of same type as coll consisting of its elements mapped with the given function:

walk(inc, {1, 2, 3}) # -> {2, 3, 4}
walk(inc, (1, 2, 3)) # -> (2, 3, 4)

When walking dict, (key, value) pairs are mapped, i.e. this lines flip() dict:

swap = lambda (k, v): (v, k)
walk(swap, {1: 10, 2: 20})

walk() works with strings too:

walk(lambda x: x * 2, 'ABC')   # -> 'AABBCC'
walk(compose(str, ord), 'ABC') # -> '656667'

One should use map() when there is no need to preserve collection type.

walk_keys(f, coll)

Walks keys of coll, mapping them with the given function. Works with mappings and collections of pairs:

walk_keys(str.upper, {'a': 1, 'b': 2}) # {'A': 1, 'B': 2}
walk_keys(int, json.loads(some_dict))  # restore key type lost in translation

Important to note that it preserves collection type whenever this is simple dict, defaultdict, OrderedDict or any other mapping class or a collection of pairs.

walk_values(f, coll)

Walks values of coll, mapping them with the given function. Works with mappings and collections of pairs.

Common use is to process values somehow:

clean_values = walk_values(int, form_values)
sorted_groups = walk_values(sorted, groups)

Hint: you can use partial(sorted, key=...) instead of sorted() to sort in non-default way.

Note that walk_values() has special handling for defaultdicts. It constructs new one with values mapped the same as for ordinary dict, but a default factory of new defaultdict would be a composition of f and old default factory:

d = defaultdict(lambda: 'default', a='hi', b='bye')
walk_values(str.upper, d)
# -> defaultdict(lambda: 'DEFAULT', a='HI', b='BYE')
select(pred, coll)

Filters elements of coll by pred constructing a collection of same type. When filtering a dict pred receives (key, value) pairs. See select_keys() and select_values() to filter it by keys or values respectively:

select(even, {1, 2, 3, 10, 20})
# -> {2, 10, 20}

select(lambda (k, v): k == v, {1: 1, 2: 3})
# -> {1: 1}
select_keys(pred, coll)

Select part of a dict or a collection of pairs with keys passing the given predicate.

This way a public part of instance attributes dictionary could be selected:

is_public = complement(re_tester('^_'))
public = select_keys(is_public, instance.__dict__)
select_values(pred, coll)

Select part of a dict or a collection of pairs with values passing the given predicate:

# Leave only str values
select_values(isa(str), values)

# Construct a dict of methods
select_values(inspect.isfunction, cls.__dict__)
compact(coll)

Removes falsy values from given collection. When compacting a dict all keys with falsy values are removed.

Extract integer data from request:

compact(walk_values(silent(int), request_dict))

Dict utils

merge_with(f, *dicts)
join_with(f, dicts)

Merge several dicts combining values for same key with given function:

merge_with(list, {1: 1}, {1: 10, 2: 2})
# -> {1: [1, 10], 2: [2]}

merge_with(sum, {1: 1}, {1: 10, 2: 2})
# -> {1: 11, 2: 2}

join_with(first, ({n % 3: n} for n in range(100, 110)))
# -> {0: 102, 1: 100, 2: 101}
zipdict(keys, vals)

Returns a dict with the keys mapped to the corresponding vals. Stops pairing on shorter sequence end:

zipdict('abcd', range(4))
# -> {'a': 0, 'b': 1, 'c': 2, 'd': 3}

zipdict('abc', count())
# -> {'a': 0, 'b': 1, 'c': 2}
flip(mapping)

Flip passed dict swapping its keys and values. Also works for sequences of pairs. Preserves collection type:

flip(OrderedDict(['aA', 'bB']))
# -> OrderedDict([('A', 'a'), ('B', 'b')])
project(mapping, keys)

Returns a dict containing only those entries in mapping whose key is in keys.

Most useful to shrink some common data or options to predefined subset. One particular case is constructing a dict of used variables:

merge(project(__builtins__, names), project(globals(), names))
omit(mapping, keys)

Returns a copy of mapping with keys omitted. Preserves collection type:

omit({'a': 1, 'b': 2, 'c': 3}, 'ac')
# -> {'b': 2}
zip_values(*dicts)

Yields tuples of corresponding values of given dicts. Skips any keys not present in all of the dicts. Comes in handy when comparing two or more dicts:

error = sum((x - y) ** 2 for x, y in zip_values(result, reference))
zip_dicts(*dicts)

Yields tuples like key, (value1, value2, ...) for each common key of all given dicts. A neat way to process several dicts at once:

changed_items = [id for id, (new, old) in zip_dicts(items, old_items)
                 if abs(new - old) >= PRECISION]

lines = {id: cnt * price for id, (cnt, price) in zip_dicts(amounts, prices)}

See also zip_values().

get_in(coll, path, default=None)

Returns a value corresponding to path in nested collection:

get_in({"a": {"b": 42}}, ["a", "b"])    # -> 42
get_in({"a": {"b": 42}}, ["c"], "foo")  # -> "foo"
set_in(coll, path, value)

Creates a nested collection with the value set at specified path. Original collection is not changed:

set_in({"a": {"b": 42}}, ["a", "b"], 10)
# -> {"a": {"b": 10}}

set_in({"a": {"b": 42}}, ["a", "c"], 10)
# -> {"a": {"b": 42, "c": 10}}
update_in(coll, path, update, default=None)

Creates a nested collection with a value at specified path updated:

update_in({"a": {}}, ["a", "cnt"], inc, default=0)
# -> {"a": {"cnt": 1}}

Data manipulation

where(mappings, **cond)
lwhere(mappings, **cond)

Looks through each value in given sequence of dicts and returns an iterator or a list of all the dicts that contain all key-value pairs in cond:

lwhere(plays, author="Shakespeare", year=1611)
# => [{"title": "Cymbeline", "author": "Shakespeare", "year": 1611},
#     {"title": "The Tempest", "author": "Shakespeare", "year": 1611}]

Iterator version could be used for efficiency or when you don’t need the whole list. E.g. you are looking for the first match:

first(where(plays, author="Shakespeare"))
# => {"title": "The Two Gentlemen of Verona", ...}
pluck(key, mappings)
lpluck(key, mappings)

Returns an iterator or a list of values for key in each mapping in the given sequence. Essentially a shortcut for:

map(operator.itemgetter(key), mappings)
pluck_attr(attr, objects)
lpluck_attr(attr, objects)

Returns an iterator or a list of values for attr in each object in the given sequence. Essentially a shortcut for:

map(operator.attrgetter(attr), objects)

Useful when dealing with collections of ORM objects:

users = User.query.all()
ids = lpluck_attr('id', users)
invoke(objects, name, *args, **kwargs)
linvoke(objects, name, *args, **kwargs)

Calls named method with given arguments for each object in objects and returns an iterator or a list of results.

Content tests

is_distinct(coll, key=identity)

Checks if all elements in the collection are different:

assert is_distinct(field_names), "All fields should be named differently"

Uses key to differentiate values. This way one can check if all first letters of words are different:

is_distinct(words, key=0)
all([pred, ]seq)

Checks if pred holds for every element in a seq. If pred is omitted checks if all elements of seq are truthy – same as in built-in all():

they_are_ints = all(is_instance(n, int) for n in seq)
they_are_even = all(even, seq)

Note that, first example could be rewritten using isa() like this:

they_are_ints = all(isa(int), seq)
any([pred, ]seq)

Returns True if pred holds for any item in given sequence. If pred is omitted checks if any element of seq is truthy.

Check if there is a needle in haystack, using extended predicate semantics:

any(r'needle', haystack_strings)
none([pred, ]seq)

Checks if none of items in given sequence pass pred or is truthy if pred is omitted.

Just a stylish way to write not any(...):

assert none(' ' in name for name in names), "Spaces in names not allowed"

# Or same using extended predicate semantics
assert none(' ', names), "..."
one([pred, ]seq)

Returns true if exactly one of items in seq passes pred. Cheks for truthiness if pred is omitted.

some([pred, ]seq)

Finds first item in seq passing pred or first that is true if pred is omitted.

Low-level helpers

empty(coll)

Returns an empty collection of the same type as coll.

iteritems(coll)

Returns an iterator of items of a coll. This means key, value pairs for any dictionaries:

list(iteritems({1, 2, 42}))
# -> [1, 42, 2]

list(iteritems({'a': 1}))
# -> [('a', 1)]
itervalues(coll)

Returns an iterator of values of a coll. This means values for any dictionaries and just elements for other collections:

list(itervalues({1, 2, 42}))
# -> [1, 42, 2]

list(itervalues({'a': 1}))
# -> [1]
count(start=0, step=1)

Makes infinite iterator of values:
start, start + step, start + 2*step, ...
cycle(seq)

Cycles passed sequence indefinitely
yielding its elements one by one.
repeat(item[, n])

Makes an iterator yielding item for n times
or indefinitely if n is omitted.
repeatedly(f[, n])

Takes a function of no args, presumably with side effects,
and returns an infinite (or length n) iterator of calls to it.
iterate(f, x)

Returns an infinite iterator of x, f(x), f(f(x)), ...
re_all(regex, s, flags=0)

Lists all matches of regex in s.
re_iter(regex, s, flags=0)

Iterates over matches of regex in s.
first(seq)

Returns the first item in the sequence.
Returns None if the sequence is empty.
second(seq)

Returns second item in the sequence.
Returns None if there are less than two items in it.
last(seq)

Returns the last item in the sequence.
Returns None if the sequence is empty.
nth(seq)

Returns nth item in the sequence
or None if no such item exists.
some([pred, ]seq)

Finds first item in seq passing pred
or first that is true if pred is omitted.
take(n, seq)

Returns a list of first n items in the sequence,
or all items if there are fewer than n.
drop(n, seq)

Skips first n items in the sequence,
yields the rest.
rest(seq)

Skips first item in the sequence, yields the rest.
butlast(seq)

Yields all elements of the sequence but last.
takewhile([pred, ]seq)

Yields seq items as long as they pass pred.
dropwhile([pred, ]seq)

Skips elements of seq while pred passes
and then yields the rest.
split_at(n, seq)
lsplit_at(n, seq)


Splits the sequence at given position,
returning a tuple of its start and tail.
split_by(pred, seq)
lsplit_by(pred, seq)


Splits the start of the sequence,
consisting of items passing pred,
from the rest of it.
map(f, *seqs)
lmap(f, *seqs)


Extended versions of map() and list(map())
mapcat(f, *seqs)
lmapcat(f, *seqs)


Maps given sequence(s) and concatenates the results.
keep([f, ]*seqs)
lkeep([f, ]*seqs)


Maps seq with f and filters out falsy results.
Simply removes falsy values in one argument version.
pluck(key, mappings)
lpluck(key, mappings)


Yields or lists values for key in each mapping.
pluck_attr(attr, objects)
lpluck_attr(attr, objects)


Yields or lists values of attr of each object.
invoke(objects, name, *args, **kwargs)
linvoke(objects, name, *args, **kwargs)


Yields or lists results of the given method call
for each object in objects.
filter(pred, seq)
lfilter(pred, seq)


Extended versions of filter() and list(filter()).
remove(pred, seq)
lremove(pred, seq)


Removes items from seq passing given predicate.
distinct(pred, key=identity)
ldistinct(pred, key=identity)


Removes items having same key from seq.
Preserves order.
where(mappings, **cond)
lwhere(mappings, **cond)


Selects mappings containing all pairs in cond.
without(seq, *items)
lwithout(seq, *items)


Returns sequence without items,
preserves order.
cat(seqs)
lcat(seqs)


Concatenates passed sequences.
concat(*seqs)
lconcat(*seqs)


Concatenates several sequences.
flatten(seq, follow=is_seqcont)
lflatten(seq, follow=is_seqcont)


Flattens arbitrary nested sequence,
dives into when follow(item) is truthy.
interleave(*seqs)

Yields first item of each sequence,
then second one and so on.
interpose(sep, seq)

Yields items of seq separated by sep.
lzip(*seqs)

List version of zip()
chunks(n, [step, ]seq)
lchunks(n, [step, ]seq)


Chunks seq into parts of length n or less.
Skips step items between chunks.
partition(n, [step, ]seq)
lpartition(n, [step, ]seq)


Partitions seq into parts of length n.
Skips step items between parts.
Non-fitting tail is ignored.
partition_by(f, seq)
lpartition_by(f, seq)


Partition seq into continuous chunks
with constant value of f.
split(pred, seq)
lsplit(pred, seq)


Splits seq items which pass pred
from the ones that don't.
count_by(f, seq)

Counts numbers of occurrences of values of f
on elements of seq.
count_reps(seq)

Counts repetitions of each value in seq.
group_by(f, seq)

Groups items of seq by f(item).
group_by_keys(get_keys, seq)

Groups elements of seq by multiple keys.
group_values(seq)

Groups values of (key, value) pairs by keys.
ilen(seq)

Consumes the given iterator and returns its length.
reductions(f, seq[, acc])
lreductions(f, seq[, acc])


Constructs intermediate reductions of seq by f.
sums(seq[, acc])
lsums(seq[, acc])


Returns a sequence of partial sums of seq.
all([pred, ]seq)

Checks if all items in seq pass pred.
any([pred, ]seq)

Checks if any item in seq passes pred.
none([pred, ]seq)

Checks if none of the items in seq pass pred.
one([pred, ]seq)

Checks if exactly one item in seq passes pred.
pairwise(seq)

Yields all pairs of neighboring items in seq.
with_prev(seq, fill=None)

Yields each item from seq with the one preceding it.
with_next(seq, fill=None)

Yields each item from seq with the next one.
zip_values(*dicts)

Yields tuples of corresponding values of given dicts.
zip_dicts(*dicts)

Yields tuples like (key, val1, val2, ...)
for each common key in all given dicts.
tree_leaves(root, follow=is_seqcont, children=iter)
ltree_leaves(root, follow=is_seqcont, children=iter)


Lists or iterates over tree leaves.
tree_nodes(root, follow=is_seqcont, children=iter)
ltree_nodes(root, follow=is_seqcont, children=iter)


Lists or iterates over tree nodes.
merge(*colls)

Merges several collections of same type into one:
dicts, sets, lists, tuples, iterators or strings
For dicts later values take precedence.
merge_with(f, *dicts)

Merges several dicts combining values with given function.
join(colls)

Joins several collections of same type into one.
Same as merge() but accepts sequence of collections.
join_with(f, *dicts)

Joins several dicts combining values with given function.
walk(f, coll)

Maps coll with f, but preserves collection type.
walk_keys(f, coll)

Walks keys of coll, mapping them with f.
Works with dicts and collections of pairs.
walk_values(f, coll)

Walks values of coll, mapping them with f.
Works with dicts and collections of pairs.
select(pred, coll)

Filters elements of coll by pred
constructing a collection of same type.
select_keys(pred, coll)

Select part of coll with keys passing pred.
Works with dicts and collections of pairs.
select_values(pred, coll)

Select part of coll with values passing pred.
Works with dicts and collections of pairs.
compact(coll)

Removes falsy values from given collection.
All collections functions work with dicts.
These are targeted specifically at them.
flip(mapping)

Flip passed dict swapping its keys and values.
zipdict(keys, vals)

Creates a dict with keys mapped to the corresponding vals.
itervalues(coll)

Yields values of the given collection.
iteritems(coll)

Yields (key, value) pairs of the given collection.
project(mapping, keys)

Leaves only given keys in mapping.
omit(mapping, keys)

Removes given keys from mapping.
empty(coll)

Returns an empty collection of the same type as coll.
get_in(coll, path, default=None)

Returns a value at path in the given nested collection.
set_in(coll, path, value)

Creates a copy of coll with the value set at path.
update_in(coll, path, update, default=None)

Creates a copy of coll with a value updated at path.
Most of functions in this section support extended semantics.
identity(x)

Returns its argument.
constantly(x)

Creates a function accepting any args, but always returning x.
func_partial(func, *args, **kwargs)

Like partial() but returns a real function.
partial(func, *args, **kwargs)

Returns partial application of func.
rpartial(func, *args)

Partially applies last arguments to func.
iffy([pred, ]action[, default=identity])

Creates a function, which conditionally applies action or default.
caller(*args, **kwargs)

Creates a function calling its argument with passed arguments.
re_finder(regex, flags=0)

Creates a function finding regex in passed string.
re_tester(regex, flags=0)

Creates a predicate testing passed strings with regex.
complement(pred)

Constructs a complementary predicate.
autocurry(func)

Creates a version of func returning its partial applications
until sufficient arguments are passed.
curry(func[, n])

Curries func into a chain of one argument functions.
Arguments are passed from left to right.
rcurry(func[, n])

Curries func from right to left.
compose(*fs)

Composes passed functions.
rcompose(*fs)

Composes fs, calling them from left to right.
juxt(*fs)
ljuxt(*fs)


Constructs a juxtaposition of the given functions.
Result returns a list or an iterator of results of fs.
all_fn(*fs)

Constructs a predicate,
which holds when all fs hold.
any_fn(*fs)

Constructs a predicate,
which holds when any of fs holds.
none_fn(*fs)

Constructs a predicate,
which holds when none of fs hold.
one_fn(*fs)

Constructs a predicate,
which holds when exactly one of fs holds.
some_fn(*fs)

Constructs a function, which calls fs one by one
and returns first truthy result.
is_distinct(*fs)

Checks if all elements in the collection are different.
isa(*types)

Creates a function checking if its argument
is of any of given types.
is_iter(value)

Checks whether value is an iterator.
is_mapping(value)

Checks whether value is a mapping.
is_set(value)

Checks whether value is a set.
is_list(value)

Checks whether value is a list.
is_tuple(value)

Checks whether value is a tuple.
is_seq(value)

Checks whether value is a Sequence.
is_mapping(value)

Checks whether value is a mapping.
is_seqcoll(value)

Checks whether value is a list or a tuple,
which are both sequences and collections.
is_seqcont(value)

Checks whether value is a list, a tuple or an iterator,
which are both sequences and containers.
iterable(value)

Checks whether value is iterable.
@decorator

Transforms a flat wrapper into a decorator.
@wraps

An utility to pass function metadata
from wrapped function to a wrapper.
unwrap(func)

Get the object wrapped by func.
@once

Let function execute only once.
Noop all subsequent calls.
@once_per(*argnames)

Call function only once for every combination
of the given arguments.
@once_per_args

Call function only once for every combination
of values of its arguments.
@collecting

Transforms a generator into list returning function.
@joining(sep)

Joins decorated function results with sep.
@post_processing(func)

Post processes decorated function result with func.
@retry(tries, errors=Exception, timeout=0)

Tries decorated function up to tries times.
Retries only on specified errors.
@silent

Alters function to ignore all exceptions.
@ignore(errors, default=None)

Alters function to ignore errors,
returning default instead.
suppress(*errors)

The context manager suppressing errors in its block.
@limit_error_rate(fails, timeout, ...)

If function fails to complete fails times in a row,
calls to it will be blocked for timeout seconds.
fallback(*approaches)

Tries several approaches until one works.
Each approach has a form of (callable, errors).
raiser(exception=Exception, *args, **kwargs)

Constructs function that raises the given exception
with given arguments on any invocation.
@reraise(errors, into)

Intercepts errors and reraises them as into exception.
tap(x, label=None)

Prints x and then returns it.
@log_calls(print_func, errors=True, stack=True)
@print_calls(errors=True, stack=True)


Logs or prints all function calls,
including arguments, results and raised exceptions.
@log_enters(print_func)
@print_enters


Logs or prints on each enter to a function.
@log_exits(print_func, errors=True, stack=True)
@print_exits(errors=True, stack=True)


Logs or prints on each exit from a function.
@log_errors(print_func, label=None, stack=True)
@print_errors(label=None, stack=True)


Logs or prints all errors within a function or block.
@log_durations(print_func, label=None)
@print_durations(label=None)


Times each function call or block execution.
log_iter_durations(seq, print_func, label=None)
print_iter_durations(seq, label=None)


Times processing of each item in seq.
@memoize

Memoizes a decorated function results.
@cache(timeout)

Caches a function results for timeout seconds.
@cached_property

Creates a property caching its result.
@cached_property

Creates a read-only property caching its result.
@make_lookuper

Creates a cached function with prefilled memory.
@silent_lookuper

Creates a cached function with prefilled memory.
Ignores memory misses, returning None.
re_find(regex, s, flags=0)

Matches regex against the given string,
returns the match in the simplest possible form.
re_test(regex, s, flags=0)

Tests whether regex matches against s.
cut_prefix(s, prefix)

Cuts prefix from given string if it's present.
cut_suffix(s, suffix)

Cuts suffix from given string if it's present.
str_join([sep="", ]seq)

Joins the given sequence with sep.
Forces stringification of seq items.
@monkey(cls_or_module, name=None)

Monkey-patches class or module.
class namespace

A base class that prevents functions turning into methods.
class LazyObject(init)

Creates an object setting itself up on first use.
isnone(x)

Checks if x is None.
notnone(x)

Checks if x is not None.
inc(x)

Increments its argument by 1.
dec(x)

Decrements its argument by 1.
even(x)

Checks if x is even.
odd(x)

Checks if x is odd.