Sequences¶
This functions are aimed at manipulating finite and infinite sequences of values. Some functions have two flavors: one returning list and other returning possibly infinite iterator, the latter ones follow convention of prepending i
before list-returning function name.
When working with sequences, see also itertools
standard module. Funcy reexports and aliases some functions from it.
Generate¶
-
repeat
(item[, n])¶ Makes an iterator yielding
item
forn
times or indefinitely ifn
is omitted.repeat
simply repeats given value, when you need to reevaluate something repeatedly userepeatedly()
instead.When you just need a length
n
list or tuple ofitem
you can use:[item] * n # or (item,) * n
-
count
(start=0, step=1)¶ Makes infinite iterator of values:
start, start + step, start + 2*step, ...
.Could be used to generate sequence:
map(lambda x: x ** 2, count(1)) # -> 1, 4, 9, 16, ...
Or annotate sequence using
zip()
:zip(count(), 'abcd') # -> (0, 'a'), (1, 'b'), (2, 'c'), (3, 'd') # print code with BASIC-style numbered lines for line in zip(count(10, 10), code.splitlines()): print '%d %s' % line
See also
enumerate()
and originalitertools.count()
documentation.
-
cycle
(seq)¶ Cycles passed
seq
indefinitely returning its elements one by one.Useful when you need to cyclically decorate some sequence:
for n, parity in zip(count(), cycle(['even', 'odd'])): print '%d is %s' % (n, parity)
-
repeatedly
(f[, n])¶ Takes a function of no args, presumably with side effects, and returns an infinite (or length
n
if supplied) iterator of calls to it.For example, this call can be used to generate 10 random numbers:
repeatedly(random.random, 10)
Or one can create a length
n
list of freshly-created objects of same type:repeatedly(list, n)
-
iterate
(f, x)¶ Returns an infinite iterator of
x, f(x), f(f(x)), ...
etc.Most common use is to generate some recursive sequence:
iterate(inc, 5) # -> 5, 6, 7, 8, 9, ... iterate(lambda x: x * 2, 1) # -> 1, 2, 4, 8, 16, ... step = lambda p: (p[0], sum(p)) map(first, iterate(step, (0, 1))) # -> 0, 1, 1, 2, 3, 5, 8, ... (Fibonacci sequence)
Manipulate¶
This section provides some robust tools for sequence slicing. Consider Slicings or itertools.islice()
for more generic cases.
-
take
(n, seq)¶ Returns a list of the first
n
items in the sequence, or all items if there are fewer thann
.take(3, [2, 3, 4, 5]) # [2, 3, 4] take(3, count(5)) # [5, 6, 7] take(3, 'ab') # ['a', 'b']
-
drop
(n, seq)¶ Skips first
n
items in the sequence, returning iterator yielding rest of its items.drop(3, [2, 3, 4, 5]) # iter([5]) drop(3, count(5)) # count(8) drop(3, 'ab') # empty iterator
-
first
(seq)¶ Returns the first item in the sequence. Returns
None
if the sequence is empty. Typical usage is choosing first of some generated variants:# Get a text message of first failed validation rule fail = first(rule.text for rule in rules if not rule.test(instance)) # Use simple pattern matching to construct form field widget TYPE_TO_WIDGET = ( [lambda f: f.choices, lambda f: Select(choices=f.choices)], [lambda f: f.type == 'int', lambda f: TextInput(coerce=int)], [lambda f: f.type == 'string', lambda f: TextInput()], [lambda f: f.type == 'text', lambda f: Textarea()], [lambda f: f.type == 'boolean', lambda f: Checkbox(f.label)], ) return first(do(field) for cond, do in TYPE_TO_WIDGET if cond(field))
Other common use case is passing to
map()
orlmap()
. See last example initerate()
for such example.
-
second
(seq)¶ Returns the second item in given sequence. Returns
None
if there are less than two items in it.Could come in handy with sequences of pairs, e.g.
dict.items()
. Following code extract values of a dict sorted by keys:map(second, sorted(some_dict.items()))
And this line constructs an ordered by value dict from a plain one:
OrderedDict(sorted(plain_dict.items(), key=second))
-
nth
(n, seq)¶ Returns nth item in sequence or
None
if no one exists. Items are counted from 0, so it’s like indexed access but works for iterators. E.g. here is how one can get 6th line of some_file:nth(5, repeatedly(open('some_file').readline))
-
last
(seq)¶ Returns the last item in the sequence. Returns
None
if the sequence is empty. Tries to be efficient when sequence supports indexed or reversed access and fallbacks to iterating over it if not.
-
rest
(seq)¶ Skips first item in the sequence, returning iterator starting just after it. A shortcut for
drop(1, seq)
.
-
butlast
(seq)¶ Returns an iterator of all elements of the sequence but last.
-
ilen
(seq)¶ Calculates length of iterator. Will consume it or hang up if it’s infinite.
Especially useful in conjunction with filtering or slicing functions, for example, this way one can find common start length of two strings:
ilen(takewhile(lambda (x, y): x == y, zip(s1, s2)))
Unite¶
-
concat
(*seqs)¶ -
lconcat
(*seqs)¶ Concats several sequences into single iterator or list.
concat()
is an alias foritertools.chain()
.
-
cat
(seqs)¶ -
lcat
(seqs)¶ Concatenates passed sequences. Useful when dealing with sequence of sequences, see
concat()
orlconcat()
to join just a few sequences.Flattening of various nested sequences is most common use:
# Flatten two level deep list lcat(list_of_lists) # Get a flat html of errors of a form errors = cat(inline.errors() for inline in form) error_text = '<br>'.join(errors) # Brace expansion on product of sums # (a + b)(t + pq)x == atx + apqx + btx + bpqx terms = [['a', 'b'], ['t', 'pq'], ['x']] lmap(lcat, product(*terms)) # [list('atx'), list('apqx'), list('btx'), list('bpqx')]
cat()
is an alias foritertools.chain.from_iterable()
.
-
flatten
(seq, follow=is_seqcont)¶ -
lflatten
(seq, follow=is_seqcont)¶ Flattens arbitrary nested sequence of values and other sequences.
follow
argument determines whether to unpack each item. By default it dives into lists, tuples and iterators, seeis_seqcont()
for further explanation.See also
cat()
orlcat()
if you need to flatten strictly two-level sequence of sequences.
-
tree_leaves
(root, follow=is_seqcont, children=iter)¶ -
ltree_leaves
(root, follow=is_seqcont, children=iter)¶ A way to iterate or list over all the tree leaves. E.g. this is how you can list all descendants of a class:
ltree_leaves(Base, children=type.__subclasses__, follow=type.__subclasses__)
-
tree_nodes
(root, follow=is_seqcont, children=iter)¶ -
ltree_nodes
(root, follow=is_seqcont, children=iter)¶ A way to iterate or list over all the tree nodes. E.g. this is how you can iterate over all classes in hierarchy:
tree_nodes(Base, children=type.__subclasses__, follow=type.__subclasses__)
-
interleave
(*seqs)¶ Returns an iterator yielding first item in each sequence, then second and so on until some sequence ends. Numbers of items taken from all sequences are always equal.
-
interpose
(sep, seq)¶ Returns an iterator yielding elements of
seq
separated bysep
.This is like
str.join()
for lists. This code is a part of a translator working with operation node:def visit_BoolOp(self, node): # ... do generic visit node.code = lmapcat(translate, interpose(node.op, node.values))
Transform and filter¶
Most of functions in this section support Extended function semantics. Among other things it allows to rewrite examples using re_tester()
and re_finder()
tighter.
-
remove
(pred, seq)¶ -
lremove
(pred, seq)¶ Returns an iterator or a list of items of
seq
that result in false when passed topred
. The results of this functions complement results offilter()
andlfilter()
.A handy use is passing
re_tester()
result aspred
. For example, this code removes any whitespace-only lines from list:remove(re_tester('^\s+$'), lines)
Note, you can rewrite it shorter using Extended function semantics:
remove('^\s+$', lines)
-
keep
([f, ]seq)¶ -
lkeep
([f, ]seq)¶ Maps
seq
with given function and then filters out falsy elements. Simply removes falsy items whenf
is absent. In fact these functions are just handy shortcuts:keep(f, seq) == filter(bool, map(f, seq)) keep(seq) == filter(bool, seq) lkeep(f, seq) == lfilter(bool, map(f, seq)) lkeep(seq) == lfilter(bool, seq)
Natural use case for
keep()
is data extraction or recognition that could eventually fail:# Extract numbers from words lkeep(re_finder(r'\d+'), words) # Recognize as many colors by name as possible lkeep(COLOR_BY_NAME.get, color_names)
An iterator version can be useful when you don’t need or not sure you need the whole sequence. For example, you can use
first()
-keep()
combo to find out first match:first(keep(COLOR_BY_NAME.get, color_name_candidates))
Alternatively, you can do the same with
some()
andmap()
.One argument variant is a simple tool to keep your data free of falsy junk. This one returns non-empty description lines:
keep(description.splitlines())
Other common case is using generator expression instead of mapping function. Consider these two lines:
keep(f.name for f in fields) # sugar generator expression keep(attrgetter('name'), fields) # pure functions
-
mapcat
(f, *seqs)¶ -
lmapcat
(f, *seqs)¶ Maps given sequence(s) and then concatenates results, essentially a shortcut for
cat(map(f, *seqs))
. Come in handy when extracting multiple values from every sequence item or transforming nested sequences:# Get all the lines of all the texts in single flat list mapcat(str.splitlines, bunch_of_texts) # Extract all numbers from strings mapcat(partial(re_all, r'\d+'), bunch_of_strings)
-
without
(seq, *items)¶ -
lwithout
(seq, *items)¶ Returns sequence with
items
removed, preserves order. Designed to work with a fewitems
, this allows removing unhashable objects:non_empty_lists = without(lists, [])
In case of large amount of unwanted elements one can use
remove()
:remove(set(unwanted_elements), seq)
Or simple set difference if order of sequence is irrelevant.
Split and chunk¶
-
split
(pred, seq)¶ -
lsplit
(pred, seq)¶ Splits sequence items which pass predicate from the ones that don’t, essentially returning a tuple
filter(pred, seq), remove(pred, seq)
.For example, this way one can separate private attributes of an instance from public ones:
private, public = lsplit(re_tester('^_'), dir(instance))
Split absolute and relative urls using extended predicate semantics:
absolute, relative = lsplit(r'^http://', urls)
-
split_at
(n, seq)¶ -
lsplit_at
(n, seq)¶ Splits sequence at given position, returning a tuple of its start and tail.
-
split_by
(pred, seq)¶ -
lsplit_by
(pred, seq)¶ Splits start of sequence, consisting of items passing predicate, from the rest of it. Works similar to
takewhile(pred, seq), dropwhile(pred, seq)
, but works with iteratorseq
correctly:lsplit_by(bool, iter([-2, -1, 0, 1, 2])) # [-2, -1], [0, 1, 2]
-
takewhile
([pred, ]seq)¶ Yeilds elements of
seq
as long as they passpred
. Stops on first one which makes predicate falsy:# Extract first paragraph of text takewhile(re_tester(r'\S'), text.splitlines()) # Build path from node to tree root takewhile(bool, iterate(attrgetter('parent'), node))
-
dropwhile
([pred, ]seq)¶ This is a mirror of
takewhile()
. Skips elements of given sequence whilepred
is true and yields the rest of it:# Skip leading whitespace-only lines dropwhile(re_tester('^\s*$'), text_lines)
-
group_by
(f, seq)¶ Groups elements of
seq
keyed by the result off
. The value at each key will be a list of the corresponding elements, in the order they appear inseq
. Returnsdefaultdict(list)
.stats = group_by(len, ['a', 'ab', 'b']) stats[1] # -> ['a', 'b'] stats[2] # -> ['ab'] stats[3] # -> [], since stats is defaultdict
One can use
split()
when grouping by boolean predicate. See alsoitertools.groupby()
.
-
group_by_keys
(get_keys, seq)¶ Groups elements of
seq
having multiple keys each intodefaultdict(list)
. Can be used to reverse grouping:posts_by_tag = group_by_keys(attrgetter('tags'), posts) sentences_with_word = group_by_keys(str.split, sentences)
-
group_values
(seq)¶ Groups values of
(key, value)
pairs. May think of it likedict()
but collecting collisions:group_values(keep(r'^--(\w+)=(.+)', sys.argv))
-
partition
(n, [step, ]seq)¶ -
lpartition
(n, [step, ]seq)¶ Iterates or lists over partitions of
n
items, at offsetsstep
apart. Ifstep
is not supplied, defaults ton
, i.e. the partitions do not overlap. Returns only full length-n
partitions, in case there are not enough elements for last partition they are ignored.Most common use is deflattening data:
# Make a dict from flat list of pairs dict(partition(2, flat_list_of_pairs)) # Structure user credentials {id: (name, password) for id, name, password in partition(3, users)}
A three argument variant of
partition()
can be used to process sequence items in context of their neighbors:# Smooth data by averaging out with a sliding window [sum(window) / n for window in partition(n, 1, data_points)]
Also look at
pairwise()
for similar use. Other use ofpartition()
is processing sequence of data elements or jobs in chunks, but take a look atchunks()
for that.
-
chunks
(n, [step, ]seq)¶ -
lchunks
(n, [step, ]seq)¶ Like
partition()
, but may include partitions with fewer thann
items at the end:chunks(2, 'abcde') # -> 'ab', 'cd', 'e' chunks(2, 4, 'abcde') # -> 'ab', 'e'
Handy for batch processing.
Data handling¶
-
distinct
(seq, key=identity)¶ -
ldistinct
(seq, key=identity)¶ Returns unique items of the sequence with order preserved. If
key
is supplied then distinguishes values by comparing their keys.Note
Elements of a sequence or their keys should be hashable.
-
with_prev
(seq, fill=None)¶ Returns an iterator of a pair of each item with one preceding it. Yields fill or None as preceding element for first item.
Great for getting rid of clunky
prev
housekeeping in for loops. This way one can indent first line of each paragraph while printing text:for line, prev in with_prev(text.splitlines()): if not prev: print ' ', print line
Use
pairwise()
to iterate only on full pairs.
-
with_next
(seq, fill=None)¶ Returns an iterator of a pair of each item with one next to it. Yields fill or None as next element for last item. See also
with_prev()
andpairwise()
.
-
pairwise
(seq)¶ Yields pairs of items in
seq
like(item0, item1), (item1, item2), ...
. A great way to process sequence items in a context of each neighbor:# Check if seq is non-descending all(left <= right for left, right in pairwise(seq))
-
count_by
(f, seq)¶ Counts numbers of occurrences of values of
f
on elements ofseq
. Returnsdefaultdict(int)
of counts.Calculating a histogram is one common use:
# Get a length histogram of given words count_by(len, words)
-
count_reps
(seq)¶ Counts number of repetitions of each value in
seq
. Returnsdefaultdict(int)
of counts. This is faster and shorter alternative tocount_by(identity, ...)
-
reductions
(f, seq[, acc])¶ -
lreductions
(f, seq[, acc])¶ Returns a sequence of the intermediate values of the reduction of
seq
byf
. In other words it yields a sequence like:reduce(f, seq[:1], [acc]), reduce(f, seq[:2], [acc]), ...
You can use
sums()
orlsums()
for a common use of getting list of partial sums.
-
sums
(seq[, acc])¶ -
lsums
(seq[, acc])¶ Same as
reductions()
orlreductions()
with reduce function fixed to addition.Find out which straw will break camels back:
first(i for i, total in enumerate(sums(straw_weights)) if total > camel_toughness)
Makes infinite iterator of values:
start, start + step, start + 2*step, ...
Cycles passed sequence indefinitely
yielding its elements one by one.
Makes an iterator yielding item for n times
or indefinitely if n is omitted.
Takes a function of no args, presumably with side effects,
and returns an infinite (or length n) iterator of calls to it.
Returns an infinite iterator of
x, f(x), f(f(x)), ...
Lists all matches of regex in s.
Iterates over matches of regex in s.
Returns the first item in the sequence.
Returns
None
if the sequence is empty.
Returns second item in the sequence.
Returns
None
if there are less than two items in it.
Returns the last item in the sequence.
Returns
None
if the sequence is empty.
Returns nth item in the sequence
or
None
if no such item exists.
Finds first item in seq passing pred
or first that is true if pred is omitted.
Returns a list of first n items in the sequence,
or all items if there are fewer than n.
Skips first n items in the sequence,
yields the rest.
Skips first item in the sequence, yields the rest.
Yields all elements of the sequence but last.
Yields seq items as long as they pass pred.
Skips elements of seq while pred passes
and then yields the rest.
lsplit_at(n, seq)
Splits the sequence at given position,
returning a tuple of its start and tail.
lsplit_by(pred, seq)
Splits the start of the sequence,
consisting of items passing pred,
from the rest of it.
lmap(f, *seqs)
Extended versions of map() and list(map())
lmapcat(f, *seqs)
Maps given sequence(s) and concatenates the results.
lkeep([f, ]*seqs)
Maps seq with f and filters out falsy results.
Simply removes falsy values in one argument version.
lpluck(key, mappings)
Yields or lists values for key in each mapping.
lpluck_attr(attr, objects)
Yields or lists values of attr of each object.
linvoke(objects, name, *args, **kwargs)
Yields or lists results of the given method call
for each object in objects.
Wrap a property accessors with ctx.
lfilter(pred, seq)
Extended versions of filter() and list(filter()).
lremove(pred, seq)
Removes items from seq passing given predicate.
ldistinct(pred, key=identity)
Removes items having same key from seq.
Preserves order.
lwhere(mappings, **cond)
Selects mappings containing all pairs in cond.
lwithout(seq, *items)
Returns sequence without items,
preserves order.
lcat(seqs)
Concatenates passed sequences.
lconcat(*seqs)
Concatenates several sequences.
lflatten(seq, follow=is_seqcont)
Flattens arbitrary nested sequence,
dives into when
follow(item)
is truthy.
Yields first item of each sequence,
then second one and so on.
Yields items of seq separated by sep.
List version of zip()
lchunks(n, [step, ]seq)
Chunks seq into parts of length n or less.
Skips step items between chunks.
lpartition(n, [step, ]seq)
Partitions seq into parts of length n.
Skips step items between parts.
Non-fitting tail is ignored.
lpartition_by(f, seq)
Partition seq into continuous chunks
with constant value of f.
lsplit(pred, seq)
Splits seq items which pass pred
from the ones that don't.
Counts numbers of occurrences of values of f
on elements of seq.
Counts repetitions of each value in seq.
Groups items of seq by
f(item)
.
Groups elements of seq by multiple keys.
Groups values of
(key, value)
pairs by keys.
Consumes the given iterator and returns its length.
lreductions(f, seq[, acc])
Constructs intermediate reductions of seq by f.
lsums(seq[, acc])
Returns a sequence of partial sums of seq.
Checks if all items in seq pass pred.
Checks if any item in seq passes pred.
Checks if none of the items in seq pass pred.
Checks if exactly one item in seq passes pred.
Yields all pairs of neighboring items in seq.
Yields each item from seq with the one preceding it.
Yields each item from seq with the next one.
Yields tuples of corresponding values of given dicts.
Yields tuples like
(key, val1, val2, ...)
for each common key in all given dicts.
ltree_leaves(root, follow=is_seqcont, children=iter)
Lists or iterates over tree leaves.
ltree_nodes(root, follow=is_seqcont, children=iter)
Lists or iterates over tree nodes.
Merges several collections of same type into one:
dicts, sets, lists, tuples, iterators or strings
For dicts later values take precedence.
Merges several dicts combining values with given function.
Joins several collections of same type into one.
Same as merge() but accepts sequence of collections.
Joins several dicts combining values with given function.
Maps coll with f, but preserves collection type.
Walks keys of coll, mapping them with f.
Works with dicts and collections of pairs.
Walks values of coll, mapping them with f.
Works with dicts and collections of pairs.
Filters elements of coll by pred
constructing a collection of same type.
Select part of coll with keys passing pred.
Works with dicts and collections of pairs.
Select part of coll with values passing pred.
Works with dicts and collections of pairs.
Removes falsy values from given collection.
These are targeted specifically at them.
Flip passed dict swapping its keys and values.
Creates a dict with keys mapped to the corresponding vals.
Yields values of the given collection.
Yields
(key, value)
pairs of the given collection.
Leaves only given keys in mapping.
Removes given keys from mapping.
Returns an empty collection of the same type as coll.
Returns a value at path in the given nested collection.
Creates a copy of coll with the value set at path.
Creates a copy of coll with a value updated at path.
Returns its argument.
Creates a function accepting any args, but always returning x.
Like partial() but returns a real function.
Returns partial application of func.
Partially applies last arguments to func.
Creates a function, which conditionally applies action or default.
Creates a function calling its argument with passed arguments.
Creates a function finding regex in passed string.
Creates a predicate testing passed strings with regex.
Constructs a complementary predicate.
Creates a version of func returning its partial applications
until sufficient arguments are passed.
Curries func into a chain of one argument functions.
Arguments are passed from left to right.
Curries func from right to left.
Composes passed functions.
Composes fs, calling them from left to right.
ljuxt(*fs)
Constructs a juxtaposition of the given functions.
Result returns a list or an iterator of results of fs.
Constructs a predicate,
which holds when all fs hold.
Constructs a predicate,
which holds when any of fs holds.
Constructs a predicate,
which holds when none of fs hold.
Constructs a predicate,
which holds when exactly one of fs holds.
Constructs a function, which calls fs one by one
and returns first truthy result.
Checks if all elements in the collection are different.
Creates a function checking if its argument
is of any of given types.
Checks whether value is an iterator.
Checks whether value is a mapping.
Checks whether value is a set.
Checks whether value is a list.
Checks whether value is a tuple.
Checks whether value is a
Sequence
.
Checks whether value is a mapping.
Checks whether value is a list or a tuple,
which are both sequences and collections.
Checks whether value is a list, a tuple or an iterator,
which are both sequences and containers.
Checks whether value is iterable.
Transforms a flat wrapper into a decorator.
An utility to pass function metadata
from wrapped function to a wrapper.
Get the object wrapped by func.
Let function execute only once.
Noop all subsequent calls.
Call function only once for every combination
of the given arguments.
Call function only once for every combination
of values of its arguments.
Transforms a generator into list returning function.
Joins decorated function results with sep.
Post processes decorated function result with func.
Turn context manager into a decorator.
A noop context manager.
Tries decorated function up to tries times.
Retries only on specified errors.
Alters function to ignore all exceptions.
Alters function to ignore errors,
returning default instead.
The context manager suppressing errors in its block.
If function fails to complete fails times in a row,
calls to it will be blocked for timeout seconds.
Tries several approaches until one works.
Each approach has a form of
(callable, errors)
.
Constructs function that raises the given exception
with given arguments on any invocation.
Intercepts errors and reraises them as into exception.
Prints x and then returns it.
@print_calls(errors=True, stack=True)
Logs or prints all function calls,
including arguments, results and raised exceptions.
@print_enters
Logs or prints on each enter to a function.
@print_exits(errors=True, stack=True)
Logs or prints on each exit from a function.
@print_errors(label=None, stack=True)
Logs or prints all errors within a function or block.
@print_durations(label=None)
Times each function call or block execution.
print_iter_durations(seq, label=None)
Times processing of each item in seq.
Memoizes a decorated function results.
Caches a function results for timeout seconds.
Creates a property caching its result.
Creates a read-only property caching its result.
Creates a cached function with prefilled memory.
Creates a cached function with prefilled memory.
Ignores memory misses, returning
None
.
Matches regex against the given string,
returns the match in the simplest possible form.
Tests whether regex matches against s.
Cuts prefix from given string if it's present.
Cuts suffix from given string if it's present.
Joins the given sequence with sep.
Forces stringification of seq items.
Monkey-patches class or module.
A base class that prevents functions turning into methods.
Creates an object setting itself up on first use.
Checks if x is None.
Checks if x is not None.
Increments its argument by 1.
Decrements its argument by 1.
Checks if x is even.
Checks if x is odd.