Created
November 10, 2015 00:16
-
-
Save bobbruno/4960c005248258a13169 to your computer and use it in GitHub Desktop.
python decorator for bufferising generator output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def bufferise(defbuf=20, defskip=0): | |
def decorate(function): | |
def wrapper(*args, **kwargs): | |
bufsize = kwargs['bufsize'] if 'bufsize' in kwargs else defbuf | |
skiplines = kwargs['skiplines'] if 'skiplines' in kwargs else defskip | |
print 'Bufsize = {}'.format(bufsize) | |
print 'Skip {} lines'.format(skiplines) | |
if skiplines: | |
for i, record in enumerate(function(*args, **kwargs), start=1): | |
if i > skiplines: | |
break | |
while True: | |
buffer = [] | |
i = 0 | |
for i, record in enumerate(function(*args, **kwargs), start = 1): | |
buffer.append(record) | |
if i > bufsize: | |
break | |
if i <= bufsize: | |
break | |
yield buffer | |
if len(buffer): | |
yield buffer | |
raise StopIteration | |
return wrapper | |
return decorate |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bufferise
This code allows to easily bufferise the output of any generator. Just decorate the generator function with
@bufferize
, and you automatically get a buffer or 20 records. You can alternatively pass adefbuf
parameter to the decorator, or even add an additionalbufsize
parameter at the call to the decorated function, to override the default.The same decorator allows you to skip leading lines (return values) as well. The
defskip
defaults to 0, but can be set to any value at the decoration point. Alternatively, you can also pass an additionalskiplines
parameter to the decorated generator.