Breaking Code

December 13, 2009

Reusing Python generator objects

Filed under: Uncategorized — Tags: , — Mario Vilas @ 9:46 pm

Generators are one of those little things that I love about Python. Never heard of generators? Well, this is all much better explained here, but in a nutshell:

    def squares( size ):
        for x in xrange( size ):
            yield x * x
        return

The above code returns a generator object. This object is iterable, and each item is returned by the yield statement. The return statement ends the iteration.

    >>> list( squares( 10 ) )
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >>> for value in squares( 5 ):
    ...     print value
    ...
    0
    1
    4
    9
    16

Now, there’s a little problem I found… I wanted an object than, when iterated, returned a certain sequence of objects. No problem so far. But here’s the catch: once it finished returning the list, I wanted to be able to reuse the generator.

    >>> gen = squares( 5 )
    >>> list( gen )        # Works the first time...
    [0, 1, 4, 9, 16]
    >>> list( gen )        # But not the second!
    []

So I came up with this solution:

    class Regenerator(object):
        """
        Calls a generator and iterates it. When it's finished iterating, the
        generator is called again. This allows you to iterate a generator more
        than once (well, sort of, since the generator is actually recreated).
        """
        def __iter__(self):
            return self
        def __init__(self, g_function, *v_args, **d_args):
            self.__g_function = g_function
            self.__v_args     = v_args
            self.__d_args     = d_args
            self.__g_object   = None
        def next(self):
            if self.__g_object is None:
                self.__g_object = self.__g_function( *self.__v_args, **self.__d_args )
            try:
                return self.__g_object.next()
            except StopIteration:
                self.__g_object = None
                raise

So, in the example we presented, this is what we’d do to be able to iterate the list of numbers more than once:

    def _squares( size ):
        for x in xrange( size ):
            yield x * x
        return
    def squares( size ):
        return Regenerator( _squares, size )

And alas, it works! 🙂

    >>> gen = squares( 10 )
    >>> list( gen )
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >>> list( gen )
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >>> list( gen )
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Download

regenerator.py

Advertisements

6 Comments »

  1. What is wrong with:
    numbers = list(gen(5))

    ?

    Comment by bobb — January 20, 2010 @ 10:32 pm

  2. That would instance the whole list in memory, defeating the entire purpose of using generators.

    You use generators when creating a list would consume too much memory or resources. Tipically you’ll use one item at a time and then discard it, and/or you bail out of the loop before you finish iterating the whole thing.

    In the example I was only building the list in the interactive console to make things simpler…

    Comment by Mario Vilas — January 20, 2010 @ 11:42 pm

  3. You should have used `itertools.tee`, from the standard library.

    Comment by dsfsdf — April 5, 2013 @ 2:31 am

  4. Well, in my opinion, itertools.tee wouldn’t quite do it. It’s not exactly the same functionality (itertools.tee can only cycle through the items a fixed number of times) and it seems to consume more resources too.

    Comment by Mario Vilas — April 5, 2013 @ 3:27 pm

  5. […] a one time deal but I know that there are workarounds to this (like this question and this post but I’m not sure what the best way to do this is. These files are quite large and I’d […]

    Pingback by “Write” method for generator in python — July 31, 2014 @ 9:43 pm

  6. […] a one time deal but I know that there are workarounds to this (like this question and this post but I’m not sure what the best way to do this is. These files are quite large and I’d […]

    Pingback by "Write" method for generator in pythonCopyQuery CopyQuery | Question & Answer Tool for your Technical Queries,CopyQuery, ejjuit, query, copyquery, copyquery.com, android doubt, ios question, sql query, sqlite query, nodejsquery, dns query, u — August 1, 2014 @ 1:56 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: