捕获生成器内的错误并随后继续

我有一个应该运行几天的迭代器。我希望捕获并报告错误，然后我希望迭代器继续。或者整个过程可以重新开始。

这是函数：

def get_units(self, scraper):
    units = scraper.get_units()
    i = 0
    while True:
        try:
            unit = units.next()
        except StopIteration:
            if i == 0:
                log.error("Scraper returned 0 units", {'scraper': scraper})
            break
        except:
            traceback.print_exc()
            log.warning("Exception occurred in get_units", extra={'scraper': scraper, 'iteration': i})
        else:
            yield unit
        i += 1

Because scraper可能是代码的许多变体之一，它不可信，我不想处理那里的错误。

但是当错误发生在units.next()，整个事情就停止了。我怀疑是因为迭代器抛出了StopIteration当其中一个迭代失败时。

这是输出（仅最后一行）

[2012-11-29 14:11:12 /home/amcat/amcat/scraping/scraper.py:135 DEBUG] Scraping unit <Element div at 0x4258c710>
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article
[2012-11-29 14:11:13 /home/amcat/amcat/scraping/scraper.py:138 DEBUG] .. yields article Counter-Strike: Global Offensive Update Released
Traceback (most recent call last):
  File "/home/amcat/amcat/scraping/controller.py", line 101, in get_units
    unit = units.next()
  File "/home/amcat/amcat/scraping/scraper.py", line 114, in get_units
    for unit in self._get_units():
  File "/home/amcat/scraping/games/steamcommunity.py", line 90, in _get_units
    app_doc = self.getdoc(url,urlencode(form))
  File "/home/amcat/amcat/scraping/scraper.py", line 231, in getdoc
    return self.opener.getdoc(url, encoding)
  File "/home/amcat/amcat/scraping/htmltools.py", line 54, in getdoc
    response = self.opener.open(url, encoding)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
[2012-11-29 14:11:14 /home/amcat/amcat/scraping/controller.py:110 WARNING] Exception occurred in get_units

...code ends...

那么如何防止迭代在发生错误时停止呢？

编辑：这是 get_units() 中的代码

def get_units(self):
    """                                                                                                                                                                                                                                  
    Split the scraping job into a number of 'units' that can be processed independently                                                                                                                                                  
    of each other.                                                                                                                                                                                                                       

    @return: a sequence of arbitrary objects to be passed to scrape_unit                                                                                                                                                                 
    """
    self._initialize()
    for unit in self._get_units():
        yield unit

这是一个简化的 _get_units()：

INDEX_URL = "http://www.steamcommunity.com"

def _get_units(self):
  doc = self.getdoc(INDEX_URL)  #returns a lxml.etree document

  for a in doc.cssselect("div.discussion a"):
    link = a.get('href')
    yield link

编辑：问题后续：更改函数中的每个 for 循环，以便在每次失败的迭代后自动执行错误处理 https://stackoverflow.com/questions/13648200/alter-for-loops-to-have-continue-and-error-handling-executed-automatically-after

StopIteration是由next()当不再有下一个项目时生成器的方法。它与生成器/迭代器内部的错误无关。

另一件需要注意的事情是，根据迭代器的类型，它可能无法在异常后恢复。如果迭代器是一个带有next方法，一定会起作用。但是，如果它实际上是一个生成器，则不会。

据我所知，这是您的迭代在出现错误后无法继续的唯一原因units.next(). I.e. units.next()失败，下次你调用它时，它无法恢复，它说它是通过抛出一个来完成的StopIteration例外。

基本上你必须向我们展示里面的代码scraper.get_units()让我们理解为什么循环在单次迭代中出现错误后无法继续。如果get_units()是作为生成器函数实现的，很清楚。如果没有，可能是其他原因阻止了它恢复。

UPDATE:解释什么是生成器函数：

class Scraper(object):
    def get_units(self):
        for i in some_stuff:
            bla = do_some_processing()
            bla *= 2  # random stuff
            yield bla

现在，当你打电话时Scraper().get_units()，它不是运行整个函数，而是返回一个生成器对象。呼唤next()就它而言，将执行到第一个yield。等等现在如果内部任何地方发生错误get_units，可以这么说，它会被污染，下次你打电话时next()，它会提高StopIteration，就好像它已经没有给你的东西了。

阅读http://www.dabeaz.com/generators/ http://www.dabeaz.com/generators/ (and http://www.dabeaz.com/coroutines/ http://www.dabeaz.com/coroutines/） 强力推荐。

UPDATE2:一个可能的解决方案https://gist.github.com/4175802 https://gist.github.com/4175802