httplib.BadStatusLine: '' 在 Linux 上，但在 Mac 上则不然

这个错误已经困扰我几个小时了。我决定编写一个单独的项目，只是为了看看我是否可以复制它，而且我可以，但只能在我的服务器上复制。这适用于我的 Mac。

• Mac：OSX El Capitan 10.11.6

• 服务器：CentOS 7.2.1511

• 两者都有 PhantomJS 版本：2.1.1

• Python Mac：Python 2.7.11

• Python服务器：2.7.5

• 两者都有 Selenium 版本：2.53.0

相同的代码在两者上运行：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.common.exceptions import NoSuchElementException
import time

dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
dcap["phantomjs.page.customHeaders.accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
dcap["phantomjs.page.customHeaders.Accept-Language"] = "en-US,en;q=0.8"
dcap["phantomjs.page.customHeaders.connection"] = "keep-alive"

driver = webdriver.PhantomJS(desired_capabilities=dcap)
driver.set_window_size(1120, 700)
driver.get("https://www.instagram.com/espn/")

while True:
    print len(driver.find_elements_by_css_selector("a[href*='/p/']"))
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        loadMore = driver.find_element_by_link_text("Load more")
        loadMore.click()
    except NoSuchElementException:
        print "No such"
        driver.save_screenshot('none.png')

Mac 输出：

12
24
No such
24
No such
36
No such
48
No such
48
No such
60
No such
72
No such
84
# This goes until I end it

服务器输出：

12
24
No such
Traceback (most recent call last):
  File "junk.py", line 27, in <module>
    driver.save_screenshot('none.png')
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 790, in get_screenshot_as_file
    png = self.get_screenshot_as_png()
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 809, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 819, in get_screenshot_as_base64
    return self.execute(Command.SCREENSHOT)['value']
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

删除屏幕截图行后的服务器输出：

12
24
No such
24
Traceback (most recent call last):
  File "junk.py", line 23, in <module>
    loadMore = driver.find_element_by_link_text("Load more")
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in find_element_by_link_text
    return self.find_element(by=By.LINK_TEXT, value=link_text)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 744, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

我发现的一个相关答案在这里：无法通过 Selenium 在 python 中运行 PhantomJS https://stackoverflow.com/questions/20435220/cant-run-phantomjs-in-python-via-selenium

所以我安装了 Selenium 2.37，它给出了同样的错误。

I read 这个答案 https://stackoverflow.com/questions/9265616/why-does-this-url-raise-badstatusline-with-httplib2-and-urllib2关于后面的问题可能与更改标头有关，因此我通过将驱动程序更改为来删除标头driver = webdriver.PhantomJS()仍然得到同样的错误。

我也在服务器上安装了2.7.12，看看是否有区别。输出为：

# python2.7 junk.py
12
24
No such
24
Traceback (most recent call last):
  File "junk.py", line 29, in <module>
    loadMore = driver.find_element_by_link_text("Load more")
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in find_element_by_link_text
    return self.find_element(by=By.LINK_TEXT, value=link_text)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 744, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/local/lib/python2.7/httplib.py", line 1136, in getresponse
    response.begin()
  File "/usr/local/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.7/httplib.py", line 417, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

检查系统上的空间。这是一个全新的 VPS，但仍然需要确认：

EDIT 3

添加以下内容：

except httplib.BadStatusLine:
    pass

EDIT 2

Python WebDriver 和 phantomJs 有问题活着 http://selenium-python.readthedocs.io/api.html。这可能是你的问题。所以添加keep_alive=False如下：

driver = webdriver.PhantomJS(desired_capabilities=dcap,keep_alive=False)

end edit