从 Google Scholar 搜索结果中抓取和解析引文信息

我有大约 20000 篇文章标题的列表，我想scrape他们来自谷歌学术的引用计数。我是 BeautifulSoup 库的新手。我有这个代码：

import requests
from bs4 import BeautifulSoup

query = ['Role for migratory wild birds in the global spread of avian 
 influenza H5N8','Uncoupling conformational states from activity in an 
 allosteric enzyme','Technological Analysis of the World’s Earliest 
 Shamanic Costume: A Multi-Scalar, Experimental Study of a Red Deer 
 Headdress from the Early Holocene Site of Star Carr, North Yorkshire, 
 UK','Oxidative potential of PM 2.5  during Atlanta rush hour: 
 Measurements of in-vehicle dithiothreitol (DTT) activity','Primary 
 Prevention of CVD','Growth and Deposition of Au Nanoclusters on Polymer- 
 wrapped Graphene and Their Oxygen Reduction Activity','Relations of 
 Preschoolers Visual-Motor and Object Manipulation Skills With Executive 
 Function and Social Behavior','We Know Who Likes Us, but Not Who Competes 
 Against Us']

url = 'https://scholar.google.com/scholar?q=' + query + '&ie=UTF-8&oe=UTF- 
       8&hl=en&btnG=Search'

content = requests.get(url).text
page = BeautifulSoup(content, 'lxml')
results = []
for entry in page.find_all("h3", attrs={"class": "gs_rt"}):
    results.append({"title": entry.a.text, "url": entry.a['href']})

但它只返回标题和网址。我不知道如何从另一个标签获取引文信息。请帮我一下。

您需要循环该列表。您可以使用 Session 来提高效率。以下是针对 bs 4.7.1 支持的:contains用于查找引用计数的伪类。看起来你可以删除h3从 css 选择器中输入选择器，然后在之前使用 classa i.e. .gs_rt a。如果您没有 4.7.1。您可以使用[title=Cite] + a选择引用计数。

import requests
from bs4 import BeautifulSoup as bs

queries = ['Role for migratory wild birds in the global spread of avian influenza H5N8',
         'Uncoupling conformational states from activity in an allosteric enzyme',
         'Technological Analysis of the World’s Earliest Shamanic Costume: A Multi-Scalar, Experimental Study of a Red Deer Headdress from the Early Holocene Site of Star Carr, North Yorkshire, UK',
         'Oxidative potential of PM 2.5  during Atlanta rush hour: Measurements of in-vehicle dithiothreitol (DTT) activity',
         'Primary Prevention of CVD','Growth and Deposition of Au Nanoclusters on Polymer-wrapped Graphene and Their Oxygen Reduction Activity',
         'Relations of Preschoolers Visual-Motor and Object Manipulation Skills With Executive Function and Social Behavior',
         'We Know Who Likes Us, but Not Who Competes Against Us']

with requests.Session() as s:
    for query in queries:
        url = 'https://scholar.google.com/scholar?q=' + query + '&ie=UTF-8&oe=UTF-8&hl=en&btnG=Search'
        r = s.get(url)
        soup = bs(r.content, 'lxml') # or 'html.parser'
        title = soup.select_one('h3.gs_rt a').text if soup.select_one('h3.gs_rt a') is not None else 'No title'
        link = soup.select_one('h3.gs_rt a')['href'] if title != 'No title' else 'No link'
        citations = soup.select_one('a:contains("Cited by")').text if soup.select_one('a:contains("Cited by")') is not None else 'No citation count'
        print(title, link, citations) 

with requests.Session() as s:
    for query in queries:
        url = 'https://scholar.google.com/scholar?q=' + query + '&ie=UTF-8&oe=UTF-8&hl=en&btnG=Search'
        r = s.get(url)
        soup = bs(r.content, 'lxml') # or 'html.parser'
        title = soup.select_one('.gs_rt a')
        if title is None:
            title = 'No title'
            link = 'No link'
        else:  
            link = title['href']
            title = title.text
        citations = soup.select_one('[title=Cite] + a')
        if citations is None:
            citations = 'No citation count'
        else:
             citations = citations.text
        print(title, link, citations)

感谢@facelessuser 的评论，底层版本被重新编写。留下的顶级版本进行比较：

在单行 if 语句中不要调用 select_one 两次可能会更有效。当模式构建被缓存时，返回的标签不会被缓存。我个人会将变量设置为 select_one 返回的任何内容，然后，仅当变量为 None 时，将其更改为 No link 或 No title 等。它不是那么紧凑，但会更有效

[...]始终检查 if tag 是否为 None: 而不仅仅是 if tag:。对于选择器来说，这并不是什么大问题，因为它们只会返回标签，但如果你曾经做过类似 for x in tag.descendants 的事情：你会得到文本节点（字符串）和标签，并且空字符串将评估为 false，即使它是一个有效的节点。在这种情况下，检查 None 是最安全的