将文本拆分为最小长度的较小段落，而不破坏给定阈值的句子

有更好的方法来完成这项任务吗？

对于 NLP 任务的预处理，我试图将大块文本分割成偶数长度的字符串列表。

通过在每个“.”处分割文本。我的句子长度会很不均匀。通过使用索引/数字，我会在中间切断句子。

目标是将句子放在长度均匀的列表中，而不会在句子结束之前截断句子。

这是我想出的解决方案，但我觉得应该存在一些更简单的解决方案。

def even_split(text):
    combined_sentences = []
    tmp_text = text.strip()
    if tmp_text[-1] != ".":
      tmp_text += "."
    while len(tmp_text) > 0:
      dots = []
      for i in range(len(tmp_text)):
        if tmp_text[i] == '.':
          dots.append(i)
      split_dot = dots[min(range(len(dots)), key=lambda i: abs(dots[i]-150))]
      combined_sentences.append(tmp_text[:split_dot+1])
      tmp_text = tmp_text[split_dot+1:].strip()
    return combined_sentences

例如，如果我输入以下字符串：

在当今过度饱和的市场中，营销产品和服务是一项艰巨而乏味的任务。特别是如果您从事 B2B 潜在客户开发业务。作为企业主或销售团队的一员，您确实需要深入了解哪些策略最有效以及如何最有效地吸引客户。潜在客户开发是您需要掌握的。了解不同类型的潜在客户将帮助您销售产品或服务并更快地扩展业务。这就是为什么我们要解释什么是热情的潜在客户以及如何轻松地将他们转变为付费客户。

这将输出：

['Marketing products and services is a demanding and tedious task in today’s overly saturated market. Especially if you’re in a B2B lead generation business.',
 'As a business owner or part of the sales team, you really need to dive deep into understanding what strategies work best and how to appeal to your customers most efficiently.',
 'Lead generation is something you need to master. Understanding different types of leads will help you sell your product or services and scale your business faster.',
 'That’s why we’re explaining what warm leads are and how you can easily turn them into paying customers.']

正如您所看到的，它们均匀分布，每个大约 150 个字符。 希望这一点是清楚的。

有更好的方法来完成这项任务吗？

Thanks!

IIUC，您想按点分割文本，但尽量保持块的最小长度，以避免句子非常短。

您可以做的是将这些点分开并再次加入，直到达到阈值（此处为 200 个字符）：

out = []
threshold = 200
for chunk in text.split('. '):
    if out and len(chunk)+len(out[-1]) < threshold:
        out[-1] += ' '+chunk+'.'
    else:
        out.append(chunk+'.')

output:

['Marketing products and services is a demanding and tedious task in today’s overly saturated market. Especially if you’re in a B2B lead generation business.',
 'As a business owner or part of the sales team, you really need to dive deep into understanding what strategies work best and how to appeal to your customers most efficiently.',
 'Lead generation is something you need to master. Understanding different types of leads will help you sell your product or services and scale your business faster.',
 'That’s why we’re explaining what warm leads are and how you can easily turn them into paying customers..']