我有一个具有以下方法的类:
def get_add_new_links(self, max_num_links):
self.get_links_m2(max_num_links)
processes = mp.cpu_count()
pool = mp.Pool(processes=processes)
func = partial(worker, self)
with open(os.path.join(self.report_path, "links.txt"), "r") as f:
reports = pool.map(func, f.readlines())
pool.close()
pool.join()
where get_links_m2
是创建文件“links.txt”的另一种方法。该工人是:
def worker(obje, link):
doc, rep = obje.get_info_m2(link)
obje.add_new_active(doc, sure_not_exists=True)
return rep
方法get_info_m2
访问该链接并提取一些信息。方法add_new_active
将信息添加到 MongoDB。
我的代码可能有什么问题?当我运行它时,我收到此错误(和回溯):
文件“controller.py”,第 234 行,在 get_add_new_links 中
reports = pool.map(func, f.readlines()) File "/home/vladimir/anaconda3/lib/python3.5/multiprocessing/pool.py", line
260,在地图上
return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/vladimir/anaconda3/lib/python3.5/multiprocessing/pool.py",
第 608 行,在 get 中
raise self._value File "/home/vladimir/anaconda3/lib/python3.5/multiprocessing/pool.py", line
385,在_handle_tasks中
put(task) File "/home/vladimir/anaconda3/lib/python3.5/multiprocessing/connection.py",
第 206 行,发送中
self._send_bytes(ForkingPickler.dumps(obj)) File "/home/vladimir/anaconda3/lib/python3.5/multiprocessing/reduction.py",
50号线,在垃圾场
cls(buf, protocol).dump(obj) TypeError: can't pickle _thread.lock objects