您的代码不适合对进程和线程之间的启动时间进行基准测试。多线程 Python 代码(在 CPython 中)意味着单核。当该线程持有全局解释器锁时,一个线程中的任何 Python 代码执行都将排除该进程中所有其他线程的执行(GIL https://wiki.python.org/moin/GlobalInterpreterLock)。这意味着只要涉及 Python 字节码,您就只能实现线程并发,而不是真正的并行。
您的示例主要是对特定的 CPU 限制工作负载性能(在紧密循环内运行计算)进行基准测试,无论如何您都不会使用线程。如果您想衡量创建开销,则必须从基准测试中删除除创建本身之外的任何内容(尽可能)。
TL; DR
启动一个线程(以 Ubuntu 18.04 为基准)比启动一个进程便宜很多倍。
与线程启动相比,使用指定 start_methods 的进程启动需要:
-
fork:约长 33 倍
-
分叉服务器:约 6693 倍长
-
spawn:约 7558 倍长
完整结果在底部。
基准
我最近升级到 Ubuntu 18.04 并测试了一个脚本的启动,希望它更接近事实。请注意,此代码是 Python 3。
一些用于格式化和比较测试结果的实用程序:
# thread_vs_proc_start_up.py
import sys
import time
import pandas as pd
from threading import Thread
import multiprocessing as mp
from multiprocessing import Process, Pipe
def format_secs(sec, decimals=2) -> str:
"""Format subseconds.
Example:
>>>format_secs(0.000_000_001)
# Out: '1.0 ns'
"""
if sec < 1e-6:
return f"{sec * 1e9:.{decimals}f} ns"
elif sec < 1e-3:
return f"{sec * 1e6:.{decimals}f} µs"
elif sec < 1:
return f"{sec * 1e3:.{decimals}f} ms"
elif sec >= 1:
return f"{sec:.{decimals}f} s"
def compare(value, base):
"""Return x-times relation of value and base."""
return f"{(value / base):.2f}x"
def display_results(executor, result_series):
"""Display results for Executor."""
exe_str = str(executor).split(".")[-1].strip('\'>')
print(f"\nresults for {exe_str}:\n")
print(result_series.describe().to_string(), "\n")
print(f"Minimum with {format_secs(result_series.min())}")
print("-" * 60)
基准测试函数如下。对于每一个测试n_runs
,创建一个新管道。
一个新的进程或线程(执行器)启动并且目标函数calc_start_up_time
立即返回时间差。就这样。
def calc_start_up_time(pipe_in, start):
pipe_in.send(time.perf_counter() - start)
pipe_in.close()
def run(executor, n_runs):
results = []
for _ in range(int(n_runs)):
pipe_out, pipe_in = Pipe(duplex=False)
exe = executor(target=calc_start_up_time, args=(pipe_in,
time.perf_counter(),))
exe.start()
# Note: Measuring only the time for exe.start() returning like:
# start = time.perf_counter()
# exe.start()
# end = time.perf_counter()
# would not include the full time a new process needs to become
# production ready.
results.append(pipe_out.recv())
pipe_out.close()
exe.join()
result_series = pd.Series(results)
display_results(executor, result_series)
return result_series.min()
它的构建是通过 start_method 和作为命令行参数传递的运行次数从终端启动的。基准测试将始终运行n_runs
使用指定的 start_method 启动进程(在 Ubuntu 18.04 上可用:fork、spawn、forkserver),然后与n_runs
线程启动。结果侧重于最小值,因为它们显示了可能的速度。
if __name__ == '__main__':
# Usage:
# ------
# Start from terminal with start_method and number of runs as arguments:
# $python thread_vs_proc_start_up.py fork 100
#
# Get all available start methods on your system with:
# >>>import multiprocessing as mp
# >>>mp.get_all_start_methods()
start_method, n_runs = sys.argv[1:]
mp.set_start_method(start_method)
mins = []
for executor in [Process, Thread]:
mins.append(run(executor, n_runs))
print(f"Minimum start-up time for processes takes "
f"{compare(*mins)} "
f"longer than for threads.")
Results
with n_runs=1000
在我生锈的机器上:
# Ubuntu 18.04 start_method: fork
# ================================
results for Process:
count 1000.000000
mean 0.002081
std 0.000288
min 0.001466
25% 0.001866
50% 0.001973
75% 0.002268
max 0.003365
Minimum with 1.47 ms
------------------------------------------------------------
results for Thread:
count 1000.000000
mean 0.000054
std 0.000013
min 0.000044
25% 0.000047
50% 0.000051
75% 0.000058
max 0.000319
Minimum with 43.89 µs
------------------------------------------------------------
Minimum start-up time for processes takes 33.41x longer than for threads.
# Ubuntu 18.04 start_method: spawn
# ================================
results for Process:
count 1000.000000
mean 0.333502
std 0.008068
min 0.321796
25% 0.328776
50% 0.331763
75% 0.336045
max 0.415568
Minimum with 321.80 ms
------------------------------------------------------------
results for Thread:
count 1000.000000
mean 0.000056
std 0.000016
min 0.000043
25% 0.000046
50% 0.000048
75% 0.000065
max 0.000231
Minimum with 42.58 µs
------------------------------------------------------------
Minimum start-up time for processes takes 7557.80x longer than for threads.
# Ubuntu 18.04 start_method: forkserver
# =====================================
results for Process:
count 1000.000000
mean 0.295011
std 0.007157
min 0.287871
25% 0.291440
50% 0.293263
75% 0.296185
max 0.361581
Minimum with 287.87 ms
------------------------------------------------------------
results for Thread:
count 1000.000000
mean 0.000055
std 0.000014
min 0.000043
25% 0.000045
50% 0.000047
75% 0.000064
max 0.000251
Minimum with 43.01 µs
------------------------------------------------------------
Minimum start-up time for processes takes 6693.44x longer than for threads.