我正在尝试使用 torch.distributed 将 PyTorch 张量从一台机器发送到另一台机器。 dist.init_process_group 函数正常工作。但是,dist.broadcast 函数中出现连接失败。这是我在节点 0 上的代码:
import torch
from torch import distributed as dist
import numpy as np
import os
master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 0
backend = 'nccl'
os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))
a = torch.from_numpy(np.random.rand(3, 3)).cuda()
dist.broadcast(tensor=a, src=0)
这是我在节点 1 上的代码:
import torch
from torch import distributed as dist
import numpy as np
import os
master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 1
backend = 'nccl'
os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))
a = torch.zeros((3,3)).cuda()
dist.broadcast(tensor=a, src=0)
I set NCCL_DEBUG=INFO
在运行代码之前。以下是我在节点 1 上获得的信息:
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143<59811> failed : Connection timed out
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:100 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:326 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:695 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:951 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO misc/group.cc:69 -> 2 [Async thread]
Traceback (most recent call last):
File "test_dist_1.py", line 25, in <module>
dist.broadcast(tensor=a, src=0)
File "/root/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 806, in broadcast
work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-req-build-58y_cjjl/torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error
Node 0 似乎卡在函数 dist.broadcast 中:
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
谁能帮我这个?如何将张量从节点 0 发送到节点 1?我真的很感激任何帮助!