Huawei uses machine translation combined with human proofreading to translate this document to different languages in order to help you better understand the content of this document.
Note: Even the most advanced machine translation cannot match the quality of professional translators.
Huawei shall not bear any responsibility for translation accuracy and it is recommended that you refer to the English document (a link for which has been provided).
Computing Tensor Nodes for Collective Communication
You can call the operator prototype APIs to compute the tensors involved in collective communication.
AllReduce
allreduce: performs reduction on tensors with the same name across ranks within a collective communication group. The Reduce operation is specified by the reduction parameter.
#---------------------AllReduce test (two devices)--------------------------------- from npu_bridge.hccl import hccl_ops tensor = tf.random_uniform((1, 3), minval=1, maxval=10, dtype=tf.float32) allreduce_test = hccl_ops.allreduce(tensor , "sum")
AllGather
allgather: Each device receives the aggregation of tensor data from all ranks in the order of the ranks.
#---------------------AllGather test (two devices)--------------------------------- from npu_bridge.hccl import hccl_ops cCon = tf.constant([1.0,2.0,3.0]) allgather_test = hccl_ops.allgather(cCon, 2) #---------- rank 0/1 allgather _test = [1.0, 2.0, 3.0, 1.0, 2.0, 3.0] ----------
Broadcast
broadcast: copies an N-element buffer on the root rank to all ranks.
#---------------------Broadcast test (two devices)--------------------------------- from npu_bridge.hccl import hccl_ops cCon = tf.Variable([1.0,2.0,3.0]) input = [cCon] broadcast_test = hccl_ops.broadcast(input, 0) #---------------- rank 0/1 broadcast_test = [1.0, 2.0, 3.0] --------------------
ReduceScatter
reduce_scatter: performs the same operation as the Reduce operation, except that each rank receives a subpart of the result. The Reduce operation is specified by the reduction parameter.
#---------------------ReduceScatter test (two devices)----------------------------- from npu_bridge.hccl import hccl_ops cCon = tf.constant([1.0,2.0,3.0,4.0]) reducescatter_test = hccl_ops.reduce_scatter(cCon, "sum", 2) #-----------------rank 0 reducescatter _test = [2.0, 4.0] ---------------------- #-----------------rank 1 reducescatter _test = [6.0, 8.0] ----------------------
Send
send: sends data to a rank within a collective communication group.
#---------------------------------Send test------------------------------------- from npu_bridge.hccl import hccl_ops sr_tag = 0 dest_rank = 1 hccl_ops.send(tensor, sr_tag, dest_rank)
Receive
receive: receives data from a rank within a collective communication group.
#---------------------Receive test (two devices)----------------------------------- from npu_bridge.hccl import hccl_ops sr_tag = 0 src_rank = 0 tensor = hccl_ops.receive(tensor.shape, tensor.dtype, sr_tag, src_rank)