Gradient Segmentation APIs
set_split_strategy_by_idx
Prototype
def set_split_strategy_by_idx(idxList, group="hccl_world_group")
Description
Sets the backward gradient segmentation policy in the collective communication group based on the gradient index ID to implement the convergence of AllReduce.
Parameters
Parameter |
Input/Output |
Description |
|---|---|---|
idxList |
Input |
A list. Index ID list of gradients.
|
group |
Input |
A string. Group name, which can be a user-defined value or hccl_world_group. Defaults to hccl_world_group. |
Returns
None
Example
from hccl.split.api import set_split_strategy_by_idx set_split_strategy_by_idx ([20, 100, 159], "group")
Restrictions
- The caller rank must be within the range defined by group in the current API. Otherwise, the API fails to be called.
- If you do not call the gradient segmentation API to set the segmentation policy, the default backward gradient segmentation policy is used.
Default segmentation policy: two segments with the first taking up 96.54% of the gradient data volume, and the second segment taking up 3.46% (In some cases, there is only one segment).
set_split_strategy_by_size
Prototype
def set_split_strategy_by_size(dataSizeList, group="hccl_world_group")
Description
Sets the backward gradient segmentation policy in the collective communication group based on the data volume percentage to implement the convergence of AllReduce.
Parameters
Parameter |
Input/Output |
Description |
|---|---|---|
dataSizeList |
Input |
A list. List of gradient parameter data volume percentages.
|
group |
Input |
A string containing a maximum of 128 bytes, including the end character. Group name, which can be a user-defined value or hccl_world_group. Defaults to hccl_world_group. |
Returns
None
Example
from hccl.split.api import set_split_strategy_by_size set_split_strategy_by_size ([60, 20, 20], "group")
Restrictions
- The caller rank must be within the range defined by group in the current API. Otherwise, the API fails to be called.
- When the backward gradient segmentation policy is set based on both the gradient data volume percentage and the gradient index ID, the setting result based on the gradient data volume percentage is preferred.
- If you do not call the gradient segmentation API to set the segmentation policy, the default backward gradient segmentation policy is used.
Default segmentation policy: The optimal segmentation location of ResNet50 is as follows: ResNet50 is divided into two segments based on the gradient data volume. The data volume of the first segment is 96.54%, and that of the second segment is 3.46%.