Skip to content

Support AWS p6 instances#3961

Merged
r4victor merged 4 commits into
masterfrom
issue_3940_aws_p6_support
Jun 15, 2026
Merged

Support AWS p6 instances#3961
r4victor merged 4 commits into
masterfrom
issue_3940_aws_p6_support

Conversation

@r4victor

@r4victor r4victor commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Closes #3940

Add support for AWS p6-b200 and p6-b300 instances. Specifically, update EFA configuration logic as every p6 type requires custom configuration (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html)

Also update EFA Docker image to include latest EFA-related software that support Blackwell. The current image didn't work.

NCCL tests on 2x p6-200:

#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)

           8             2     float     sum      -1    50.58    0.00    0.00       0    54.46    0.00    0.00       0
          16             4     float     sum      -1    54.55    0.00    0.00       0    54.52    0.00    0.00       0
          32             8     float     sum      -1    54.59    0.00    0.00       0    55.34    0.00    0.00       0
          64            16     float     sum      -1    55.60    0.00    0.00       0    55.51    0.00    0.00       0
         128            32     float     sum      -1    56.27    0.00    0.00       0    56.26    0.00    0.00       0
         256            64     float     sum      -1    57.14    0.00    0.01       0    56.54    0.00    0.01       0
         512           128     float     sum      -1    54.89    0.01    0.02       0    53.72    0.01    0.02       0
        1024           256     float     sum      -1    56.52    0.02    0.03       0    55.72    0.02    0.03       0
        2048           512     float     sum      -1    57.42    0.04    0.07       0    58.40    0.04    0.07       0
        4096          1024     float     sum      -1    65.08    0.06    0.12       0    62.18    0.07    0.12       0
        8192          2048     float     sum      -1    62.92    0.13    0.24       0    65.11    0.13    0.24       0
       16384          4096     float     sum      -1    63.52    0.26    0.48       0    63.80    0.26    0.48       0
       32768          8192     float     sum      -1    68.55    0.48    0.90       0    63.88    0.51    0.96       0
       65536         16384     float     sum      -1    65.94    0.99    1.86       0    68.60    0.96    1.79       0
      131072         32768     float     sum      -1    69.73    1.88    3.52       0    72.51    1.81    3.39       0
      262144         65536     float     sum      -1    88.11    2.98    5.58       0    88.38    2.97    5.56       0
      524288        131072     float     sum      -1    92.30    5.68   10.65       0    86.44    6.07   11.37       0
     1048576        262144     float     sum      -1    95.76   10.95   20.53       0    97.25   10.78   20.22       0
     2097152        524288     float     sum      -1   107.75   19.46   36.49       0   110.37   19.00   35.63       0
     4194304       1048576     float     sum      -1   138.53   30.28   56.77       0   138.33   30.32   56.85       0
     8388608       2097152     float     sum      -1   154.79   54.19  101.61       0   154.86   54.17  101.57       0
    16777216       4194304     float     sum      -1   190.75   87.95  164.91       0   188.86   88.84  166.57       0
    33554432       8388608     float     sum      -1   292.61  114.67  215.01       0   291.07  115.28  216.15       0
    67108864      16777216     float     sum      -1   481.39  139.40  261.38       0   481.32  139.43  261.43       0
   134217728      33554432     float     sum      -1   744.86  180.19  337.86       0   744.39  180.31  338.07       0
   268435456      67108864     float     sum      -1  1209.14  222.00  416.26       0  1198.05  224.06  420.11       0
   536870912     134217728     float     sum      -1  2401.34  223.57  419.20       0  2352.59  228.20  427.88       0
  1073741824     268435456     float     sum      -1  3742.18  286.93  537.99       0  3725.04  288.25  540.47       0
  2147483648     536870912     float     sum      -1  6696.75  320.68  601.27       0  6694.57  320.78  601.46       0
  4294967296    1073741824     float     sum      -1  12714.6  337.80  633.37       0  12653.1  339.44  636.45       0
  8589934592    2147483648     float     sum      -1  24589.9  349.33  654.99       0  24577.6  349.50  655.32       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 144.893
#
# Collective test concluded: all_reduce_perf
#

NCCL tests on 2x p6-300:

           8             2     float     sum      -1    52.59    0.00    0.00       0    55.36    0.00    0.00       0
          16             4     float     sum      -1    55.83    0.00    0.00       0    55.54    0.00    0.00       0
          32             8     float     sum      -1    63.49    0.00    0.00       0    57.03    0.00    0.00       0
          64            16     float     sum      -1    56.09    0.00    0.00       0    56.66    0.00    0.00       0
         128            32     float     sum      -1    58.28    0.00    0.00       0    57.42    0.00    0.00       0
         256            64     float     sum      -1    57.46    0.00    0.01       0    59.40    0.00    0.01       0
         512           128     float     sum      -1    57.19    0.01    0.02       0    59.40    0.01    0.02       0
        1024           256     float     sum      -1    58.32    0.02    0.03       0    58.76    0.02    0.03       0
        2048           512     float     sum      -1    61.75    0.03    0.06       0    61.44    0.03    0.06       0
        4096          1024     float     sum      -1    63.86    0.06    0.12       0    63.24    0.06    0.12       0
        8192          2048     float     sum      -1    67.63    0.12    0.23       0    66.69    0.12    0.23       0
       16384          4096     float     sum      -1    66.20    0.25    0.46       0    68.05    0.24    0.45       0
       32768          8192     float     sum      -1    69.04    0.47    0.89       0    71.83    0.46    0.86       0
       65536         16384     float     sum      -1    71.60    0.92    1.72       0    70.87    0.92    1.73       0
      131072         32768     float     sum      -1    73.05    1.79    3.36       0    72.36    1.81    3.40       0
      262144         65536     float     sum      -1    87.76    2.99    5.60       0    87.54    2.99    5.62       0
      524288        131072     float     sum      -1    90.57    5.79   10.85       0    91.32    5.74   10.76       0
     1048576        262144     float     sum      -1    97.55   10.75   20.16       0    97.00   10.81   20.27       0
     2097152        524288     float     sum      -1   109.56   19.14   35.89       0   110.80   18.93   35.49       0
     4194304       1048576     float     sum      -1   139.79   30.00   56.26       0   141.73   29.59   55.49       0
     8388608       2097152     float     sum      -1   153.14   54.78  102.71       0   155.10   54.08  101.41       0
    16777216       4194304     float     sum      -1   189.00   88.77  166.44       0   193.26   86.81  162.77       0
    33554432       8388608     float     sum      -1   283.43  118.39  221.98       0   283.00  118.57  222.31       0
    67108864      16777216     float     sum      -1   394.06  170.30  319.31       0   392.35  171.04  320.71       0
   134217728      33554432     float     sum      -1   596.39  225.05  421.97       0   597.03  224.81  421.51       0
   268435456      67108864     float     sum      -1  1023.87  262.18  491.58       0  1025.24  261.83  490.93       0
   536870912     134217728     float     sum      -1  1988.76  269.95  506.16       0  1990.46  269.72  505.73       0
  1073741824     268435456     float     sum      -1  3910.37  274.59  514.85       0  3910.56  274.57  514.83       0
  2147483648     536870912     float     sum      -1  5606.43  383.04  718.20       0  5615.47  382.42  717.04       0
  4294967296    1073741824     float     sum      -1  10827.2  396.68  743.78       0  10911.1  393.63  738.06       0
  8589934592    2147483648     float     sum      -1  21327.9  402.76  755.17       0  21412.5  401.16  752.18       0

# Out of bounds values : 0 OK
# Avg bus bandwidth    : 164.191
#
@r4victor r4victor merged commit c21b686 into master Jun 15, 2026
24 checks passed
@r4victor r4victor deleted the issue_3940_aws_p6_support branch June 15, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant