Understanding scale-up and scale-out networks

1 00:00:00,000 --> 00:00:01,680 Hi, I'm Spencer Giacalone, 2 00:00:01,680 --> 00:00:04,230 Principal CTO Architect at Ciena. 3 00:00:04,230 --> 00:00:06,675 If you're interested in AI infrastructure, 4 00:00:06,675 --> 00:00:09,210 you've probably heard of scale-up and scale-out. 5 00:00:09,210 --> 00:00:11,655 But what are they, and why do they exist? 6 00:00:11,655 --> 00:00:14,700 AI training involves huge mathematical models 7 00:00:14,700 --> 00:00:16,305 and large amounts of data. 8 00:00:16,305 --> 00:00:18,165 To fit these into GPU memory, 9 00:00:18,165 --> 00:00:19,740 while minimizing training time, 10 00:00:19,740 --> 00:00:21,150 the model and data are 11 00:00:21,150 --> 00:00:23,370 split across sets of GPUs. 12 00:00:23,370 --> 00:00:25,020 But during training, the model needs 13 00:00:25,020 --> 00:00:27,030 to be iteratively consolidated. 14 00:00:27,030 --> 00:00:29,340 And that imposes various traffic flows on 15 00:00:29,340 --> 00:00:30,720 the network–each of which 16 00:00:30,720 --> 00:00:32,370 has different performance requirements, 17 00:00:32,370 --> 00:00:35,220 but all of which are intolerant to jitter and loss. 18 00:00:35,220 --> 00:00:37,800 Congestion or packet drops can 19 00:00:37,800 --> 00:00:40,905 cause training corruption, stalls, or restarts. 20 00:00:40,905 --> 00:00:44,100 These training issues are costly and time consuming, 21 00:00:44,100 --> 00:00:46,260 and so we place training flows onto 22 00:00:46,260 --> 00:00:49,605 dedicated AI networks—scale-up and scale-out. 23 00:00:49,605 --> 00:00:51,660 Scale-up and scale-out exist in 24 00:00:51,660 --> 00:00:54,915 AI clusters to protect training flows with high capacity, 25 00:00:54,915 --> 00:00:57,780 low latency, lossless network connectivity. 26 00:00:57,780 --> 00:00:59,370 Scale-up and scale-out have 27 00:00:59,370 --> 00:01:00,780 different performance and size 28 00:01:00,780 --> 00:01:03,810 characteristics—and they use different technologies. 29 00:01:03,810 --> 00:01:07,720 Scale-up is the smaller but high-capacity AI network. 30 00:01:07,720 --> 00:01:10,325 For example, in NVIDIA's GB200 design, 31 00:01:10,325 --> 00:01:12,350 it provides 14.4T of 32 00:01:12,350 --> 00:01:16,280 capacity to each of 72 GPUs in a single rack. 33 00:01:16,280 --> 00:01:17,920 Larger, faster, scale-up designs 34 00:01:17,920 --> 00:01:19,650 will be coming soon, 35 00:01:19,650 --> 00:01:21,990 and in some cases have already been done. 36 00:01:21,990 --> 00:01:24,390 In NVIDIA's architecture, scale-up uses 37 00:01:24,390 --> 00:01:27,570 the bookended, or proprietary, NVLink protocol, 38 00:01:27,570 --> 00:01:29,580 which provides integrated congestion 39 00:01:29,580 --> 00:01:32,160 and buffer management at high speed. 40 00:01:32,160 --> 00:01:35,400 However, new standards such as UALink and 41 00:01:35,400 --> 00:01:37,070 ESUN are being launched 42 00:01:37,070 --> 00:01:39,270 to open the scale-up ecosystem. 43 00:01:39,270 --> 00:01:41,025 Scale-out, on the other hand, 44 00:01:41,025 --> 00:01:44,190 is the larger, but slower dedicated AI network. 45 00:01:44,190 --> 00:01:46,860 Scale-out can support hundreds of thousands 46 00:01:46,860 --> 00:01:48,735 of GPUs in a single campus, 47 00:01:48,735 --> 00:01:50,115 but at ethernet speeds. 48 00:01:50,115 --> 00:01:52,530 Most scale-out networks run RDMA over 49 00:01:52,530 --> 00:01:55,650 converged Ethernet, or RoCEv2, protocol, and that 50 00:01:55,650 --> 00:01:58,650 encapsulates the InfiniBand transport layer 51 00:01:58,650 --> 00:02:00,135 in IP and Ethernet. 52 00:02:00,135 --> 00:02:02,400 To make ethernet lossless for AI, 53 00:02:02,400 --> 00:02:04,560 features like Priority Flow Control and 54 00:02:04,560 --> 00:02:06,600 Explicit Congestion Notification are 55 00:02:06,600 --> 00:02:08,250 used to mitigate drops. 56 00:02:08,250 --> 00:02:11,070 As with scale-up, new standards are evolving, 57 00:02:11,070 --> 00:02:12,765 with the most notable being those 58 00:02:12,765 --> 00:02:14,665 from the Ultra Ethernet Consortium. 59 00:02:14,665 --> 00:02:17,855 In summary, AI models and data are very large. 60 00:02:17,855 --> 00:02:20,615 So, we split them up across groups of GPUs. 61 00:02:20,615 --> 00:02:23,000 However, consolidation is required 62 00:02:23,000 --> 00:02:25,010 to build a single consistent model. 63 00:02:25,010 --> 00:02:27,325 That creates various traffic flows, each of 64 00:02:27,325 --> 00:02:29,730 which have different performance requirements. 65 00:02:29,730 --> 00:02:33,510 And so we map those flows onto our two dedicated AI networks, 66 00:02:33,510 --> 00:02:35,010 scale-up and scale-out, 67 00:02:35,010 --> 00:02:36,585 both of which have different 68 00:02:36,585 --> 00:02:39,045 AI focused performance characteristics. 69 00:02:39,045 --> 00:02:41,095 Ciena is addressing scale-up and 70 00:02:41,095 --> 00:02:44,010 scale-out markets with our Nitro and Vesta products. 71 00:02:44,010 --> 00:02:46,980 Nitro is our linear ACC redriver that 72 00:02:46,980 --> 00:02:49,425 enables ultra-low-power inter-rack scale-up 73 00:02:49,425 --> 00:02:51,135 with no added latency. 74 00:02:51,135 --> 00:02:53,805 Vesta is our co-packaged optical engine, 75 00:02:53,805 --> 00:02:56,775 with the best density and lowest power in its class. 76 00:02:56,775 --> 00:03:00,390 Vesta is also part of the open CPX MSA. 77 00:03:00,390 --> 00:03:03,220 Contact Ciena to find out more.