Advanced algorithms implemented in coherent optical Digital Signal Processor (DSP) design have enabled the deployment of 800G-capable optical transport systems—offering unprecedented reach-capacity capabilities and better network economics. In addition to programmable capacity rates, 800G coherent DSP multi-baud technology is a key enabler for efficient network upgrade paths and for maximizing spectral efficiency for certain submarine applications. This paper explains the algorithms developed and employed in Ciena’s WaveLogic™ 5 Extreme (WL5e) which underpin the performance characteristics of this latest generation of 7 nm FinFET-based coherent optical DSP.