Debate on IP and DWDM continued in early 2023

LightCounting interviewed leading service providers on their priorities for DWDM networks last year and released a white paper in December 2022. One of the topics discussed in this study is the adoption of IP over DWDM technology, enabled by 400ZR/ZR+ pluggable transceivers. While all service providers are interested, they are not ready to adopt it yet. The debate continued at MWC and OFC this year.

Telefonica’s Oscar Gonzalez de Dios discussed progress in deployments of IP over DWDM in his OFC presentation. Despite being commercially available for more than 10 years, IP over DWDM has not been widely adopted in telecom networks. This approach is well suited for point-to-point links and DCI networks, but telco networks have a long list of concerns about it. The introduction of 400ZR/ZR+ modules shortened this list, but it still includes two important points:

• Operational complexity – different management teams and software systems for IP and DWDM transport.

• Speed – optical transport is always ahead of IP routers in terms of data rates of the optical interfaces.

Dr. Gonzalez de Dios addressed progress at Telefonica on a unified control of IP routers and DWDM transport equipment. He expects that Multilayer Software Defined Networks (SDN) could offer a solution. OIF runs a project along these lines. Nokia announced its unified IP/DWDM transport management software at MWC. Ciena also offers multi-layer management on their growing portfolio of routers, as illustrated in Figure 1.

In a conversation with LightCounting at OFC, Cisco disclosed that more than 40 service providers are deploying IP over DWDM now, but all them are small Tier 2 and Tier 3 companies. Simple networks can indeed use IP over DWDM, but complex optical networks, operated by larger telecom service providers, are unlikely to deploy this solution at scale.

Tad Hofmeister of Google reported on successful deployments of 400ZR on L3 switches in their networks. These deployments were limited to regional (metro) connectivity in just one of several networks run by Google, but they plan to deploy more this year. Saving cost by not using grey optics helped, but Google also runs an SDN to control the whole network – a key enabling factor.

Google is planning to deploy 800ZR next and hopes to have 1.6T pluggable modules in the future. However, different approaches will be needed, if 1.6T pluggable consume too much power. Distributed disaggregated (DDC) routers, composed of multiple chassis, is one of the options to consider. These systems can accommodate larger line cards, relaxing requirements on size and power consumption of DWDM optics.

In April 2023, Ciena introduced WaveRouter, designed to support up to 1.6T per wavelength DWDM optics. This metro-core router is designed to scale up to 192T maximum capacity in multiple chassis, interconnected with direct-attached copper cabling. It is effectively a DDC system with an integrated software and hardware.

One of the challenges for implementation of DDC systems is low cost and low power connectivity. Copper or optical cabling are the primary choices now, but we have seen board-mounted optical transceivers such as CXPs used in the past. We expect that co-packaged optics (CPO) will play a role in future DDC systems. This effectively brings the cost of grey optics, used to connect routers with DWDM transport systems, back into the picture.

The cost of grey optics has declined substantially over the last decade. Deployments of grey optics with the first 100G routers started with 100G CFP LR4 transceivers, priced around $5,000. This was almost a decade ago. Current designs of 100G LR4 (10km reach) are based on QSFP28 form factor, and are priced below $300. More than a million of these transceivers were sold last year. Given large volumes, transceiver suppliers can often reduce prices to levels unimaginable a decade ago.

Pricing of 400G FR4 (2km reach) transceivers, shipped in decent volumes last year, declined to below $500 by the end of 2022. It offers a good alternative to pricier 400G LR4, which are just starting to ship now.

According to our research, network operators value the flexibility enabled by a separation of DWDM transport and core routers, referred to as “IP+DWDM”, instead of “IP over DWDM”. This helps to manage different data rates, resolving the issue listed as the second bullet point above. It also enables deployments of flex-rate DWDM ports to find the right balance between the data rate and reach for a specific link in a network.

Dr. Maxim Kuschnerov of Huawei discussed this advantage in one of his slides at OFC 2023, presented in Figure 2 below. This analysis includes not only the cost of pluggable transceivers, but also cost of the equipment boards for it.

It clearly shows that cost savings of IP over DWDM are minimal and it would not justify increasing operational complexity, which is a major headache for large telecom operators. This is one of the reasons why network automation was ranked number one issue by service providers, as discussed in our white paper on that topic. It is the best option for supporting new applications and reducing network operation cost at the same time.

---