Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments - Equipe Télécommunications Optiques Access content directly
Theses Year : 2023

Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments

Quantification des égaliseurs de réseaux neuronaux dans les expériences de transmission par fibre optique

Abstract

The advent of the coherent detection paved the away for the compensation of the fibertransmission effects in the electrical domain using the digital signal processing (DSP).While the chromatic and polarization mode dispersion can be efficiently compensated withDSP, the compensation of the nonlinear distortions remains challenging.In this work, we consider neural networks (NNs) for nonlinearity mitigation in dualpolarization optical fiber transmission. Compared to the digital back-propagation (DBP),NNs do not require the fiber link parameters, and may mitigate the impairments withlower complexity.We propose two low-complexity NN equalizers: a convolutional-dense and an LSTM-dense model, placed at the end of the linear DSP to compensate the nonlinearities. Theseequalizers are evaluated in the context of three dual-polarization transmission experiments:a 9x50km true-wave classic fiber link, a 9x110km standard single-mode fiber link, and a17x70km LEAF fiber link. It is shown that the proposed NNs and DBP achieve about thesame Q-factors, both outperforming the linear DSP.We use quantization in order to reduce the computational complexity, storage sizeand energy consumption of the NN equalizers. We compare a number of post-trainingquantization (PTQ) and training-aware quantization (TAQ) algorithms for casting theweights and activations of the NN in few bits. For quantization above 5 bits, we showthat TAQ with straight-through estimation (STE) outperforms PTQ, since it mitigatesthe quantization noise during the training to some extent. For a Q-factor drop of less than0.5 dB compared to the unquantized NN, the storage and computational complexity of theNN can be typically reduced by over 90%. However, there is a bit width cut-off value ofaround 5 bits below which TAQ fails to outperform the linear DSP. This is because, theapproximation of the derivative of the quantizer in the STE is not sufficiently accurate atlow bit widths. Further, the proposed low-complexity models are not overparameterized,so that the quantization noise can be mitigated during the training at low bit widths. Itis shown that the quantization of the activations has a greater impact on the performancecompared to the quantization of the weights.Finally, we study extreme quantization of the NN equalizers below 5 bits. For thiscase, we propose three novel algorithms: successive PTQ (SPTQ), alpha-blending (AB)and successive AB (SAB) which is a hybrid algorithm that combines the SPTQ with AB.These algorithms are iterative, and incorporate ideas from PTQ and TAQ. We demonstrateiiithat the weights of the NN can be quantized up to one bit, if the activations are notquantized. Further, it is shown that both weights and activations can be quantized at 2-3bits, while still notably outperforming the linear equalization. Furthermore, we quantifythe impact of the quantization noise arising separately from the weights and activationson the Q-factor performance of the model. The results demonstrate for the first time thatlow-complexity binary NNs can mitigate nonlinearities in optical fiber communication.This PhD thesis is in the frame of a European Union's Horizon 2020 MSCA-ITN-EID REAL-NET project, grant agreement no. 813144, in collaboration with Infinera inGermany and Portugal
L'avènement de la détection cohérente a ouvert la voie à la compensation des effets liés à la propagation dans les fibres optiques en utilisant le traitement numérique du signal (”DSP”). Alors que les effets linéaires, tels que la dispersion chromatique et la dispersion modale de polarisation, peuvent être compensées efficacement, la compensation des distorsions non linéaires reste aujourd'hui un défi compte-tenu des complexités d'implémentation. Dans ce travail, nous considérons les réseaux de neurones (”NN”) pour l'égalisation dans la trans- mission par fibre optique à double polarisation. Par rapport aux égaliseurs conventionnels tels que la rétropropagation numérique (”DBP”), les NN ne nécessitent pas d'informations sur l' état du canal, et peuvent atténuer les dégradations du signal avec une moindre complexité. Nous proposons un certain nombre d'algorithmes de quantification ”post-training” et ”training-aware” pour représenter les poids et les activations du NN en quelques bits, ceci afin de réduire la complexité de calcul, l'espace mémoire et la consommation d'énergie du DSP. Une analyse de performance et de complexité montrent que les algorithmes proposes surpassent les algorithmes d'égalisation linéaire et DBP dans plusieurs expériences de transmission. Cette thèse est réalisée dans le cadre du projet H2020 MSCA-ITN-EID REAL-NET, financée par la Commission Européenne (en collaboration avec le partenaire industriel, Infinera Corporation, en Allemagne et au Portugal)
Fichier principal
Vignette du fichier
121877_DARWEESH_2023_archivage.pdf (4.63 Mo) Télécharger le fichier
Origin : Version validated by the jury (STAR)

Dates and versions

tel-04274898 , version 1 (08-11-2023)

Identifiers

  • HAL Id : tel-04274898 , version 1

Cite

Jamal Darweesh. Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments. Networking and Internet Architecture [cs.NI]. Institut Polytechnique de Paris, 2023. English. ⟨NNT : 2023IPPAT025⟩. ⟨tel-04274898⟩
89 View
45 Download

Share

Gmail Facebook X LinkedIn More