Abstract : The research work presented in this dissertation deals with the application of modulation and digital signal processing techniques in the field of optical communications. It focuses on the design of a 40Gb/s short-reach optical transmission system. For a given optical wavelength, capacity increase is generally achieved by pushing technological limits in order to augment the serial line rate of simple on-off keying modulations. This approach is not compatible with the economics of high-capacity short-reach applications. The use of sub-carrier multiplexing with multilevel signaling reduces bandwidth occupation and signaling rate. Components taken from existing 10Gb/s systems can thus be used to transmit 40Gb/s. At reduced signaling rates, CMOS circuit technology can be exploited to integrate elaborate signal processing functions. The adaptive signal processing structures presented in this dissertation relax analog front-end design requirements. They can also compensate for linear impairments due to the channel as well as fabrication process variations. The system is analyzed by deriving theoretical and numerical models agreeing with experimental validation. Simulation results show that a low-cost 40Gb/s system reaching beyond 10km is indeed feasible. We present a system prototype based on a CMOS test chip implementing modulation/demodulation functions for the highest-frequency and most challenging RF channel. This transmission system prototype operates over more than 30km of single mode fiber.