Maximum Likelihood Sequence Estimation (MLSE) over an FIR channel with unknown coefficients can be formulated as either a channel estimation problem (followed by MLSE), a joint sequence/channel estimation problem, or a direct sequence estimation problem. Since MLSE is our primary concern, we consider only the latter two formulations here. For joint estimation of both the transmitted sequence and channel coefficients, optimum methods have been developed both for non-time-varying (e.g. [1,2,3]) and slowly time-varying (e.g. [4]) channels. This approach can be effective, and the use of non-time-varying block processing algorithms has been proposed for a fast-changing channel (under the assumption that the channel is constant over the processing block). Direct estimation of the sequence (only) can result in improved estimator performance. However, the ML formulation for optimal sequence estimation over random channels, which requires marginalization over the unknown channel parameters, is generally intractable to solve directly.
The Expectation Maximization (EM) method is an approach to development of iterative ML and Maximum a Posterior (MAP) estimation algorithms, which was introduced into the signal processing community by Dempster, et al. [5]. For digital communications, the EM approach has been applied to the channel estimation problem [6,7,8]. It has also been employed for MLSE. For example, Georghiades, et al. [9] use EM to account for unknown phase given unsynchronized reception. Nelson and Poor [10] employ EM in multiuser applications where the other users' symbols are unknown. Zeger and Kobayashi [11] estimate GMSK modulated symbols in the presence of unknown multipath parameters. Cozzo and Hughes [12] consider multiple transmitter/receiver antenna systems and memoryless channels, and investigate the use of interlaced pilot signals. Zamiri-Jafarian and Pasupathy [13] derive an interesting on-line, suboptimal, joint channel/symbol estimator, and relate it to per-survivor and per-branch processing. In these examples, either simple prior distributions on the unknown secondary parameters are assumed in order to develop algorithms, or approximations (e.g. high SNR or on-line) are derived.
In this paper we describe a general EM algorithmic approach to MLSE over unknown random ISI channels. We use EM to marginalize over unknown channel parameters, incorporating knowledge of the FIR channel coefficients in the form of a prior distribution on the channel parameters. In [14] we describe how to incorporate linear channel constraints into this EM formulation. Here, we focus on a Gauss-Markov model for random FIR channels which can have significant variation over a short temporal block.