Performance Optimization of Nonorthogonal MFSK for Symbol-by-Symbol Coherent Detection

M-ary frequency-shift keying (MFSK) is employed for several applications, including Internet-of-Things (IoT) and sensor-based communications. Previous studies have demonstrated that coherent detection of well-designed nonorthogonal MFSK signals outperforms orthogonal MFSK. This paper optimizes the error performance of nonorthogonal MFSK signals when the receiver uses a simple coherent detector on a symbol-by-symbol basis. First, we derive the theoretical conditions on the frequency separations to produce M symbol waveforms with negative crosscorrelation. Second, assuming equispaced frequencies, we analytically determine the optimum modulation index that maximizes the minimum distance among the symbol waveforms. Third, assuming non-equispaced frequencies, we optimize both nonorthogonal 4FSK and 8FSK signal sets. The optimized signal waveforms reduce the symbol error probability with respect to the current-best MFSK schemes existing in the literature, at the price of a bandwidth increase. For additive white Gaussian noise (AWGN) channels, an accurate expression for the symbol error probability of nonorthogonal 4FSK is also proposed.