The modulation mode recognition of co-channel signals is an important issue in modern electronic countermeasure and communication reconnaissance. The empirical mode decomposition provides a new method for signal processing from a non-stationary viewpoint, but one of the major drawbacks is mode mixing. A new method based on EMD method combined with Hilbert transform for modulation recognition was presented in this paper, that is, inserting different masking signals for several times during the decomposition process of co-channel signals. In the end, the necessary conditions of masking signal were also pointed out. The experimental results show that the proposed method performs well in single carrier signal separation and modulation recognition.