We evaluate comprehensively neural network language models (NNLMs) and hybrid NNLMs in handwritten Chinese text recognition. We apply CNNs to over-segmentation and geometric context modeling in addition to character recognition. By training NNLMs on large corpus and integrating CNN shape models, we achieve new state-of-the-art performance on standard datasets. We analyze the upper bound of performance of the text recognition system by calculating the lattice error rate.