Colorectal cancer (CRC) is the third most common cancer and is usually diagnosed using colonoscopy and biopsy. Diagnosis of pathological biopsy requires professional knowledge and technology. Computer-aided gland and lesion segmentation systems have been proposed to help pathologists in diagnosis of CRC. However, to the best of our knowledge, there has not been a literature work trying to segment different levels of intraepithelial neoplasia in CRC pathological image. To reduce such a research gap, in this paper, we firstly collect a colorectal cancer biopsy histopathology whole slide image (WSI) dataset, named Histo-CRC Biopsy dataset, for algorithm evaluation. We further propose a PPC-UNet network to segment high level, low level intraepithelial neoplasia and normal tissues. The proposed PPC-UNet consists of two modules i.e., a UNet-based network for segmentation, and a pixel-to-propagation consistency (PPC) contrastive learning-based network for UNet encoder pre-training. As the important feature can be learned from the unannotated data during pre-training, our approach can consistently improve the Dice of UNet by around 2% when different ratios of the training data are labeled.