Objective and accurate evaluation of cataract surgery is a necessary way to improve the operative level of resident and shorten the learning curve. Our objective in this study is to construct quantifiable evaluation indicators through deep learning techniques to assist experts in the implementation of evaluation and verify the reliability of the evaluation indicators. We use a data set of 98 videos of incision, which is a critical step in cataract surgery. According to the visual characteristics of incision evaluation indicators specified in the International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubric: phacoemulsification (ICO-OSCAR: phaco), we propose using the ResNet and ResUnet to obtain the keratome tip position and the pupil shape to construct the quantifiable evaluation indexes, such as the tool trajectory, the size and shape of incision, and the scaling of a pupil. Referring to the motion of microscope and eye movement caused by keratome pushing during the video recording, we use the center of the pupil as a reference point to calculate the exact relative motion trajectory of the surgical instrument and the incision size, which can be used to directly evaluate surgical skill. The experiment shows that the evaluation indexes we constructed have high accuracy, which is highly consistent with the evaluation of the expert surgeons group.