Cataract presents the leading cause of preventable blindness in the world. The degraded image quality of cataract fundus increases the risk of misdiagnosis and the uncertainty in preoperative planning. Unfortunately, the absence of annotated data, which should consist of cataract images and the corresponding clear ones from the same patients after surgery, limits the development of restoration algorithms for cataract images. In this paper, we propose an end-to-end unsupervised restoration method of cataract images to enhance the clinical observation of cataract fundus. The proposed method begins with constructing an annotated source domain through simulating cataract-like images. Then a restoration model for cataract images is designed based on pix2pix framework and trained via unsupervised domain adaptation to generalize the restoration mapping from simulated data to real one. In the experiment, the proposed method is validated in an ablation study and a comparison with previous methods. A favorable performance is presented by the proposed method against the previous methods. The code of of this paper will be released at https://github.com/liamheng/Restoration-of-Cataract-Images-via-Domain-Adaptation.