Retinal image analysis is used by clinicians to diagnose and identify, if any, pathologies present in a patient's eye. The developments and applications of computer-aided diagnosis (CAD) systems in medical imaging have been rapidly increasing over the years. In this paper, we propose a system to classify left and right eye retinal images automatically. This paper describes our two-pronged approach to classify left and right retinal images by using the position of the central retinal vessel within the optic disc, and by the location of the macula with respect to the optic nerve head. We present a framework to automatically identify the locations of the key anatomical structures of the eye-macula, optic disc, central retinal vessels within the optic disc and the ISNT regions. A SVM model for left and right eye retinal image classification is trained based on the features from the detection and segmentation. An advantage of this is that other image processing algorithms can be focused on regions where diseases or pathologies and more likely to occur, thereby increasing the efficiency and accuracy of the retinal CAD system/pathology detection. We have tested our system on 102 retinal images, consisting of 51 left and right images each and achieved and accuracy of 94.1176%. The high experimental accuracy and robustness of this system demonstrates that there is potential for this system to be integrated and applied with other retinal CAD system, such as ARGALI, for a priori information in automatic mass screening and diagnosis of retinal diseases.