Automated classification of retinal artery (A) and vein (V) is of great importance for the management of eye diseases and systemic diseases. Traditional colour fundus images usually provide a large field of view of the retina in color, but often fail to capture the finer vessels and capillaries. In contrast, the new Optical Coherence Tomography Angiography (OCT-A) images can provide clear view of the retinal microvascular structure in gray scale down to capillary levels but cannot provide A/V information alone. For the first time, this study presents a new approach for the classification of A/V in OCT-A images, guided by the corresponding fundus images, so that the strengths of both modalities can be integrated together. To this end, we first estimate the vascular topologies of paired color fundus and OCT-A images respectively, then we propose a topological message passing algorithm to register the OCT-A onto color fundus images, and finally the integrated vascular topology map is categorized into arteries and veins by a clustering approach. The proposed method has been applied to a local dataset contains both fundus image and OCT-A, and it reliably identified individual arteries and veins in OCT-A. The experimental results show that despite lack of color and intensity information, it produces promising results. In addition, we will release our database to the public.