It is believed that there are between 1000 to 2000 skin conditions, and about 20% are difficult to diagnose. An intelligent system capable of making accurate diagnosis not only helps patients in places where access to health services are scarce, but also benefits typical general practitioners who have received minimal dermatology training. In this paper, we introduce a challenging dataset developed by gathering 2309 images from 44 different skin conditions, and collecting answers to simple perceptual questions from 361 'Amazon Mechanical Turk' workers. We also propose a method based on random forest technology that combines visual features of the skin lesion images with user provided answers to achieve promising recognition rates. We believe that our solution can be potentially improved and installed on smart phones and tablets to enhance quality of life in patients across the world.