Toward extracting and exploiting generalizable knowledge of deep 2D transformations in computer vision

Existing deep learning models suffer from out-of-distribution (o.o.d.) performance drop in computer vision tasks. In comparison, humans have a remarkable ability to interpret images, even if the scenes in the images are rare, thanks to the generalizability of acquired knowledge. This work attempts to answer two research questions: (1) the acquisition and (2) the utilization of generalizable knowledge about 2D transformations. To answer the first question, we demonstrate that deep neural networks can learn generalizable knowledge with a new training methodology based on synthetic datasets. The generalizability is reflected in the results that, even when the knowledge is learned from random noise, the networks can still achieve stable performance in parameter estimation tasks. To answer the second question, a novel architecture called “InterpretNet” is devised to utilize the learned knowledge in image classification tasks. The architecture consists of an ESTIMATOR and an IDENTIFIER, in addition to a CLASSIFIER. By emulating the “hypothesis-verification” process in human visual perception, our InterpretNet improves classification accuracy by 21.1%.