How Can We Evaluate Object Recognition Algorithms Using a Public Object Image Database?

Authors: Soochahn Lee, Duck Hoon Kim, Sang Uk Lee, Il Dong Yun

TL;DR: We propose an evaluation framework for object recognition algorithms that synthesizes realistic “Virtual Scenes” from the Amsterdam Library of Object Images, enabling controlled benchmarking under varying viewpoint, illumination color, and illumination direction.

What this paper is about

How do you fairly test whether an object recognition algorithm actually works? In 2005, public object image databases like the Amsterdam Library of Object Images (ALOI) existed, but there was no systematic way to use them for controlled evaluation. Simply running a detector on individual object images doesn’t tell you how it performs in cluttered, realistic scenes.

Key idea

We build synthetic natural scenes – called “Virtual Scenes” – by compositing object images from ALOI. To test a specific condition (say, changing illumination direction), we select a target object and place it alongside randomly chosen distractor objects, all rendered under that condition. By generating many Virtual Scenes per condition, we can statistically analyze how a recognition algorithm degrades under each factor. As a demonstration, we evaluate a SIFT-based recognition method across variations in viewing direction, illumination color, and illumination direction.

(Left) An example of a virtual 3-D plane projected onto a Virtual Scene. (Right) An example of the area on which 3-D instances are placed in the virtual 3-D plane.

Why it matters

This framework turns a static object database into a flexible evaluation testbed, letting researchers isolate exactly which real-world factors trip up their algorithms – something that was hard to do with existing benchmarks at the time.