Birdsnap

Researchers at Columbia Engineering and the University of Maryland have taken bird-watching to a new level. Using computer vision and machine learning techniques, they have developed Birdsnap, a new iPhone app that is an electronic field guide featuring 500 of the most common North American bird species. The free app, which enables users to identify bird species through uploaded photos, accompanies a visually beautiful, comprehensive website that includes some 50,000 images. Birdsnap, which also features birdcalls for each species, offers users several ways to organize species -- alphabetically, by their phylogenetic relationship, and by the frequency with which they are sighted at a particular place and season. 

The researchers realized that many of the techniques that were developed for face recognition could also be applied to automatic species identification. State-of-the-art face recognition algorithms rely on methods that find correspondences between comparable parts of different faces, so that, for example, a nose is compared to a nose, and an eye to an eye. Birdsnap works the same way, detecting the parts of a bird so that it can examine the visual similarity of its comparable parts (each species is labeled through the location of 17 parts). It automatically discovers visually similar species and makes visual suggestions for how they can be distinguished. The app can also identify which parts of the bird the algorithm used to identify each species. It then automatically annotates images of the bird to show these distinctive parts often called 'field marks'. That way a user can learn what to look for.

The team also took advantage of the fact that modern cameras, especially those on phones, embed the date and location in their images and used that information to improve classification accuracy. Not only did they come up with a fully automatic method to teach users how to identify visually similar species, but they also designed a system that can pinpoint which birds are arriving, departing, or migrating.

The first in a series of electronic field guides was Leafsnap an app to identify tree species from photographs of their leaves. It was developed two years ago by the same group of researchers and institutions.

The group hopes next to work with Columbia Engineering colleagues on adding the ability to recognize bird songs, bringing audio and visual recognition together. They also wants to create "smart" binoculars that use this technology to identify and tag species within the field of view.