Annotation Replay: Images and Labour in Machine Vision

On 28 October 2021, Nicolas Malevé will deliver a collaborative paper as part of L’économie des images en sciences. Enjeux, modalités et impacts sur la production et la circulation des savoirs (XVIIIe-XXIe siècles) | The Economy of Images in the Sciences. How does it affect the production and circulation of knowledge? (18th-21st centuries) | 28 et 29 octobre 2021 – Institut National d’Histoire de l’Art, Paris.

Today, many scientific disciplines, increasingly large sectors of the industry and the Internet economy rely on computer vision algorithms to classify, filter, label, censor, augment, optimise, organize and take decisions. The recent breed of algorithms performing these tasks are often based on a deep learning framework. In this context, machine training is of uttermost importance. The training consists of feeding a program with huge curated sets of data from which it “learns” regularities. The production of these data sets requires an infrastructure at web scale. Billions of images are culled from the internet. And a large population of precarious workers, recruited on crowdsourcing platforms, annotate this flood of images to describe their contents to machines.

The presentation analyses this double outsourcing. Where computer scientists were once producing the visual data sets in house or by commissioning photoshoots, they now rely increasingly on popular platforms of photo sharing. Where computer scientists were annotating themselves their collections of images or hiring domain experts, a significant portion of the annotation work is now delegated to crowdworkers. This double outsourcing entails the design of a large assemblage of acquisition and classification of images produced for free by amateurs and semi-professionals, and a new division of labour for the field of computer vision where platform workers take decisions that define the boundaries of what machines will be able to “see”.

To address these issues, the presentation draws upon an experiment of annotation replay conducted in the frame of the research project Ways of Machine Seeing started at the University of Cambridge and developed at the Centre for the Research of the Networked Image at London South Bank University. An annotation replay experiment involves the new annotation of a popular data set produced in a teaching environment where primary and secondary school students revisit the descriptions, tags, categories and image selection as well as experience the workflow underlying machine vision.

Based on the findings of this experiment, the presentation will reflect on the epistemic and political consequences of the change in the visual supply chain of computer vision and its dependency to the click work economy. It will ask where the relevant knowledge is produced in this vast assemblage. What is the nature of this knowledge and who contributes to it? How is it enabled through the various instantiations of the image from online communities to data sets? Following the trajectories of images offers a different way of interrogating machine vision’s relation to knowledge and opens up a different pedagogical approach where what counts is to question how both humans and machines are trained to see.