Art objects access

Introduction

Access to art and culture for people with visual impairments (VI) is often complex, with the majority of works exhibited in museums based on the visual dimension.

Solutions to overcome this problem currently exist, such as audio descriptions or the creation of creation of 3D models allowing tactile exploration of the work. However, these solutions bring their share of limitations: audio descriptions are sequential and passive because they require listening and monopolize the attention of the user. 3D printing or thermoformed media is generally expensive to manufacture and produces too much detail to the touch, which requires the assistance of a person to assist in understanding.

Autonomous tactile exploration constitutes a current challenge and requires a simplification of forms. However, each artwork being specific, classical automatic methods do not provide a universal adaptive solution. It is more important that a tactile exploration (guided by the glance or the finger) allows a mental representation of the object, which audio descriptions do not always offer because they are often designed by people without visual impairment.

In order to improve accessibility to art and culture for people with visual impairment, we are developing a material solution allowing to display "transformed" artworks combining tactile, kinesthetic and audio perceptions, and allowing their active exploration, as well as software solutions to facilitate simplification towards this multimodal representation.

Tactile representation of an artwork

Tactile representations are based on the contours (borders) of objects because they allow to locate them and delimit them in a scene. Following the contours, it is possible to imagine the object (create your mental image) and recognize it. In figure 1 (right), we presented the outlines of the scene "the raven and the fox", taken from the Bayeux Tapestry (left image). It can be observed that the tactile representations of different elements of this scene are simplified, we tried to preserve the essential characteristics allowing them to be recognized (a tactile gist).

The raven and the fox, original (left) and outlines (right) Figure 1. Scene "the raven and the fox": image of the Bayeux Tapestry and its tactile representation with the outlines.

By recognizing the objects represented, their number and their relative locations it is possible to understand a scene (to create a mental representation of a scene).

How to get a tactile gist ?

Our first approach to creating a tactile gist is based on semantic segmentation of a scene. This segmentation allows parts of the image to be grouped into categories of generic (simplified) objects. Figure 2 illustrates this concept.

Segmentation sémantique (à gauche) et segmentation d’instance (à droite) Figure 2. Semantic segmentation and instance segmentation (source : https://ichi.pro/fr/segmentation-d-instance-en-une-seule-etape-un-examen-33996392156233).

Two approaches are most often considered to semantically segment an image:

  1. Segmentation into contours
  2. Segmentation into regions
These two classes of methods aim to simplify the image by eliminating or simplifying greatly the information contained (colors, details, etc.). Separating the content of a scene in regions (rough areas, interior of objects), and in contours (boundaries of objects) aims to facilitate the interpretation of the spatial organization of the content of the image and find atactile representation.

Figure 3 (Image on the left) shows scene 37 of the Bayeux Tapestry, the image in the center gives the results of its segmentation into regions (split-and-merge) and the one on the right gives its representation in contours (the HED algorithm, Holistically Nested Edge Detection, a convolutional neural network).

Segmentation in regions (center) et in contours (right) of the scene 37 of Bayeux Tapestry(left) Figure 3. Segmentation into regions (center image) and outlines (right image) of scene 37 of the Bayeux Tapestry.

Creation of 3D and 2.5D models

3D and 2.5D (or raised) models are another way to access the content of painting images. Our approach to mock-up is based on additive manufacturing (or 3D printing).

Our object model to be printed results from a combination of 3D reconstruction of a work from its 2D image acquired with a conventional camera, and 3D information obtained using the stereophotometric technique extended to image acquisition through glass.

Figure 4 (taken from scene 23 of the Bayeux Tapestry) shows Duke William of Normandy (image on the left), the visual/tactile gist of the figure of Guillaume (image in center) and a relief representation (2.5D) of Duke William (image on the right).

Presentations of Duke William the Conqueror Figure 4. Presentations of Duke William the Conqueror (from left to right): image of the Tapestry, visual gist and relief image (2.5D).

Figure 5 (taken from scene 1 of Bayeux Tapestry) shows King Edward(image on the left), the visual & tactile gist of the king's person and the relief card (image on the right) estimated by stereophotometry (15 million pixels each).

Different representations of King Edward Figure 5. Different representations of King Edward (scene 1 from Bayeux Tapestry).

Figure 6 taken from the painting kept at the Martainville Museum (image on the left) allowed to create the model (in Blender, image in the center) of the spinning wheel printed in 3D(image on the right).

Spinning wheel Figure 6. Spinning wheel (Martainville museum) and its different representations.

Towards a multimodal presentation of artworks

We present a solution developed for Museums to improve accessibility to artworks for visually impaired persons. Based on an innovative audio-tactile interface, this solution allows the active and independent exploration of artworks simplified into a more intuitive representation for visually impaired persons, by a solution combining edge detection and semantic segmentation. The research continues with the segmentation process to make it collaborative and interactive. In particular, we are planning the development of intuitive graphical interfaces, so that segmentation can be manually adjusted (Superpixel method) by specialists such as museum curators, to better reflect the intentions of the author of the work.