Use of Force Feedback Pointing Devices for Blind Users

Bertrand Tornil and Nadine Baptiste-Jessel
Institut de Recherche en Informatique de Toulouse
Université Paul Sabatier - Toulouse 3
31062 Toulouse Cedex, France
bertrand.tornil@gmail.com

Abstract

In this paper, we discuss the possible uses, for a blind user, of force feedback pointing devices, such as the mouse Wingman Force Feedback Mouse and the PHANTOM, associated with a sound feedback. We set the definitions of the gesture interaction and propose the interaction loop relative to these devices. Related works enable us to raise the limit of their use and thus to specify within which framework they are adapted: the relative localization of elements compared to the others. The applications that we are developing are based on this context of use. We present our application of geography which indicates the relative positions of the areas on a map, via a force feedback mouse and a voice synthesis. Lastly, we present the prototype of our three-dimensional application which accounts for the relative position of the human body elements. In order to automate the treatment as much as possible, we based our applications on data files in XML: the SVG for the geographical maps and the X3D will be retained for the format of the forms in 3D.

Keywords: accessibility, force feedback pointing devices, gestural interaction

1  Introduction

In front of a computer, a blind user uses the keyboard to operate the machine which answers him by a voice synthesis and/or a braille display. The text processing is well adapted to these methods. However, a graphic document will be presented to him by long and tiresome descriptions. The force feedback devices are used within the framework of the accessiblity for the blind users because they authorize a more direct interaction based on sensory capacities. We first of all will set the main definitions of the gestural interaction and will locate the perceptual mechanisms needed in order to use these devices. We concentrate here more specifically on force feedack pointing devices like the mouse. While reviewing related works within the framework of the accessibility for the blind users, as well as the limits observed, we propose a specific context of use for these devices: relative localization. We present then two applications of the relative localization. In 2D, it is a programme of reading of geographical maps which allows a better accessibility for the blind men; and in 3D, our prototype makes it possible to a blind man to deduce the position from a human body model displayed on the screen. Lastly, we will conclude by presenting the outlooks which we consider.

2  The gestural interaction

We set the general standards which relate to the gestural interaction between the user and the machine. A action-reaction loop (or interaction loop) could then be defined in our applicative context.

2.1  Human gesture

The tactilo-kinesthetic or " haptic" [Rev50] system is the synthesis of the movements of exploration of the motor system and of perceptions of the tactile system. The haptic sense is thus both effector and receptor. [LK87] classified these two aspects in the following way:
  1. movements of exploration of the hand:
    1. side friction (movement on both sides of the surface of an object)
    2. envelopment
    3. the static contact (positioning of the palm of the hand on the surface of an object)
    4. the following of contours
    5. the pressure (regular force applied to a given place of the object)
    6. the rising of an object
  2. sensory capacities related to the gestural modality:
    1. the cutaneous sense: it is the touch sense. It allows to feel the temperature, the pressure or the pain, and is relayed by sensory receptors located under the skin.
    2. the kinesthetic sense: it is the sense related on the position and the movements of the body. It enables us for example to know the weight of an object we're handling and its position. It is relayed by receptors based in the muscles, the tendons and the articulations.
All these human capacities must have their equivalent on the machine, in order to accomplish the interaction loop. We will see further which sensory and explorative capacities are stimulated according to the devices that we use.

2.2  Computer "gesture"

There are numerous force feedback and/or tactile devices. In our study, we focused more specifically on the force feedback pointing devices. These devices handle only one pointer in the virtual space of the machine: the position of the device is then translated into a couple of coordinates (x,y) in 2D or a triplet (x, y, z) in 3D. The devices are the Wingman Force Feedback Mouse for the 2D and the PHANToM for the 3D.

2.2.1  The Wingman Forces Feedback Mouse

This force feedback mouse (figure 1), was created by Immersion Corporation and marketed by Logitech. The mouse is interdependent of its base. The effective surface of work is of 1.9 X 2.5 cm and the forces can reach 1N in peak. The Wingman formely was a game device, but its use was diverted toward research on the accessibility.
Figure 1: Wingman force feedback mouse
Figure 1: Wingman force feedback mouse

2.2.2  The PHANTOM

The PHANTOM (figure 2) was created and is marketed by Sensable Technology. It is the most popular device in research on the haptic interaction. The volume of work is of 13 X 16 X 13cm and the force feedback can reach 8.5N.
Figure 2: the PHANTOM device
Figure 2: The PHANTOM device

2.3  The interaction loop for the pointing

The handling of these two pointing devices makes use of the movements of exploration of the arm. Articulations of the shoulder, the elbow and the wrist, and their associated muscles are thus implemented. The feedback operates on the same parts of the body. It is thus kinesthetic perception related to the arm via the shoulder, the elbow and the wrist which is requested.

3  Force feedback for blind users

3.1  Related Works

The use of the force feeback for blind users aims to make up, as much as reasonably possible, for the absence of the visual channel. Several approaches exist.
  1. Haptic feedback of a graphic interface. Thanks to force feedback mice, [Ram96] and [Ros97] transcribed the the graphic interactors of the interface in force feedback.
  2. Haptic feedback of the contents and the layout of a document. The translation of mathematical figures or tables was studied by [FB96]. [FB99] finally carried out a haptic system of visualization based on the PHANToM for people with visual handicap. [OT01] developed a programming library making it possible to identify the layout of a document and to guide the hand of the user on this document.
  3. Description of graphic documents. [HFR98] studied the possibilities of a force fedback on VRML (Virtual Reality Markup Language), a 3D file format on the Web. [GB01] used the SVG (Scalable Vector Graphics) to enable blind users to read geographical maps.
  4. Apprehension of shapes or textures. [FB96] worked on the synthesis of haptic textures. [CPK+98] used the Impulse Engine 3000 to study perception by the blind users of textures and virtual shapes. Finally [YRB01] studied the perception of mathematical graphics by blind users using the PHANToM.
Moreover, [DMR95] showed the interest of the audio-haptic bimodality for blind users. The table 1 indicates the percentage of good answers for three modal situations for 12 sighted users and 12 blind users.
Blind Users: 12 Sighted Users: 12 Total:24
Audio 68 % 62 % 64 %
Haptic 78 % 71 % 74 %
Bimodale 83 % 78 % 80 %
Total 76 % 70 % 73 %
Table 1: Bi-modal results

3.2  Limits

Limits were raised in the use of force feedback pointing devices for the blind users. As follows:
  1. the use of these devices for the perception of textures is inadequate, as pointed [YRB01]. Indeed, the cutaneous perceptors of the skin are not stimulated.
  2. [MRGSD02] and [CPK+98] pointed that the single contact point of the PHANToM does not allow the recognition of a three-dimensional complex shape. The gestures of envelopment of the hand would allow such a recognition, but that would need a device activating the kinesthesic feedback on the fingers.

3.3  Relative localization

The use of force feedback pointing devices must be based on the properties of the kinesthesic interaction of the arm. Kinesthesic perception related to the arm enables us to visualize the position of the hand in space. Thus, if an haptic event, like a vibration or a shock, occurs during a move of the arm, we can mentally represent the position that the hand had when the event occurred. Associated with a voice synthesis, this approach will allow the rebuilding of a mental image of an object from the relative positions of the elements of this object. The two applications which we will present use the force feedback in this context.

4  Applications of the relative localization

4.1  2D application: Géogr'Haptic

In this application, which run in a Internet browser, we display a map indicating the American states. A blind user handle the Wingman Force Feedback Mouse to explore the surface of the screen. When the pointer of the mouse passes on an area, it is "magnetized" toward its center. A sound feedback gives the name of the state, via a screen reader and a voice synthesis. The figure 3 illustrates this operation.
Figure 3: Geogr'Haptic, before the move
Figure 3: Geogr'Haptic, before the move
It is then necessary for the user to force his way out of the area, and either:
  1. to fall into a state bordering and to hear the name of this state, as the figure 4 shows it;
  2. to leave the map, and then to feel an effect of texture.
Figure 4: Geogr'Haptic, after the move
Figure 4: Geogr'Haptic, after the move
The format of the map is the SVG [SVG03] which is the implementation in XML of the vectorial pictures. The interest to use this format is multiple:
  1. its contents can be indexed by the search engines on the Web.
  2. the SVG supports the DOM (Document Object Model) and is therefore entirely scriptable. Geogr'Haptic is coded in Javascript.
  3. graphics in SVG can react to the users events such as onMouseOver() when the mouse passes on an area or onMouseClic() when the user clicks.
  4. the SVG can be displayed perfectly on all platforms, all output resolutions, with various bandwidths.
The first tests which we carried out with blind users are encouraging: they are able to quote the frontier states of Canada, or which states one must cross to go from a point A to a point B, which would not be possible by using traditional pictures on the Web.

4.2  3D application

This application uses the same principle as géogr'Haptic but in three dimensions thanks to the use of the PHANToM. We plan to base the application on the 3D XML format: the X3D [X3D02]. However, for the prototype, we currently use the POSER file format [Cur04] which contains, like the X3D, some meta-datas. The figure 5 shows the prototype of the application after the loading of a model of a human skeleton.
Figure 5: Body'Haptic
Figure 5: Body'Haptic
When we load the 3D model, the bounding boxes of each element are computed. A 3D cursor moved by the PHANToM allows to navigate into the 3D scene. A force feedback then attracts the pointer in the center of the nearest bounding box, while a voice synthesis reads the name of this element. Nowadays, our prototype allows to blind user to deduce the 3D model position. For instance, on the figure 5 it is standing with the arms in cross.

5  Outlooks

We should soon propose a test protocol which aims to confront our prototypes with the existing tools of access to the graphic documents for blind people. The prototype of our 3D application is still in an early stage development. For complex 3D models, our application will have to filter informations to be handled by the force feedback in instance to produce a scene with a good haptic legibility. We're going to use the XML 3D file format for the Web: the X3D [X3D02]. Just like the SVG, the X3D is completely scriptable, and support the DOM, which enables us to consider an exploitation in a Internet browser. Moreover its specifications include the management of the objects displayed by pointing devices. We must also study adjustments of the force feedback, which would be specific with the 3D model loaded: an effect could thus guide the user hand along the elements and the effects could be characteristic of the various parts of the body (the intensity of the effect would be different if we are in a bone or in an organ). This leads us to our last objective: we are about to build a 3D model of a human body including the organs. The haptic reading of such a model would be useful for blind and sighted users, in a pedagogical context.

6  Aknowlegdment

The authors would like to thank Frederic Gianni for providing the handling libraries of POSER file, and for its constant support.

References

[CPK+98]
C. Colwell, H. Petrie, D. Kornbrot, A. Hardwick, and Furner S. Use of a haptic device by blind and sighted people: perception of virtual textures and objects. In Improving the quality of life for the European citizen: technology for inclusive design and equality. Amsterdam: IOS Press, i. placencia-porrero and e. ballabio edition, 1998.
[Cur04]
Curious Labs. Poser 5. http://www.curiouslabs.com/, 2004.
[DMR95]
A. Dufresne, O/ Martial, and C. Ramstein. Multimodal user interface system for blind and 'visually occupied' users: Ergonomic evaluation of the haptic and auditive dimensions. In Proceedings of IFIP International Conference Interaction'95, Lillehammer, Norway, pages 163-168, 1995.
[FB96]
J.P. Fritz and K.E. Barner. Design of haptic graphing method. In Proceedings of RESNA-96 Annual Conference, Salt Lake City, USA, pages 158-160, 1996.
[FB99]
J.P. Fritz and K.E. Barner. Design of a haptic visualisation system for people with visual impairments. In IEEE Transactions on Rehabilitaion Engineering, pages 372-384, 1999.
[GB01]
J.A. Gardner and V. Bulatov. Smart figures, svg, and accessible web graphics. In Proceedings of Technology And Persons With Disabilities Conference 2001, Los Angeles, 2001.
[HFR98]
A. Hardwick, S. Further, and J. Rush. Tacile display of virtual reality from the world wide web - a potential access method for blind people. Display, 18, Issue 3:151-161, 1998.
[LK87]
S. J. Lederman and R. L. Klatzky. Haptic exploration in humans and machines: An initial overview. Technical report, Office of Naval Research, 1987.
[MRGSD02]
C. Magnusson, K. Rassmus-Gröhn, C. Sjöström, and H. Danielsson. Navigation and recognition in complex haptic environments - reports from an extensive study with blind users. Présenté à EuroHaptics, Edinburgh, UK, July 8-10, 2002.
[OT01]
D. Offen and B. Thomlinson. Good vibrations: Using a tactile mouse to convey page layout information to visually impaired computer users. In Proceedings of CSUN'S Sixteenth Annual International Conference :"Technology and Persons with Disabilities", Los Angeles, 2001.
[Ram96]
C. Ramstein. Combining Hpatic and Braille Technologies, Design Issues and Pilot Study. In ASSET'96, ACM/SIGCAPH. In 2nd Annual ACM Conference on Assistive Technologies, Vancouver, BC, Canada, pages 37-44, 1996.
[Rev50]
G. Revesz. Psychology an art of the blind. New York: Longmans, 1950.
[Ros97]
L. Rosenberg. Feelit mouse: Adding a realistic sense of feel to the computing experience, 1997.
[SVG03]
SVG. Scalable vector graphics (svg) 1.1 specification. http://www.w3.org/TR/SVG/, 2003.
[X3D02]
X3D. X3d working group overview. http://www.web3d.org/x3d.html, 2002.
[YRB01]
Wai Yu, Ramesh Ramloll, and Stephen Brewster. Haptic graphs for blind computer users. Lecture Notes in Computer Science, 2058:41-??, 2001.