Design Guidelines for Web Interfaces of Home Automation Systems Accessible via Screen Reader

Marina Buzzi1, Barbara Leporini2 and Clara Meattini2

1 IIT–CNR, Pisa, Italy

2 ISTI–CNR, Pisa, Italy

E-mail: marina.buzzi@iit.cnr.it; barbara.leporini@isti.cnr.it; meattini.clara@gmail.com

Received 15 April 2019;
Accepted 30 June 2019

Abstract

Home Automation Systems (HAS) – also referred to as smart homes – exploit multiple components such as sensors, RFID readers, wireless devices, and remote control systems to enable easy interaction with smart appliances and devices, and to automate performing sequences of tasks to make human-device interaction simpler and life more comfortable. For people with vision impairment, especially those who are unable to see at all, smart homes can be a powerful tool for enhancing personal autonomy, provided that the system offers suitable device integration and accessible interfaces with a simple interaction via keyboard, assistive technology and other modalities such as voice and gestures.

This paper investigates the accessibility of web interfaces when interacting with HAS components via screen reader assistive technology, in order to propose potential suggestions to developers. Web interfaces are particularly considered in this study in order to support screen reader users who are not yet skilled in using touch-screen devices. Specifically, based on collected accessibility and usability issues, as well as users’ expectations and preferences, a Web-based prototype has been designed and optimized especially for interaction via screen reader. After describing an evaluation conducted with a small group of skilled screen reader users, several guidelines are suggested for designers of HAS interfaces.

Keywords: Smart homes, home automation, accessible interfaces, blind users.

1 Introduction

Autonomy is crucial for people with disability. The ability to move, interact, and perform everyday tasks increases personal freedom, independence and self-esteem. Domestic routines can be facilitated by a mix of emerging technologies including smart sensors and appliances connected throughout the Internet (IoT, Internet of Things), Home Automation Systems to check and control home devices, Ambient Intelligence to check on what is happening in one’s environment, Artificial Intelligence empowering personal assistants, and even more. Home Automation systems are increasingly popular. Thanks to ICT progress and mobile device pervasiveness, today all smart home components can be easily controlled via tablet and/or smartphone, both using touchscreen interaction or voice control. However, web user interfaces are still important for users who are not yet skilled with touchinteraction or for people with disability, such as blind users. Despite the increasing impact of smart home services in our society, few studies have investigated the motivations and obstacles experienced by blind users when using smart home services to empower their lives [1]. Smart homes can be a powerful way to enhance personal autonomy for people with disability, provided the system offers suitable device integration and truly accessible user interfaces (UI) that are easy to use via assistive technology (AT). Two decades ago accessibility and usability guidelines were proposed in order to enhance people’s interaction with web and desktop interfaces [2]. Nevertheless, accessibility problems still hinder interaction via assistive technology [3]. Research focuses on methodologies and tools for improving accessibility of user interfaces and related services. Despite an increasing focus on smart home environments in the human-computer interaction (HCI) field, there are few studies in this context for people with special needs.Although many studies address the design of effective and usable user interfaces [4–6], HASs have not yet been investigated focusing on UIs. This study aims to help overcome this gap. Our study investigates how to simplify and ease interaction via screen reader by blind people with Smart Home (SH) web interfaces, analyzing possible barriers and proposing guidelines for design from the very beginning of a project. Visually-impaired people can experience obstacles and issues when interacting with digital components. Speech technology, screen reading software and multimodal user interfaces have been proposed to overcome those access barriers [7]. The main goal of our study is designing accessible Smart Home Web UIs for screen reader users. This paper is an extension of previous work [8], introducing the following new content:

(1) Three scenarios of use that illustrate motivations and needs expressed by the users in the smart homes context;

(2) Additional functionalities of the prototype;

(3) An evaluation of the enhanced web interfaces prototype;

(4) A set of guidelines resulting from this study to help HAS developers make their products more usable via screen reader.

The main contribution of this work is to provide practical suggestions for designers enabling blind people to fully enjoy the potential of SH technology, both delivered services and functions, and logical organization of the interfaces’ graphic components, to make interaction via screen reader simple and satisfactory when performing actions autonomously. The paper is organized into nine sections. Sections 2 and 3 introduce related work and the methods applied to this study, respectively. Section 4 focuses on the accessibility of HAS interfaces, summarizing previous results [1, 9]. Section 5 concerns the design of smart homes; after introducing scenarios motivating this study, the system requirements are introduced. Section 6 describes the web prototype in detail. Section 7 describes an initial evaluation of the prototype in order to receive feedback early and improve overall design usability. Section 8 details specific accessibility guidelines for simplifying web interaction via screen reader. Conclusions and future work end the paper.

2 Context and Background

Smart homes can increase people’s comfort by simplifying control and management of domestic appliances and devices, seamlessly integrating different domotics components and automating repetitive tasks. The home and our living environment become aware of, adapt and respond to personal comfort needs, individual moods and information requirements. Gestures or vocal commands enable the user to modify the surrounding environment [10]. To answer these needs many HASs have been implemented in both commercial and open systems. The enormous spread of the Internet which now offers interoperability between different networks, components and systems together with the simplicity of user-friendly web interfaces led to HAS implementing more and more user interaction via web, with the possibility of aggregating and setting up different system components in a familiar environment. For this reason, web accessibility is crucial in order to guarantee satisfying access for all. Blind users are greatly interested in technology that can enhance their interaction and increase their autonomy. Smart homes offer all-in-one remote control services for managing and controlling all equipment and devices installed in the house: apps, facilities, and utilities such as electricity, gas and water supply, air conditioning, refrigerators, TVs [11]. They rely on a set of technologies for connecting smart objects (equipment, appliance, sensors, devices) present in the house and software control components (apps) [12].

Focusing more on monitoring health and personal well-being, Ambient Assisting Living (AAL) is a research field that exploits information and communication technology (ICT) to build systems and environments assisting people with disability and the elderly. Multiple disabilities progressively undermine the independence of older persons, affecting their motor, perceptive and cognitive systems. Safety and health monitoring support the independence of vulnerable people. Nowadays persons hope to live autonomously for as long as possible, maximizing comfort as aging progresses. Smart homes can contribute to Aging-In-Place, enabling the elderly to maintain their autonomy by controlling their home, activities, and health, thus enhancing their dignity and promoting well-being [13]. Old people still want to be independent and make decisions, so it is crucial to deliver non-obtrusive assistance since literature indicates that a decreased sense of control might have significant adverse effects on their health [14]. Internet empowers home automation systems by making them simpler and smarter, accessible anytime, anywhere. Smart Homes are a pervasive assistive technology for people with disability, a way to achieve independence, which encompasses usability and embraces personal care and safety [9]. B-Live is an early example of a system designed for the motor-impaired and elderly to automate routine tasks such as turning on/off lights, and opening/closing shutters and doors [15].

Since smart homes are still expensive it would be more valuable to use them in our everyday life before the need for assistance appears and be able to easily extend the home automation system by plugging in specific assistive technologies to make the surrounding environment suitable for the user when some ability is lost or reduced (making the adaptation process easier) [16]. Park et al. [17] investigated factors affecting the adoption of smart home services. They proposed an acceptance model incorporating enjoyment, compatibility, perceived connectedness, control, system reliability, security, and cost. Results suggest that perceived usefulness is the most influential predictor of intention and attitude. Furthermore, compatibility was the greatest motivation, showing the importance of providing compatible services incorporating traditional devices and appliances.

According to the WHO the number of visually impaired people worldwide is 285 million, of whom 39 million are totally blind. Visual disability is more frequent among older age groups: 65% of visually impaired people and 82% of the totally blind are aged 50 years and older [18]. The difficulties provoked by blindness can have a considerable impact on their daily functioning [19]. The impact of blindness and aging on independent living was investigated by Gallager and Jackson [20]. The psychological condition of being unable to carry out daily activities reduces the ability to perform physical exercise, thus increasing the risk of falls. A strong association exists between aging, impaired vision and falling. A number of very helpful aids are currently available on the market to improve the quality of life of blind people at home, including liquid level indicators, signature guides, talking appliances (watches, microwaves, weighing scales, etc.), and large-button phones [20].

The Internet of Things is shaping our society. Objects can now receive an IP address, becoming a digital entity on the Internet [21]. This enables new ways of communication between people and objects and between things themselves [22]. Smart objects such as devices and sensors, wearable health systems, RFID readers/tags, and beacons, make interaction smart by activating services either on demand or automatically event-driven. Context-aware services can be triggered specifically when the user enters in proximity of smart objects. The integration of RFID in the smart home environment is also essential for identification and tracking purposes [23]. An RFID architecture was proposed by Darianian and Michael [24] to enable control of home appliances: (a) washing machine –i.e., tagged clothes deliver information about colors, materials and suitable washing programs, and the reader also checks for laundering compatibility, (b) fridge and shelves communicate to a reader in the kitchen to suggest recipes based on food items detected, user preference and needs (his/her health conditions), by searching online recipes. Some studies propose architecture for implementing smart kitchens exploiting RFID and multimodal interfaces for assisting people with disability, but they are not focused on the totally blind but rather on the visually impaired who may have different needs as well as different assistive technologies [25, 26].

Smart homes equipped with IoT objects facilitate daily life for people with disability, increasing their autonomy and self-confidence. Being independent in personal daily life is the highest priority for the visually impaired [27]. Unfortunately there is little information about how mobile applications for home automation control can be made accessible via screen reader. If accessibility guidelines have been not been incorporated in such applications, users with visual disabilities may not be able to use them to help users lead fully independent lives [28]. A 3-year study carried out by participants in their own homes revealed resistance to proactive actions of technology. Users are afraid of losing control of life in the home. Thus authors recommend hiding proactive technology seamlessly within the users’ environment in order to avoid disturbing their daily lives. Smart home proactive actions can include programming and controls of parameters, but not making a phone call without the user command [29]. A common observation among many studies is that for smart home systems to become successful, users should be included in the design phase at the beginning of the project [29, 30]. Acceptability is the key factor in integrating new technologies in homes, especially for elderly or unskilled users [16].

Speech recognition and natural language interaction have great potential for empowering vulnerable people such as the elderly and persons with disability. A few years ago, speech recognition was rarely included in the design of smart homes. The audio channel was a promising but unexplored area for smart home interaction, but today technology is finally ready. Accurate speech recognition engines are available, intelligent systems exploiting artificial intelligence and neural networks make adaptation systems more sophisticated and able to answer people’s questions more accurately. Today the availability of commercial products that can immediately be plugged into home automation systems and controlled via popular personal assistants such as Alexa o Google Home has opened new avenues for smart home control and services. A vocal control system can manage nearly everything. Unfortunately, this approach may be not suitable for handling all smart home activities of a blind person. For example, via a vocal assistant like Alexa, it is possible to give a command to switch on/switch off a device, but up until now it was not possible to check its status. Even when a list of devices with a specific status or characteristic is vocally provided, user interaction requires the considerable time and cognitive load required to remember each item listed. This type of interaction has not yet been adequately investigated for use for smart homes.

Exploiting sounds in home automation for delivering information quickly may enhance the user experience [31]. Despite the improved security and safety ensured by smart home systems designed to offer greater independence, a big concern of older persons is the fear that vocal interaction might encourage a lazy lifestyle that could lead to the rapid worsening of their health. To address this concern, the technology design must evolve to promote a healthy way of living [16]. Designing for the elderly is fundamental today, considering the aging population rate in Europe, and it is crucial to promote a healthy way of life for these users [32]. However, a recent review of smart homes focused on home health monitoring technologies show low technology readiness and lack of strong evidence on smart homes’ effectiveness as a prevention tool for assisting the elderly [31].

Various solutions have been proposed to overcome difficulties for blind people [33]. IoT, sensors and actuators embedded in everyday objects, smart appliances and devices, and RFID systems are incorporated in the smart home environment, to mention a few technologies that are seamlessly integrated. Smart homes can empower people with disabilities when a usable design is applied to interfaces and to the logical information flow, encouraging social activity and monitoring health parameters over time [33]. Alerts advise caregivers when anomalies are detected by the monitoring system. A multimodal (gestural and vocal) interface for controlling smart home appliances has been proposed by Jeet et al. [34], to provide hands-free operation to people with motor disability and simplify interaction for blind people.

3 Method

In our study we investigated how to design accessible and usable web User Interfaces for a home automation system, especially for screen reader users. We focused on Web technologies since (1) Several home automation systems are still handled via Web interfaces and a mobile app version is also available, (2) not all blind users are fully familiar with touch-screen devices, and (3) various mobile apps are based on Web layout or on rendering engines of HTML + CSS + JavaScript (e.g. React, Cordova, etc.).

We carried out participative design by involving the end users in all phases of the project: collecting user requirements as well as designing the user interfaces and performing code-sign sessions with two blind users [1, 9]. Starting from this knowledge, we identified several crucial needs and consequently features to include in the HAS design, and defined a simple template for enabling a usable interaction via screen reader. For instance, rather than providing a single and comprehensive interface overview for showing that the lights are on/off, different simplified views are made available for a more compact rendering, better navigable via screen reader and keyboard. Briefly, our methodology can be summarized in the following procedure:

  1. Collection of the users’ requirements through users’ feedback and preferences as well as main accessibility and usability issues experienced via screen reader when interacting with a popular commercial system for Home Automation (Fibaro);
  2. Analysis of collected data and identification of the main features as well as functions to include in the design of user interface of a HAS;
  3. Selection of the components to include in the user interfaces, especially in terms of arrangement and organization;
  4. Design and development of a web-based prototype with special focus on the main functionalities to include in the user interface to be tested;
  5. Evaluation of the proposed prototype of Web-based interface for HAS;
  6. Generalization of the learned experience to propose a set of potential guidelines for the design.

For Step 4, the main accessibility and usability guidelines and suggestions were considered: the W3C WCAG (www.w3.org/WAI/ standards-guidelines/wcag/) and WAI-ARIA techniques (https://www.w3.org/WAI/standards-guidelines/aria/), specific usability design suggestions proposed in [1], and [9]. During the entire design cycle two totally blind users were involved in various pilot tests to evaluate the interface prototype. The Internet Explorer and Mozilla Firefox Browsers, and the screen reader Jaws for Windows (http://www.freedomscientific.com/) have been used to test the Web interfaces.

4 Accessibility of HAS Interfaces

4.1 Inspection Evaluation of HAS Interfaces via Screen Reader

As a preliminary step in our study, in [1] we analyzed the web interface of Fibaro (https://www.fibaro.com), a popular commercial Home Automation System, to understand the main issues encountered by a blind user when interacting via JAWS screen reader. This case study highlighted that when navigating the Fibaro standard web interfaces, a blind user is unable to obtain information about the status of devices and sensors, and easily perform any action with them. Since basic accessibility guidelines were not implemented, content appeared unclear, disorienting the user [1]. This result is similar to previous work. The accessibility of six web and mobile apps for home automation control was evaluated, all showing very low degrees of accessibility. These serious obstacles and issues might prevent users with visual disabilities from performing a number of tasks in the smart home environment [28]. The main accessibility issues can be summarized thus:

In summary, for accessible design it is important to keep in mind that totally blind people perform tasks and explore the content and elements differently from sighted people, when the screen reader is running. A blind user does not have the same opportunity to obtain an overview of the interface; consequently the contents and elements should be arranged and designed so that navigation via keyboard and voice synthesizer is feasible and easy to perform. Our work explores how the user interface can be arranged and designed so that the screen reader can quickly detect the elements and information.

4.2 Users’ Preferences and Expectations

Since user experience is crucial to understanding problems encountered in practice and fully understand needs still unexplored, for reader convenience we summarize the results of [9], considered when designing the prototype features and the new functional improvements.

A 10-item survey was created using Google Docs to obtain an overview of current use of SH technology and collect preferences and suggestions. The sample comprised forty-two visually-impaired people: 15 females (35.7%) and 27 males (64.3%), aged from 18 years to 70+ years (well-distributed between age ranges, apart from only 1 person older than 70 years), who answered the questionnaire. Of these subjects 32 are totally blind (76.2%) and 10 are low-vision people (23.8%).

Smart apps and devices are spreading fast throughout the visually impaired community. Although mobile touchscreen devices still pose some challenges in non-visual interaction, the smartphone (25%) was slightly preferred over the computer (17%) while most of them (52%) believe it is valuable to have both possibilities. More than half of the participants exploit technology to facilitate daily life, using tools for interacting with lighting and smart objects such as talking scales, digital objects labellers, and kitchen aids. The most interesting remote control systems for the participants include heating (95%), lights (81%) and water/gas valve control systems (64%), followed by air conditioning (40%), electrical system (40%), presence sensors (36%), and shutters (36%). The presence sensors could be particularly useful for a blind individual in many contexts, not only for detecting intruders. For example, they enable the user to understand whether, and for how long, a domestic worker is in the home. Participants suggest that accessibility and usability are still poor when interacting via screen reader. Personalization and scalability are missing features. These results are similar to those of sighted users investigated by [30], except that blind people do not express interest in TV and visually based entertainment (e.g., PlayStations).

5 Smart Home Design

5.1 Motivating Scenarios

A Home Automation System usually offers many functions and commands to manage and customize home control. Not all those features are used frequently and every day. Every so often visually impaired people have problems even accessing the most frequently used functions, such as switching devices on/off, or checking their status. Every day a number of tasks are routinely repeated, especially at home. Opening and closing doors/windows/curtains, on/off heating/cooling systems, cooking, activating/deactivating intrusion control systems, open/close gas valves, and garden watering are key examples. Simplifying these repeated tasks can greatly enhance the interaction for a screen reader user. These features should be considered when designing the main features and functionalities, in order to provide simpler interaction as well as shortcuts to make them easier to use by screen reader users.

Customization and personalization of the user interface as well as of the features and especially the routines offered by the system can simplify the system’s use. Tasks and triggering actions can be directly set up by the user according to his/her personal preferences and needs. A blind person could be interested in switching off all the lights to be sure that no light is still on (e.g., when going to sleep or going out); a person with motor impairment might be interested in closing or opening all the shutters. Being able to perform these tasks quickly and easily facilitates daily life. Instead, others can be automatically activated. For instance, potential time-based tasks are related to (1) going bed or waking up, (2) preparing lunch and dinner, (3) watering the garden (early morning or evening), and so on. Below we introduce some possible scenarios, which motivated us in proposing and designing new functionalities introduced in the prototype.

5.1.1 Scenario 1

Alice is a 54-year-old blind woman who works at the municipal office as phone operator. For her job she regularly uses the computer via a desktop screen reader for searching phone numbers and writing down notes. She uses the smartphone only for calls, because she is not very familiar with the touch-screen device. It is Tuesday afternoon and Alice is still in the office and would like to check her current home status: the current temperature, if the heating is turned on and if the washing machine scheduled for the afternoon is still working or if it has finished. By accessing her Home Automation System, she is able to read immediately on the home page that the current temperature is 17.9°C and that the heating system is on since it has been set at 18.5°C. The washing machine has finished. However, Alice notices that the oven is on. Therefore she supposes that her daughter has arrived home and is preparing something for dinner. Alice is satisfied with the HAS web interface: fortunately she can interact with it via PC and her screen reader and she is able to get all the main information as soon as the home page opens, with minimal effort. In fact, the interaction is easy and does not require numerous steps and clicks, nor is there extra content to be read via screen reader before arriving at the desired information. This is very different from the interfaces offered by the popular system, which was not truly accessible via screen reader (e.g., the Fibaro system).

Such system use guides the design in choosing a web interface and in arranging the main information in the home page (e.g., heating temperature, devices switched on, etc.).

5.1.2 Scenario 2

Brian is a blind 70-year-old man who lives with his daughter, son-in-law and grandchildren in a big house. Often in the evening or on the weekend he stays home alone, when the rest of the family go out for dinner, visit friends or attend other events. Before going to bed, Brian wants to make sure all windows are closed and to check which lights are still on. Thanks to the system recently installed by his son-in-law, he can autonomously control the status of the smart devices through his PC. He can therefore check the status of all the windows, and learn which lights are still on. Fortunately, the new system installed offers a simple interface that requires few actions. Brian uses the computer to read, access his mail and search for simple information. He is not a particularly skilled screen reader user, but thanks to the few and simple tasks required, checking the windows and lights is now easy. Brian is able also to close or switch off the desired windows and lights.

This use scenario motivated us to design a web interface and specially to introduce simple routines to check the windows and lights by a single click.

5.1.3 Scenario 3

Charlotte is a blind girl who lives alone in Rome. She studies Humanities at the university, and she spends a lot of time on the computer to take notes and for studying. When going out, Charlotte would like to close all the windows and turn off all the lights, except one that is placed near the sofa in the living room. Fortunately, the smart home system allows Charlotte to customize a set of ready functions, such as “Close all windows” and “Turn off all lights”. These functions are customizable from the “settings panel”, which allows personalizing various features. In this way the function “Turn off all lights” can be personalized by adding the lights to be turned off, leaving out those to be kept turned on (in this case the light in the living room).

This type of use motivated us to introduce customization and personalization of pre-set functions which can be associated with everyday routines.

5.2 System Requirements

The preliminary results of our study lead us to suggest recommendations for the designers of domotics applications. A summary of general functional requirements is reported. See [9] for more details:

5.2.1 Web Interfaces

The usability definition by ISO 9241 (www.iso.org) takes into account the specified users when interacting with a system in a context of use to reach specified goals. Our goals are to be able to easily and satisfactorily check the status of the devices and to set/perform specific actions such as turning lights on/off or activating predefined scenarios. Accordingly, the main tasks and objectives to meet in the proposed interfaces are identified in:

These requirements have been used to design the prototype Web interfaces.

6 The Prototype

The following functionalities have been included in the UI design:

  1. Home summary and room details. The user interface offers a quick summary of the home status (i.e., how many lights are on, which devices are active, etc.); the user also has the opportunity to visualize the details of a single room.
  2. Global search and “ready-functions” for checking device status. The user can perform any search about the status of devices, but to simplify the interaction for frequent actions, the user interface provides “pre-set functions”.
  3. Scenarios to perform according to user preference or triggered by specific events for performing a set of set tasks. Specifically, in this case we will consider the interfaces and labels to use for a clearer perception via screen reader.

In this perspective, the prototype includes the following user interfaces:

In the following we illustrate these features showing how the proposed solutions can simplify the web interface interaction via screen reader.

6.1 Case 1: Map View

Map views enable a friendly graphic interaction with the home structure but usually are poorly accessible via screen reader. We show how to apply accessibility design to a map. The user interface design is aimed at (1) reproducing an attractive map for sighted people, (2) creating a room list clearly detectable by the screen reader, and (3) showing a summary of the home status, to provide a quick global overview, or room by room to achieve more details. When the user selects the link “Home” from the “menu”, the home map shows a summary of the current status of the most important home devices (See Figure 1).

Table 1 shows the content announced by Jaws when accessing the home map. The italics indicate the content added by the screen reader to indicate the kind of user interface element (e.g., link, button, heading level, etc.). By navigating the map via Tab key, Jaws announces the type of the focused element. The user can obtain more details by navigating via arrow keys. In this way the map is detected by the screen reader as a list of elements, i.e., the room names. Each room is announced in a single line with its name followed by the ‘clickable’ attribute. The user can quickly go along the list items with the arrows and select the desired room by just pressing the space bar key (corresponding to clicking on that item). For a global overview, on the home page we included an interface enhancement: the summary of the home status, such as temperature, heating system status, number of the total lights on, number of opened shutters, status of the washing and dishwasher machines (Table 1). This summary should be configurable according to user preferences. Analogously, by clicking on a room, its detailed state is displayed. This provides the screen reader user with a quick overview of the home status at any time.

images

Figure 1 Home map view: status summary.

Table 1 Text announced by Jaws: the home status (reading by column)

My home: StatusBathroom clickable
Menu navigation region list of 4 itemsTerrace clickable
Link HomeHeading level 2 Summary
Link Deviceslist of 9 items
Link ScenariosInternal Temperature: 16.5°
Link SettingsExternal Temperature: 18°
list endHeating System: OFF, away 15°
Menu navigation region endLights ON: 10 Windows Open: 2
main regionDishwasher: ON
Home Map clickableOven: ON
Double bedroom clickableFridge: ON
Kitchen clickableTelevision: ON
Living room clickablelist end
Bedroom clickablemain region end

6.2 Case 2: View by Status & Devices

The user may want (1) the status related to homogeneous devices (e.g., all shutters), or (2) an overview of all the devices in a specific status (e.g., lights turned on). To this end, the user interface should enable search by “category” or “status”. Flexible interfaces aid screen reader users to make interaction faster and more satisfying. Thus, in the proposed prototype, the panel “Devices” offers the opportunity to perform structured searches (by category, state, room) or predefined quick searches (Figure 2(a)).

images

Figure 2 (a) Quick search – (b) “Lights on” results.

This saves time and especially increases the opportunity to catch the desired information. Generally speaking, the user interface should provide a very quick way to carry out these frequent daily activities. Thus, our prototype has been designed to offer this opportunity: (1) performing a search with three parameters (device, status and room pull-down menu), and (2) exploiting the results to quickly change device/element status in one click. The user interface also offers buttons to perform high-frequent queries very quickly, such as obtaining the list of lights on, open shutters and windows, and so on (Figure 2(b)). The user can immediately read it via the arrow keys. To turn off a light it is enough to press the space bar on the corresponding element.

6.3 Case 3: The Ready Scenarios

Some activities performed frequently and repeatedly every day can be encapsulated in ready-to-use scripts. These should be grouped on a single tab/page clearly identifiable by the screen reader. For example, a tab entitled “scenarios” could include options already packaged and just a simple click can be used to activate one. The prototype offers few ready-made options to be selected quickly. Figure 3 shows a possible list of scenarios available to the user. The system offers two type of scenarios:

images

Figure 3 Customized and triggered scenarios.

6.4 Case 4: Settings and Customization

Customization enhances the functionalities offered by the system and allows blind users to maximize flexibility when using the system. For example, the system provides a “turn off all lights” button. Indeed, the user (such as the woman in Scenario 3) has to be able to select which lights to turn off when this function is activated. The user should be able to create new configured scenarios matching their needs. This type of customization would make the system effectively flexible. Figure 4 shows the interface for creating a new scenario. The scenarios can be also modified by adding and removing actions. For instance, the “Away” scenario can be modified adding the action “turn lights OFF”, including the command to switch off the list of lights personalized by the user.

images

Figure 4 Creating the customized scenarios “Away”: Arming the IDS system.

6.5 Design Features: Partitioning and Additional Info

Some design techniques have been added to facilitate screen reader interaction:

7 Evaluation

A small group of blind people was involved in gathering feedback on the proposed web-based interfaces by the end-users. Thus, five visually impaired users (3 males and 2 females, age range 26–57 years) were selected from a local association for the blind, and no remuneration occurred.All participants involved in testing the proposed Web interfaces were skilled in using the screen reader Jaws for Windows (3 users) and VoiceOver for OS X (2 users).

The tests were conducted using the user’s desktop, one at time at their home or office. Each test was monitored via Skype by the authors. We decided on this approach for two reasons: (1) the user could navigate the prototype in a familiar environment (i.e., screen reader settings, operating system customization, preferred browser, etc.); (2) The user was not required to travel to reach the test location. Furthermore, the blind author was able to listen to and “understand” the screen reader speaking while the user was interacting with the prototype.

The users were asked to interact with the system by assigning them three tasks: (a) Check (if any) what devices are ON in the living room; (b) Turn off all the lights, and (c) Detect the home internal temperature. These tasks were selected according to the (I) expectations expressed by the interviewed users (i.e., the desired activities) and (II) main accessibility issues and lacks observed in the inspection testing on a popular HAS [1] in performing everyday tasks and actions. Furthermore, the three tasks were also selected by taking into account the main actions to be performed on the HAS prototype: (a) getting the list of devices with a specific status (e.g., ‘on’); (b) checking the status of a ‘simple’ parameter available on the home summary page; and (c) performing an action on a specific device category. The first task allowed us to evaluate whether the user was able to perform a search by status in a room. The second one allowed us to assess whether the user was able to easily detect the pre-set button to perform an action (switch off all the lights) on a specific device type. The third task allowed us to evaluate whether the user was able to navigate the home map and show the home status summary, rather than perform a search to obtain this information.

The users accomplished three tasks in a natural setting. A think-aloud protocol was applied and researchers observed and recorded any comments. Since the aim was to detect navigation issues encountered during task and satisfaction feedback and collecting user feedback, no quantitative measures were collected.

All users completed the three proposed tasks successfully, but two of them took longer to explore the interfaces for performing the tasks. People may use different exploration and navigation strategies. Therefore it is important to expose various features and techniques to navigate the web interface quickly. In performing the second task (turn all the lights on) we observed that one user started by performing a search by device (light), but finally changed the action to searching for a potential pre-set button designed to the purpose.

Interesting suggestions regarding the interfaces emerged during the first pilot evaluation conducted with three skilled people [8], useful for enhancing their usability. Those suggestions included shortcuts and additional predefined buttons to improve the use of the system; specific links to simplify the selection in order to enhance a search by device and status, and so on. These suggestions – as emerged from the first evaluation – were used to design an enhanced version of the prototype, which is presented in this work. In this version, links such as “Lights off” were designed by also simplifying the labels. Briefly, additional scenarios, customizable features and clearer labels were introduced in our enhanced prototype, which was further tested by the small group of users. The new version of the prototype resulted simplified and easier to use by everyone, as emerged from this second evaluation. However, further comments were provided by the users: (a) some commands such as “turn off all lights” available near the results of a search in order to switch off the devices just listed (i.e., those listed in the results); (b) some further scenarios to set features could be very useful to enhance navigation with the user interface. (c) shortcuts could be customizable and associated with different and various functions available in the HAS and personalizable by the user. An interesting suggestion that emerged from this evaluation was related to (d) the opportunity to exploit sounds or short audio messages to provide additional information. Most of the users declared they would be interested in this feature in order to have quick feedback from the system when accessing it. For example, a given short sound associated with certain events or status as soon as the user accesses the system could significantly improve the perception, avoiding the need to explore the user interface. An opportunity to customize these sounds or audio messages could further enhance the usability and effectiveness of the user interface (and thus of the system). For instance, a given sound could be heard by the user when the internal temperature is higher (or lower) than a certain threshold set by the user himself: when the user accesses the system, the sound is immediately played and so it is not necessary to explore the user interface to check it. Also, when showing a list of devices (lights, appliances and devices in a room, etc.), a short sound could inform the user that at least one of them is on. This would be very useful to simplify the notifications to the user. For the user, it is important for these sounds to be customizable in order to make the perception easier and more intuitive. This can reduce the need to explore the user interface just to obtain quick information on the status. This comment was expressed by four out of five users.

8 Proposed Design Guidelines

To the best of our knowledge, no specific criteria or guidelines have been proposed for an inclusive Home Automation System designed to be accessible and usable, especially for screen reader users. Based on the experience of this study, we propose a set of eight guidelines for designers and developers of Home Automation System interfaces.

Although this study is mainly based on web user interfaces, these suggestions can be generalized for any interface designed for a Home Automation System in the smart home context. Briefly, three key aspects have driven our proposal of the design guidelines:

(1) Avoidance or reduction of irrelevant information; this can positively affect the sequential reading and interaction via screen reader and keyboard.

(2) Visualization of content related only to the current goal; this enhances the user interaction, especially with common commands. In addition, the user can focus on the useful functionalities and elements related to the context.

(3) Simplicity for repeated tasks. Activities carried out frequently and repeatedly should be made simpler to use via screen reader and keyboard.

The proposed guidelines have been arranged according to the three principles defined by the W3C in the Web Content Accessibil-ity Guidelines: perceivable, operable and understandable. For each principle, a short description to introduce the essential meaning is reported.

Principle 1: Perceivable. Information and user interface components must be presentable to users in ways they can perceive.

G1. Content and functions semantic groups. Group information, functions and commands semantically into separate pages, well-detectable via screen reader according to precise goals and activities. For example, consider the sections “summary”, “devices”, “scenarios” and “settings” to create pages including the related functions and contents. Each page should be well-perceived by the screen reader via:

  1. Clear Links (labels): each page should be linked in the navigation bar/menu via well-detectable clear labels. Labels such as “home”, “devices”, “scenarios” and “settings” are suggested as a minimum typical set for the menu items in a well-perceivable HAS.
  2. Clear title. Each page should be equipped with a clear title able to inform the user immediately about its content, i.e., with a clear content in the <title> tag. For example, the title could include the label of the link pointing to that page (e.g., “My home: Scenarios”), or a very short summary of the page (e.g., “my home: Device status via different views”).

G2. Status summary. Make available a clear and easily reached summary of the comprehensive current status. A possible design can include:

  1. Home status summary: comprehensive information on the home status, such as appliances, devices and any other useful data available within easy reach on the home page.
  2. Room status summary: a summary status on appliances, devices and any other useful data can be visualized for each room.

G3. Different status views. Let the user have different opportunities to show the devices and their status in order to obtain information or interact with them. Different views help to rapidly accommodate different user needs and preferences. This can be designed through:

  1. Search Options: various modalities to limit the visualization to specific devices, rooms or categories can help provide a suitable view via screen reader. In fact, in this way the sequential reading would focus only on a restricted group of elements with specific features (e.g., all the devices on, or all the devices in a single room, all the devices of a specific typology, and so on).
  2. Interactive graphical map: a graphical map can offer a quick overview of the home structure and navigation among the rooms.

An accessible graphical map can show the screen reader the rooms in a list of clickable items. This allows the user to explore room by room in a sequential way via screen reader. When a room is selected, the relative summary can be displayed on the same or another page, or in a dialogue window. In sum, graphical maps can be very useful for sighted users and so they should be used in a HAS, but they must be accessible for screen reader users via textual labels clearly expressing the names of the rooms and making each label interactive via keyboard. This allows a blind person to quickly understand the home structure and easily obtain information on a room.

G4. Feedback. Provide clear and immediate feedback and notification about status and events. These notifications should require minimal effort by the user. For instance, a short sound or a dialogue window used to provide a short message or quick information can significantly enhance the perception by the user, who is not asked to perform any action to get the same content. This can be implemented by:

  1. Dialogue window or live region. A brief information or message should be provided to the user via a dialogue window or a live region, which only includes the information to deliver. In this way, the screen reader can quickly access the information (e.g., that the washing machine has just finished) and announce it to the user. In this case, no action is required by the user.

2. Short sound or auditory message. For certain events or status notifications, a short sound or an auditory message can be very useful. Give the user the opportunity to assign sounds or auditory content to the events or customizable functions. This could be better implemented by making available some rules to be used for customizable functions. For instance, a rule could be “When the status of a certain threshold is becoming lower, make a sound (selectable by the user).”

Principle 2: Operable. User interface components and navigation must be operable. The user should be able to interact easily with the Home Automation System functions and commands. Special attention should be focused on those are repeatedly used every day.

G5. Shortcuts. Make functions and commands easily operable via key-board and screen reader. Repeated tasks and frequently used commands should be made available in a compact and agile way. In fact, screen reader users who do not have an overview of the user interface (i.e., the web page) cannot quickly select the desired elements by simply clicking on them; on the contrary, several steps via Tab or arrow keys must be carried out, requiring considerable effort. Shortcuts can enhance the screen reader interaction. The design includes:

1. Pre-set functions and routines: for frequently repeated tasks the sequence should be compacted in a pre-set function. For example:

(1) “Lights off” for quickly turning off a set of lights customized by the user; (2) “Lights on” to get the list of the lights currently on and turn off some of them (by clicking on the specified light); (3) “Windows Open” to check which windows are still open. Similarly, other routines can be defined and associated with a specific button/link.

  1. Keyboard shortcuts: for main functions and commands, a key as shortcut can greatly aid the user’s effort to activate the associated command or routine.

G6. Content partitioning. Use techniques and strategies to arrange the contents on a page and allow the user to operate properly. This guideline refers to each page, and not for a specific case (as instead occurred for the graphical map). This criterion allows a screen reader user to better focus on the main parts and contents when exploring any page via keyboard and assistive technology. The guideline suggests using:

  1. Section headings: heading levels assigned to each section on the page can give a screen reader user a quick overview and help them navigate easily among the sections. For instance, in the “Scenarios” panel, the “Customized Scenarios” and “Triggered Scenarios” sections could be added to better arrange the various options (e.g., “lights off”, etc.). Furthermore, on the home page use the “Summary” heading for the section showing the home or room summary.
  2. Titled regions: organize the content into meaningful regions and give an expressive title to each region. For instance, on the “Home” page consider dividing the content into “menu”, “home map” and “summary”. Thanks to this approach, the screen reader can give the user information on the main sections available on the page and quickly skip from one to another.

Principle 3: Understandable. Information and the operation of user interface must be understandable. Content, links and buttons should be clear and easily detectable by the screen reader.

G7. Clear functions, commands and actions. Design self-explanatory links and buttons (labels) to facilitate understanding the context. For a Home Automation System use self-explanatory and clear labels for the links or buttons to handle

  1. Status labels: for links and buttons used to inform about and change the status, use complete labels composed of the device name and its status, such as “kitchen chandelier off” or “bathroom shutter open” rather than “off”, “on” or “open” placed next to the device name (as occurs in many HAS).
  2. Actions on devices: for links and buttons used to perform actions, use complete labels composed of the device name and its status, such as “lights off”, “heating on” or “close shutters” rather than simply “off”, “on” or “close” placed next to the device name (e.g., “Close).

G8. Focus on main content. Avoid or limit unnecessary content and focus on the main information related to the current context and goal. Do not include too much information on a page; too much information or content can make navigation confusing and tiring.

9 Conclusions

This study investigates the opportunities offered by smart homes for visually-impaired people. After introducing the context and related work, motivation and use cases have been presented to show the users’ needs and the opportunities offered by the technology to overcome barriers and also enable unskilled users to become more familiar and autonomous at home. Since the user interfaces play a crucial role in the accessibility of the Home Automation Systems, herein we have proposed a web prototype with the aim of simplifying the interfaces in order to reduce the content shown, and avoid non-useful elements, choosing a minimal design approach. Advantages of this design are highlighted by Giraud et al. [35]; by avoiding redundant and irrelevant information we can lighten participants’ cognitive load, increasing their performance and satisfaction. In this context, we have shown how to optimize the user interface for this aim and how to simplify the execution of repetitive tasks such as checking the current status or turning off lights or closing shutters. Thus, we especially focused on Web Interface design to propose several solutions and approaches with the aim of addressing many suggestions and preferences expressed by the interviewed people. Finally, we propose a set of guidelines to consider while developing a Home Automation System in terms of User Interfaces. In conclusion, technology can empower users with disability or special needs, provided accessibility and usability are also guaranteed via assistive technology or for those who are not particularly familiar with modern devices and interaction modalities. Further studies should investigate how ICT could be better combined and designed in order to improve user interaction independently from personal abilities and preferences.

References

[1] Buzzi, M., Gennai, F., & Leporini, B. (2017). How Blind People Can Manage a Remote Control System: A Case Study. In International Conference on Smart Objects and Technologies for Social Good, (pp. 71–81). Springer, Cham.

[2] Boldú, M., Paris, P., Térmens i Graells, M., Porras Serrano, M., Ribera, M., & Sulé, A. (2017). Web content accessibility guidelines: from 1.0 to 2.0.

[3] Power, C., Freire, A., Petrie, H., & Swallow, D. (2012, May). Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proc. of the SIGCHI Conference, (pp. 433–442). ACM.

[4] Almeida, L. D., & Baranauskas, M. C. C. (2018). A Roadmap on Awareness of Others in Accessible Collaborative Rich Internet Applications. In Application Development and Design: Concepts, Methodologies, Tools, and Applications, (pp. 479–500). IGI Global.

[5] Carvalho, L. P., Ferreira, L. P., & Freire, A. P. (2016). Accessibility evaluation of rich Internet applications interface components for mobile screen readers. In Proc. of the 31st Annual ACM Symp. on Applied Computing, (pp. 181–186). ACM.

[6] Velasco, C. A., Denev, D., Stegemann, D., & Mohamad, Y. (2008, April). A web compliance engineering framework to support the development of accessible rich internet applications. In Proc. of W4A, (pp. 45–49). ACM.

[7] Stephanidis, C. (Ed.) (2009). The Universal Access Handbook. CRC Press, 2009. pp. 1034.

[8] Buzzi, M., Leporini, B., & Meattini, C. Simple Smart Homes Web Interfaces for Blind People. In Proceedings of WEBIST 2018: 223–230.

[9] Leporini, B., & Buzzi, M. (2018). Home Automation for an Independent Living: Investigating the Needs of Visually Impaired People. In Proceedings of the Internet of Accessible Things, (p. 15). ACM.

[10] Rogers, Y. (2006). Moving on from weiser’s vision of calm computing: Engaging ubicomp experiences. In International Conference on Ubiquitous Computing, (pp. 404–421). Springer, Berlin, Heidelberg.

[11] Alam, M. R., Reaz, M. B. I., & Ali, M. A. M. (2012). A review of smart homes – Past, present, and future. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6), 1190–1203.

[12] Chan, M., Campo, E., Estève, D., & Fourniols, J. Y. (2009). Smart homes – current features and future perspectives. Maturitas, 64(2), 90–97.

[13] Marek, K. D., & Rantz, M. J. (2000). Aging in place: a new model for long-term care. Nursing Administration Quarterly, 24(3), 1–11.

[14] Rodin, J. (1986). Aging and health: Effects of the sense of control. Science, 233(4770), 1271–1276.

[15] Santos V., Bartolomeu P., Fonseca J., & Mota A. (2007, July). B-live-a home automation system for disabled and elderly people. In Industrial Embedded Systems, 2007. SIES’07. Int. Symposium on, (pp. 333–336). IEEE.

[16] Portet, F., Vacher, M., Golanski, C., Roux, C., & Meillon, B. (2013). Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects. Personal and Ubiquitous Computing, 17(1), 127–144.

[17] Park, E., Kim, S., Kim, Y., & Kwon, S. J. (2018). Smart home services as the next mainstream of the ICT industry: determinants of the adoption of smart home services. Universal Access in the Information Society, 17(1), 175–190.

[18] Who. Global data on visual impairment 2010, https://www.who. int/blindness/publications/globaldata/en/, retrieved May 2019.

[19] Swagerty, J. D. (1995). The impact of age-related visual impairment on functional independence in the elderly. Kansas Medicine: the Journal of the Kansas Medical Society, 96(1), 24–26.

[20] Gallagher, B., & Jackson, J. (2012). Ageing and the impact of vision loss on independent living and mobility. Optometry in Practice, 13(2), 45–54.

[21] Espada, J. P., Martínez, O. S., Lovelle, J., G-Bustelo, B. C. P., Álvarez, M.Á., & García, A. G. (2011). Modeling architecture for collaborative virtual objects based on services. Journal of Network and Computer Applications, 34(5), 1634–1647.

[22] Tan, L., & Wang, N. (2010, August). Future internet: The internet of things. In 2010 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE), (Vol. 5, pp. V5–376). IEEE.

[23] Domingo, M. C. (2012). An overview of the Internet of Things for people with disabilities. Journal of Network and Computer Applications, 35(2), 584–596.

[24] Darianian, M., & Michael, M. P. (2008, December). Smart home mobile RFID-based Internet-of-Things systems and services. In 2008 International Conference on Advanced Computer Theory and Engineering, (pp. 116–120). IEEE.

[25] Blasco, R., Marco, Á., Casas, R., Cirujano, D., & Picking, R. (2014). A smart kitchen for ambient assisted living. Sensors, 14(1), 1629–1653.

[26] Gullà, F., Ceccacci, S., Menghi, R., & Germani, M. (2016, June). An adaptive smart system to foster disabled and elderly people in kitchen-related task. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, (p. 27). ACM.

[27] Lanigan, P. E., Paulos, A. M., Williams, A. W., Rossi, D., & Narasimhan, P. (2006, October). Trinetra: Assistive Technologies for Grocery Shopping for the Blind. In ISWC, (pp. 147–148).

[28] de Oliveira, G. A. A., de Bettio, R. W., & Freire, A. P. (2016, October). Accessibility of the smart home for users with visual disabilities: an evaluation of open source mobile applications for home automation. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, (p. 29). ACM.

[29] Mäyrä, F., Soronen, A., Koskinen, I., Kuusela, K., Mikkonen, J., Vanhala, J., & Zakrzewski, M. (2006). Probing a proactive home: Challenges in researching and designing everyday smart environments. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments.

[30] Callejas Z., & Lopez-Cozar R. (2009). Designing smart home interfaces for the elderly. SIGACCESS Newsletter 95.

[31] Liu, L., Stroulia, E., Nikolaidis, I., Miguel-Cruz, A., & Rincon, A. R. (2016). Smart homes and home health monitoring technologies for older adults: A systematic review. Int. Journal of Medical Informatics, 91, 44–59.

[32] Vacher, M., Lecouteux, B., Istrate, D., Joubert, T., Portet, F., Sehili, M., & Chahuara, P. (2013). Experimental evaluation of speech recognition technologies for voice-based home automation control in a smart home. In 4th workshop on Speech and Language Processing for Assistive Technologies, (pp. 99–105).

[33] Brady E., Morris M. R., Zhong Y., White S., & Bigham J. P. (2013). Visual challenges in the everyday lives of blind people. In Proc. of the SIGCHI Conference, (pp. 2117–2126). ACM.

[34] Jeet, V., Dhillon, H.S., & Bhatia, S. (2015,April). Radio frequency home appliance control based on head tracking and voice control for disabled person. In Communication Systems and Network Technologies (CSNT), 2015 Fifth Int. Conf. on, (pp. 559–563). IEEE.

[35] Giraud, S., Thérouanne, P., & Steiner, D. D. (2018). Web accessibility: Filtering redundant and irrelevant information improves website usability for blind users. Int. Journal of Human-Computer Studies, 111, 23–35.

Biographies

images

Marina Buzzi, technologist at the Italian National Research Council (CNR), leads the Web Accessibility and Usability group @ the Institute for Informatics and Telematics (IIT-CNR). She received her M.Sc. degree in Computer Science from the University of Pisa in 1991. Expert in networking design, Internet Services Management, and Web applications, in 2004 she shifted her research focus to accessible and participative design techniques to create usable applications for users with special needs. Since 2010 she has investigated how to exploit IoT and RFID systems to improve people’s lives. She has coordinated technological and research projects, designing accessible games, training and rehabilitation systems for people with cognitive impairment and web apps for teaching children with autism. She is coauthor of more than one hundred papers.

images

Barbara Leporini earned her PhD in Computer Science at University of Pisa in 2003 with a dissertation on accessibility and usability subject related to websites. She is now a researcher in Human-Computer Interaction (HCI) at the CNR in Pisa (ISTI). Barbara investigates techniques and methods to make user interfaces accessible and usable to users with special needs. She is the author of numerous publications and participates in many projects in the field. In addition to research, Barbara has been teaching computer science classes and participating in boards and groups working on accessibility applied to various areas. In particular, she has been collaborating with the working group for the requirements defined by Italian law on accessibility. She is also the director of the Institute for Research, Training and Rehabilitation (I.Ri.Fo.R.) for the visually-impaired of the Region of Tuscany.

images

Clara Meattini earned her B.Sc. and M.Sc. degrees in Digital Humanities from the University of Pisa, with a thesis focusing on accessible publishing and assistive technologies. She has acquired solid experience in smart programming solutions and a thorough knowledge of accessibility standards. Clara is passionate about ICTs and their application in human and social fields. She developed this work during her curricular stage at the ISTI-CNR.

Abstract

1 Introduction

2 Context and Background

3 Method

4 Accessibility of HAS Interfaces

4.1 Inspection Evaluation of HAS Interfaces via Screen Reader

4.2 Users’ Preferences and Expectations

5 Smart Home Design

5.1 Motivating Scenarios

5.1.1 Scenario 1

5.1.2 Scenario 2

5.1.3 Scenario 3

5.2 System Requirements

5.2.1 Web Interfaces

6 The Prototype

6.1 Case 1: Map View

images

6.2 Case 2: View by Status & Devices

images

6.3 Case 3: The Ready Scenarios

images

6.4 Case 4: Settings and Customization

images

6.5 Design Features: Partitioning and Additional Info

7 Evaluation

8 Proposed Design Guidelines

9 Conclusions

References

Biographies