RetroFab: A Design Tool for Retrofitting Physical Interfaces using Actuators, Sensors and 3D Printing

RetroFab is an end-to-end design and fabrication environment that allows non-experts to retrofit physical interfaces. Our approach allows for changing the layout and behavior of physical interfaces. Unlike customizing software interfaces, physical interfaces are often challenging to adapt because of their rigidity. With RetroFab, a new physical interface is designed that serves as a proxy interface for the legacy controls that are now operated by actuators. RetroFab makes this concept of retrofitting devices available to non-experts by automatically generating an enclosure structure from an annotated 3D scan. This enclosure structure holds together actuators, sensors as well as components for the redesigned interface. To allow retrofitting a wide variety of legacy devices, the RetroFab design tool comes with a toolkit of 12 components. We demonstrate the versatility and novel opportunities of our approach by retrofitting five domestic objects and exploring their use cases. Preliminary user feedback reports on the experience of retrofitting devices with RetroFab.

RetroFab goes beyond the traditional IoT vision of interconnecting and monitoring multiple heterogeneous devices, and enables users to change the behavior of devices and rearrange the layout of user interfaces. Novel opportunities include, resolving design flaws in physical interfaces or enabling shortcuts to frequently used or personalized actions.

Scientific Publication:

Raf Ramakers, Fraser Anderson, Tovi Grossman, George Fitzmaurice. RetroFab: A Design Tool for Retrofitting Physical Interfaces using Actuators, Sensors and 3D Printing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16).
Paper (Best Paper Honorable Mention Award at CHI2016)

Share:

Patent:

Raf Ramakers, Fraser Anderson, Tovi Grossman, George Fitzmaurice. RetroFab: A Design Tool for Retrofitting Physical Interfaces using Actuators, Sensors and 3D Printing. (pending)

Press:




The key idea behind this work is to refactor physical interfaces by mounting a redesigned proxy interface over top of the existing form factor that is able to intercept user input and redirect it to the original object using mechanical actuators, while also intercepting device output and redirecting it to the user.

Walkthrough: Refactoring a Toaster

The following walkthrough illustrates the process of retrofitting a legacy interface using RetroFab (Figure 1). For the toaster, the legacy interface is composed of the various buttons (cancel, bagel, defrost, and reheat), a dial for temperature control, a lever to push the toast up and down, and a set of indication LEDs. The following example shows how RetroFab can be used to intercept all interactions and add one extra button on the toaster as a shortcut to a preferred setting – a defrost cycle to thaw the bread, then a mild toasting.

Step 1: 3D Scanning and Annotating Controls
The user starts by 3D scanning the toaster using the Skanect 3D scanning software and a Microsoft Kinect. Before the scan, the buttons and LEDs are highlighted by covering them with tape to ensure their visibility even in low quality 3D scans by novices (Figure 1a). The user loads the 3D model in the RetroFab design tool and annotates the position of the legacy controls and indicators using the brushes in the toolbar on the left. The system contains one brush for every type of supported legacy control and indicator (Figure 1b).

Step 2: Automated Enclosure Design
Once the annotations are finished, RetroFab positions the housings for all actuators and sensors: linear actuators for the pushbuttons and lever, stepper motors for dials and light sensors for LEDs (Figure 3a). RetroFab highlights the region that will be redesigned, and thus covered by the new physical enclosure. The user extends this region to include the area where the new shortcut button will be positioned (Figure 3b). RetroFab responds by generating the 3D model of the enclosure (Figure 1c). Finally, the user specifies the preferred location of the mounting brackets (using a brush) that attaches the enclosure structure to the toaster (Figure 4).

Step 3: Redesigning the Interface and Behavior
RetroFab automatically integrates a retrofit interface in the front panel of the enclosure structure that serves as a proxy for the legacy interface. If desired, the user can redesign this front panel by repositioning the retrofit components or by replacing them with alternative components available in the toolbar. Figure 1d shows an additional pushbutton being added to the toaster that will serve as a shortcut to the user’s favorite toast setting.

By default, retrofit controls are configured to mirror the actions of their associated legacy controls: pushing the new retrofit defrost button on the enclosure structure will cause the legacy defrost button on the toaster to be pushed. Similarly, output is redirected from the toaster to the enclosure structure: when the legacy LED representing the defrost state on the toaster turns on, the corresponding retrofit LED on the enclosure lights up.

The user can alter this default behavior or add extra logic for new components using a Programming by Demonstration paradigm. Users demonstrate actions directly on top of the 3D models of the respective retrofit components and can also record functional relationships between these actions. Figure 5 shows the user specifying the logic of the new button on the toaster: If pressed, the lever goes down, and the defrost button is pushed. Additionally, the user can specify that when defrosting is finished (sensed by the LED next to the defrost button), the toast goes back in for a second time at a higher temperature, resulting in perfectly crisp toast!

Step 4: Fabrication and Assembly
When the design is complete, RetroFab generates: (1) STL files to be printed out using a 3D printer (FDM printer for our prototypes), (2) Microcontroller code that can be directly uploaded to an Arduino platform, and (3) Assembly instructions to guide users in connecting the actuators and sensors to the microcontroller.

Figure 1e shows the assembled 3D printed enclosure and its embedded actuators, sensors, and retrofit components. Finally, the enclosure structure is attached to the toaster by gluing the feet of the mounting brackets to the appliance (Figure 1f). The mounting bracket is designed so that the enclosure structure is easy to detach using screws, leaving only the feet of the mounting brackets behind (Figure 1e). Once the enclosure structure is attached, RetroFab guides the user through a process to calibrate all the actuators and sensors.

Step 5: Deployment
All logic defined using RetroFab runs on a central PC which communicates continuously to the retrofit interface. This makes it possible to reconfigure the behavior of devices at run time and allow for interconnectivity. Figure 6 shows the user linking the “snooze” button of his retrofitted alarm clock to his personal “perfect toast” shortcut button on the toaster.

When the user launches the companion RetroFab mobile application, all retrofitted devices are automatically loaded, and display their functionalities to allow for remote control (Figure 7).

RetroFab Toolkit

To enable retrofitting a wide variety of devices, RetroFab comes with a set of electrical and mechanical primitives to retrofit common physical interface components, such as pushbuttons, rotary dials, rocker and wall switches, and LEDs. Figure 8 shows the full RetroFab toolkit, consisting of (a) actuators, (b) sensors, (c) controls, and (d) indicators. The actuators and sensors are positioned inside the enclosure and concealed, while the controls and indicators are positioned on the outside of the surface, forming the retrofit interface. For every component in the toolkit, the RetroFab design tool has a specific component housing that is integrated in the enclosure structure to facilitate assembly and ensure precise positioning. This is particularly important for the actuators, which operate the underlying legacy controls.

To make it possible for users without electronics knowledge to use the RetroFab toolkit, wires have a color coding scheme and integrate the necessary electronic components, such as resistors, in them. Our toolkit is easiest to deploy using the Adafruit Motor Shield , which avoids complex H-bridge electronic constructions. As such, components are connected directly to the microcontroller by following instructions provided in the RetroFab design tool, avoiding the need for complex electronic wiring designs on breadboards.

Other Example Designs and Use Cases:

Using the RetroFab design tool, 5 legacy interfaces were retrofitted:

(a) A wall switch, exposing a rocker switch on the retrofitted interface: Every retrofit interface created by the user is available through the RetroFab mobile application. This makes it possible to control devices and appliances remotely, such as the light switch when one forgets to turn off the lights. A retrofit interface can also serve as a remote for another retrofit interface. Turning the lamp off when going to bed, turns off the lighting in the room as well, using the retrofitted wall switches.

(b) A lamp, converting a legacy rocker switch into a push button: People with disabilities are often unable to operate controls that are found on most devices, as they require considerable amounts of force or fall outside the range of motion they are capable of. This retrofitted desk lamp illustrates how a rocker switch can be converted to a lower force control, such as a push button. A similar push button is used in the retrofit interface of the toaster to replace the heavy mechanical lever.

(c) A toaster: As highlighted in the walkthrough, the retrofit interface for the toaster can integrate a personalized button for automatically toasting bread to one’s favorite toast settings.

(d) An oven: RetroFab also facilitates the process of resolving poor design decisions found in physical interfaces. Controlling the dials located on the back panel of the stove requires moving one’s arm over a number of elements, which could have pots or frying pans on them. RetroFab allows for repositioning these controls to a more convenient or safe location, such as the side panel of the stove.

(e) An alarm clock: Setting the time and alarm on an ordinary alarm clock is often tiring. By retrofitting the interface using RetroFab, a shortcut can be designed for automatically setting the current time after the lock is unplugged or a power outage occurs.