unconventional controllers
Extending the screen into physical space through tangible interfaces, computer vision & machine learning
00
overview
The team’s research objective was to extend the screen into physical space, in order to prototype unconventional forms of analog input through computer vision and machine learning applications. The team explored embodied interactions with analog objects that communicated with the screen space through computer vision. Through an iterative process, the team explored individual code sketches based on the research and ideation phase, the results of which were developed and combined into the final installation.
behind the work
This project pushed coding skills by delving deeper into pose detection and tools like COCO. It wasn't just about making things look cool—it was about digging into interaction principles and understanding how users engage with objects. Working with a small, interdisciplinary team led to a rich exchange of ideas, blending technical knowledge with creative intuition. Beyond the technical explorations, the project sparked bigger questions: What happens when multiple people interact with multiple objects in a shared space? It marked a shift toward a more technical, systems-based approach to human-computer interaction, laying the groundwork for future explorations.
The research investigated collapsing the distance between the digital (screen space) & the physical (space inhabited by the user) and how this might be conducive to increased feelings of presence. The installation aimed to inspire the user to make the connection between their physical bodies with the digital avatar, engendering a sense of surprise and delight. The larger vision involved a geometric installation with everyday objects and multiple affordances, allowing users to create micro-narratives based on their personal associations.
The design aimed to create a space that would not require instructions, instead relying on the affordances of the physical design. It was essential for the installation to feel alive upon user entry, with the screen’s camera displaying the user’s image alongside a grid of moving tennis balls. The analog object and its position were designed to afford interaction, with a tennis ball placed on a lit pedestal to encourage engagement.
The initial prototype focused on an interaction with one object only, while also incorporating a surprise second object to test the logic of the programmatic approach (state machines) and object recognition library. The prototype utilized P5.js in conjunction with ml5.js, the PoseNet machine learning model, and the COCO dataset. The installation concept was segmented into features, with code sketches created for each—object recognition and state machine, digital object using GIFs, and the GIFs’ interactions with the skeleton. These separate code bases were then integrated for testing and debugging.
The user response to the experience was positive, with most engaging intuitively without instructions. The key moment of transformation into a "tennis ball person" was well received, though some users desired more complex system behaviours, such as a digital tennis ball following their analog ball when thrown or bouncing off surfaces. If the project were to be further developed, the next steps would involve formal user testing, additional research, and iterative design refinements.
This project was submitted in partial fulfilment of the Creation and Computation course at OCAD University’s Digital Futures graduate program, by Shipra Balasubramani, Tamika Yamamoto and David Oppenheim. This project’s description and writing are a collaborative effort between all team members and taken from the primary project documentation.
year
2022
timeframe
2 weeks
location
Toronto
analog tools
Research + Prototyping + Installation Design
digital tools
Pose Estimation + Object Detection + CV + COCO + p5.js
category
Computation + Interaction Design
01
Ideation & Exploration sketches along with teammates Tamika Yamamoto and David Oppenheim
02
Through an iterative process the team developed individual sketches that were combined to form the final prototype
03
Layout testing & Components of the installation
04
User flow diagram representing the various stages of interaction by David Oppenheim
05
Installation presented to the Digital Futures cohort as a part of the Creation & Computation course at OCAD University
06
User enters the play space and sees a grid of digital tennis balls on the screen. A physical tennis ball displayed in front of them, a web camera captures and mirrors their movements; As the user picks up the tennis ball, a parallax black background appears, and the digital grid deconstructs to form the user's body
07
As the user moves, the Ball Person on screen mirrors their motion, with a parallax background enhancing the sense of a shared 3D environment between the digital screen and physical playspace; When the user releases the tennis ball—either dropping it to the ground or returning it to the pedestal (tripod)—the Ball Person deconstructs, causing tennis balls to fall on screen. After five seconds, the original ball grid reappears
08
A dynamic installation where users interact with a digital tennis ball grid, their movements mirrored by a "Ball Person" on screen. This video showcases multiple users exploring the play between physical and digital spaces.
