Design and semantics of form and movement

Lin-Lin Chen, Tom Djajadiningrat, Loe Feijs, Simon Fraser, Steven Kyffin, Dagmar Steffen
Design and semantics
of form and movement
DeSForM 2012: MEANING.MATTER.MAKING
Principal Sponsor
Principal Sponsor
Sponsors
Sponsors
3D Printing Systems
Industry Partners
Industry Partners
3D Printing Systems
www.objet.com
www.objet.com
Affiliated Affiliated
Research Organisations
Research Organisations
DeSForM DeSForM
Academic Affi
Academic
liations Affiliations
Cover: ‘Hylozoic
Cover:
Ground’
‘Hylozoic
VeniceGround’
2010 © Venice
PBA 2010 © PBA
Page 9: Port Hills,
PageChristchurch,
9: Port Hills,December
Christchurch,
2009.
December
Image: Matt
2009.
Paterson
Image: Matt Paterson
RMIT University
RMIT University
Document: School logo JPG version.doc
Author: Winnie Ha
Save Date: 17/03/2012
Page 1 of 1
Document: School logo JPG version.doc
Author: Winnie Ha
Save Date: 17/03/2012
Page 1 of 1
Design and semantics
of form and movement
DeSForM 2012: MEANING.MATTER.MAKING
DeSForM 2012
Programme Committee
Lin-Lin Chen, National Taiwan University of Science and Technology, Taiwan
Tom Djajadiningrat, Philips Design, Eindhoven, The Netherlands
Loe Feijs, Technical University Eindhoven, The Netherlands
Simon Fraser, Victoria University of Wellington, NZ
Steven Kyffin, Northumbria University, Newcastle upon Tyne, UK
Dagmar Steffen, Lucerne School of Art and Design, Switzerland
Organising Committee
School of Design, Victoria University of Wellington, NZ
Co-Convenors: Simon Fraser, Margaret Maile Petty
Academic Reviews: Jules Moloney
Editor Proceedings: Edgar Rodríguez Ramírez
Proofreader: Beth Thomas
Website Co-ordinator: Kah Chan
Promotion/Media Relations: Margaret Maile Petty
Media Consultant: Helen Baxter, Mohawk Media
Professional Programme: Doug Easterly, Leon Gurevitch
DeSForM2012 Exhibition: Bettina Neu, Tim Miller
Demonstrations Co-ordinator: Kah Chan
Hylozoic Series: Vesica/City Gallery Wellington
Coordinators Hylozoic Series: Maxe Fisher, Bernard Guy
Design Led Futures/Wellington2040: Ross Stevens
Conference Administrator: Eli Feth
Funding/Sponsors: Simon Fraser
Technical Coordinator: Mark Shaw
Thanks to: All academic, technical and administrative staff
who have contributed to this event in so many different ways,
the Victoria Foundation and Victoria University of Wellington.
2
Design and semantics of form and movement
Programme DeSForM 2012
Wednesday, April 18
11.00 am – 12.00 pm
12.00 pm – 12.15 pm
12.15 pm – 12.30 pm
Registration and Lunch
Welcoming Ceremony
Opening Address
Prof. Simon Fraser, Head, School of Design, Victoria University of Wellington
Prof. Steven Kyffin, Dean, School of Design, Northumbria University, UK
01.40 pm – 02.00 pm
02.00 pm – 02.10 pm
Paper Presentation I: Immersive Environments:
Smart Systems and Interactive Spaces
Semantic Connections: A New Interaction Paradigm by Bram van der Vlist,
Gerrit Niezen, Jun Hu, Loe Feijs
Table Manners: The Influence of Context on Gestural Meaning
by Tom Djajadiningrat, Luc Geurts, Jeanne De Bont
Discussion
Persuasive Design for Energy Saving Behavior through Social Gaming
by Vaijayanthi Iyengar and Madhusudhan Marur
Four Installations Inviting Playful Interaction by Daniel Cermak-Sassenrath
Discussion
02.10 pm – 02.40 pm
Afternoon Tea
12.30 pm – 12.50 pm
12.50 pm – 01.10 pm
01.10 pm – 01.20 pm
01.20 pm – 01.40 pm
03.50 pm – 04.10 pm
04.10 pm – 04.20 pm
Paper Presentation II: The Emotive
Fluenci: The Expression of Expressing
by Jaap Knoester, Tom Djajadiningrat, Philip Ross
Fictional Emotions within Emotion Driven Design by Eva Knutz
Discussion
Co-Authored Narrative Experience: Affective, Embodied Interaction through
Combining the Diachronic with the Synchronistic by Carol MacGillivray
The Semantics of Surprise in Industrial Design by Edgar Rodríguez Ramírez
Discussion
05.30 pm
06.30 pm
06.40 pm
07.40 pm
Exhibition and Keynote Address
City Gallery Wellington, drinks and preview, Hylozoic series:Vesica
City Gallery Opening address: Mayor Celia Wade-Brown
City Gallery Keynote Address: Philip Beesley
City Gallery Design Led Futures/Wellington 2040: Ross Stevens
02.40 pm – 03.00 pm
03.00 pm – 03.20 pm
03.20 pm – 03.30 pm
03.30 pm – 03.50 pm
–
–
–
–
06.30 pm
06.40 pm
07.40 pm
08.00 pm
Design and semantics of form and movement
3
Programmme DeSForM 2012
Thursday, April 19
08.30am – 09.00am
10.50 am – 11.00 am
Paper Presentation III: Metaphors, Agency and Semantics
The Role of Designer Expertise in Source Selection During Product
Metaphor Generation by Nazli Cila, Paul Hekkert, Valentijn Visch
The Actions of Things: Design, Materiality and Agency of Things
by Jorn Guldberg
Discussion
Embracing Relational Agency in Design Process by A. Baki Kocaballi,
Petra Gemeinboeck, Rob Saunders, Lian Loke, Andy Dong
Beyond Metaphor in Product Use and Interaction
by Thomas Markussen, Elif Özcan, Nazli Cila
Graphic Design and the Tyranny of Connoisseurship: An Argument
for a Semiotic Approach to Graphic Design Pedagogy by Alan Young
Discussion
11.00 am – 11.20 am
Morning Tea
09.00 am – 09.20 am
09.20 am – 09.40 am
09.40 am – 09.50 am
09.50 am – 10.10 am
10.10 am – 10.30 am
10.30 am – 10.50 am
12.50 pm – 01.00 pm
Paper Presentation IV: Immaterial Forces: Light, Sound, Gesture
Grace: A Gesture Controlled Wake-Up Light by Tom Djajadiningrat,
Luc Geurts, Jeanne De Bont, Pei-Yin Chao
The Aesthetics of Immateriality in Design by Mads Folkmann
Discussion
In Pursuit of the Extraordinary by Cassie Hester
Tradition and Innovation: A Study on Reconfiguration of Product Language
Through Innovative Approaches by Ozge Merzali Celikoglu
Discussion
01.00 pm – 01.40 pm
Lunch
11.20 am – 11.40 am
11.40 am
12.00 pm
12.10 pm
12.30 pm
–
–
–
–
12.00 pm
12.10 pm
12.30 pm
12.50 pm
03.10 pm – 03.20 pm
Paper Presentation V: Gestures and Touch
Controlling Smart Home Environments with Semantic Connections
by Bram van der Vlist, Gerrit Niezen, Stefan Rapp, Jun Hu, Loe Feijs
An aesthetics of touch: Investigating the language of design relating to form
by Victoria Teinaki, Bruce Montgomery, Nicholas Spencer, Gilbert Cockton
Discussion
Learn to Make, Make to Learn: Reflections from Sketching Haptics Workshops
by Camille Moussette
A Study on a Tangible Interaction Approach to Managing Wireless Connections
in a Smart Home Environment by Jeroen Peeters, Bram van der Vlist, Gerrit
Niezen, Jun Hu, Loe Feijs
Discussion
03.20 pm – 03.40 pm
Afternoon Tea
01.40 pm – 2.00 pm
02.00 pm – 02.20 pm
02.20 pm – 02.30 pm
02.30 pm – 02.50 pm
02.50 pm – 03.10 pm
4
Coffee and Refreshments
Design and semantics of form and movement
03.40 pm – 04.00 pm
04.00 pm – 04.20 pm
04.20 pm – 04.30 pm
04.30 pm – 04.50 pm
04.50 pm – 05.10 pm
05.10 pm – 05.20 pm
05.20 pm – 06.00 pm
06.00 pm – 07.00 pm
07.00 pm – 09.00 pm
Paper Presentation VI: Fabricating Futures: Digital Manipulation,
Fabrication and the Craft of Design
Yeti: Designing Geometric Tools with Interactive Programming
by Daniel Davis, Jane Burry, Mark Burry
Towards a Responsive Architectural Morphing Skin by Chin Koi Khoo
Discussion
Digital-Physical Hybrid Design: Harmonizing the Real World and the Virtual
World by Mizuki Sakamoto, Tatsuo Nakajima, Todorka Alexandrova
Digital Craft in Digital Space: A Paradigm Shift in the Making by Maxe Fisher,
Simon Fraser, Tim Miller, Ross Stevens, Jerad Tinnin, Annelies Zwaan
Discussion
Reception Keynote Address
Neil Leach, Keynote Address
Optional visit: late night opening, Museum of New Zealand
Te Papa Tongarewa
Design and semantics of form and movement
5
Programme DeSForM 2012
Friday, April 20: Professional Presentations
08:30 am – 09:00 am
10:00 am – 10:20 am
10:20 am – 10:40 am
10:40 am – 10:50 am
Professional Presentations
Opening Address
Session I: Interaction Design
Unlimited Realities - http://www.unlimitedrealities.com/
Click Suite - http://www.clicksuite.co.nz
Discussion
Session II: Transmedia, Technology and Entertainment
Weta Digital - http://www.wetafx.co.nz
Pik Pok - http://pikpok.com/
Discussion
10:50 am – 11:20 am
Morning Tea
09:00 am – 09:10 am
09:10 am – 09:30 am
09:30 am – 09:50 am
09:50 am – 10:00 am
11:40 am – 12:00 pm
12:00 pm – 12:20 pm
12:20 pm – 12:30 pm
Session III: Immersive Environments
Spatial Information Architecture Lab (SIAL) RMIT Melbourne, Australia http://www.sial.rmit.edu.au/
HIT Lab NZ, Canterbury University, NZ - http://www.hitlabnz.org/
The Gibson Group - http://www.gibson.co.nz/
Discussion
12:30 pm – 01:30 pm
Lunch
11:20 am – 11:40 am
01:30 pm – 01:50 pm
01:50 pm – 02:10 pm
02:10 pm – 02:20 pm
02:20 pm
02:40 pm
03:00 pm
03:20 pm
6
Coffee and Refreshments
–
–
–
–
02:40 pm
03:00 pm
03:20 pm
03:30 pm
Session IV: Culture Jamming
RESN - www.resn.co.nz/
SWAMP (USA) - http://www.swamp.nu/
Discussion
Session V: Fabricating Futures
diatom - http://diatom.cc/
77Pieces - http://77-pieces.com/
Ponoko - http://www.ponoko.com/
Discussion
03:30 pm – 04:00 pm
Afternoon Tea
04:00 pm – 04:40 pm
04:45 pm – 05:25 pm
05:30 pm – 05:40 pm
Panel Discussion I Blurring the Physical and the Digital
Chair: Prof. Steven Kyffin, Dean, School of Design, Northumbria University
Panel Discussion II Designing True Interactions; Chair: Helen Baxter, Mohawk Media
Closing Address
Prof. Simon Fraser, Head, School of Design, Victoria University of Wellington
Prof. Steven Kyffin, Dean, School of Design, Northumbria University, UK
06:30 pm – 09:00 pm
Conference Dinner The Waka House Te Raukura Te Wharewaka o Poneke
Design and semantics of form and movement
Design and semantics of form and movement
7
Foreword
DeSForM 2012
It is encouraging to witness the growing global influence of the DeSForM conference and
workshop series, and indeed to be an active part of this growth. Following on from the
2009 conference in Taipei, Taiwan and the 2010 conference in Lucerne, Switzerland, the
School of Design at Victoria University of Wellington, New Zealand is honoured to bring
DeSForM’s sphere of influence to the Southern Hemisphere for the first time.
DeSForM 2012: MEANING.MATTER.MAKING continues the tradition of diversity
and exchange. In reaching out to the many innovative schools, businesses, designers,
researchers and cultural organisations in the Southern Hemisphere and beyond, DeSForM
2012 will expand the organisation’s community, offering new horizons, insights and audiences.
The relative isolation of New Zealand confers real advantages. Being at the edge brings
freedom to improvise, to invent and to imagine. The School and its location in New
Zealand’s ‘Creative Capital’ embodies many of the core principles of DeSForM as a
pioneering forum for the exchange of knowledge and ideas, unburdened by tradition and
accepted ways of doing things. Our South Pacific location has inspired a commitment to
digital technologies, which offer real opportunities to overcome distance with new forms of
communication, interaction and enhanced experiences. As a result the city has a colourful
and growing creative community of technology developers, software specialists, game and
interface designers, film makers and special effects researchers, usability experts, social
networking service providers and innovators in digital design, fabrication and distribution.
Against this backdrop, DeSForM 2012: MEANING.MATTER.MAKING is pleased to
welcome a diversity of practice-led researchers and research-led practitioners from
academia and industry, as well as representatives from the many disciplines, perspectives
and interpretations integral to the design and semantics of form and movement.
Design semantics – or the making of meaning through ‘matter’ – is of increasing relevance
to the practice of design in the twenty-first century. Rapidly evolving techniques and
processes are creating pathways for new modes of interactivity, expression, and experience.
These modalities offer expanded opportunities and contexts for design to communicate
meaning through form, movement and experience, extending the parameters of semantic
interpretation. Swift, on-going shifts in the ‘vocabulary’ of design suggest that it is time to
reevaluate our understanding of ‘semantics’. DeSForM2012 poses new questions for present
and future practice: What new engagements are being created and what boundaries are
being crossed or blurred in the digital age? How is digital design and manufacturing changing
relationships between designers, producers, and users? What role does the ‘material’ play
in an increasingly ‘immaterial’ culture of design? Can we maintain notions of creator and
ownership when all objects are digital and effortlessly duplicated? How will the products of
design be valued when the engines of creation become both powerful and commonplace?
With such questions at the fore, DeSForM 2012: MEANING.MATTER.MAKING explores
8
Design and semantics of form and movement
the design of physical and digital things, systems and environments, seeking novel
approaches to how meaning is both created and conveyed in the twenty-first century.
Emphasising future-oriented design, MEANING.MATTER.MAKING highlights a broad
range of scales and approaches, from the singular to the multiple, the hand to the machine,
the conceptual to the self-organising, and the sensory to the theoretical. DeSForM 2012:
MEANING.MATTER.MAKING brings together researchers, designers and industry
partners to share ideas, methods, and theories about the creation and communication
of meaning through design semantics.
We are honoured to have two world renowned keynote speakers joining us for the
conference:
Philip Beesley is Professor in the School of Architecture, University of Waterloo.
His work is widely cited as a pioneer in the rapidly expanding technology of responsive
architecture and spans a diverse range of expressive media, from that of architecture to
sculpture and installation.
Neil Leach, architect, theorist and Professor at the University of Southern California,
School of Architecture, has contributed significantly to the theorization of digital design,
with numerous exhibitions, edited volumes and monographs broadly addressing and
merging critical theory and digital design.
Another first, we are delighted to have been able to bring the premier exhibition of work
by Philip Beesley to New Zealand for DeSForM 2012. Beesley’s project, Hylozoic Series:
Vesica, references the tradition in medieval painting of circumscribing holy figures in a
luminous aura, forming a kind of liminal zone between the body and the outside world.
The exhibition will be held in one of the city's key cultural institutions, The City Gallery
Wellington, te whare toi, and will be shown in the prestigious Hirschfeld Gallery.
The installation features interactive kinetic systems composed of dense arrays of microprocessors, sensors and actuator systems arranged within lightweight ‘textile’ structures
that explore emerging new technologies with digitally-fabricated components.The work
conveys a subtle quality of hovering, organic movement and with gently vibrating fields
of skeletal elements captures the otherwise imperceptible air currents produced by
participant viewers. The integration of chemistry and biology within the composition of
the sculptural body is an expression of Beesley’s evolving concept of ‘living architecture’.
This work integrates shared values of human caring, environmental and urban sustainability
with immersive creativity and experimentation. These values are framed within a philosophy
that sees humanity as participant in the complex negotiations between nature, culture
and technology that increase in complexity as we look into the future. In this last respect,
the exhibition aligns strongly with the City of Wellington's 2040 initiative that addresses
fundamental questions of future living and city making in the 21st century.
We would like to thank our principle sponsor, the Ministry of Science and Innovation
for supporting this event and for their enthusiastic facilitation of research links between
academia and industry in the digital creative sector. This support has been reciprocated in
turn by our industry partners whose generous collaboration in the form of presentations
and demonstrations adds significantly to the scope and impact of the conference.
Design and semantics of form and movement
9
We are also very grateful for the support of the Wellington City Council, not only as a
sponsor but also for recognising the need to develop productive and innovative knowledge
industries to support the vision of Wellington as a knowledge economy and New Zealand’s
creative capital; a vision that is shared and supported by Victoria University of Wellington.
We hereby offer the 7th DeSForM Proceedings. We would like to thank all the authors
who have submitted their work to DeSForM 2012:Meaning.Matter.Making as well as the
reviewers for their constructive and critical comments. We hope the presentations and
demonstrations will raise many questions and inspire lively discussion and debate about
the increasing sophistication and complexity of design research, theory and practice in
a constantly changing global context.
Wellington, April 18 2010
Programme Committee
Lin-Lin Chen, National Taiwan University of Science and Technology, Taiwan
Tom Djajadiningrat, Philips Design, Eindhoven, the Netherlands
Loe Feijs, Technical University Eindhoven, the Netherlands
Simon Fraser, Victoria University of Wellington, NZ
Steven Kyffin, Northumbria University, Newcastle upon Tyne, UK
Dagmar Steffen, Lucerne School of Art and Design, Switzerland
10
Design and semantics of form and movement
Introduction
Interaction requires the involvement of two entities. In a world in which the development of
computational capacity, digital manufacture and ever advancing sensing technologies allows
for ever more complex digital and physical interactions, people and objects cannot be seen
as passive ‘users’ or static designs but as truly immersed participants and even co-creators.
Design, auspiciously living in between technology and people, has the opportunity and
responsibility to answer fundamental questions about co-designed, content rich, connected
and intelligent objects within adaptive systems. It is thrilling to see the response from designers,
researchers, artists, musicians and architects to these questions from the starting point of
the semantics of form and movement. The studies presented in these proceedings reflect a
gamut of design knowledge that embraces rigour in professional practice and beauty in research.
Immersive environments within adaptive systems and interactive devices are studied
through the design and analysis of several haptic and gesture controlled environments
and objects. We learn that while gesture control can be regarded as an exciting, beautiful
and useful feature in interactive devices, it can also be considered as intrusive through the
connotations it brings and even violate personal space. The appropriateness of gestures is
strongly dependent on socio-cultural contexts, which authors urge designers to evaluate.
Fortunately, authors also suggest ways in which designers can use theories from Gestalt
psychology to help people create accurate representations of tangible interactions.
Several papers make a strong case for the importance of digital manipulation, fabrication
and the craft of design. A ‘soft’ approach to responsive architecture suggests the use
of lightweight form-changing materials for architectural morphing skins. Interactive
programming is suggested as a way to design geometric tools. A documentation of industrial
design projects describes how digital technologies offer designers the opportunity to
experiment with new ways of expressing and manufacturing while reviving and expanding
traditional notions of craftsmanship.
Human emotions offer a vast area for investigation, which authors address through suggesting
co-authored narrative experiences in which audiences to affective and embodied interactions
move from being users to being co-creators. Papers also suggest strategies experienced
designers use to elicit surprise in people. A study on the design of a breastfeeding pump that
mimics many of the stimulating triggers provided by a baby and allows for a natural, intimate
and emotional experience suggests concepts such as ‘anthropomorphic form’. This invites
people to engage with a product as if it were human, without what some people would
consider potentially disturbing anthropomorphic product appearance.
Papers also undertake the essential study of metaphors in semantics and suggest revealing
differences on how novice and experienced designers select sources of metaphors. The
concept of ‘semantic connections’ is suggested as an approach to overcome the move from
device-oriented paradigms to more task-oriented paradigms.
The topics, questions and findings addressed in these proceedings show state-of-the-art
research that is just scratching the surface of a rapidly evolving area. It can only make us
excited about what we will experience in our future and in subsequent DesForm workshops.
Edgar Rodríguez Ramírez, Editor, Proceedings, Victoria University of Wellington, NZ
Design and semantics of form and movement
11
Keynote speaker - Philip Beesley
Diffusive form and
near-living environments
Abstract
Philip Beesley's Hylozoic Series explores how synthetic
environments might offer near-living qualities. New installations
in Wellington, Sydney, Madrid and Trondheim demonstrate
the technical progress and design methods currently being
developed at his studio which further pursue integrated
protocell chemistry, distributed kinetics and acoustics, and
evolutionary control systems. The presentation will trace the
origins of Beesley's Hylozoic Series, from archeological sites
beneath the Palatine Hill in Rome to land based geotextile
installations in the woods of Maine and coastal Nova Scotia.
Links will be drawn between these works, conceived as
regenerative ground coverings, and the more recent focus
on suspended, interactive environments. Illustrated projects
will include Beesley's entry into the 12th Venice Biennale in
Architecture, Hylozoic Ground (2010), the light-based Aurora
Series (2010), and the Epithelium Series (2008-2009) which
included air-muscle powered suspended meshworks populated
with densely massed whisker-like mechanisms.
A detailed tour through of this work will discuss design
methodology in relation to the components and systems that
constitute the anatomy of this expanded physiology. These
hovering filter environments, composed of tiny laser cut acrylic
elements, create diffusive boundaries between occupants
and the surrounding milieu. The structures prefer deeply
reticulated skins, turning away from the minimum surface
exposures of reductive crystal forms as they seek to maximize
interchange with the atmosphere and other occupants.
The discussion will also position this body of work within cultural
and theoretical discourse, drawing connections to Roman and
Greek atomists and Romantic theory. Building upon French art
historian George Didi Huberman’s sensitive analysis of marginal
spaces seen within mediaeval paintings, Beesley will discuss
how the behaviours of his installations challenge Humanist
conceptions of environment and expansion of human domain.
The presentation will argue that diffusive, deeply reticulated
forms and unapologetically sentimental empathetic responses
can serve as effective design models for renewed public
architecture.
12
Design and semantics of form and movement
Philip Beesley is a Professor in the School of
Architecture at the University of Waterloo.
A practitioner of architecture and digital media art,
he was educated in visual art at Queen’s
University, in technology at Humber College, and
in architecture at the University of Toronto. At
Waterloo he serves as Director for the Integrated
Group for Visualization, Design and Manufacturing,
and as Director for Riverside Architectural
Press. He also holds the position of Examiner at
University College London. His Toronto-based
practice PBAI is an interdisciplinary design firm that
combines public buildings with exhibition design,
stage and lighting projects. The studio’s methods
incorporate industrial design, digital prototyping,
and mechatronics engineering.
Philip Beesley’s work is widely cited in the rapidly
expanding technology of responsive architecture.
He has authored and edited eight books and
appeared on the cover of Artificial Life (MIT),
LEONARDO and AD journals. Features include
national CBC news, Casa Vogue, WIRED, and
a series of TED talks. Distinctions include Prix
de Rome in Architecture (Canada), VIDA 11.0,
FEIDAD, Katerva finalist, and the Canada pavilion
at the 2010 Venice Biennale for Architecture.
Beesley’s funding includes core CFI, SSHRC,
NSERC and Canada Council for the Arts grants.
Keynote speaker - Neil Leach
Desiring machines
Abstract
In his 1968 classic science fiction movie, 2001: A Space Odyssey,
Stanley Kubrick depicts a maverick computer attempting to take
charge of a mission to Jupiter, and oust the human occupants
from their spaceship. The 1960s were a time of great technological advances, and within a year of the launch of Kubrick’s
movie the US had successfully sent the Apollo 11 crew to
the Moon. Yet such was the suspicion of technology – and
computation in particular – that Kubrick’s apparent technophobia
was no isolated incident. Indeed for several decades afterwards
many schools of architecture were dominated by a phenomenological outlook, which saw technology as symptomatic of our
alienated condition in the world today, and the computer as antithetical to human creativity. Indeed in some schools computers
were even banned from the design studio.
With the advent of the new millennium, however, the full impact
of technology on our lives has become only too evident. We now
live within a hyper-technological environment with computer
systems so advanced that even our individual mobile phones have
more computational power than the Apollo 11 mission to the
Moon. Our homes have become veritable ‘machines for living in’.
It is time, perhaps, to formulate a more sympathetic theoretical
approach towards technology.
This paper argues for a new theoretical approach towards
technology based on the materialist philosophies of Gilles
Deleuze and Manuel DeLanda, looking in particular at Deleuze’s
concepts of ‘machinic processes’ and ‘desiring machines’ –
concepts that move beyond the earlier distinction between
the mechanical and the organic, and that recognize the
human potential to absorb the new and the unfamiliar, and to
appropriate technology as a prosthesis to human operations.
The paper then goes on to look at design itself, and considers
various robotic fabrication technologies that may be understood
within this logic of ‘machinic processes’, especially 3-D printing.
The paper argues that while items within the home – chairs,
tables, shoes and even clothes – are being printed these days,
technologies for printing the home itself – architectural 3-D
printing – are less advanced. The paper considers three rival
attempts to develop the world’s first commercially available 3-D
concrete printer, evaluating the merits of each. It then goes on to
explore the race to print buildings not just on earth, but also on
the Moon, focusing in particular on the NASA sponsored project
to adapt the robotic concrete fabrication technique, Contour
Crafting, for printing structures out of regolith on the Moon.
Architect and theorist Neil Leach is currently
Professor at the University of Southern California,
School of Architecture. He previously has taught
at the University of Brighton, University of Bath,
Architectural Association School of Architecture,
University of Nottingham, Columbia University,
Cornell University, SCI-Arc, Royal Danish Academy
of Art, Dessau Institute of Architecture, and
Institute for Advanced Architecture of Catalonia.
Since 2004 Leach has co-curated exhibitions at
the Architecture Biennial Beijing with Xu WeiGuo, including Fast Forward>> (2004), Emerging
Talents, Emerging Technologies (2006), (Im)
material Processes: New Digital Techniques for
Architecture (2008), and Machinic Processes
(2010). With Roland Snooks he also co-curated
Swarm Intelligence: Architectures of Multi-Agent
Systems (2010) in Shanghai.
Leach has contributed significantly to the
theorization of digital design, with numerous
edited volumes and monographs broadly
addressing and merging critical theory and digital
design. These include Rethinking Architecture:
A Reader in Cultural Theory (Routledge, 1997);
The Anaesthetics of Architecture (MIT Press,
1999); Millennium Culture (Ellipsis, 1999);
Architecture and Revolution: Contemporary
Perspectives on Central and Eastern Europe
(Routledge, 1999); The Hieroglyphics of Space:
Reading and Experiencing the Modern Metropolis
(Routledge, 2002); Designing for a Digital World
(Wiley, 2002); China (Map Office, 2004); Digital
Tectonics (Wiley, 2004); Camouflage (MIT Press,
2006); Digital Cities (Wiley, 2009); Machinic
Processes (CABP, 2010); Scripting the Future
(Tongji, forthcoming); Fabricating the Future
(Tongji, forthcoming).
With Philip Yuan he also co-curated
DigitalFUTURE (2011) in Shanghai.
He is currently working on a research project
sponsored by NASA to design robots for
fabricating structures on the Moon.
Design and semantics of form and movement
13
Contents
14
16
Semantic connections: A new interaction paradigm for smart environments
Bram van der Vlist, Gerrit Niezen, Jun Hu, Loe Feijs
27
Table manners: The influence of context on gestural meaning
Tom Djajadiningrat, Luc Geurts, Jeanne de Bont
33
Persuasive design for energy saving behavior through social gaming
Iyengar Vaijayanthi, Madhusudhan Marur
42
Four installations inviting playful interaction
Daniel Cermak-Sassenrath
46
Fluenci: The expression of expressing
Jaap Knoester, Tom Djajadiningrat, Philip Ross
52
Fictional emotions within emotion driven design
Eva Knutz
62
Co-authored narrative experience: Affective, embodied interaction through
combining the diachronic with the synchronistic
Carol MacGillivray, Bruno Mathez
70
Semantics of surprise: Strategies for eliciting surprise through design
Edgar Rodríguez Ramírez
79
The role of designer expertise in source selection during product metaphor generation
Nazli Cila, Paul Hekkert, Valentijn Visch
88
Design and iconicity: Design, materiality, and the meaning of things
Jorn Guldberg
99
Embracing relational agency in design process
Baki Kocaballi, Petra Gemeinboeck, Rob Saunders, Lian Loke, Andy Dong
110
Beyond metaphor in product use and interaction
Thomas Markussen, Elif Özcan, Nazli Cila
120
Graphic design and the tyranny of connoisseurship: An argument for a semiotic
approach to graphic design pedagogy
Alan Young
Design and semantics of form and movement
130
Grace: A gesture-controlled wake-up light
Tom Djajadiningrat, Luc Geurts, Jeanne de Bont, Pei-Yin Chao
137
The aesthetics of immateriality in design: Smartphones as digital design artifacts
Mads Folkmann
148
Elevating the everyday: Designed objects as companions
Cassie Hester
150
Tradition and innovation: A study on reconfiguration of product language
through innovative approaches
Ozge Merzali Celikoglu
160
Controlling smart home environments with semantic connections:
A tangible and an AR approach
Bram van der Vlist, Gerrit Niezen, Stefan Rapp, Jun Hu, Loe Feijs
170
An aesthetics of touch: Investigating the language of design relating to form
Vicktoria Teinaki, Bruce Montgomery, Nicholas Spencer, Gilbert Cockton
180
Learn to make, make to learn: Reflections from sketching haptics workshops
Camille Moussette
187
A study on a tangible interaction approach to managing wireless connections
in a smart home environment
Jeroen Peeters, Bram van der Vlist, Gerrit Niezen, Jun Hu, Loe Feijs
196
Yeti: Designing geometric tools with interactive programming
Daniel Davis, Jane Burry, Mark Burry
203
Towards a responsive architectural morphing skin
Chin Koi Khoo
211
Digital-physical hybrid design: harmonizing the real world and the virtual world
Mizuki Sakamoto, Tatsuo Nakajima, Todorka Alexandrova
223
Digital craft in digital space: A paradigm shift in the making
Maxe Fisher, Simon Fraser, Tim Miller, Ross Stevens, Jerad Tinnin, Annelies Zwaan
Design and semantics of form and movement
15
Bram van der Vlist, Gerrit Niezen, Jun Hu, Loe Feijs
Department of Industrial Design, Eindhoven University of Technology, Eindhoven, Netherlands
{b.j.j.v.d.vlist, g.niezen, j.hu, l.m.g.feijs}@tue.nl
Semantic connections: A new interaction
paradigm for smart environments
Abstract
Keywords
As the environments we inhabit contain a growing
number of networked, interactive products, both
users and designers need a better understanding of
how these products can potentially work together.
User interaction is changing from interaction with
single products into interaction with a larger system
of products. This trend faces designers with a
challenge: to create meaningful interactions for users
to deal with the complexity of the larger ecosystem
of technologies users function in. In this article
we introduce an interaction paradigm, where we
view smart environments in terms of connections
and associations between the actors and artefacts
within the environment. In this notion of Semantic
Connections, meaning is pivotal. We report on a
search for a theoretical foundation for our approach
in existing semantic theories. We attempt to use and
extend these theories beyond their traditional focus
on the appearance of objects and interaction with
them in isolation, towards designing for systems of
interoperating products. We illustrate our contribution
by providing examples of products and design
prototypes that implement our ideas. Although our
research is ongoing and the theory unfinished, we
believe that sharing our work can fuel the discussion
on how designers may deal with the challenges in
contemporary interaction design.
Product semantics, interaction design, smart home.
16
Design and semantics of form and movement
1 Introduction
The environments that people inhabit are occupied by
a growing number of digital devices and gadgets. Many
of these devices may be connected to the Internet,
wireless networks or other devices. Interaction with
networked devices is changing from interaction with
a single device, to interaction with a larger system
of devices. Some of these devices are becoming
portals to information stored somewhere else (e.g.
online services). Others have the potential to share
information like multimedia content, data, device
capabilities and services. However, we have not yet
succeeded in seamlessly operating among these devices.
Especially when we consider the way user interaction
was envisioned in paradigms like Ambient Intelligence
[1], Pervasive Computing, Ubiquitous Computing [2]
and the more recent notion of an Internet of Things [3].
The key goal of ubiquitous computing1 is “serendipitous
interoperability”, where devices which were not
necessarily designed to work together (e.g. built for
different purposes by different manufacturers at
different times) should be able to discover each others’
functionality and be able to make use of it [4]. Future
ubiquitous computing scenarios involve hundreds of
devices, appearing and disappearing as their owners
carry them from one room or building to another.
1
In this article we
adopt the paradigm of
ubiquitous computing,
as this matches our
understanding of a
smart environment
the closest.
Therefore, standardizing all the devices and usage
scenarios a priori is an unmanageable task.
Besides the technological challenges, there also lies
a challenge ahead for designing user interactions with
these ecosystems of interconnected devices. When
moving away from interaction with a single device
towards interactions with systems of devices, designers
need to find ways to communicate the relationships
between the devices and the larger system they are
part of. Additionally, designers need to find ways to
communicate the action possibilities of new, “emergent
functionalities”, that emerge when devices are being
interconnected. As Bill Buxton stated at the 2010
Design by Fire conference:
The real problems are not with any single device, but
in the complexity, the potential complexity of the larger
ecosystem of technologies that we function in. [...]
It’s about how this device works with that device, whether
it’s from the same or a different manufacturers; it’s the
complexity of the ecosystem. Why aren’t those things about
the interoperability taking more of a point? It’s about the
society of appliances and how they work together, which is
the new frontier [5].
An important problem that arises when designing
for these systems of interactive objects is their highly
interactive and dynamic nature [6]. The inherent everchanging nature of these systems and the severely
limited overview of the ecosystem in its entirety is one
of the most important challenges a designer faces when
designing for such systems. Additionally, such a system
comprises many different “nodes” that the designer, at
the time of designing, has no control over. Yet, when
designing and adding new nodes to the system, making
them interoperable is crucial for success.
In this article we introduce an approach to systems,
focusing on the inter-device relations and connections
that exist or may potentially exist. We see these relations
as both real “physical” connections (e.g. wired or wireless
connections that exist in the real world) and “mental”
conceptual connections that seem to be there from a
user’s perspective. The context of the connections and
the things that they connect are pivotal for their meaning.
Previous work has resulted in similar approaches.
Newman, Sedivy, Neuwirth, Edwards, Hong, Izadi, et
al. have developed an approach, which they named
recombinant computing [7].
How objects of design acquire meaning throughout
their use has been the subject of design research for
many years. The process of making sense of artefacts
is described by theories such as product semantics [8],
product language [9], semiotics [10] and the theory of
affordances (a term originally coined by Gibson, but
introduced to the design community by Norman [11]).
While these theories provide handles for designers
when designing (simple) products and to some extent
also for designing interactive products, they have not
yet shown their potential for providing handles for the
design of systems of interoperating devices.
In this article we present an approach to designing for
user interaction in smart environments called Semantic
Connections [12]. Central to this approach is the focus
on the semantics – or meaning – of the connections
between artefacts in such a smart environment. We
report on a search for a theoretical foundation for our
approach in existing design and semantics theory, and
re-apply the theories to our notion of connections or
associations between artefacts.
2 Semantic Connections
To address the problems as outlined in the introduction,
this section introduces an approach to interaction with
a system of devices in which the connections and
associations between the devices play a central role.
Before we give an extensive review of existing semantic
theories and discuss their implications for our approach,
we first introduce our semantic connections interaction
model.
Semantic connections is a term for meaningful
connections and relationships between artefacts and
entities in an ecosystem of interconnected and interoperating devices. These connections can be viewed
as both the real, physical connections (e.g. wired or
wireless connections that exist between devices) and
mental or conceptual connections that seem to be
there from a user’s perspective. The context of the
connections (what things they connect) is pivotal for
their meaning. The term “semantic” refers to the
meaningfulness of the connections. We consider the
type of connection, which currently often has the
emphasis when interconnecting devices (e.g. WiFi,
Bluetooth, USB) not to be the most relevant, but what
the connection can do for someone – its functionality
(e.g. stream music, share files) – even more. Semantic
connections exist in both the physical world and the
Design and semantics of form and movement
17
digital domain. They have informative properties,
i.e. they are perceivable in the physical world and
have sensory qualities that inform about their uses.
However, these physical qualities might be hidden at
some times, or only accessed on demand. We envision
semantic connections to exist between objects, people
and places. Not only objects and devices have meaning
in a system of networked devices. According to [13],
physical location within the home and device ownership
(or usage) are of central importance for understanding
and describing home networks by users. Amongst
places, people and objects, we specifically consider
semantic connections to exist between:
- artefacts;
- smart objects;
- sensors;
- UI elements;
- places;
- (smart) spaces; and
- persons.
Semantic connections have properties like directionality,
transitivity and modality (i.e. what things they carry).
Connections can be one-to-one, one-to-many, manyto-one and many-to-many. Connections can be
persistent or temporary.
The rationale behind Semantic Connections is to rely on:
- the meaning of existing objects to provide meaning for
the relationships between the objects and the resulting
meaning of the networked objects.
- the power of natural mapping and locality, using real
objects and locations to provide meaning for the
connections that are created between the objects
and (object) locations.
- inherent, augmented and functional feedback and
feedforward to strengthen the meaning of the
connections and the emerging functionality [14].
The interactions with the connections and the objects
that are connected are the carriers of meaning.
This meaning may be supported or augmented with
informative concepts like symbols, icons and indication
functions [15]. We may need to rely on metaphors and
symbolic and iconic meaning, because they provide the
flexibility and expressiveness of language. Affordances
are crucial but limited. They invite for a certain action,
but only communicate the purpose of the action to a
certain extent. Communicating what will be the result
of an action – feed-forward – is the real challenge as the
action itself is not the goal of the user.
18
Design and semantics of form and movement
Crucial to our approach is to make the gap between
user goal and action smaller. If we consider streaming
music from one device to another, “streaming” now
consists of multiple steps (actions) that do not necessarily
make sense. In our view, this single high-level goal
should have one (or at least as few as possible) single
high-level action(s). That single action should carry
the meaning of its goal. By using the physical world
as interaction space and using the real location of the
objects, we are reducing the need to identify the devices
from a list with names or rely on other forms
of representation.
2.1 Semantic Connections Interaction Model
A user interaction model for semantic connections
is shown in figure 1. It describes the various concepts
that are involved in the interaction in a smart space
and shows how these concepts work together.
The interaction model was inspired by the Tangible
Interaction model (MCRpd) by Ullmer and Ishii [16],
which in turn was based on the Model View Controller
(MVC) model. We distinguish between the physical part
of the user interaction and the part that takes place
in the digital domain. A user cannot directly observe
what is happening in the digital domain (and should not)
but experiences the effect it has in the physical world,
by interacting with the various smart objects and the
(semantic) connections that exist in-between them.
In doing so, users create a mental model of the objects/
system they are interacting with, which only partly (or
not at all) includes the digital part. Digital information
manifests itself in the physical world as data, media
and services. When a user interacts with a smart
object connected to the smart space, he/she senses
feedback and feedforward, directly from and inherent
to the controls of the device (inherent feedback),
digital information augmented onto the physical world
(augmented feedback) and perceives the functional
effect of the interactions (functional feedback).
The terminology, inherent, augmented and functional
feedforward and feedback is adopted from [14].
The user actions in the physical world are transformed
into interaction events and events/state changes, using
semantic transformations. This interaction data in terms
of user intentions is stored in the smart space2 , possibly
together with user preferences, defaults and context
information.
2
The notion of smart
space means that data
is stored centrally, and
can be accessed by the
various smart objects
in the smart space. For
more information on
these concepts refer
to [17].
SMART
OBJECT
SEMANTIC
CONNECTION
AUGMENTED
FEEDBACK
USER1...n
B1...n
A1...n
FUNCTIONAL
FEEDBACK
PHYSICAL
MENTAL
MODEL
INTERACTION
INHERENT
FEEDBACK
INTERACTION
EVENTS
DATA
MEDIA
SERVICES
DIGITAL
a 1...n
b 1...n
SMART
OBJECT
Fig. 1. Semantic Connections user interaction model.
3 Design Semantics Theory
3.1 Direct Approach – Interaction Frogger
Framework
The Frogger framework, as was introduced by
Wensveen [18], describes user interaction in terms
of the information a user perceives, (like feedback
and feedforward) and the nature of this information.
It distinguishes between inherent, augmented and
functional information. These types of information
can serve as couplings between user actions and
the products’ functions in time, location, direction,
modality, dynamics and expression. Although the
framework was designed to describe the interaction
with electronic devices and their interfaces, many of
the concepts in the framework are applicable to our
semantic connections concept as well.
When a user performs an action and the device
responds with information that is directly related
to the function of that product (lighting switching
on when a light switch is operated), we speak of
functional feedback. When a device has more than one
functionality, functional feedback should be viewed
with respect to the users’ intentions and goals when
performing the action. If there is no direct link between
a user’s action and the direct function of the product,
or when there is a delay, augmented feedback can be
considered to confirm a user’s action. This feedback is
usually presented in the form of lights, sounds or labels.
Inherent feedback is directly coupled (inherently) to
the action itself, like the feeling of displacement, or
the sound of a button that is pressed.
While feedback is information that occurs after or
during the interaction, feedforward is the information
provided to the user before any action has taken place.
Inherent feedforward communicates what kind of action
is possible, and how one is able to carry out this action.
Inherent feedforward is in many ways similar to the
concept of affordances, revealing the action possibilities
of the product or its controls [18]. When an additional
source of information communicates what kind of action
is possible it is considered augmented feedforward.
Functional feedforward communicates the more general
purpose of a product. This type of information often
relies on association, metaphors and the sign function
Design and semantics of form and movement
19
of products, which are described by theories such as
product semantics [8] and product language. Good
practice in creating inherent feedforward is making the
functional parts of a product visible, informing users
about the functionality of the product [11].
Implications for Semantic Connections.
If we view semantic connections in terms of the Interaction Frogger framework, the following interesting
insights emerge:
Feedback: When we consider multiple interconnected
devices and the functionalities and services they provide,
information like feedback and feedforward gets spatially
distributed. A user may operate a device, receiving
inherent feedback locally, but receiving augmented and/
or functional feedback remotely. In figure 1, the several
types of feedback are indicated. As inherent feedback
is inherent to the operational controls of the device,
these reside only in the physical world and are local
to the device. Augmented feedback is feedback that is
augmented from the digital domain onto the physical
world. This type of feedback is subject to change when
devices get connected to other devices. In the domain
of networked digital artefacts, functional feedback is
of a digital nature. Data, media and services that
exist in the digital domain become available in the
physical world, through the various devices and their
connections. Although many functionalities of digital
devices can be regarded as (displaying) media, data or
services, for some simple functionalities this seems
problematic. If we, for example, look at functional
lighting, it seems that the presence of light as the
functionality of a lighting device is not a very digital
concept. However, if we view a lighting device as a
networked smart device, the presence of lighting,
based on some sensor data, can be considered the
functionality of a digital service.
But what about the semantic connections themselves,
do they have these types of feedback as well? When
we approach the connections as if they were physical
entities with which one can interact, be it through
an interaction device, they do provide these types
of information as well. However, how this happens
and what kind of information it is, is slightly more
complicated. Inherent feedback is feedback that is
mediated through an interaction device, as one cannot
manipulate a connection directly. This inherent feedback
20
Design and semantics of form and movement
may however be closely related to the action of making or
breaking a physical connection, like a snap or click when
the connection is made or broken. Augmented feedback
to indicate a connection may be in the form of lights, or
in the form of projected or displayed lines. Functional
feedback is information about the actual function of the
connection, like the sound from a speaker that was just
connected to a media player. This type of feedback always
reaches the user through the devices being connected.
Figure 2 shows examples of these types of feedback in
designs that were created for this research.
Feedforward: Inherent feedforward, conceptually similar
to the notion of affordances, provides information
about the action possibilities with the devices or the
individual controls of an interface. Similar to this are
also informatives [8, p. 117] and partially also indication
or marking functions as defined in the theory of product
language [15]. Inherent feedforward is always physical
and locally on the device. However, when devices or
objects are part of a larger system, feedforward also
emerges where interaction possibilities between objects
exist (e.g. a key that fits a lock, a connector of one
device or cable that fits another). The same holds for
augmented feedforward, lights, icons, symbols and
labels that provide additional information about the
action possibilities. These may concern the action
possibilities locally at the device, as well as action
possibilities that concern the interaction with other
devices in the environment. While inherent and
augmented information are primarily concerned with
“the how”, functional feedforward communicates “the
what”, the general function of the device or the function
of a control. This type of information often relies on
association, metaphors and the sign function of products,
and is described in theories such as product semantics
and product language. With multifunctional digital
artefacts, and even more with networked artefacts,
this becomes increasingly difficult. Introducing the
concept of semantic connections tries to address these
problems; therefore the functional feedforward is the
main challenge when designing semantic connections.
Functional feedforward should give information
about the function of the semantic connection
before the interaction takes place. Properly designing
functional feedforward is therefore the crucial part of
understanding semantic connections, smart services and
smart environments.
Wensveen [14] further proposes that in interaction,
these types of information can link action and function
together in time, location, direction, modality, dynamics
and expression. Strengthening these couplings between
action and function will lead to richer and more
intuitive interactions [18]. We can also view semantic
connections in the Frogger framework in more general
terms. Although semantic connections are not a physical
device or product, but rather describe the structure
or configuration of a system of devices, the Frogger
framework can teach us important lessons. When
we look at the link between action and functional
information in time or location, a strong link would
mean they coincide in time and location. For location
this would mean that the connection that is made
between devices corresponds to the location of the
actual devices in physical space. Additionally, the
direction of the action of connecting/disconnecting
devices, being moving devices towards or away from
each other, would strengthen the coupling in terms of
direction. Also, the direction of the action could have a
link to the directionality of the semantic connection that
is made. This is similar to the couplings in dynamics.
Fig. 2. Examples of the different types of feedback:
(a) Augmented feedback; (green) lights showing a
connection currently exists. (b) Inherent feedback;
the feeling of a “snap” when two tiles are aligned. (c)
Functional feedback; a light rendering the mood of the
music when a music player is connected to it.
3.2 Product Semantics
As discussed previously (to some extent), the theory
of product semantics describes and analyzes the
meaning of products in terms of what a product is and
to a certain extent how it can be operated. Product
semantics is a theory about how products acquire
meaning. Krippendorff states in his work The Semantic
Turn [8]: “Humans do not see and act on the physical
qualities of things but on what they mean to them”
[8, p. 47] and “One always acts according to the
meaning of whatever one faces. [...] It always concerns
sets of possibilities and presupposes human agency”
[8, p. 58].
Krippendorff [8] thus differentiates between the
intended meaning of the designer, leading to the
design, and the meaning it eventually acquires after
interpretation and reinterpretation by the user during
use. These two meanings are different things and the
meaning that a design has for its user may be a different
one than the meaning the designer intended. This
concept of meaning is in accordance with information
theory, where the designer is viewed as a communicator
of a message in the form of a product and the user as
a receiver of that message [19].
Design and semantics of form and movement
21
Krippendorff’s semantic theory has, as briefly discussed
before, a very human-centered approach; as he states:
“meanings are always someone’s construction [...]
meanings are always embodied in their beholder”
[8, p. 56]. He also argues for conceptual openness, as
meaning emerges in the process of human interaction
with artefacts. “Meanings are neither intrinsic to the
physical or material qualities of things, nor can they
be located within the human mind. [...] Meanings are
constructed from previous experiences, expanded on
them and drift, much like imagination does” [11, p. 56].
All meanings are context-dependent as usually many
meanings are possible, but only few of them make
practical sense. Artifacts may mean different things in
different contexts and may mean different things to
different people. Contexts limit the number of meanings
as “artifacts mean what their contexts permit” [8, p. 59].
Contexts work in two directions, in the sense that
one thing provides the context for the other and vice
versa. For artefacts this means that “the meaning of
an artefact’s parts depends on the meaning of their
arrangements, just as the meaning of its arrangements
depends on that of its parts” [8, p. 61]. Krippendorff
compares understanding complex artefacts with
reading texts, with the distinction that one can interact
physically with an artefact, in contrast with only visually
perceiving a text.
Krippendorff [8] speaks of four main mechanisms of
how artefacts acquire meaning: meaning of artefacts in
use, meaning of artefacts in language, meaning in the
lives of artefacts, and meaning in an ecology of artefacts.
For our semantic connections, both the first and the last
of these mechanisms invite a closer look.
Meanings of artefacts in use: Norman distinguishes
between surface artefacts (what you see is all you
get) and internal artefacts, of which the latter needs
interfaces to represent and allow control over its
internals. The majority of problems with usability
and the constructions of meaning occur with internal
artefacts. Krippendorff describes interfaces and
states that: “Humans always act so as to preserve the
meaningfulness of their interfaces” [8, p. 84]. When
using a well-designed interface users go through the
stages of:
Recognition: correctly identifying what something is
and what it can be used for;
Exploration: figuring out how to face something, how
22
Design and semantics of form and movement
it works, what to do to achieve particular effects, and
Reliance: handling something so naturally that
attention can be on the sensed consequences of its use.
For recognition, (product) categories, (visual)
metaphors and attractiveness play an important role.
By finding resemblances in form and finding closeness to
ideal types of a product category, people can recognise
artefacts for what they are. Artefacts deviate from ideal
types in dimensions, varying within certain boundaries
of dimensions that define an artefact. They may also
vary in features, dispensable additions to an artefact
that do not alter its identity. As an example consider
a smart phone. With or without many of its features
it would still be a phone, as long as its core function is
preserved. When we have to recognise new artefacts
we can rely on the meaning of existing artefacts by
using metaphors. Central to the stage of exploration
are User Conceptual Models (UCMs), which are mental
models of how artefacts could work, when to do what,
and what to expect as a consequence of one’s actions.
Affordances and (physical) constraints are important
mechanisms to invite users into actions and guide users
in an artefact’s possible use. Other conceptual handles
for designing interfaces are informatives and semantic
layering. Informatives are similar to the concept of
“indication functions” in the theory of product language
and essentially guide and inform users about the flow
of the interaction. Informatives include: signals, state
indicators, progress reports, confirmings, affordings,
discontinuities, correlates, maps of possibilities, error
messages and instructions.
Meanings in an ecology of artefacts: Looking at artefacts
as a species, that are part of an ecology of things, is
an interesting viewpoint. There is a crucial difference
between ecologies of things and biological species
however, as is pointed out by Krippendorff “biological
species interact on their own terms; artefacts interact
on human terms” [8, p. 195]. Technological artefacts
do not know of each other but “interact with each
other on account of the designer’s specifications and/or
users’ desire to connect them” [8, p. 195]. Krippendorff
[8] distinguishes between diachronic accounts and
synchronic accounts to analyse ecologies of artefacts.
While for a diachronic account artefacts are being
traced according to their evolutions, a synchronic
account “describes the network of concurrent
connections between artefacts that co-determine their
use” [8, p. 197]. Important here are: causal connections
(actual physical connections); family resemblances
(belonging to the same product family, part-whole
relationships); metaphorical connections (carry meaning
between one, more familiar species of artefacts to
another species) and institutional liaisons (different
institutions are depending on the same species of
artefacts).
Within the context of smart environments, an
increasing amount of automation and increasing
interconnectedness may have a negative impact on the
meaningfulness of products. Artefacts can no longer
be considered in isolation, as they are part of a larger
ecosystem of technologies that we interact with.
Fig. 3. Example from our research; using a spotlight
metaphor to project connections into the physical
world.
The Spotlight Navigation device (top); and projecting
Therefore, designers need to provide users with handles
and clues to make them understand and enable them
to be effective in such an ecosystem of technologies,
to understand what is happening and allow them to
be and feel in control.
Implications for Semantic Connections.
Considering the theory of product semantics, and in
particular Krippendorff’s view on semantics, we can
start defining what implications this has for our concept
of semantic connections.
Building on Krippendorff’s user-centred approach to
meaning, we should be careful when indicating that a
certain connection has a certain meaning. Although
it might have a certain predefined functionality, what
it will come to mean for its users is not entirely for
the designer to control. By taking a second-order
viewpoint, and using principles such as metaphor,
affordances and informatives to support the phases
of recognition, exploration and reliance, designers
can, however, provide circumstances that increase the
probability of the intended meaning to come across.
For semantic connections this might mean that we have
to look for reliable metaphors like physical cables and
the interactions with them. Or like using a spotlight
metaphor to explore connections that are invisible
without using it (figure 3).
Physical constraints and informatives like signals, state
indicators, affordances or discontinuities in form might
help to indicate where and how to act; how to make
or break connections, and which devices allow (and
which do not allow) to be connected. Additionally, the
notion of causal connections that link artefacts together,
like wired or wireless networks, that is known and
understood, provide helpful clues. Also the notion
of family resemblances, where portable media players,
stereo sets and speakers belonging to the same product
families, might provide practical understanding of
what a connection, connecting products of this family
(with music playing capabilities) together, might mean
and what the emerging functionality will be. Looking
from an ecological perspective the following should be
considered:
- The meaning of a semantic connection depends on
the meaning of the artefacts it connects.
- Semantic connections work in mutual cooperation.
They depend on other species (smart objects) and
also support them.
the wireless connections between devices (bottom).
Design and semantics of form and movement
23
- Semantic connections might also have competitive
interactions with other artefacts. Emergent
functionalities through interoperability between
artefacts could eventually lead to less objects around
us. By combining the functionality of several artefacts,
others might become obsolete (e.g. combining a
printer and a scanner gives copying functionality).
- Semantic connections may also have a cooperative
relationship with other artefacts, because more smart
objects might result in more semantic connections
being made.
3.3 Ecological Perception
Although the theory of ecological perception and the
concept of affordance has been briefly discussed in some
of the previous sections, we would like to discuss the
theory and its implications a bit further. While many of
the semantic theories discussed depart from a semiotic/
linguistic and communication perspective, the ecological
approach to perception has an entirely different
theoretical foundation. Despite these differences, it will
also become clear that on a practical level, the resulting
designs might rely on similar perceptual qualities.
Affordance, which is a central concept of ecological
perception theory, is the property of an object that
appeals to our sensory-motor skills, like a door-handle
that “affords” to be grabbed and a chair that “affords”
to be sat upon. When the insights of ecological
perception were introduced into design by Norman
[11], it fuelled the design community to try and solve
many usability problems. Whereas on a practical and
application level not necessarily relevant, Norman’s
view of affordances is slightly different from the original
thoughts of Gibson and many like-minded psychologists
[11, p. 219]. Central to the notion of affordances is the
inseparability of humans and their environments, as
humans have always dealt with their environments going
through evolution. Affordances can thus neither be seen
independently from humans, nor can they be viewed
independently from the environment. For affordances
to be detected, they need to be available as information
that can be perceived by the human perceptive system.
Secondly, they need to be viewed in relation to the
bodily properties of every individual. While chairs may
afford seating for adults, it may afford something else for
children that might play underneath it [20].
Furthermore, when designing complex products and
interfaces, affordances often work well for inviting users
24
Design and semantics of form and movement
to perform certain actions that the controls
allow for. This does not necessarily indicate what the
results of such an action will be. This is acknowledged
by Djajadiningrat [21]; however, he also successfully
shows that the notion of affordances can be used as
a framework for design.
Implications for Semantic Connections.
Because connections/relationships between networked
artefacts are not physical, and perhaps only mental
constructions, affordances are a difficult concept in this
context. We can create affordances for the control
over these connections, but they will most likely only
reveal how to manipulate the connections/relations,
and not be very informative about the nature of these
connections. Here, associations and meanings of the
artefacts and their capabilities are important, which
are learnt and rooted in convention and previous
encounters with products. However, affordances can
be used to invite users to perform certain actions, and
these actions can carry meaning. To give a few examples
of possibilities - the affordance of a control to make
a connection can be shaped in such a way that it invites
an action that may associate it with permanent or
non-permanent connections, like a locking action after
inserting a connector into a socket. Furthermore, there
can be the affordance that invites the movement of
a control in a certain direction (e.g. a sliding switch).
This direction may in turn translate into the directionality
of a connection. Some of these ideas have been
implemented in the design of a digital camera and
a VCR controller as described in [21].
4 Discussion
In this article we have discussed various theories of
design and sense making. Much of the design theory
described is, however, about the meaning of objects
(or sometimes language) and originates from the era of
non-interactive, mechanical and electric products and
machines. With the introduction of microelectronics
and digital electronics, many of these theories have
been reconsidered to accommodate for interfaces
and interactivity, and some have evolved into new
ones. Now that we have entered the era of digital
networked artefacts, which introduces additional
concepts and complexity, these theories may need to
be reconsidered; especially when considering that the
networking technologies that connect these devices are
wireless and thus invisible. Even if we find ways to shape
objects in such a way that they reveal their connectivity,
how will they inform users about the possible
connection types and the emerging functionality?
Today networked objects are often recognised by their
LCD screens, as part of a product category of “smart
objects”, or desktop, portable or wearable computers.
Developing a form language and interaction paradigms
for such products is a challenge that a large part of
the (interaction) design community is and has been
working on. Despite these efforts, today’s products
remain mainly GUI-based and these GUI’s are the most
important means for controlling connectivity.
The semantic connections interaction model and
underlying theory proposes to reveal these invisible
connections and allow direct physical control over
them, like we have control over many physical wired
connections. To support this, part of our semantic
connections interaction model also proposes a software
architecture to solve the current interoperability
problems to a certain extent. This software
architecture enables networked devices to exchange
information and share device capabilities. Together this
is expected to enable users to interact with the various
devices in the system on a higher, more goal-oriented
level, moving away from the current device-oriented
way of interaction. Even though our approach still has
to prove itself in practice, our experimental prototypes
and setups show potential.
Wensveen et al. [14] propose an approach they refer
to as the direct approach, which departs from the idea
that not only the physical appearance of a product, but
also the actions it invites users to perform, are carriers
of meaning. They argue for a strong link between the
qualities of an action and the result of that action, as is
described in the Frogger framework [18]. The notion of
feed-forward is pivotal in this direct approach, especially
functional feedforward (as described in section 3.1). For
our notion of semantic connections, we rely on several
mechanisms to provide this (functional) feedforward.
First of all we rely on natural mappings [11]. The
connections or associative links are created between
devices, places, persons or interactive parts of devices,
that all exist in the physical reality. Instead of relying on
identifying networked devices by name or other types
of representation, we identify them by their physical
location, where users can perceive them, point at them
and touch them. Secondly, we rely on the meaning
of the devices that are being connected, in particular
the resemblances in meaning of the devices being
connected. Important here is the change in meaning
that might occur, when users view the device no longer
in isolation, but as part of a larger system. Krippendorff
[8] discusses these part-whole relationships (as is
described in section 3.2). Thirdly we rely on feedback
and feedforward being provided by a mediating device
or service, which has the special purpose to enable
exploring and manipulating the – otherwise invisible –
connections. We not only consider which things are
connected, but also how these connections are made.
This is where we have the freedom to carefully craft
the way we discover and manipulate these connections,
to provide additional information about what the
connection will mean once it is made. Once the
connection is active, in many cases the functional result
in the physical environment will give additional feedback
on the success and functionality of the connection that
was made.
Although we believe that our approach will contribute
to the necessary paradigm shift in user interaction,
needed to accommodate interaction with systems of
devices in contrast with single-device interactions, we
realise our contribution is only a start. However, we are
convinced that sharing this viewpoint and its theoretical
foundation with the design community can be beneficial
for starting the discussion amongst a larger audience
than primarily the ubiquitous computing research
communities and interaction designers working in
that area.
Acknowledgments
SOFIA is funded by the European Artemis programme
under the subprogramme SP3 Smart environments and
scalable digital service.
References
[1] Aarts, E., & Marzano, S. (2003). The New Everyday:
Views on Ambient Intelligence. Rotterdam: 010 Publishers.
[2] Weiser, M. (1991). The computer for the 21st century.
Scientific American, 265(3), 66-75.
[3] Kranenburg, R. van. (2008). The Internet of Things:
A Critique of Ambient Technology and the All-Seeing
Network of RFID. Institute of Network Cultures.
[4] W3C. (2004). OWL Web Ontology Language Use
Cases and Requirements. Retrieved January 6, 2011, from
http://www.w3.org/TR/webont-req/#usecase-ubiquitous
Design and semantics of form and movement
25
[5] Buxton, B. (2010, October). Nui: What’s in a name? Audio
IEEE symposium on Computers and Communications,
recording. Available from http://www.designbyfire.nl/2010/
(pp. 1029 -1034). IEEE.
program/
[18] Wensveen, S.A.G. (2005). A Tangibility Approach
[6] Frens, J., & Overbeeke, C. (2009). Setting the Stage for
to Affective Interaction. Unpublished doctoral dissertation,
the Design of Highly Interactive Systems. In Proceedings of
TU Delft, Delft, The Netherlands.
international association of societies of design research 2009
[19] Crilly, N., Maier, A., & Clarkson, P.J. (2008). Representing
- IASDR’09 (pp. 1-10). Seoul, Korea: Korean Society of Design
artefacts as media: Modelling the relationship between designer
Science.
intent and consumer experience. International Journal of Design,
[7] Newman, M.W., Sedivy, J.Z., Neuwirth, C.M., Edwards,
2(3). Available from http://www.ijdesign.org/ojs/index.php/
W.K., Hong, J.I., Izadi, S., et al. (2002). Designing for serendipity:
IJDesign/article/view/429
Supporting end-user configuration of ubiquitous computing
[20] Djajadiningrat, J.P. (1998). Cubby: What you See is
environments. In Proceedings of the 4th Conference on
Where you Act. Unpublished doctoral dissertation, TU Delft,
Designing Interactive Systems: Processes, Practices, Methods,
Delft, The Netherlands.
and Techniques (pp. 147–156). New York: ACM. Available from
[21] Djajadiningrat, J.P., Wensveen, S.A.G., Frens, J., &
http://doi.acm.org/10.1145/778712.778736
Overbeeke, C.J. (2004). Tangible products: Redressing
[8] Krippendorff, K. (2006). The Semantic Turn: A New
the balance between appearance and action. Personal
Foundation for Design. Boca Raton: CRC Press.
and Ubiquitous Computing, 8(5), 294–309.
[9] Bürdek, B.E. (2005). Design: History, theory and practice
of product design. Basel, Switzerland: Birkhäuser.
[10] Chandler, D. (2002). Semiotics: The Basics. London:
Routledge.
[11] Norman, D.A. (1998). The Design of Everyday Things
(illustrated edition). Cambridge, MA: MIT Press Ltd.
[12] Vlist, B.J.J. van der, Niezen, G., Hu, J., & Feijs, L. (2010).
Design semantics of connections in a smart home environment.
In L.L. Chen, T. Djajadiningrat, L. Feijs, S. Kyffin, D. Steffen, &
B. Young (Eds.), Proceedings of Design and Semantics of
Form and Movement (desform) 2010 (pp. 48-56). Lucerne,
Switzerland: Koninklijke Philips Electronics N.V.
[13] Poole, E.S., Chetty, M., Grinter, R.E., & Edwards, W.K.
(2008). More than meets the eye: Transforming the user
experience of home network management. In Proceedings of
the 7th ACM Conference on Designing Interactive Systems
(pp. 455–464). New York: ACM. Available from http://doi.acm.
org/10.1145/1394445.1394494
[14] Wensveen, S.A.G., Djajadiningrat, J.P., & Overbeeke,
C.J. (2004). Interaction frogger: A design framework to couple
action and function through feedback and feedforward. In
Proceedings of the 5th Conference on Designing Interactive
Bram van der Vlist,
Systems: Processes, Practices, Methods, and Techniques
Gerrit Niezen,
(pp. 177–184).
Jun Hu and Loe Feijs
[15] Gros, J. (1983). Grundlagen einer theorie der
Department of
Industrial Design
Eindhoven,
University of
Technology
Eindhoven,
The Netherlands
productsprache. Vol 1. HfG Offenbach, Offenbach/Main.
[16] Ullmer, B., & Ishii, H. (2000). Emerging frameworks for
tangible user interfaces. IBM Systems Journal, 39(3-4), 915–931.
[17] Niezen, G., Vlist, B. van der, Hu, J., & Feijs, L. (2010). From
events to goals: Supporting semantic interaction in smart
environments. In Computers and Communications (ISCC), 2010
26
Design and semantics of form and movement
Tom Djajadiningrat, Luc Geurts, Jeanne de Bont
[email protected]
Table manners: The influence
of context on gestural meaning
Abstract
We investigated the activation of gesture control
for a dining room table lamp. Using video scenarios
which show a user interacting with the lamp during a
dinner with friends, we carried out an online user test
comparing the existing activation through hand gesture
with three new alternatives: clapping, finger snapping
and voice. Though we had expected users to prefer
these alternatives for being more fluid, easier and faster
than the existing initiation through hand gesture, the
opposite turned out to be true. They were considered
intrusive since they disturb through sound, through
their connotations and by violating the personal space of
others. We argue that the appropriateness of gestures
is strongly dependent upon socio-cultural context and
should be evaluated in the final use context.
1 Introduction
In this project, we focused on an interaction style called
deviceless gesture control. Deviceless gesture control
allows users to operate devices from a distance without
the need for physical remote controls, body-worn
electronics or markers [1], [2].
One of the challenges with deviceless gesture control
from a distance is that it is difficult to distinguish
between gestures which are intended to control
the product and gestures which are not. That is, if
gesture control were continuously enabled, users
may unintentionally trigger a function. For example,
if the brightness of a dining room table lamp were
continuously controllable through an up-down gesture,
any vertical movement such as picking up a pan would
influence the lighting. To prevent such frustrating,
unintentional triggering, gesture control should only
become active after users have explicitly activated
gesture recognition. This activation is called the
initiation action.
2 Use Case: A Dining Room Table Lamp
One of our business divisions developed a gesturecontrolled prototype version of a Philips Arcitone dining
room table fixture (Figure 1). With this suspension
light, brightness can be controlled through a vertical
hand gesture. Moving the hand upwards increases the
fixture’s brightness, moving the hand downwards dims
it. The advantages of using gesture control for this type
of pendant light is that the user need not walk to a wall
dimmer, use a remote control nor touch the fixture
which would lead to it to sway. This Arcitone pendant
was first publicly shown to critical acclaim at Light +
Building 2010, a trade fair on architectural lighting [3].
The gesture initiation movement for this pendant
requires the user to move a hand horizontally from
either end of the lamp to just below the middle, then
move this hand downwards vertically, to finally stop and
wait for two seconds for the luminaire to confirm its
initiation through blinking (Figure 2).
Design and semantics of form and movement
27
Fig. 1. The Arcitone gesture-controlled pendant.
2.1 The Gesture Control Paradox:
The Unnatural Initiation Gesture
The difficulty in deciding upon an initiation procedure
based on movement only, is that the movement should
not occur in daily ritual as gesture control may then be
activated unintentionally, thereby defeating the point of
having an initiation gesture.
The paradox then is that while gesture control is often
described – as is voice control – as a form of natural
inter action, the required initiation is unnatural by
definition.
Indeed a user test conducted by the business division
concluded that, although the initiation was easily
learnable, it was not intuitive. Our team was asked to
investigate alternative ways of initiating gesture control.
the manual. This is difficult to achieve with deviceless
gesture control, especially if no changes can be made
to the product design. We therefore focused on three
aspects in which the initiation gesture clearly fell short:
naturalness, repeatability and speed.
3.1 Naturalness
The original, movement-only initiation gesture has a
rather mechanical, even robotic expression, as there
is no flow to the movement. Due to cost-driven
technological choices, the user is required to move
orthogonally (first horizontally, then vertically) and then
pause (Figure 2). As a result, the initiation action felt like
an unlocking procedure and rather contrived.
2.2 Project Scope
When we started our project, the product design and
gesture sensor technology for the dining room table
fixture had already been frozen. Therefore it was not
possible to change the form factor. This blocked the
possibility to add use cues to the existing form or to
incorporate additional visual guidance through LEDs
or displays. Alternative initiation actions would have to
work in combination with the existing product design
and sensors.
3 Critiquing the Existing Initiation Action
In its strictest definition, intuitive interaction implies
that it is immediately clear to the user how to act
in order to achieve a certain goal. In other words,
users should not need to learn the right action by
trial and error, by instruction, or by having to read
28
Design and semantics of form and movement
Fig. 2. Diagram showing the dining room table fixture
suspended above a table. The original initiation gesture
required users to: (1) move their hand horizontally from
the side to just below the middle of the lamp, (2) move it
vertically downwards, (3) hold it still for two seconds.
3.2 Repeatability
Even experienced users did not always manage to
initiate gesture control successfully every time. Initiation
would fail if the user moved too quickly or too close
to the sensors. So, even though the gesture is easy to
remember in broad terms, it can be difficult to execute
it exactly correctly. The main reason for this is that
users only get feedback after they have completed the
initiation movement, not during the movement. Since
the interaction lacks real-time feedback, users cannot
correct their actions mid-movement. Their only option
is to carry out the movement and hope that it will turn
out to be correct. If not, they have to start all over
again which causes frustration.
3.3 Speed
Because of the change of direction and the pause, the
initiation gesture feels long-winded, even though the
initiation may only take seconds in absolute terms.
The emphasis is more on the initiation gesture than on
the functional gesture which changes the brightness.
3.4 Design Brief
Ideally, an initiation action should be accompanied by
continuous feedforward and feedback, which makes
users understand what is going wrong in case of errors
and which allows them to correct their actions [4],
[5]. However, with the product design already finished,
our options to provide such continuous guidance were
limited. We therefore aimed for an initiation action
which was short, so that in case of a false negative users
could try again without becoming too annoyed. The
initiation action should:
r'FFMáVJEBOEOPUCFEJTDPOUJOVPVTJOFJUIFSTQBDFPS
time. The user should be able to continue immediately
with the up-down gesture to control the brightness.
r#FTVGàDJFOUMZVOVTVBMUPTDPSFMPXPOGBMTFQPTJUJWF
(i.e. the product mistakenly flagging an event as an
initiation action), yet be simple enough not to result in
many false negatives.
r#FFBTZUPHFUSJHIUFWFSZUJNFXJUIPVUNBOZGBMTF
negatives (i.e. the product mistakenly ignoring an
intentional initiation action) and without requiring
additional guidance.
r'FFMJOTUBOUBOFPVT
4 Initiation Alternatives
Through bodystorming we explored three initiation
alternatives: finger snap, double hand clap and voice.
Finger snapping and clapping were already suggested
repeatedly by test participants during earlier evaluations
of gesture-controlled lamps. All three alternatives are
multi-modal, require the recognition of both sound and
movement, and allow more fluid, simpler and quicker
initiation whilst managing the risk of false positives.
Simpler initiation is also interesting from a
communication perspective. As it is not self-evident
that a product can be controlled through gestures,
marketing gesture-controlled products is likely to
require an explanation of this new technology and how
to interact with these products. There are many touch
points at which both brand and user benefit from a
simple initiation gesture which can be explained in a split
second: in advertising, at point of sale, on the packaging,
in the manual etc.
4.1 Finger Snap, Double Clap and Voice
With finger snap initiation, after users have snapped
their fingers underneath the luminaire they can
immediately change the brightness by moving their hand
up-down. The initiation procedure through double
clapping is similar, with the only difference being that
instead of having to snap their fingers, users have to
clap their hands twice underneath the luminaire. With
voice initiation, users first say the name of the luminaire
to activate gesture control. This is similar to humanhuman interaction in which we first attract a person
by saying their name to get his or her attention before
we enter into conversation. The risk of false positives
with voice initiation is increased if the word which is
used as the voice command does not occur in everyday
conversation. As the concept was to be tested in
Holland, we chose the name ‘Linea’, meaning ‘line’ in
Italian, as the name befits the minimalist rectilinear form
factor and because the word 'Linea' is not part of the
Dutch language. Clearly, this name would not be a good
choice in Italy since the word 'linea' occurs commonly in
Italian conversation.
4.2 Robustness
The term robustness refers to the ability of the product
to withstand adverse, confusing conditions. With
sound recognition, various noises may be mistaken for
the initiation sound. Though finger snapping has the
Design and semantics of form and movement
29
Fig. 3a. Initiation by hand movement.
Fig. 3b. Initiation by finger snapping.
Linea!
Fig. 3c. Initiation by double clapping.
advantage that it can be done one-handedly, double
clapping is likely to be more robust than a single finger
snap. A single sound spike can result from all sorts
of actions (e.g. putting the lid on a pan, throwing a
magazine on the table), possibly resulting in false
positives, whilst two loud noises in succession are
far more unlikely to occur. Whilst clearly it would
be possible to analyse the waveform of the sound to
determine its origin, this puts higher demands on the
processing power of the embedded microcontroller
and thus raises the product’s complexity and bill of
materials.
The robustness of all three initiation methods can be
increased by requiring that sound and movement have
to be detected simultaneously. That is, the luminaire
only initiates if it ‘hears’ finger snapping, double clapping
or voice whilst detecting movement in the active zone.
If people make an initiation sound anywhere in the room
without movement being detected underneath the lamp,
gesture control will not be activated.
5 Online, Video-Based User Tests
To obtain user feedback we shot video scenarios of
30
Design and semantics of form and movement
Fig. 3d. Initiation by voice.
the initiation options being used during a dinner party
(Figures 3a-3d). The resulting video clips were evaluated
in an online user test. The original, movement-based
initiation was the only method which was technically
operational. The other three (finger snap, double clap
and voice) were executed using a Wizard-of-Oz set-up.
As the actor carried out the gestures, the brightness of
the lamp was controlled from behind the scenes.
5.1 Hypothesis
Our hypothesis was that the alternative initiation
procedures would be preferred over the original
movement-only initiation, as the alternatives were less
contrived, less convoluted and faster.
5.2 Experimental Design
Twelve participants (5F, 7M) took part in the evaluation.
For the user evaluation AttrakDiff was used [6].
First, the participants watched the four movie clips
in randomized, counter-balanced order. Then they
watched them one by one, completing a questionnaire
after each viewing. They also sent us their comments
and critique.
too
self
oriented
Finger snap
selforiented
P
neutral
task
oriented
too
taskoriented
superfluous
too
self
oriented
desired
Hedonic quality (HQ)
Hedonic quality (HQ)
Movement
task
oriented
too
taskoriented
Voice
selforiented
desired
neutral
task
oriented
P
too
taskoriented
too
self
oriented
Hedonic quality (HQ)
Hedonic quality (HQ)
neutralP
Pragmatic quality (PQ)
Double hand clap
superfluous
desired
superfluous
Pragmatic quality (PQ)
too
self
oriented
selforiented
selforiented
desired
neutral
task
oriented
P
superfluous
Pragmatic quality (PQ)
too
taskoriented
Pragmatic quality (PQ)
Fig. 4. AttrakDiff test results for the four initiation actions.
5.3 Results
The results are shown in Figure 4. Since the comparison
was based on video rather than on interaction with
working prototypes, all initiation methods offered
apparently perfect recognition: in all four videos
the user makes the correct gesture and the product
initiates correctly. Once implemented, some methods
will have a higher recognition rate than others. Such
recognition errors were not taken into account in this
test. The pragmatic quality scores are therefore not a
realistic reflection of real-world usability, but rather an
indication of perceived usability. On the hedonic axis
we see that finger snapping, double clapping and voice
all score significantly lower than the original movementbased initiation. Hedonically, only finger snapping
approaches movement-based initiation.
5.4 Discussion
For an explanation of these results, consider some
of the comments provided by our participants.
Finger snap. It’s considered impolite to attract attention
by snapping your fingers, but may be acceptable to a
device. Still, guests at the table may be disturbed.
Voice. Too loud and noisy.
Movement. In my opinion this is the friendliest way to
control the lamp, with the least hindrance on what is
going on around the table.
Double clap. This is too loud and noisy (e.g. when
you’re in a conversation), people will be shaken up,
conversations might stop and all attention will go to
controlling a lamp. / I don’t want to clap my hands in
front of my guests’ noses nor make some funny noises
just to activate a dining lamp. / Gesture-controlled
Design and semantics of form and movement
31
activation seems most logical to me (…once you know
how to use it and get used to it). In the movie it still
comes over a bit unnatural and complicated to me.
The recurring critique appears to be that the alternative
initiation methods are too intrusive for three reasons.
First, in a dining setting double clapping and finger
snapping both have the connotation of attracting
attention with an air of superiority (e.g. calling a waiter,
reproving children). Second, suddenly speaking to the
lamp, the sound of finger snapping and double clapping
may interrupt conversation. Third, by making these
gestures close to other people’s faces, the user is
intruding upon their personal space. Considering that
a table may only be 100cm wide and that the lamp is
suspended above the middle of the table, the user may
be finger snapping or double clapping as close a 50cm
from the face of a table companion.
In terms of hedonic quality, only finger snapping comes
close to movement-based initiation. However, not
everyone can snap their fingers sufficiently loudly,
resulting in consumers being locked out from using the
product.
In this particular use context, the original movementbased initiation therefore remains – at least from a
hedonic point of view – the preferred initiation method.
The only way we may achieve a simpler initiation is
by reconsidering the product design and concurrently
designing the product’s form and required gestures.
Gestures carry not only functional information but
also socio-cultural and even emotional connotations.
Therefore, gestures which are simple to execute may
still be inappropriate in particular contexts. The three
multi-modal initiation actions were both simpler and
more robust than the initiation by movement only.
Yet users disliked them, as they were considered to
be in violation of good table manners.
This case study shows that we need to consider the
connotations or feelings that gestures might invoke,
something which can only be judged in combination
with the use context. Looking back, we did act out the
alternative multi-modal initiation actions in the early
stages of the project, but only in a meeting room setting
with a single user operating a task light. It was not until
we tried these initiation actions in a setting with dinner
guests and the fixture above the dining room table, that
we discovered the negative associations of the gestures.
Acknowledgments
We extend our gratitude to our colleagues at Philips
Lighting and Philips Research. We also thank our
project manager and lead actress, Anja Janssen, our
‘extras’, Jolande Bouma and Maarten Nijhuis, and our
proofreader Aletta Stevens.
References
[1] Flikkert, W. (2010). Gesture Interaction at a Distance. PhD
dissertation. Media Interaction (HMI) research group at the
University of Twente, The Netherlands.
6 Conclusions
[2] Wigdor, D., & Wixon, D. (2011). Brave NUI World: Designing
Our experiences with the dining room table fixture
resulted in the following recommendations for the
design of gesture-controlled products:
r6TFBIPMJTUJDBQQSPBDIQSPEVDUGPSNJOUFSBDUJPO
and gestures should be designed concurrently as they
are strongly interdependent. Premature decisions on
form without considering interaction and gestures can
result in ‘digging yourself a hole’ and the loss of design
opportunities.
r1SPWJEFDPOUJOVPVTHVJEBODFUIFQSPEVDUTIPVME
guide users during, rather than after, a gesture through
feedforward and feedback, thereby allowing users to
correct their actions.
r&WBMVBUFJODPOUFYUHFTUVSFTTIPVMECFFWBMVBUFEXJUI
a product’s final form factor and in the intended sociocultural context.
Natural User Interfaces for Touch and Gesture. Morgan Kaufmann.
32
Design and semantics of form and movement
[3] Light + Building (2011). Retrieved November 18, 2011, from
http://light-building.messefrankfurt.com/frankfurt/en/besucher/
willkommen.html
[4] Wensveen, S.A.G., Djajadiningrat, J.P., & Overbeeke, C.J.
(2004). Interaction frogger: A design framework to couple
action and function through feedback and feedforward.
DIS2004, pp.177-184.
Tom Djajadiningrat,
[5] Djajadiningrat, J.P., Wensveen, S.A.G., Frens, J.W., &
Luc Geurts and
Overbeeke, C.J. (2004). Tangible products: Redressing the
Jeanne de Bont
balance between appearance and action. Special Issue on
Philips Design
High Tech Campus
Building HTC-33
5656 AE Eindhoven
P.O. Box 218,
5600 MD Eindhoven
The Netherlands
Tangible Interaction of the Journal for Personal and Ubiquitous
Computing, 8, 294-309.
[6] Hassenzahl, M. (2006). Hedonic, emotional and experiential
perspectives on product quality. In C. Ghaoui (Ed.), Encyclopedia
of HCI, 266-272.
Iyengar Vaijayanthi, Madhusudhan Marur
[email protected]; [email protected]
Persuasive design for energy saving
behavior through social gaming
Abstract
1 Introduction
This paper proposes to use social network gaming
as a persuasive tool to adapt energy saving practices
amongst the Millennial Generation while using
connected devices or appliances. The proposed solution
leverages the power of connected devices and social
networking principles to bring behavioral changes
for efficient energy consumption among users aged
18-25 years. The paper is based on the insights gained
through the research we conducted to understand the
motivations of users regarding energy saving in devices,
capabilities of connected devices and the role of future
web. Insights from our research revealed that a solution
that is an extension of user’s daily energy consumption
behavior, could lead to sustained behavioral changes
for energy conservation. We suggest a system that
captures the energy consumption of users, analyses and
provides recommendations for efficient consumption
on a social networking game. This paper reports on the
work in progress of a gaming solution called ‘Wattever’.
Wattever is a social networking game which maps users’
devices in the real world to a game in the virtual world.
Connected and smart environments make it possible
for users to be always ‘ON’ their devices. The inventory
of devices and appliances we use in our everyday life
continues to grow: Smartphones, Tablets, microwave,
smart TVs, washing machines, electric vehicles, etc [1].
A lot of these appliances consume a significant amount
of energy. There is a high demand for energy and the
supply is not meeting the equivalent to this demand [2].
Therefore it is essential to enable users to take steps
that support energy conservation in connected devices.
We adopted a three-pronged approach to propose an
energy conservation method for connected devices. We
studied the following factors
r.PUJWBUJPOTPG.JMMFOOJBMTSFHBSEJOHFOFSHZ
conservation
r$BQBCJMJUJFTPGDPOOFDUFEEFWJDFT
r'VUVSFXFCBOEJUTJNQBDUPOVTFST
Using emergent technologies; Governments, Energy
providers, Device Manufacturers and other Energy
sector players are able to track and communicate
about energy consumption to users [3]. Although these
players are able to monitor, and analyze the energy
consumption information to the users, moderate
success has been achieved for the intended outcome of
changing the energy consumption behavior among users.
We conducted a user study to understand the inherent
motivations and issues in Millennials about adopting
existing energy conservation solutions.
Keywords
Connected devices, social networking and gaming,
energy conservation, millennials, virtual worlds.
Design and semantics of form and movement
33
2 Millennials and Energy Saving
The target audience for this solution is aged 18-25
years. They are also known as “Millennials” or the
Net Generation. Millennials have been chosen for
this study since they are characterized by heightened
use and familiarity with Communications, Media and
Digital technology [4]. The term "Digital Native" is
used to refer to the Millennials. In addition, their
constant exposure to the Internet and other digital
media has shaped how they receive information and
how they learn. To develop a thorough understanding
of Millennials behaviors and attitudes towards energy
saving, we conducted user research. Our research
included an online survey of 30 participants, one-onone interviews with ten participants and an extensive
literature study. The focus of this study was to
understand lifestyle, energy consumption awareness,
energy saving practices followed and issues faced, and
energy saving information needs required by them.
2.1 Insights
Based on our research user study and literature
study, we identified some key issues in existing
energy saving solutions. Figure 1 shows some existing
energy management solution interfaces that indicate
consumption of energy.
users about energy wastage, they are little affected
by the repeated energy recommendations proposed
to them [5]. Studies reveal that [6] Millennials find
routine boring and unchallenging and are driven by a
need for novelty. Their preferences tend towards new
experimental actions, use of new technology and group
activities. Millennials believe that it is cool to be smart,
are tech savvy and goal oriented. This goal constitutes
major focus of their attention. For any other practice
or behavior to be adopted, they need a non-serious,
tangential motivation which does not divert them from
their main goal. We have identified this attribute, of
not developing a tangential approach for addressing
energy issues, as one of the reasons for the failure of
some existing energy saving behavior models. We also
found that energy cost saving is not the key motivating
factor for Millennials. Most users felt that the electricity
prices were reasonable and not significantly expensive
to motivate them to save energy. People generally
were unaware of the electricity costs and value for
money it provides [7]. Our research further indicated
that real-time feedback of energy consumption is
effective in promoting a sustained response and users
feel accountable for their actions. Therefore real-time
feedback can lead to reduced energy consumption [8],
[9]. We observed some personas enthusiastic about
saving energy were keen to know what to do ‘now’
rather than getting a delayed feedback about their
overall energy consumption.
3 Capabilities of Connected Devices and
Future Web
Fig. 1. Use of complex graphs in existing energy
management systems.
Complex graphs and visuals make energy information
dull and difficult to comprehend. We learnt that
frequent addressing of Energy issues as a crisis situation
has made it routine information and the alarming nature
of this propagation desensitizes the users to act on the
issue. Although there is a heightened awareness among
34
Design and semantics of form and movement
Trends indicate that devices and appliances will have
a unique way of identification in future as more and
more machines and people will get connected through
the internet. Future devices, like Smartphones, tablets,
electrical appliances, vehicles, etc. will be equipped
with an IP, hence capable of communicating with each
other and the user. Internet connected appliances will
empower users and energy service providers with more
access and control [10]. As new technologies like Cloud
computing enhance the addressability of appliances and
advanced computing capability will bring in ubiquity
of devices. The scope of future web also includes
networking within environment; hence the smart grid,
energy service providers, home environment and users’
devices, users’ social networks will be connected [11].
The omnipresence of the internet across systems can
be leveraged to bring together variable systems like
social networking, online gaming and connected devices.
The connectedness of devices will not only provide
information to users about energy consumption in their
devices, but also that of other users in their networks.
Moreover, when the users’ environment along with the
appliances will be connected to user’s social network,
they will have an involvement in the user’s decision
making. This reach of the internet through connected
devices, can be used to bring about a change in user
behavior, leveraging on its power of persuasion.
An important aspect of future web is social networking.
Insights from our user research indicate that social
media has been an important factor among users to
communicate and share information. Users choose
social networking sites to express themselves and
to reflect their personality. B.J. Fogg mentions that
social networks have an ability to trigger an inherent
motivation within users and hence persuade them to
take action [12]. Millennials have a notable presence in
social networks and social networking games. Currently
online social gaming is rapidly growing as a main form of
digital entertainment among Millennials. Online social
games like Sims Online, Cityville and Second Life are
good examples of the power of social games to engage
Millennials [13]. Studies show that the target audience
spends an average of 10.6 hours a day accessing social
networking [14]. Online multiplayer games add a social
dimension to the game, which manifests in changes in
existing behavior and/or emergence of new behaviors.
Users choose these games to reflect their personality
or aspirations through avatars. The word avatar
originated from Sanskrit and means ‘a form of self’.
An avatar is used for self-representation or the alter ego
of users [15], [16]. The behavior of a user’s redirection
of their feelings and desires through an avatar in a game
is referred to as ‘Transference’ [17]. An example of this
phenomenon is explained using the popular online game
Sims Social. Sims Social gives an opportunity for players
to interact with their friends in new ways. In addition to
taking care of the basic needs of one’s avatar, a user can
also enrich relationships with other online friends. This
game leverages on the existing relationship of friends on
Facebook to climb up in the game hierarchy. In order to
reach a relationship milestone within the game, the user
has to involve approval of the real Facebook friends.
Instead of choosing from predefined interactions with an
AI-controlled character, the user has to ponder about
the nature of the relationship to be established with
the online friend. These relationships range from being
friends, lovers, flirting, making out and even making
enemies. The avatars visit each other’s home and use
the objects in the home to explore new interactions,
like playing a guitar, looking at stars through a telescope,
etc. In order to customize and decorate the avatar’s
home, the user has to send requests to friends or spend
money to buy virtual merchandise. The eagerness of the
user to send many gift requests to each other and spend
real money to buy virtual merchandise in these games is
notable.
The entropy of parameters like the psychological
effects of Social Gaming, the capability of Connected
Devices and Web and principles of Sustainability can
result in a new learning leading to new behaviors. The
understanding of this learning can be used to develop
new approaches for promoting sustainable behaviors
among Millennials [18]. Our solution is therefore
built on social network gaming which is an already
established platform for communication and interaction
among Millennials.
3.1 Related Work: Energy Saving
We studied existing efforts to promote energy saving
behavior. An interesting solution is ‘Power House’
by Stanford University which gives users the task of
managing the power usage in a game world home and
incorporates their actual home’s utility data using a
smart meter [19]. Nissan’s Leaf line of electric vehicles
have “Eco Mode” software that keeps track of variables
like speed and power usage and provides feedback to
the drivers to improve driving efficiency. It also provides
online profiles to these drivers, so people can compete
with other drivers.
Another solution from OPOWER, a smart grid
software firm for energy conservation, provides
reports for users with monthly household energy
consumption. It also has a feature to compare individual
energy consumption over a period and also with other
households in the neighborhood. A successful case
study of OPOWER is in a California municipal utility
where they managed up to 4.1% energy saving during
peak season, using this solution [20].
The competition aspect of these solutions is a major
motivating factor for customers to consistently
consume less energy than those that do not.
Design and semantics of form and movement
35
4 Design Proposal: A Social Gaming
Solution for Energy Saving
As established through the above studies the internet
makes it possible to monitor, control, regulate and
inform about the Energy supplied and consumed.
Leveraging this device capability and future web
capabilities, a solution is proposed to encourage
energy conservation behavior among users while using
electrical appliances and smart devices in connected
environments. Our proposed solution is called
‘Wattever’. This is a multiplayer online social game that
enlists players worldwide to save energy in using their
connected devices and appliances. Users’ real world
connected devices are mapped to a Green Plot in the
virtual world. The game world reflects the energy
consumption of the devices in the real world.
The virtual equivalent of users and their devices in the
game are the avatars and a green plot respectively. The
health of the plot of a user is directly proportional
to efficient energy consumption by devices in the
real world. The objective of the game is to efficiently
consume energy in connected devices in the real world
in order to maintain a flourishing green plot in their
virtual world equivalents. This real-virtual mapping
system is illustrated in Figure 2.
and compete with other players to obtain the highestscoring (lowest-energy using) models.
4.1 Game Play as a Single Player and a Multiplayer
The energy consumed by a player in the real world
is translated to the health of user’s green plot in the
game. Over-consumption or misuse of energy leads
to deterioration and efficient consumption results in
higher scores resulting in a flourishing green plot. The
challenge in the game is for the players to plan and
consume energy in real life effectively so as to score
higher and maintain the health of their plots. In case
of surplus energy consumption, there is an energy loss
which leads to negative scoring and the corresponding
wilting of plants in the player’s plot. Users are motivated
to decrease their surplus energy consumption by
noticing a corresponding weakening of health in the
game. Apart from this, in order to inculcate new energy
saving practices among users, they are given tasks in the
game interface. Tasks are activities to be undertaken by
the players which will result in energy saving. The tasks
are devised based on multiple factors such as: Energy
standards, device capabilities, information on energy
from the Smart Grid and the user’s profile information.
These tasks are either to be achieved as an individual
or as a group. Table 1 describes the type of tasks that
can be undertaken by players in this game, the challenge
posed to the user and their examples.
Task
Challenge
Example
General Tasks
To decrease overconsumption of
energy in daily
usage of connected
devices
Turning the switch
of electrical
appliances off, even
when they have
been switched off
Task as a single
player
To decrease overconsumption of
energy by adopting
new practices in
individual usage of
devices
Washing clothes in
washing machine
during non-peak
hours based on
Demand Response
cycles
Task as a multiplayer
To decrease
over-consumption
of energy as a
community
Organized
carpooling in an
internet connected
Electric vehicle
Fig. 2. A system where users’ real world entities like
connected appliances, law making bodies, friends and
social circle are metaphorically mapped into a virtual
gaming world with their virtual surrogates.
Lower energy consumption by a user in the real world
yields a higher game score in the virtual world. Since
this is a multiplayer social game, players can collaborate
36
Design and semantics of form and movement
Table 1. Tasks and challenges based on single and
multiplayer options.
Fig. 3. Wattever interface with player’s avatar
Fig. 4. Recommendations for energy conservation.
and green plot.
Each task will have corresponding recommendations
for saving energy. A user can choose a suitable energy
saving recommendation and complete the task. Since
learning of energy efficient practices is paramount to
the success of the game, an increasing learning curve
will be maintained in these tasks, as users proceed to
subsequent levels. Some tasks will be easy and lead to
marginal energy saving, whereas completion of tougher
tasks will result in higher scores and therefore more
energy saving. Upon completion of these tasks players
are rewarded with scores, collectables or titles. These
rewards are an incentive for players to keep saving
energy (see Player engagement mechanism: Section 4.4).
Apart from earning scores and rewards, players will also
be able to publish their energy saving achievements on
their social networking sites.
4.2 Game Interface
Wattever provides a playful interface through which users
can monitor and control the energy consumption of their
devices. Figure 3 illustrates the basic screen of this game.
The interface primarily has a plot with green plants.
Other interface elements on the screen are tasks,
scores and user’s energy consumption data. In Figure
3 the user has been given a task to restore a wilting
plant. There is an avatar of the player who tends this
plot, restores the wilting plants and maintains its health.
The greenery of a player’s plot is a visual indicator of
how efficiently the player is consuming energy. Since
Wattever is accessible from multiple user devices,
like PC, Smartphone or tablet, a common interface
is proposed. This interface provides a unique feature,
where users will be able to see their devices as part of
a feedback loop of the game (see Recommendation and
Feedback mechanism: section 4.3, 4.4).
4.3 Recommendations in Wattever
Wattever provides energy saving recommendations to
the users to regulate their energy consumption. These
recommendations are based on tasks undertaken by
players. The recommendations are contextual to users.
These recommendations are formed taking into account
the following factors:
r5IFFOFSHZUPCFTBWFEGPSUIFUBTLDIPTFOCZVTFSTQMBZFST
r6TFSTEFWJDFTUIBUBSFDPOTVNJOHTVSQMVTFOFSHZ
r3FDPNNFOEFE&OFSHZ4UBOEBSETGPSEFWJDFTCZ
Energy Saving Bodies and Government policies
r$PNQBSBUJWFFOFSHZDPOTVNQUJPOPGPUIFSQMBZFST
in the game network.
These recommendations are real time by default
but can be based on timings set by users. The real
time recommendation appears as notifications on
users’ device screens. Users can interact with these
notifications to take energy saving actions. Users
will get multiple choices of recommendations to save
energy and complete a task. Users can choose to act
on a suitable recommendation. These actions need
not be taken only when users are actively playing the
game. The ubiquity of devices facilitates users to act
on given recommendations from any of their devices.
These recommendations are given in game vocabulary.
For example, in Figure 4 the user has a task to restore
a wilting plant. The reason for a plant to wilt is due to
surplus energy consumption amongst the user’s devices.
Figure 4 shows a recommendation pop up in the game
interface. This recommendation consists of choices like:
r4XJUDIJOHPGGBHBNJOHDPOTPMF
r'PMMPXJOHSFDPNNFOEFEEFWJDFTFUUJOHTPGBTUFBNJOHJSPO
r5VSOJOHPGGBQPSDIMJHIU
The details of the recommendations mention the
devices that are consuming extra energy, their individual
energy consumption in kilowatts and the energy cost
of each device. Users can implement the suggested
recommendations via the game interface. Interactions
with the game interface will enable users to control
their devices any time and anywhere. Users are prompted
to switch a device off through the game, using the visual
cues. A wilting plant will prompt a user to take action,
and the user will follow the suggested recommendation
through the game. For example in figure 4, the user
switches off the gaming console while not using it.
4.4 Feedback, Scores and Rewards in Wattever
Once players have taken the recommended action,
they are provided with feedback. Feedback loops are
employed by many social games to give compelling
information to the users about the tasks they have
Design and semantics of form and movement
37
undertaken. These can be positive or negative feedback
based on the consequence of a player’s action. The
positive feedback is in the form of scores and rewards.
The scores help to maintain the point system in the
game. Higher scores lead to better rewards and lower
scores lead to a deterioration of health of the green
plot. Hence players are motivated to earn high scores
in order to unlock new rewards in forthcoming game
levels and maintain the plot’s health.
The calculation of scores is made based on the
recommended energy consumption levels of users’
devices. The average energy consumption level is not
just based on energy consumed by devices but also
various external parameters. Energy requirement per
devices, time of day, age of the device and efficiency
standards of a device are all taken into consideration
while evaluating a device’s energy consumption.
Earning a particular total of scores corresponds to
rewards in Wattever. These rewards provide incentives
to players in the real and virtual worlds. The real world
incentives can include cash discounts while buying
energy efficient devices from device manufacturers, tax
incentives from government, Wattever game badges,
goodies and so on. Virtual rewards, as the name
suggests are not tangible. These rewards can be used by
the players to enhance their existing game experience.
These rewards include Game Titles, collectibles, virtual
gifts and more. The rewards are not only meant for
a single player but involve multiple players in a user’s
network. For example rewarding a player with an
ability to challenge other players, game tokens that help
players to buy collectibles in the game that help through
difficult challenges. These reward systems offer various
forms of addictive gaming. Negative feedback is given
when a user fails to perform the given task. Examples of
negative feedback include negative scoring, loss or turn.
Wattever will employ a feedback loop that encourages
its players to perform energy saving actions. The
priority will be given to positive feedback, so that
players are encouraged by rewards of playing the game.
Players can earn ‘in game’ tokens which will enable
them to decorate their game environment. Negative
feedback will be subtle unlike many social games. Since
the primary goal of this game is Energy Saving and not
entertainment, player’s inability to complete the task
should not be dealt as a punishment. Failure to complete
a task will result in a lower score and affect the health
of elements in a player’s game plot.
38
Design and semantics of form and movement
Feedback also provides an opportunity for players to
brag about their achievements in a social networking
platform. Users can use this to project their green facet
on a social network. Wattever uses player engagement
mechanisms to engage the users to undertake new
energy saving tasks, without losing the excitement
of performing the task. The playful interaction and the
inbuilt sociability of this game will be used to develop
a closer bond between users and their appliances. Figure
5 displays feedback to the player about earning
a high score for switching off an unused gaming console.
Fig. 5. Positive feedback saving energy saving.
4.5 Social Gameplay of Wattever
Wattever reflects the energy consumption in devices
of an individual and a social community who are part of
the game. A user is not only motivated as an individual
but as a social being to save energy. The social nature of
this game gives a platform to users to take up activities
together. Many online social games involve the users and
their network to cooperate within a game.
For example in the popular social game Cityville, if
a player has cultivated some crops and is unable to
harvest them in the right time, player’s friends can come
and harvest it for them, saving them from withering. The
player who has been helped by a friend feels obligated to
help the other friend in an hour of need. This causes the
effect of reciprocity [21], that is, when people give you
something, you feel the need to give something back.
Reciprocity is employed in Wattever to enable users
to help each other save energy. Wattever proposes
the following method by which a community can play
together in the game to save energy:
r6TFSTGSJFOETDBOSFEVDFUIFJSSFHVMBSDPOTVNQUJPO
in order to gain game scores and share it to maintain
the health of user’s game plot
r6TFSTDBOCFHJWFOBUBTLDPNQMFUJPOPGXIJDI
depends upon actions of multiple players
r6TFSTDBOHJWFEFWJDFDPOUSPMSJHIUTUPPUIFSUSVTUFE
players, and manage energy consumption of each
other’s devices.
If the user is consuming surplus energy due to an
unavoidable reason, the user’s social network can come
to the rescue by choosing from the above mentioned
options. Users can make individual contributions to
saving energy, in order to achieve a community goal in the
game. Figure 6 displays how different players adapt energy
saving practices to complete a group task in the game.
used to retain the interest of players for a sustained
period through powerful player engagement strategies.
Few examples of these mechanisms include enhancing
entertainment value, encouraging cooperation and
competition between players, giving bragging rights
for achievements and rewarding completion of tasks.
Many games have employed these tactics to keep the
existing players tied to the game and also to entice
the less active players back to the game. A notable
mechanism developed by the online games is that of
gifting. These gift announcements are made on the
user’s social networking profile and the user’s network
is notified about this. These gifts usually are useful to
their recipients to complete some game task. Figure 7
indicates the player engagement features employed in a
social game design to make lasting behavioral changes in
Millennials.
Fig. 7. Player
Fig. 6. Multiple players acting together to complete a task.
engagement features
employed in a social
In fig. 6 users adapt sustainable practices like car pooling,
using solar power and washing in water saving mode,
in order to unlock a community park in the game. A
community park adds to the visual appeal of the neighborhood. A neighborhood is formed by a network of friends
in Wattever. Therefore networks of friends can compete
against each other to have a better virtual neighborhood.
The multiplayer nature of Wattever encourages
cooperation and healthy competition thus leading to
gamification of energy saving practices. The intent of
gamification is to inculcate competition in an otherwise
mundane task. At the end of completing a task in the
game, the users are given positive feedback and also
reminded of how their other friends are faring in the
game. When users of similar lifestyle and life stage
compare each other’s progress the sense of competition
is invoked where each player tries to outscore
one another. The scores of players are displayed in
scoreboards. Wattever will display the scores of users
along with their corresponding game title and rank. In
this game, players are encouraged to compare their
appliance usage statistics with other users who are
faring better using a similar appliance.
4.6 Player Engagement Mechanisms
Popular social networking games imperatively focus on
Player Engagement Mechanisms. These mechanisms are
game design.
4.6 Wattever: The Game’s Integration with
Connected Devices
Wattever players leverage addressability of devices
to use and control them. The real time device usage
information of a user’s connected devices is pushed to
the internet. Increasing the number of sensors in the
devices, device to device communication and DLNA
capabilities are additional enablers of connected devices
to collect energy information. The ease of access of
energy information facilitates enhances the monitoring
and tracking of energy consumption by appliances of
different users. A system where the web encompasses
energy information from different entities like legal bodies,
smart homes, smart grid, external services providers and
users on social networks is shown in Figure 8.
A system like this works on the participation of multiple
stakeholders for energy saving. Device providers can
enable the device with sensors so that energy consumed
by the devices is pushed via the internet to information
providers. Sustainability Service providers like AMEE
[22] can aggregate this energy data and distill this
into meaningful information. Smart energy standards
Design and semantics of form and movement
39
Fig. 8. A system that analyzes energy
consumption based on inputs from
various information providers.
bodies like ZigBee alliance [23] can share protocols for
interoperable products that monitor, control, inform
and automate the delivery and use of energy. Smart grid
can predict and intelligently respond to the behavior
and actions of all electric power users connected to
it in order to efficiently deliver sustainable electricity
services. Social networks can provide data about
individual users and help in the profiling of energy
consumption data. The aggregated data is analyzed
by our proposed system and mapped to its virtual
equivalent in the game world. For example, if the user
has consumed an amount of surplus energy, then a
corresponding amount of damage will be incurred to an
element in the game world. The devices are not mapped
one to one in the virtual world whereas the health of
the virtual elements in the game corresponds to the
total energy consumption by real world devices. This
damage can be rectified by adopting energy efficient
practices while using devices and appliances and gaining
game scores. These scores are considered to restore
the health of the game world element.
4.7 Why Will this Work?
Wattever presents an informal environment to learn
energy saving practices. This energy saving Social
Game can result in a system that subtly assists users to
consume energy efficiently. Online Social games already
have an established popularity among Millennials. Hence
this approach will provide a lower learning barrier to
users than adopting a complex energy saving practice.
The multisensory experiences that games can provide
increase the attention span on energy conservation
of Millennials [24].
The key differentiator of this approach is that a user
is adopting energy saving practices through other users
instead of an impersonal third party recommendation.
The metaphorical representation of one’s own devices
in a game world establishes closer relation between
users and their devices. Studies show that users perform
better in the presence of others [25] and follow people
with lifestyle similarities and for information [26].
40
Design and semantics of form and movement
Therefore players can learn the best energy saving
practices from each other to prosper in the game.
The internet provides a ‘Never off/always on’
experience. Wattever can leverage this to engage
players in energy saving games through time zones.
This capability also provides for the possibility of
people from different cultures and countries coming
together under the cause of energy saving through the
game. Because of this ‘Never off/always on’ nature of the
internet, multiplayer online games can have players across
the globe engaged in heated competition with teams
comprising of players across time zones and geographies.
4.8 Future Direction of Using a Social Gaming
Approach for Inculcating Desired Behaviors
Changes in technology provide creative ways to reach
the target population. The proposed solution harnesses
the ability of future web and connected devices to create
a virtual world for the user. This solution caters to core
motivation and inner needs among the target age group.
A role playing game gives a user the freedom of choice;
this is exercised when users strategize about survival
and the success of their avatars in the game world.
By addressing inherent needs of Millennials like the need
for power, affiliation and a sense of achievement though
an energy saving online game; a new approach is created
to communicate about the seriousness of the energy
issue. An online social game removes the alarming
nature of this situation and makes the learning more
enjoyable and active. The learning curve in a simulated
virtual environment is lesser than other traditional
methods of addressing the energy crisis. Wattever
can cater to a broader audience who are already
playing a variety of online social games and provide
a complement and supplement to other traditional
methods of energy saving.
5 Conclusion
We have proposed the social gaming approach for
energy saving based on our insights from research
aimed at Millennials. Wattever’s game design principles
and scoring algorithms have to be further articulated.
However, this research helped us develop a new
approach of a tangential and casual motivation for
encouraging desired behavior in Millennials. The learning
from this project can be employed for causes other
than sustainability like, education, healthcare, driving
behavior, abstinence from anti-social practices and so on.
Social games provide a rich learning platform which can
be embraced by device manufacturers, energy solution
providers and governments to keep up with today’s and
future generations’ thinking and behaviors.
Appliance_Report_Zpryme_Smart_Grid_Insights.pdf
[12] Fogg, B.J. (2009). A behavior model for persuasive design.
In Proceedings of Persuasive 09, Proceedings of the 4th
International Conference on Persuasive Technology, New York:
ACM. Retrieved MArch 15, 2012, from, http://www.bjfogg.com/
fbm_files/page4_1.pdf
[13] Lewis, J., & Moorman, G.B (2007). Adolescent Literacy
Instruction: Policies and Promising Practices. International
Reading Association.
[14] WorldOne Research, (2008). Lexis Nexis Technology Gap
Survey. Retrieved March 15, 2012, from, http://www.lexisnexis.
References
com/media/pdfs/LexisNexis-Technology-Gap-Survey-4-09.pdf
[1] Lundberg, S., & Forbes, R. (2010). Ericsson Connected
[15] Derryberry, A. (2007). White paper: Serious games: Online
Devices - The Energy Efficient Way, http://www.ericsson.com/
games for learning. Retrieved March 15, 2012, from www.adobe.
res/docs/whitepapers/wp-50-billions.pdf
com/resources/elearning/pdfs/serious_games_wp.pdf
[2] Electricity Sector Overview (figure 8). Retrieved March
[16] Lim, S., & Reeves, B. (2009). Being in the game: Effects of
15, 2012, from http://www.pewclimate.org/docUploads/
avatar choice and point of view on psycho physiological responses
ElectricitySectorOverview.pdf
during play. Journal of Media Psychology, 12(4), 348-370.
[3] Labelle, R. (2011). Connectivity Trends and Issues. Retrieved
[17] Detweiler, C. (2010). Halos and avatars: Playing video games
March 15, 2012, from http://www.itu.int/dms_pub/itu-t/
with god. Westminster John Knox.
oth/06/45/T06450000020061PDFE.pdf
[18] Greitzer, F.L., Kuchar, O., & Huston, K. (2007). Cognitive
[4] Howe, N., & Strauss, W. (2000). Millennials Rising: The Next
science implications for enhancing training effectiveness in a
Generation. New York: Vintage Books.
serious gaming context. Journal on Educational Resources in
[5] Hanks, K., Odom, W., Roedl, D., & Blevis, E. (2008). Sustainable
Computing (JERIC), 7(3), article no. 2.
millennials: Attitudes towards sustainability and the material effects
[19] Stanford University and Powerhouse (2012). Kumasi games.
if interactive technologies. In CHI '08, Proceedings of the twenty-
Retrieved March 15, 2012, from https://www.freeenergygame.
sixth annual SIGCHI conference on Human factors in computing
com/portal/
systems. M. Czerwinski, A. Lund, & D. Tan, (Eds.). 333-342.
[20] Given, K. (2010). Green marketing: The gamification
[6] Oblinger, D. (2003). Boomers, gen-xers and millennials:
of social change. Retrieved March 15, 2012, from http://
Understanding the new students. Educause Review, July/August.
www.greenmarketing.tv/2010/12/31/gaming-for-good-the-
[7] Randolph, B., & Troy, P. (2008). Energy consumption and the
gamefication-of-social-change/
built environment: A social and behavioural analysis. City Futures
[21] Wohn, D.Y. (2010). Reciprocity in social network games
Research Paper, 7, March.
and generation of social capital. Retrieved March 15, 2012, from
[8] Using real-time feedback to reduce electricity consumption.
http://src.acm.org/2011/D.YvetteWohn.pdf
Research Reports International. http://www.marketresearch.
[22] AMEE (2012). Retrieved March 15, 2012, from www.amee.com
com/Research-Reports-International-v3284/Real-Time-
[23] ZigBee Alliance (2012). Retrieved March 15, 2012, from http:/
Feedback-Reduce-Electricity-2091630/
www.zigbee.org/Standards/ZigBeeSmartEnergy/Overview.aspx
[9] Mountain, D. (2006). The Impact of Real-time Feedback on
[24] Blunt, R. (2007). Does Game-Based Learning Work? Results
Residential Electricity Consumption: The Hydro One. Retrieved
from Three Recent Studies. Retrieved March 15, 2012, from http://
March 15, 2012 from, http://www.mendeley.com/research/
patrickdunn.squarespace.com/storage/blunt_game_studies.pdf
impact-realtime-feedback-residential-electricity-consumption-
[25] Rosenbloom, T., Shahar, A., Perlman, A., Estreich, D., &
hydro-one-pilot/
Kirzner, E. (2007). Success on a practical driver's license test
[10] CNN International. 2011, LG Thinq, Retrieved March 15,
with and without the presence of another testee. Accident
Iyengar Vaijayanthi,
2012, from, http://cnn.com/video/?/video/tech/2011/01/07/CES.
Analysis and Prevention, 39(6), 1296-1301.
Madhusudhan Marur
texting.tweeting.appliances.cnn
[26] Brzozwski, M., & Romero, D. (2010). Who should I follow?
UX Design, Samsung
India Software
Operations
[11] Zpryme Research and Consulting. (2010). Smart Grid
Recommending people in directed social networks. Retrieved
Insights: Smart Appliances. Retrieved March 15, 2012, from,
March 15, 2012, from http://www.hpl.hp.com/research/scl/
http://www.zpryme.com/SmartGridInsights/2010_Smart_
papers/follow/follow.pdf
Design and semantics of form and movement
41
Daniel Cermak-Sassenrath
[email protected]
Four installations inviting
playful interaction
Abstract
The course Embodiment, Tangible Interaction and
Games was offered as an elective in the Bachelor of
Creative Technologies (BCT) programme at Auckland
University of Technology, New Zealand in semester
1/2011. It was open to year two and three students.
We discussed ideas of phenomenology and attempted to
apply them to the design of interactive installations. The
installations integrated space, movement and artefacts
in collaborative and competitive settings and in playful
ways. A number of the installations appear suited for
exhibition and are presented here. The Eight-Bit Mirror,
Deimous and [sol] are student works1, Box Me Dumb
Human was developed by the author while preparing
the course. The installations focus on different sides of
the bodily experience, and on different aspects of playful
interaction.
Keywords
Student works, phenomenology, tangible interaction, play.
1 Introduction
The course Embodiment, Tangible Interaction and
Games was offered as an elective in the Bachelor
of Creative Technologies (BCT) programme at
Auckland University of Technology, New Zealand, in
semester 1/2011. The programme was only established
a few years ago in 2008 and is located within the
Interdisciplinary Unit which was formed to develop
42
Design and semantics of form and movement
new experimental alliances, research collaborations
and student-centred learning experiences across the
overlapping disciplines of Art & Design, Computing
& Mathematical Sciences, Communications & Media
Studies, and Engineering. The research-led BCT is seen
as a key component of this interdisciplinary project.
The course was open to year two and three students.
We discussed ideas of phenomenology in relation
to software systems (e.g. [1], [2]) and attempted to
develop holistic concepts for interactive, playable
installations. The student works were conceived and
created in a process that attempted to include and
connect theory and practice, writing and building,
reflection and action. Lectures were accompanied
by technical tutorials and exercises. Students were
asked to research theory, implement a classic computer
game, create several designs for their project, build
mock-ups and prototypes, and present, discuss and
critique their works.
The installations incorporated the arrangement
of physical objects in space, and bodily action and
movement in a collaborative or competitive setting of
multiple users as outlined in [3]. The students came up
with very different ideas for their projects which varied
considerably in the ways they established connections
to phenomenology. Some works were open-ended and
invited free exploration, some had explicitly defined and
formulated goals and were organised in levels or rounds;
some had background stories, some incorporated
1
The Eight-Bit Mirror
was developed by
Felicity Powell, Nick
Redwood and Dylan
Turney; Deimous
by Anneke Crouse,
Brandon Dawson and
Sahil Vallabh; [sol] by
Tarei King, Zak Henry
and Thorsten Ziller.
2
This quote and the
following quotes are
taken from design
documents written
by the students.
abstract entities; some took the notion of playful
interaction in the direction of action games, some
moved towards artistic reflection.
2 Eight-Bit Mirror
Fig. 1. Eight-Bit
Mirror (top)
and Deimous
(bottom).
The Eight-Bit Mirror installation (Fig. 1) is an interactive
mirror, inviting passing people to interact playfully with
its representation. It mixes recordings and live view, and
it shows a roughly pixelated image in black and white,
similar to [4], [5] and [6].
The installation was inspired by J.J. Gibson’s studies
in the field of visual perception and aims at creating
a visual instigator to prompt moments of instinctual
reaction in the visitor. ‘The original idea came from
observing both animals and infants' reactions to
mirrors. [...] Reflections and mirrors are often met
with a raw curiosity, and perhaps in some cases a mild
alarm.’2 The Eight-Bit Mirror attempts to create a visual
anomaly 'to examine, experiment, or provide insight
into overall visual behaviours'. It is something like a
magic mirror, that augments or diminishes what it sees
and reflects. It creates ‘a sort of alternate universe
in which the projection is still a mirror image of the
person[s] interacting with the installation, but it does
not behave as an everyday mirror image is expected to’.
The software reads the visitors’ silhouettes for a few
seconds and makes ‘the projection behave in anomalous
ways by rewinding, fast-forwarding, and flipping it in
the vertical plane’. The distortions are comprehensible
enough that the visitors recognise their silhouettes but
are piqued in their curiosity as to just what the mirror
image is doing next.
3 Deimous
The Deimous installation (Fig. 1) was conceived to
induce fear and manage an interactive experience
through sound alone. It is about fighting creepy
creatures in a dark room. The creatures make noise as
they approach the player, and the player slashes them by
waving a plastic meat cleaver at their position in space.
Having only sound as a means to perceive the
environment around him, the player is made acutely
aware of this sense and transforms his or her reception
of the surroundings during and even after the experience.
A 5.1 surround sound system is used to play sounds
triggered by MaxMSP on different speakers/at different
locations around the player. Constant background
noises create an ambient sound environment ‘that
Design and semantics of form and movement
43
stimulates emotional reactions of fear’. The enemies
have no definite entity, although the sounds draw on
conventions of horror movies. By not knowing what or
who exactly the enemy is, a sense of the unknown is
established – a key concept in creating fear. To make the
experience more challenging and interesting, the player
needs to learn to differentiate between bad and good
sounds. While the creatures sound “bad”, health packs
emit ‘good’ sounds, which the player needs to locate
and collect.
4 [sol]
The [sol] installation (Fig. 2) invites collaboration between
strangers who share a space. A stream of virtual particles
is to be directed from emitters around obstacles to
receivers. People attract particles and deflect their course
by their bodily presence and movement. This work is
intended as a large top-down projection.
It was inspired by the Beacon installation [7] and by
the AntiVJ installations [8] which both use technology
to build interactive live performances. Patterns of
struggle for leadership and collaborative teamwork are
erupting. [sol] focuses ‘less on the relationship between
the player and the game world, but more on the interrelationship between the players’. The interaction in
the virtual space is influenced by and influences the
real-world interaction of the actual people. To provide
for an easy entry and smooth immersion of the players,
the installation is not using any physical interaction
devices, ‘save for the bodily interaction of being in the
space’. A Microsoft Kinect is used to scan the space for
participants’ locations.
5 Box Me Dumb Human
In the Box Me Dumb Human installation (Fig. 2), a large
44
Design and semantics of form and movement
leather bunny puppet with red glowing eyes (a boxing
bag for now) is boxed by the player, while insulting/
motivating him with arguments taken from the AI debate.
The player is in a way fighting for all that is good about
humanity, what distinguishes it from abstract mechanism,
and he is fighting against the machine or system.
The puppet announces to the player the next boxing
hit (i.e. jab, straight, hook) to be executed to a specific
position, and the player has to react as fast as possible.
The order is fixed and makes sense boxing-wise.
Each round is 60 seconds long. If the player stops hitting
the puppet or is too slow, it will start to insult him
(e.g. ‘Humans are slow’, ‘Humans are irrational’,
‘Fight like a man’, ‘The machines will win’). After each
hit the puppet reacts (e.g. ‘Uff’, ‘Ahh’, ‘That wouldn't
stop me from telling what I know’) and then announces
the next hit (e.g. ‘Right cross to face’). The computer
voice is recorded with the Amiga's standard speech
tool, which has the classic monotone pronunciation
that marked computer speech for a long time. After the
match is over, a photograph is taken for the high score
list. Instead of pushing an OK button the player hits (i.e.
boxes) the robot rabbit on the nose.
The installation is intended as a contribution to the
debate that is coming to the front every few years about
being able to construct intelligent machines. It draws
on Norman White’s Helpless Robot [9], being a playful
comment on the often uneasy relationships between
machines and people, taking the everyday struggles
with e.g. telephone answering systems, ticket machines,
network connections or computer applications to a
visible, tangible level. Paradoxically, the conflict unfolds
at the same time as the player/user/human is trying to
do exactly what the machine is telling him.
In playing the game, the player takes sides, and fights
Fig. 2. [sol] (left and
middle) and Box
Me Dumb Human
(right).
against the machine. This action can spark participants'
reflection on what makes us human, and on the question
if we are losing out to the system. The installation
attempts to point out that living is not about abstract
existence, reasoning, problem solving and sensory
perception; but about bodily existence and lived
experience; activities which are always connected to
location and situation, and appropriating these (J. J.
Gibson, e.g. [10]). Living is neither pre-planned [1]
nor clear-cut, but creatively, associatively, initiatively
moving on and creating meaning along the way. Being,
experiencing, acting, reasoning, learning, understanding
and making belongs together, naturally – how could we
imagine otherwise? The dynamic process of having a
world is intimately connected to concrete, direct and
immediate practice. Acting in and experiencing the
world (Debray, e.g. [11]), changing it, change oneself,
being changed, experiencing time and space (Heidegger,
e.g. [12]), living and dying, is making us human. Our
bodies do not limit our being, but enable us to be.
They are not only our vehicle or tool in the world, but
we are our bodies (Merleau-Ponty, e.g. [13]), we are our
actions, we are what we do, we are our opinions and
emotions. Don’t we feel so alive when we do certain
things, or when we experience certain situations?
When we are fully in-tune with our surroundings,
with what is happening [14], with other people? Our
being is movement, rhythm, tone, tension, relaxation,
ambivalence, play, success, failure, trying, texture,
material, images, dreams, love, chaos and structure.
This is what we as humans are concerned with, what
we thoroughly enjoy (and despise), and it’s called life.
Little exploration into the unknown was conducted.
In this respect, many works were interesting, while
not being very provocative or experimental.
The course appears to have worked to get basic ideas
of phenomenology across to students and awaken their
interest. It identified the main protagonists, made them
and their theories accessible, and showed the relevance
of concepts of embodiment for students’ practice.
This led to a new perspective on the relationship of
reasoning and action with relevance and consequences
for interactive system design. Of course, the full
potential of the approach has not been realised yet,
but it was exciting to see the initial results in students’
creative work.
References
[1] Suchman, L. (1987). Plans and Situated Action. New York:
Cambridge University Press.
[2] Norman, D. A. (1988). The Design of Everyday Things.
New York: Currency-Doubleday.
[3] Dourish, P. (2001). Where the Action Is: The Foundations
of Embodied Interaction. Cambridge: MIT Press.
[4] Lightinstallation & Deep Wall (n.d.). Retrieved July 7, 2011, from
http://usermx.wordpress.com/2010/01/28/lightinstallation-deep-wall
[5] You Fade To Light (n.d.). Retrieved July 18, 2011, from http://
usermx.wordpress.com/2010/01/28/you-fade-to-light
[6] Interaction of light (n.d.). Retrieved July 18, 2011, from http://
topdonyblog.blogspot.com/2011/03/interaction-of-light.html
[7] Cinimod Studio & O'Shea, C. (2009). Beacon installation.
Retrieved May 4, 2011, from http://www.beaconinstallation.com,
unavailable on July 5, 2011
[8] AntiVJ (n.d.). Retrieved July 5, 2011, from http://www.antivj.com
[9] Lähnemann, I. (2009). We fix toasters. Norman White's
Daniel CermakSassenrath
Interdisciplinary
Unit, Faculty of
Design & Creative
Technologies
Auckland University
of Technology,
New Zealand
6 Conclusion
machines. In W. Herzogenrath & I. Lähnemann (Eds.), Norman
This article described installations that were produced
in a course on tangible interaction. The aim of the
course was to motivate students to connect phenomenological ideas with their own work. While
developing their practical projects, students were
invited to uncover the underlying theories, to explore
them on their own and to (re-) connect their findings
to their practice. We attempted to establish a feedback
loop of practical work and theoretical reasoning, in
a natural way, a holistic approach.
The approach led to some strong results, although
it appeared to be a common trend to avoid (artistic)
risks, conflicts and controversial topics, and to default
to quite conservative messages or meanings.
White. We Fix Toasters (pp. 12–33). Bremen: Kunsthalle Bremen.
[10] Gibson, J. J. (1950). The Perception of the Visual World.
Boston: Houghton Mifflin.
[11] Debray, R. (2004). Transmitting Culture. New York:
University of Columbia Press.
[12] Heidegger, M. (2010). Being and Time. Albany: State
University of New York Press.
[13] Merleau-Ponty, M. (2002). Phenomenology of Perception.
New York: Humanities Press.
[14] Alexander, C. (1979). The Timeless Way of Building.
New York: Oxford University Press.
Design and semantics of form and movement
45
Jaap Knoester, Tom Djajadiningrat, Philip Ross
[email protected], [email protected], [email protected]
Fluenci: The expression of expressing
Abstract
In this paper, we discuss the design process of a breast
pump which celebrates expressing as a natural, intimate
and emotional experience. Through interviews with
mothers and breastfeeding experts, the psychological
and physiological factors which may inhibit or stimulate
the milk let-down reflex were identified. Based on this
information, a concept called Fluenci was developed
which, in its interaction and behavior, mimics many
of the stimulating triggers provided by a baby. In an
evaluation of an experience prototype of this concept,
users preferred Fluenci over a conventional breast
pump both on a pragmatic and a hedonic level. We
discuss our design rationale in which the product's
functionality benefits from anthropomorphic interaction
and behavior yet does not result in a potentially
disturbing anthropomorphic product appearance. Lastly
we propose an additional form of anthropomorphism,
Embodied Anthropomorphic Form, which invites the
user to engage with a product as if it were human.
Keywords. Expressing, anthropomorphism, product
behavior, prototyping, interaction design, product design.
1 Introduction
Especially in Western culture, an increasing number
of mothers start to combine employment and
parenting within months after birth. The World Health
Organization promotes mother milk as having benefits
for both baby and mother [1]. However, breastfeeding
46
Design and semantics of form and movement
during working hours is often not possible. Also, the
mother and baby might not be physically able to breast
feed. An alternative is expressing mother milk at work
using a breast pump and feeding the baby the expressed
milk. There is much room for improvement in the design
of breast pumps, especially regarding the unpleasant
experience mothers have while expressing. This paper
describes the design process for a breast pump, in
which we focused on the aesthetics of interaction and
took the intimate experiential dimensions of normal
breastfeeding as point of departure.
2 Inhibitors and Stimulants in Expressing
Seven mothers were interviewed on their experiences
with expressing. The interviewees were asked to
act out their expressing ritual and show the location
and breast pump used. In addition, two lactation
consultants provided expert knowledge on expressing.
The interviews showed that the let-down reflex, which
causes the ejection of milk, is influenced by a number of
interdependent physiological and psychological factors.
2.1 Inhibitors
Oxytocine, the hormone that triggers the let-down
reflex, and adrenaline, the hormone associated with fear
and pain, are antagonists [2]. Discomfort and anxiety
therefore inhibit lactation. Sources of discomfort can
include work-related stress and suboptimal expressing
tools. Most mothers indicated that they ‘feel like a
cow’ during expressing and described the experience
as a very ‘mechanical’, even degrading ritual (figure 1).
Sources of anxiety include a lack of privacy: many work
places lack a breastfeeding room and mothers fear
that colleagues will see them partially unclothed and
expressing. Secondly, fear of not being able to express
can lead to a vicious circle.
tiring. Electric breast pumps (figures 2b & 2c) provide
suction using a (separate) motor. The efficacy of electric
pumps is higher than that of the manually operated
ones, but the electric pumps are more expensive, larger,
noisier and require a connection to a power socket or
regular charging.
Current pumps address expressing on physical level but
do not provide additional emotional or physiological
triggers. On the contrary, the interviews revealed that
the users related both product appearance and sound
to cow milking equipment. The interaction is strongly
machine-like in feel, with controls that are difficult to
operate or even out of reach when holding the pump.
Fig. 1. Illustration of mother using a breast pump
at work.
2.2 Stimulants
Whilst the primary stimulant for the let-down reflex is
the suckling on the nipple and areola, the mother seeing,
hearing and smelling the baby and feeling its warmth
and trampling also contribute to the reflex. To mimic
these stimuli during expressing, mothers use warm
compresses, massage their breasts and rely on simulated
‘triggers’ from their baby. For example, mothers
may look at photos or videos of their baby, listen to
recordings of its sounds or bring pieces of clothing that
carry its smell. These stimuli help to mentally visualize
the baby and provide a conditioned trigger for the letdown reflex [3], [4].
Fig. 2a. Hand pump.
Fig. 2b. Electric pump.
Fig. 2c. Double electric pump.
3 Benchmarking Existing Expressing
Methods
There are several ways to express milk. The simplest
way is through massaging the breast with the hands.
This method requires quite some practice and can be
physically heavy. Therefore mechanical devices were
developed: manual breast pumps. Breast pumps are
designed to imitate the physical suckling of the baby
on the nipple. Such manual breast pumps (figure 2a) are
usually powered by squeezing a handle. Though manual
pumps are quiet and relatively cheap, their use can be
4 Concept Design
Based on interviews with mothers and experts, a breast
pump concept called Fluenci was developed and built
(figure 3). The interaction with Fluenci was designed to
be evocative of breastfeeding a baby and to avoid the
machine-like connotations of traditional breast pumps.
Fluenci features automated, baby-like behavior which
eliminates the need for physical controls, contributing
to a calm, object-like appearance.
Design and semantics of form and movement
47
Fig. 3. The handheld part of the breast pump.
Note the thermofoil wires in the breastshield
and the circular tapping area.
4.1 Baby-like Interaction
The interaction with Fluenci mimics many of the
sensorial triggers provided by a baby. When the user
holds Fluenci’s soft, organic shape, her posture and
gesture resemble the cradling of the back of a baby’s
head. The product’s form contributes to reducing
anxiety by hiding the nipple and most of the breast from
view, thus providing privacy from incidental glances.
Through electric thermofoils, the breast-shield is heated
to body temperature, which widens the mammary ducts
– thus stimulating the let-down reflex [5]. Similarly the
back of the unit is heated to simulate the warmth of a
baby’s head. A digital sound recorder allows mothers
to record and playback their baby’s sounds or listen to
music that they also listened to during breastfeeding,
thus triggering conditioned reflexes. The baby’s sounds
are played back from the handheld pump, nearby and
directed solely to the mother. Since the music supports
creating a more ambient atmosphere, it is played from
the base unit which houses the pump motor, which is
more distant, both physically as well as psychologically.
4.2 Baby-Like Behavior
Fluenci starts pumping when the breast shield is
placed correctly on the breast and stops when it is
moved away. This behavior resembles that of the baby
(figure 5). Fluenci’s milk flow sensor allows it to mimic
a baby’s changes in suckling frequency. A baby starts
with suckling quickly and superficially to stimulate
the let-down reflex. Once the milk starts flowing,
the baby’s suckling slows down as it drinks the milk.
Similarly, Fluenci starts pumping stronger and at a lower
frequency once it detects the flow of milk (figure 6).
This optimizes the milk production and minimizes the
chance of sore nipples. In case the mother wishes to
override this automated process, a touch sensor on the
product allows the mother to determine the pumping
speed through three taps in the desired frequency,
without having to reach over to the pump unit.
Fig. 5a. Pump switches on when placed on breast.
Fig. 5b. Pump switches off when removed from breast.
Fig. 6. The frequency and amplitude of the pump
Fig. 4. System overview: recorder, pump motor and
automatically change after the let down reflex has
breast pump.
been detected.
48
Design and semantics of form and movement
5 Experience Prototyping
6 Concept Evaluation
Experience prototyping was an essential element during
the iterative design process. Initially, low-fi wooden
prototypes with an integrated speaker were made, which
could play sounds from an mp3 player. These were used
to get feedback from the users in the first concepts.
Six mothers were asked to compare the final Fluenci
experience prototype against a traditional, high-end
electric breast pump (figure 10). Both pumps were
preassembled and accompanied by custom-made quickstarting guides. The participants were asked to
Fig. 9. Part of the final prototype.
Fig. 7. User with one of the low-fi prototypes
combining basic electronics and Phidgets within
a housing of laser cut MDF.
The final prototype was fully functional, including
an Arduino prototyping board, speakers, sensors,
heating elements and a ‘hacked’ Avent breast pump,
all programmed in Cycling74’s MaxMSP software.
The final model was 3D printed. For hygiene reasons,
the prototype could not be used to express milk.
act out the expressing ritual – evoking a ‘let-down
reflex’, adjusting the pump frequency, and so on, after
which they ranked the pumps using AttrakDiff semantic
differential scales [6]. The results (figure 11) indicate
that Fluenci is rated higher than a traditional electric
pump, both in terms of pragmatic and hedonic quality.
On a side note, the Fluenci concept is rated as more
isolating (as opposed to connected), which influences
the hedonic identity dimension negatively, though
actually ‘isolation’ can be a positive attribute since it
might indicate that Fluenci provides privacy.
Fig. 8. Making the final prototype, using McNeel’s
Rhino 3D modelling software, 3D printing, an Arduino
microcontroller board and Cycling74’s MaxMSP
programming environment.
Fig. 11. AttrakDiff results, A is the Avent
pump, B is the Fluenci pump.
Design and semantics of form and movement
49
Fig. 10. Concept evaluation with end users and prototype.
Semi-structured interviews confirmed the positive
outcome of the AttrakDiff test. The way of holding
Fluenci, the warmth and the optional playback of
sounds were much appreciated. Aspects which received
criticism were the dimensions of the milk bottle which
should be increased to allow capturing more milk and
the control of the pump frequency which should be
more precise and direct.
2. Gestural (imitating the ways people communicate
with and through the human body with a focus on
human behavior)
3. Aware (imitating the traits, roles or functions of
people as a social role)
4. Anthropomorphic Form of Character (imitating the
common human capacity for thought, intentionality,
or inquiry).
7 Discussion of Semantic Considerations
Due to Fluenci’s multi-sensorial nature, it is interesting
to see how it can be positioned in their framework.
In general, Fluenci can be seen as an Anthropomorphic
Form of Character. The playback of baby sounds, the
suckling behavior and responsiveness to the user are
expressive of a ‘needing-role’. Fluenci’s size, human-like
warmth and organic ‘neck-like’ curvature invite it to
be cradled. The pump hereby not only defines its own
role, but potentially also changes the (self) image of the
mother in her social context.
The suckling of Fluenci would be Gestural Anthropomorphic Form – the suckling can be seen as motions
that suggest human action. The suckling behavior
contains elements of Aware Anthropomorphic Forms
as well – Fluenci seems to be aware of being held to the
breast and attentively adapts the suckling rhythm to the
milk supply from the mother, memorizing the favorite
settings. This behaviour suggests intelligence and
thereby minimizes the feeling of operating a machine.
The warmth Fluenci gives can be seen as an example
of Structural Anthropomorphic Form; it imitates the
warmth of a human body, resulting from its ‘operation’.
A breast pump is an intimate product of which the
functionality is strongly influenced by its look and feel.
How the user feels she is perceived by the outside
world and the feel of the product’s interaction and
behavior have a direct psychological and physiological
influence on its core functionality: the stimulation of the
let-down reflex. Here semantics are not limited to the
product’s appearance in isolation but include the userproduct interaction.
The central semantic question in the design of
Fluenci was to which extent the product should be
anthropomorphic. Anthropomorphism is defined as
the attribution of human-like qualities to inanimate
objects or animals [7]. They focus on a part of
anthropomorphism, which they call ‘Anthropomorphic
Form’: the imitation of human form by designers
as manifested in the objects they create – be it
static or dynamic. DiSalvo et al. define four kinds of
anthropomorphic forms:
1. Structural (imitating the appearance or functioning
of the human body with a focus on its materiality)
50
Design and semantics of form and movement
The warmth relaxes the milk ducts and provides
comfort when holding it against unclothed skin, hereby
stimulating the let-down reflex.
It is difficult to position Fluenci in DiSalvo’s framework
when focusing on its form. Fluenci’s form is not Aware
Anthropomorphic, since it does not express
consciousness or awareness. It is not an Anthropomorphic Form of Character, because it does not
per se display qualities or habits that define a role
like a baby. Fluenci’s form is not recognizable as an
imitation of the appearance of a baby’s head (structural
anthropomorphism). A breast pump with the looks
of a real baby would project a bizarre, possibly even
disturbing image, the so-called uncanny valley effect [8].
Similarly, a breast pump with a video screen showing the
baby’s face would turn the product into a robotic piece
of machinery. Gestural Anthropomorphic Form is using
motions or poses that suggest human action to express
meaning, but the emphasis is on actions performed by
the anthropomorphic object – not on the actions from
the user of the object.
Therefore we suggest a fifth kind: Embodied Anthropomorphic Form. Though the product itself does not
visually resemble (parts of) a human body, its design
invites the user to interact with it as if it were human.
Hereby the meaning arises from the (inter)action with
the product, both in physical and in social terms.
In the case of Fluenci, its shape affords to be held
by the mother in a particular, cradling manner. Once
the mother holds Fluenci in this way, the similarity to
feeding a baby can be experienced.
Acknowledgments
The authors would like to thank the mothers who
shared their experiences during the user research
phase and gave their valuable feedback on the concepts.
Their pride for – and dedication towards – their babies,
themselves and their work was a great source of
inspiration. Making the final prototype would not have
been possible without the help of Chet Bangaru and
Joep Frens of Eindhoven University of Technology. Last
but not least we would like to thank Philips Design for
being the client of this project.
References
[1] World Health Organization (2011), Breastfeeding.
Retrieved November 6, 2011, from
http://www.who.int/topics/breastfeeding/en/
[2] Riordan, J., & Wambach, K. (2005). Breastfeeding and
human lactation (4th ed.) Sudbury: Jones & Bartlett Publishers.
[3] Feher, S.D. (1989). Increasing breast milk production
for premature infants with a relaxation/imagery audiotape.
Pediatrics 83: 57-60.
[4] Gormezano, I., Porkasy, W.F., & Thompson, R.F. (Eds.).
(1987). Classical Conditioning (3rd ed.). Routledge. Hillsdale,
NJ and London: Lawrence Erlbaum Associates.
[5] Neifert, M. (2009) Great Expectations: The Essential Guide
to Breastfeeding. New York: Stirling Publishing Co. Inc., 205.
[6] Hassenzahl, M. (2006). Hedonic, emotional and experiential
perspectives on product quality, In: C. Ghaoui (Ed.),
Encyclopedia of Human Computer Interaction, 266-272,
http://www.attrakdiff.de
[7] DiSalvo, C., Gemperle, F., & Forlizzi, J. (2007). Imitating the Human
Form: Four Kinds of Anthropomorphic Form. New York: ACM.
Jaap Knoester,
Tom Djajadiningrat,
8 Conclusion
[8] Masahiro, M. (1970). Bukimi no tani The uncanny valley
We set out to design a breast pump that would make
expressing a more enjoyable experience for the mother,
by restoring the mother’s intimate multi-sensorial
engagement during expressing and triggering her
instinctive milk reflex. Our evaluation of Fluenci shows
that we made considerable improvements compared
to existing breast pumps. Instrumental for the final
result was the iterative, hands-on design process and
focus on the interaction experience, making use of
anthropomorphic form that goes beyond simple product
appearance, entering the realm of embodied interaction.
(K. F. MacDorman & T. Minato, Trans.). Energy, 7(4), 33–35.
(Originally in Japanese).
Philip Ross
Philips, Technical
University of
Eindhoven
Design and semantics of form and movement
51
Eva Knutz
[email protected]
Fictional emotions within
emotion driven design
Abstract
The aim of this paper is to address imaginative
experiences of emotions by drawing Kendall Walton’s
theory of make-believe. Moreover, we use a design case
as means for investigating how a child’s felt emotions
towards a hospital situation relates to his or her
imaginative experiences of emotions towards a fictive
character in a computer game simulating the real-world
situation. In so doing, we contribute with new insights
to existing theories of emotions in design, which tend
to focus narrowly on felt and measurable emotions.
Keywords
Fictional Emotion, Emotion Driven Design, Interaction
design
1 Introduction
Theories of emotion’s role in design have been
successful in explaining many new levels of product
meaning and product experience that until a few years
ago were hardly understood [1], [2], [3]. However, a
number of knowledge gaps inherent in these theories
become evident when working with new modes of
interaction and time-based experiences in game design.
While theories of emotions offer exhaustive
frameworks for describing how emotion driven design
is able to elicit emotions, the questions as to how such
emotions may change and evolve over time has only
received little attention.
52
Design and semantics of form and movement
Secondly, theories of emotions in design have a
tendency to focus too narrowly on the user’s felt and
sensed emotions, while the question of how imaginative
experiences of emotion relates to an emotional state is
left largely unaddressed.
The aim of this paper is to address imaginative
experiences of emotions by drawing upon Kendall
Walton's theory of make-believe [4]. More specifically,
we will use a design case to investigate how a child's
felt emotion (towards a hospital situation) relates to
a child's imaginative experiences of emotion towards
a fictive character in a computer game.
In this paper we will first introduce the notion of
‘fictional emotions’ as a new key concept in design
research that accounts for how a persons felt emotions
relate to a persons response to a fictional world.
Secondly we will introduce our design case, which is
using the design of an experimental computer game
(called the Child Patient Game) designed especially for
hospitalized children as research artefact. The purpose
of the Child Patient game is to design a computer game
environment that can map the subjective feelings felt by
children being in hospital. Here we will look especially
into the relation between Character Experience (that of
the fictive character in the game) and Player experience
(that of a Patient).
Thirdly we will explain the method of inquiry; the
overall method of this project, which is built up from
a research through design method [5], [6] as well as
the specific method of testing the Child Patient Game
in a hospital environment. A part of this method has
been to organize the data by using "visual mapping".
Visual mapping is a way of structuring and organizing
information to see patterns and relationships.
Fourthly, we will take a deep look into the data that this
inquiry has produced; how does a child's felt emotion
(towards a real hospital situation) relate to persons
imaginary experiences with a fictive character? How
are these make-believe states of mind charged with
emotions? Which ones are not charged with emotions?
Can we get a more fine-grained understanding of
how mental states of fictional emotions, imaginary
experiences and make-believe states are expressed?
Kendall Walton's make-believe theory will be used as a
framework for analyzing the relation between an actual,
felt emotion (referred to as "Real Emotion") and the
emotions we have towards fictional characters (referred
to as "Fictional Emotion").
Finally we will discuss the outcome of our findings and
where this outcome will lead us. Here a position will be
taken in the favour of narratives and gaming to be used
as a communicative practice that might give us valuable
knowledge and new insight about the emotional lives of
paediatric patients. In doing so we will widen the perspective on emotion elicitation by adding fictional emotions
to the prevailing emotion measurement paradigm.
2 Key Concept: Fictional Emotion
Within emotion research (theories on emotion) there
is general agreement on that emotion implicates
Feelings, Actions and Thought [7]. This means that
emotions are body related (somatic) as well as rational,
meaningful and logic (involves cognition).
Fictional emotion also implicates Feelings, Actions and
Thought - but the stimuli is to be found within the
fictional world: we can cry when we experience a sad
movie wherein the character we sympathize with will
die. Such an experiences belong to the game of makebelieve according to Kendall Walton [4] and must be
distinguished from emotions exposed during a real
situation that we actually cannot escape, for instance
a situation that involves the death of a real friend.
Walton’s make-believe theory is interesting because
it deals with the question how "remote" are fictional
worlds are from the real world? What is the role of
imagination?
Kendall Walton presents the following well-known
experiment in his article Fearing Fictions from 1978 [8, p.1]:
Charles is watching a horror movie about a terrible
green slime. He cringes in his seat as the slime oozes
slowly but relentlessly over the earth destroying
everything in its path.
Soon a greasy head emerges from the undulating mass,
and two beady eyes roll around, finally fixing on the
camera. The slime, picking up speed, oozes on a new
course straight toward the viewers. Charles emits a
shriek and clutches desperately at his chair. Afterwards,
still shaken, Charles confesses that he was "terrified"
of the slime. Was he?
According to Walton the answer is "No". To Walton,
it is only “make-believe-kind-of" true - that we have
feelings for certain characters in films, books or games.
He agrees on that these characters can have a great
affect upon us - and that fictional worlds and fictional
characters move us both physically and psychologically
and that the similarities to real fear, real sadness or real
happiness are close – but regardless of what our body
responses are or what we might say, think, or believe we
are feeling towards a fictional character it is not a real
emotion. It is a fictional emotion we are experiencing also referred to as “quasi-emotions” [9].
Quasi-emotions are very similar to real emotions but
still different because they are generated by "secondorder" believes about what is fictionally true. Walton
explains: “Charles believes (he knows) that makebelievably the green slime is bearing down on him and
he is in danger of being destroyed by it. His quasi-fear
results from this belief” [8, p.14].
Proponents of the Theory of Make-believe suggest that
our (quasi) emotional attitude towards the fictional are
strong related to our real (genuine) emotional states,
except that the make-belief overrides the role that
would otherwise be occupied by belief. So Charles
doesn't believe that the green slime is dangerous:
He imagines or makes-believe that it is.
It is not our aim establish weather fictional emotions
(and imaginary experiences) are equal to real emotions
(and real experiences) since this depends on how
we understand emotions. The aim of this paper is to
investigate - through game design - how persons felt
emotion (towards a specific situation) relates to the
persons imaginary experiences with a fictive character
(going through a similar situation).
Design and semantics of form and movement
53
2.1 Player Experience and Character Experience
When Children involve them selves in a game like for
instance playing "mum and dad" they involve them selves
in a game of make-believe (using Waltons expression).
They decide the rules for their game; who is playing
who, what kind of "mum" is in their game and what
objects must act as props. They are pretending that
a cartoon-box is their "house", that stones are "food
"and they acts as players in a fictional world, playing
out "mum" and "dad" in various imaginary scenarios imitating the real world, seen from their perspective.
It’s "just" a game. We think.
In the design case presented in this paper, we also
deal with children’s responses to fiction and how they
interact with that. It’s both similar and very different
from the example presented above. It’s similar because
we are inviting children to take part (as players) in a
game of make-believe. And it’s similar because it’s a
game that includes a fictional world and has certain
rules. Its different because in our case they cannot
make their own rules, the story is narrated in a way
that the main character (an animated character) must
undergo a fixed number of experiences. Its different
because the children must play the game under certain
circumstances, when they are not well at all; when
they have just been undergoing and uncomfortable
(emotional) medical examination. So we are asking them
to take part in a game of make believe that involves a
situation that they have just experienced them selves.
That’s why it makes sense to talk about what the
character (a fictive child patient) is experiencing in the
fictive world - and what the player (who’s a patient in
the real world) is experiencing while playing the game.
3 Design Case: The Child Patient Game
(CPgame)
The Child Patient Game (CPgame) is an experimental
computer game for hospitalized children (age 4 to 6).
The purpose of the game is to design a computer game
environment for young patients that can map emotional
experiences - and hereby allow hospitalized children to
inform staff and researchers about their emotional lives.
The concept of the game is developed by designer /
researcher Eva Knutz in cooperation with the children
ward and research unit at Kolding Hospital. The design
of the CPgame is strongly shaped by emotion theory,
as described in an earlier paper [10].
54
Design and semantics of form and movement
3.1 The Design of the CPgame
The CPgame is about a young child's journey through
a healthcare system. This child figure is the main
character that the player (the patient) will follow and
must control in the game. The player can attach certain
Emotions to the child figure (fig.1) or give the child
figure certain Secret Powers (fig.2) that can help him
through unpleasant things, such as the blood test.
Fig. 1. Screenshot from the CPgame: the player is
applying the emotion "afraid" to the main figure.
© Knutz 2010.
The player must attach one out of five emotions: three
negative, one positive and one less articulated (Angry,
Afraid, Happy, Sad or Uncertain) that the player thinks
fits to the child figure in the game. The emotions are
visualized as five animated characters, with five different
colours (fig. 1). This action of game play results in the
continuation of the story. For instance, if the player
attaches the emotion "anger" to the child figure in a
particular situation, the child figure will act angry. If the
player thinks that "sadness" is more appropriate, the
player can change emotion. When the player has arrived
at the "right" emotion, the game can continue.
Fig. 2. Screenshot from the CPgame: Catching
a "Secret Power" © Knutz 2010.
The "Secret Powers" consist of five different objects
that the player can catch (fig.2) before going into the
fictitious blood test. If an object is caught, the "Power"
becomes visible: A teddy bear that the child figure can
hug; an iron armour that the child figure can wear; a
magic cape that makes the child figure invisible; a bottle
of light liquid that makes the child figure shrink when he
drinks it; and a bottle of dark liquid that makes the child
figure grow up when he drinks it.
When the player has caught the Secret Power, that he
or she think will help the child figure the best, the game
can continue with that particular power attached to the
game story. So the child figure keeps the secret power
in the narrative sequence that follows. All choices of
the player are stored in a database, fixing the players
pattern of choices (of emotions and secret powers)
throughout the game.
3.2 Game World, Game Story, Rules and Players
Fictional Worlds (in books, films or games) are by
definition incomplete because it is not possible to
specify all the details about any world [11]. This is where
the player comes in; the player transmits his or her
intentions into the game world. The sequence of action
that unfolds in a game occurs as a result of the player’s
interaction with the game. Since it is a game - and not
just a story - the fictional world represented in a game
cannot function without the rules [11].
The way the player transmits her intentions into the
game world is through gameplay. The gameplay lays
down the rules for the player’s interaction with the
game world and the game story. Understanding the
gameplay, is understanding the pattern you need to
play with in order to perceive the fictional world of the
game as a whole. A fictional world that can be described
through the concepts of:
rGame World: An imagined world, inhabited with fictive
characters (objects, sound);
rGame Story: A sequence of events within the game
world containing beginning, middle, end;
rRules: What the player "must do" in the Game world,
constituted through gameplay;
rPlayers: Who are imagining the fictional world,
interacts with game, make choices.
If we drop these concepts upon our case - the CPgame
- we have a fictional world that (schematic) can been
described as:
rGame World: The universe of a hospital, experienced
by the main character: a child patient;
rGame Story: The journey of a child patient (a fictive
costumer) going through a fictive healthcare system,
with several events taking place over time: the child
is sick (begin of game) the child (now a patient) gets
treatment (middle) the child becomes well again (end
of game);
rRules: The player must control the child character, lead
him through the hospital and must attach "emotions"
and "secret powers" to the child patient (during his
journey) that the player feels must be the right ones;
rPlayers: Hospitalized children who are in the middle of
a similar "journey".
3.3 Player Experience and Character Experience
in relation to CPgame
The game is build up that way that the main character
(the child figure) must go through certain experiences;
he will become sick, he must overnight at the hospital,
he will be treated and he will become well again
(Character Experience). But it is also a game that allows
the player to interact with the game world, through
gameplay (Player Experience). Table 3 explains the
Character Experience and the Player Experience in
relation to the CPgame.
Character Experience
Player Experience
The main character (a
child figure) is at home; he
becomes ill and is brought to
the hospital by his parents.
Here different things are
happening to him; he is
being treated (examined,
measured, having a blood
test) and he has to spend the
night in the hospital (where
he falls into a strange dream).
When he wakes up, he feels
well again and can be brought
back home. It's like a 24-hour
circle story, where the end is
connected to the beginning.
During this journey (the stay
at the hospital) the player
must attach "emotions" and
"secret powers" to the child
figure, that the player feels
must be the right ones (the
emotions) or the ones that
can help the child figure the
best way (the secret powers).
The game is build up so that
that it increasingly becomes
more difficult and challenging
and that the player slowly
gets more control over the
main character. What the
player does in the game and
the way he interacts with the
game (the choices he makes
in the game, the movement
of his play pattern) will form
his Player Experience.
Table 1. Character Experience and Player Experience
in relation to the CPgame.
Design and semantics of form and movement
55
In the next section of this paper, we will take a closer
look at how the players (children patients) interact
with the Child Patient game. We are investigating
how persons felt emotion (towards a real hospital
situation) relates to the persons imaginary experiences
with a fictive character (experiencing a fictive hospital
situation). For the sake of clarity, let us define all
experiences, before we move on:
Character Experience: What the character (a fictive
child patient) is experiencing in the fictional world
(of the CPgame)
r Player Experience: What the player (a child patient)
is experiencing while playing (the CPgame)
r Real Emotion: Emotional Experience towards a real
situation.
r Fictional Emotion: Emotional Experience towards
a fictional situation.
r
4 The Method: Using Game Design as
a Method for Exploring the Relation
between Real Emotion and Fictional
Emotion
In this section we will explain the method of inquiry;
how the CPgame has been tested and how real emotion
and fictional emotions are mapped into so-called
"CPcards". But since the argument of the paper is built
up from a research through design method [5], [6], [12]
this need to be explained briefly.
4.1 Overall method: Research Through Design
This research project has been undergoing two
Experiments: First of all a computer game has been
designed that allow hospitalized children to inform
staff and researchers about their emotional experience
(Experiment 1) as explained in the previous chapter.
Secondly, the computer game has been tested upon a
group of patients and non-patients, using the game as
an alternative method of inquiry (Experiment 2). In the
next section we will focus on the second experiment.
4.2 Specific method: Using the CPgame as an
alternative method of inquiry
The CPgame was tested at the children’s ward of
Kolding Hospital from February 2011 until October
2011. The overall scope of Experiment 2 was to account
for relation between Real Emotions (expressed by
56
Design and semantics of form and movement
the child during hospitalization) and Fictional Emotion
(expressed by the child during gaming). This is done by
letting a group of 12 patients (aged four to six) play the
CPgame, right after having had a blood test. During the
blood test, the emotional states of the patients were
observed by staff, parents and researcher. The staff
were asked to observe the bodily states of the patients,
the parents were asked to judge the emotional state (of
their own children) and the researcher observed more
general issues, such as moods and attitudes.
Immediately after the blood test the game session
took place. During the game session the patients had
a "game-dialog" with the researcher while playing the
CPgame. In these dialogs the researcher would ask
about the choices that the patients made in the game.
For instance if a child attached "sadness" to the childfigure in the game, the researcher would ask "Why
is the child figure sad?". In the game dialogs, child and
researcher stayed within imaginary world of the game and
the researcher was observing as well as asking questions.
During the game session, the play pattern of the players
were obtained within the database of the CPgame. The
table below shows how the different sources of data,
collected during the experiment, addresses particular
aspects of emotion - or aspects of the imaginative
experiences of emotions.
Observations
forms, filled in
by staff
Emotional state / body evoked
feelings. The staff’s observation of
body states and behaviour during
blood test. How do the children feel
during blood test? Which emotions
are involved?
Real Emotions
Ratings
systems, filled
in by parents
Emotional state / appraisals.
The parent’s judgement of their
children’s emotional state during
blood test. How do the children feel
during blood test? Which emotions
are involved?
Real Emotions
Database of
CPgame
Play pattern. Which secret powers
do they choose in the CPgame?
Do they choose emotions and secret
powers directly or play around for
long?
Fictional emotion or
Imaginative Experiences
expressed through play
Game-Dialogs,
between
researcher and
child
Motivation and play behaviour
How do the children play?
What do they say about their choices
in the CPgame.
Fictional emotion or
Imaginative Experiences
expressed through verbal or
non-verbal accounts.
Table 2. Shows how different sources of data address
particular aspects of emotion.
4.3 Control Group of Non-patients
The experiment involved a control group of nonpatients. Here a group of children (aged four to six)
were chosen from a kindergarten, matching age and
gender. These children all played the CPgame in a
separate office of a kindergarten. Obviously, the control
group did not undergo any medical procedures they
simply just played the CPgame and were involved in a
game-dialog under same conditions as the patients.
4.4 Discussion of Inquiry (Experiment 2)
Experiment 2 was aiming at collecting knowledge
from different sources at the same time; sources that
supported the emotional aspects as well as the narrative
and imaginary aspects of a hospital experience. This is
not an easy task. An hospital experience involves many
sorts of experiences; the children must be away from
home, they must sleep in strange place, eat at certain
hours, they are being examined and tested and must
undergo treatment in various forms (medicine, scanning,
operations etc.). In consultation with the research group
at Kolding Hospital, we chose from the beginning to deal
with a specific experience that all children patients must
go through; the blood test. So we chose to put emphasis
on the blood test in the CPgame - and let the blood test
play a particular role in the testing of the CPgame.
Fig. 5. The Child Patient during testing © Knutz 2011.
It is a non-linear method that makes it easier to see
patterns, relationships, hierarchies and dependencies
that might otherwise remain hidden.
The data was filtered before it was mapped: the ratings
from the parents and the observations from the staff
was colour-coded to show if there was a relation
between the real (actual, felt) emotion and play-pattern
(the children's choices of emotion obtained in the game
database). Furthermore the recordings and notations
from the game-dialogs were transcribed to include only
verbal expressions and catch words, representing each
respondent’s imaginative experiences or play behaviour.
After organizing all data from both groups the data was
mapped into so called CPcards, representing each player
of the CPgame.
Figure 9 represents the visual mapping of information
of a patient into a CPcard:
The cards are divided into a left side, representing
the real emotions - and a right side representing the
fictional emotions. In the middle of the card we have
the player of the CPgame. The colour indicates if there
is a link between the emotions observed and rated by
staff/parents (left side) and the emotions chosen by the
player in the game (right side).
Fig. 9. Visual mapping of information into CP card.
© Knutz 2011.
4.5 Organizing the Data through Visual Mapping
After the testing of the CPgame all information, from
rating system (made by parents), from observation
(made by staff), from game dialogs (between researcher
and children), and from the database (of CPgame) - were
organized into a large visual map. Visual mapping is a
way of structuring, organizing, arranging information.
The CPcard above gives us information about the
patient "Gustav". The card tells us that Gustav's
emotional state during the blood test (left side of card)
evolved in sequence that can be described as Passive
(just before blood test) to Nervous (during blood test)
and to Sad (right after blood test). The right side of the
Design and semantics of form and movement
57
card tells us that Gustav first gave the child figure the
emotion "Sad" when the child figure (in the game) was
hospitalized. Later in the game (when the child figure,
had to have a blood test) Gustav attached the emotion
"Afraid". In order to give the child figure a "secret
power" before the blood test, Gustav gave the figure
a "grow-drink" and Gustav explained that the reason
for giving the figure such a power was: "because if he
becomes an adult he wont be afraid" (quote Gustav,
CPcard right side, fig. 9).
Gustav also verbally expressed the reason for choosing
"afraid" as fitting for the child figure in the CPgame:
"I don’t think he is happy" (quote Gustav, CPcard right
corner below, fig.9).
For every player a CPcard has been made. This gives
us a visual framework of how persons felt emotion
(towards a real hospital situation) relates to the
persons imaginary experiences with a fictive character
(experiencing a fictive hospital situation). The CPcards
also gives us information about how emotions may
change and evolve over time.
5 Outcome and Discussion: Exploring the
Relation between Real Emotion and
Fictional Emotion
Fig. 10. Relation between choice of “Emotion” (Angry,
Afraid, Sad, Uncertain and Happy) chosen by Patients
and Non-Patients in the CPgame. © Knutz 2011.
Fig. 11. Relation between choice of “Secret Power”
Figures 10 and 11 gives us a picture of how patients and
non-patients (the control group) chose emotions and
secret powers within the CPgame.
(Cape, Bear, Shrink-Drink, Grow-Drink, and Armour)
5.1 Explaining Player Choices
Figure 10 tells us that not a single non-patient chose a
positive emotion, where as the positive emotions were
chosen several times by the patients.
Figure 11 tells us that eight out of the twelve nonpatients chose the teddy bear, where as only four of
the patients chose the bear. This is interesting since
it is only the (teddy) bear that actually belongs to the
real world (some of the patients do actually bring their
teddies) whereas the other powers (becoming invisible,
wearing an amour or growing big or small) truly belong
to the world of fiction. We will speculate on that in
the discussion section. For now we can just conclude
that the 12 patients played the CPgame rather
differently than the 12 non-patients, who functioned
as a control group.
experimental game test. The 12 cards are organized
in such way that the children who felt most anxious,
insecure or uncomfortable during the blood test are
placed at first (number 1) and the patients feeling at
least anxious are placed at last (number 12).
The CPcards (Fig. 12) reveals several insights in relation
to the "the real" and "the fictional" world making.
First of all, if we look at the second emotion (below,
at the right side if the card) which is the emotion that
the player attach to the fictitious child patient during
the fictitious blood test - we can see that 7 of the 12
players (player no. 2, 3, 7 8, 9, 11, 12) chose an emotion
very similar to the real emotion (left side of the card).
That will say, in most cases the children's own (real)
emotion is close to the emotion the children think the
fictional child patient "must feel". This is particularly the
case when we look at the two (only) players, Player 11
and 12, who exposed a mainly positive sequence of real
emotions during the blood test session; they expressed
Figure 12 gives an overview of all the CPcards drawn
from each patient playing the CPgame during the
58
Design and semantics of form and movement
chosen by Patients and Non-Patients in the CPgame.
© Knutz 2011.
Fig. 12. CPcards
of all patients
(seven girls, five
boys) playing the
CPgame. © Knutz
2011.
both positive emotions within the real world and
choose only positive emotions as fitting to the fictional
character in the game world.
What is significant, is that 6 of these 7 players (player:
2, 7, 8, 9, 11 and 12) were able to verbalize and motivate
their imaginative experiences of why the child figure in
the game needed a specific Secret Power; Like player 8
and 9 (who both seemed very self assured about there
choices) - both chose the bear "because then he has
someone to hug". Player 2 (who choose Invisible Cape)
motivates her choice with saying; "because then they
can’t see him, then they can’t find him, then he doesn’t
get scared". Player 7 chose Grow-Drink "because if
he becomes an adult he wont be afraid". Player 11
also chose the Grow-Drink, but here the imaginative
experience is different. Player 11 chose the Grow-drink
because "then you don’t get sick so much" (if you are an
adult). These experiences says clearly something about
the children’s emotions on a narrative and imaginative
level; thinking that in the future, when you grow up, you
won’t be afraid anymore, or get sick.
If we look at the remaining five players that chose
fictional emotions, a bit more remote to their real
emotions (player: 1, 4, 5, 6, and 10) then it is as if we
have a different set of players: These Players have in
general very little to say. They were very willing to
play the game, but were difficult to get in contact with
during the game dialogs. Player 4, 5 and 6 had a similar
pattern - both in having the same narrative sequence
of real emotions (Passive-Afraid-Sad), and they all
choose the positive emotion as the first emotion: so
they believed that fictional child patient felt happy in the
beginning of the game (arriving at the hospital and being
hospitalized)
When it comes to the second emotion (the emotion
related to the fictional blood test), this group of
children all chose an emotion in the game, close to
their real emotion: Player 5 choose (in a self assured
manner) the emotion "afraid"; an emotion identical to
the real emotion he felt during the blood test. Player 4
and 6 chose "uncertain"; an emotion a bit remote from
their real emotion during the blood test but similar to
their play behaviour (being uncertain and insecure about
playing the game).
Player 1 and 10 both chose fictional emotions, very
remote to their real emotions. Player 1 expressed a
high degree of anger and fear during the real blood
test, but still chose "happy" as the emotion fitting best
to the character in the game, when he had to go the
through the fictitious blood test. When I asked him
why, he said, "Because he thinks its funny". Player 1
played the CPgame for the longest time of all. He tried
all emotions and secret powers out many times - and
kept on choosing the positive emotion in relation to the
fictional blood test. Did he want the animation figure to
be happy? We will discuss that in the next section.
Player 10 seemed calm about the blood test and had
no objection towards medical procedures. Player 10's
narrative sequence of real emotions is described as:
Passive in the beginning then surprised during the
blood test, and hereafter passive again. But the playpattern at the left side of the card reveals some quite
different emotions towards the fictional situation of the
hospitalization and the blood test: Sadness, Anger and a
Secret Power in the form of a teddy bear. Also this we
will discuss in the next section
We can - on the basis of the CPcards - sketch out a
more detailed picture of how a young person’s felt
emotion (towards a real hospital situation) relates
to the persons imaginary experiences with a fictive
character (experiencing a fictive hospital situation).
Design and semantics of form and movement
59
Player type A:
Fictional emotion relates to Real emotion
(player no. 2, 3, 7, 8, 9, 11 and 12)
Characteristics: Fictional emotion relates strongly
to Real emotion. The players can express their
imaginative experiences verbally.
Player type B:
Fictional emotion partly relates to Real emotion
(player no. 4, 5 and 6)
Characteristics: Fictional emotions are different
from their real emotions, but not entirely remote.
The players can't (or won't) express their imaginative
experiences verbally. Only through gaming.
Player type C:
Fictional emotion is very remote to Real emotion
(player no. 1 and 10).
Characteristics: Fictional emotions are completely
different from their real emotions. The players can't
(or won't) express their imaginative experiences
verbally. Only through gaming.
This is where the CPgame distinguish itself from
other products or tools that seeks to measure or
communicate emotion, such as the Product Emotion
Measurement Instrument "PrEmo" [1], the hospital
game "SiSom" [13] or the Self-Assessment Manikin
"SAM" [14]. This, because the CPgame contextualizes
the emotional experience it is meant to measure and
brings in the notion of Fictional Emotion.
Temporality and modulation play an important role
at both sides of the CPcard; on the left side ("Real
Emotion") integrated in the narrative sequence of
emotion - and on the right side of the card ("Fictional
Emotion") as an emotion-modulation process, taking
place during the interaction with the fictional characters
in the game ("in making the animation figures feel
certain ways"). This narrative and visual form of sensemaking is our main argument for developing a narrative
framework for emotion driven design that is actually
capable of providing us with valuable knowledge
and new insights about the emotional lives of
paediatric patients.
6 Conclusion
5.2 Discussions of Findings
Why does only the patients (and not the control group)
choose positive emotions in the game world?? Why
does some patients, who are clearly very distressed
about the blood test insist on giving the animation figure
a positive emotion? Why do the CPcards sometimes
reveals two different emotional worlds, as was the case
with Player type C?
To understand this we have to return to the central idea
of seeing "emotions" and "secret powers" in the game
world, as a set of reactions that the player can apply to
the animation figure "as if they where him". This might
explain, why its only the patients who wants the little
figure in the game "to be happy" - and not the children
who are not hospitalized; the CPgame may offer the
patient opportunities for communicating and modulating
its emotions by playing with fictional characters, finding
themselves in a "as-if scenario" similar to that of the
patients own situation.
So even though a child, who feels insecure about the
hospital treatment, chooses positive emotions in the
CPgame, they are just as important because it indicates
that the Player Experience may act as a (positive)
modulation process to the actual (negative) situation.
60
Design and semantics of form and movement
In this paper we have demonstrated that by drawing
upon Walton’s theories of make-believe it is possible
to address two unexplored areas within emotion
driven design; a possible framework for describing how
emotions may change and evolve over time, as well as
addressing imaginative experiences of emotion. In doing
so we seek to merge existing emotion theories with a
narrative approach that accepts imaginary experiences
and fictional emotions, as meaningful activities (or
perceptions) that contributes to our emotional
experience of the real world.
Furthermore this paper will contribute to the field
of interaction design and emotion driven design with
a visual and playful method for communicating with
paediatric patients that can increase the understanding
of children's unique emotional experiences during
hospitalization.
References
[1] Desmet, P.M.A. (2002). Designing emotion, Delft University
of Technology.
[2] McDonagh, D. et al. (2004). Design and Emotion:
The experience of everyday things, CRC.
[3] Norman, D.A. (2004). Emotional Design: Why We Love
(or Hate) Everyday Things, Basic Civitas Books.
[4] Walton, K. (1990). Mimesis as Make-Believe: On the
Foundations of the Representational Arts (pp. 209-289).
Cambridge, Mass.: Harvard University Press.
[5] Brandt, E. & Binder, T. (2007). Experimental design research:
Genealogy intervention-argument. In the Proceedings of the International Association of Societies of Design Research, Hong Kong.
[6] Redström, J. (2011). Some notes on program/experiment
dialectics. The Nordic Design Research Conference, 2011.
[7] Prinz, J. (2004). Gut reactions: A perceptual theory of
emotion, Oxford University Press, USA.
[8] Walton, K. (1978). Fearing Fictions, The Journal of Philosophy, 75(1).
[9] Walton, K. (1990). Mimesis as Make-Believe: On the
Foundations of the Representational Arts (pp. 195). Cambridge,
Mass.: Harvard University Press.
[10] Knutz, E. & Markussen, T. (2010). Measuring and
Communicating Emotions through Game Design. In proceedings
of the 7th International Conference on Design & Emotion, Illinois
Institute of Technology, Chicago.
[11] Juul, J. (2005). Half-Real. Video Games between Real Rules
and Fictional Worlds, London: The MIT Press.
[12] Markussen, T., Knutz, E. & Rind, P. (2011). Making Theory
Come Alive through Practice-based Design Research.
Swissdesignnetwork design research symposium Practicing
Theory or: Did Practice Kill Theory? Geneva University of Art
and Design (HEAD), Geneva.
[13] Ruland, C.M., Starren, J., & Vatne, T.M. (2008). Participatory
design with children in the development of a support system
for patient-centered care in pediatric oncology,” Journal of
Biomedical Informatics, 41, pp. 624-635.
[14] Bradley, & Lang, P.J. (1994). Measuring emotion: the selfassessment manikin and the semantic differential,” Journal of
behavior therapy and experimental psychiatry, 25, pp. 49-59.
Eva Knutz
Kolding School
of Design,
Department of
Communication
Design, Denmark
Design and semantics of form and movement
61
Carol MacGillivray, Bruno Mathez
[email protected]
Co-authored narrative experience:
Affective, embodied interaction
through combining the diachronic
with the synchronistic
Abstract
1 Introduction
Examination of perception tends to look at senses
in isolation, but Neuroaesthetics and Gestalt design
principles treat perception as an embodied synaesthetic
experience. The Diasynchronoscope project takes timebased techniques from animation and converts them
to spatiality, animating static objects through projected
light and creating transient visual cues that, when
combined with sound, demand selective attention. The
work challenges the use of passive nouns to describe
participants such as ‘user’ or ‘audience’ or ‘viewer’
and instead asks participants to recognise their true
position to be that of ‘exegete’, reading and interpreting
the gaps in space and time. Their interaction with the
work provides narrative meaning transcending the static
and hidden, to create a Gestalt systemic whole, making
each participant a truly immersed co-creator. Using
audience analysis, the paper evaluates the work against
more traditional media such as screen-based visuals
and makes the case for further research into somatic
perception of dynamics.
By removing the mediation of a screen, and suppressing
image recognition factors to create a narrative based
primarily on movement, the Diasynchronoscope is
designed to test the hypothesis that change is a key
signifier in perception [1] and that all change can only
be perceived through attention. In visual terms, this last
has been demonstrated in a number of reproducible
psychological tests such as studies in ‘change blindness’ [2].
The creators of the project are from audiovisual
industry backgrounds, and felt that ideas on continuity
of perception and how to ‘direct’ the eye had not
moved on greatly from psychologist Richard Gregory’s
observation that perceptions can be seen as predictive
hypotheses [3]. Indeed the powerful influence of the
‘just seen/heard/perceived’ is an essential tool for all
screen media. Marchant et al of Leeds University, UK,
showed through a three-year eye-tracking experiment
in attention mapping of audiences watching scenes from
Hitchcock’s Vertigo [4] that attention can be directed
about a screen with a surprising commonality [5]. This
commonality of viewing could be attributable to a
number of factors (perhaps not least to Hitchcock’s skill
in directing an audience’s attention and that Vertigo is
generally regarded as the best film made by a virtuoso
film director), but also appears to be physiologically
attributable to the way our visual cortex divides and
processes four separate parallel pathways of perception:
form, color, movement and depth. This paper looks
Keywords
Gestalt, animation, attention, projection, audience,
perception.
62
Design and semantics of form and movement
at the first phase of the Diasynchronoscope project
where experimental media designed by the authors to
explore notions of directed attention in an audience
unmediated by screen, and with suppressed cultural and
contextual content, is documented. It is the intention
that this first phase should be scaled up to a larger space
and to incorporate longer narrative elements, where
some current restraints such as audience number and
restricted viewpoints will be removed.
1.1 What is a Diasynchronoscope?
The name Diasynchronoscope comes from combining
Diachronic, (the study of a phenomenon as it changes
through time) with synchronous and scope (view). In
being so named, it evokes the early animation simulators
such as the phenakistoscope and the zoetrope which we
regarded as direct ancestors of the piece acting both as
art object and experimental media.
Used in a performance environment, the Diasynchronoscope is a real architectural space where prepared
3D objects are arranged in a way that they change
incrementally in shape and/or position (figure 1).
Although nothing in the room actually moves, the
objects are revealed through serial illumination in a dark
room, sequenced using the technique of projection
mapping. This technique enables creation of a
dramatization over time, encouraging the common
perceptual shortcuts of a participant to create
sequences and a narrative from static abstract objects.
Fig. 1. A range of objects in the Diasynchronoscope.
The closest practical examples for the Diasynchronoscope are the 3D zoetropes created by Pixar and studio
Ghibli. Here a rotating turntable is lit by strobe lighting
to give the illusion of 3D characters as real objects
performing in loops in a real space. Because the illusion
relies on a turntable, there is no narrative just looped
action. Nonetheless, the 3D zoetropes are objects that
create great wonder in modern spectators. It appears
that medium does impact significantly on the qualia of
experiencing movement and that there is something
special about seeing 3D unmediated by screen or
camera [6]. Because the Diasynchronoscope uses masks
to create specifically applied, sequenced projected
illumination, a narrative is possible, as is synchronization
of the effect with audio. In this first phase we showed
a restricted chunk of narrative to an audience of ten
participants from various backgrounds and ages and
recorded their perceptual responses.
1.2 Physiology Versus Personal Schemas
Although today’s audiences may not share the same
geographical or cultural space, they do share the same
cognitive neural architecture and perceptual shortcuts.
The interesting question is if they would create the
same narrative from abstract movement in an embodied
experience [7]. This introduces two main, at first
sight competing, areas for investigation: physiological,
universal shared cognition; and individual responses
drawn from personal schemas of movement.
The four separate parallel pathways of perception
in our visual cortex transmit signals devoted to form,
color, movement and depth. Of these four the least
explored appears to be movement, perhaps because it
is so hard to separate from other attributable contexts.
The first experiments in phi phenomenon were made
by Gestalt founder Max Wertheimer in 1912, where
he observed that rows of flashing lights created the
illusion of motion even when there was none [8]. The
discoveries of Wertheimer are credited with launching
a perceptual revolution giving designers the Gestalt laws
of similarity, pragnanz, proximity, continuity and closure.
Although as Filip Pizlo of Purdue University points
out, the literature on the phi phenomena or ‘pure’
movement and Beta movement, or ‘apparent’ movement
is very confused indeed [9], both require attention in
order for cognition to take place. Further, although Beta
movement has been divided into short-range and longrange apparent motion the definitions of what these
terms actually constitute when translated to cognitive
perception is still opaque in most literature with, as
Joseph and Barbara Anderson remark in their seminal
paper The Myth of Persistence of Vision Revisited, little
consensus on the perceptual differences between Phi
and Beta [10]. The Diasynchronoscope explores the
two physiological functions (Phi and Beta) in a new way,
Design and semantics of form and movement
63
by including synchronization with audio and embodied
phenomenological observation. The experiment exploits
Gestalt laws and human cognition shortcuts to achieve
an illusion of movement through selected attention.
Put simply, a personal schema is what makes us all
respond differently, eg. One man’s famine is another’s
feast. Each person’s upbringing and experience means
we cannot help but bring our ‘personal baggage’ along
with us when we encounter any phenomena. Personal
schemas are shaped by ‘relevance’ [11], and relevance
can be defined in terms of a cost-benefit analysis
weighting effort against effect. The more information
processing effort it would take to bear x in mind in
the context of y, the more costly it would be, giving an
equation of High cost = Low relevance. It is not that
we pre-compute just what effort and effect would be
involved in considering this or that connection/belief
before picking the most economical one, but that we
have evolved unconscious psychological mechanisms
that have much the same result [12]. Salience is what
makes us apply attention to a stimulus, and salience can
be either personal, or an evolutionary imperative. For
any set of sensory data, there are generally multiple
interpretations possible, but we need to rapidly
come to a perceptual conclusion if we are to spot an
aggressor or potential food source: Better to duck
a fleeting shadow than be hit by a rock. If a common
narrative were to be interpreted from an abstract
movement it would indicate salience taking precedence
over personal schema, but this could only be tested
in an embodied way. By combining Gestalt principles
with media theory and cognitive neuropsychology we
have developed a new form of kinetic perception, with
novel implementations of sequenced images in time
unmediated by the camera or screen. It is the first step
in a new way of looking at attention in perception,
communication and action that poses the question:
Does the unconscious perception of movement provoke
a stronger emotional response than image recognition,
particularly if synchronized with sound?
2 Methodology
Because the Diasynchronoscope used projected light, it
was essential that the sculpture took place in a blacked
out space. The creators achieved this by erecting a
2200 sq mm gazebo indoors and covering the frame
64
Design and semantics of form and movement
with heavy black velvet. After some experimentation it
was decided that the optimum material for creating the
objects was white medium density polystyrene cut to
shape. The polystyrene had to be painted black so that
it would not be visible to the viewer in the blacked-out
space. The lit result was pleasingly hard to register as
material, imbuing the finished artifact with a level of
ambiguity (figure 2).
Fig. 2. A single
polystyrene object
from the sequence.
As no object can
really be construed and labeled as ‘abstract’, an early
challenge was to design objects that deliberately
emphasized movement and were less likely to be
culturally loaded than recognizable. This led to
considerable debate about the character/object.
We finally settled on a basic ‘hero’ object/block
measuring 60 x 100 x 30mm. Aside from practical
considerations, we were very clear that we did not
want anything that resembled either a ball or a 3D
pixel, i.e. cube. As when we developed the movement in
virtual 3D, we decided we wanted to include a level of
animated squash and stretch.
Because the artwork required the dimension of time to
be translated into spatial dimensions, the work could
not have been made without using 3D visualization
(figure 3). In fact, all animation requires the animator
to translate the dimension of time to that of space but
this phenomenon is rarely made visible to the consumer,
who only receives the data when it is translated back
into time. Animators automatically switch between
time/space when animating, always aware of how,
for instance, higher speeds translate to greater gaps
between objects in space. Because the objects were
solid and the movement was so rapid, it would have
been very difficult to have accomplished a reasonable
animation without testing and adjusting it in 3D first.
The creators first worked out the sequence in Autodesk’s
3DS Max at 12 frames per second (fps). The normal rate
of film in the UK is 25 fps, but there is a long tradition
in model animation and claymation of animating on 12
fps and holding each frame for two frames to make up
a second’s worth of animation. This is clearly less labor
intensive and we trusted it not to look too jerky if the
animation held an optimum path. We also could not ‘overlap’ action frames, so we knew that the animation would
have to consist of fast movement and that we could not
stay on one plane but wanted to explore all axes of the
3D space. Because of space restrictions, the animation
was kept very simple and short using just 23 objects that
translated into exactly two seconds of animation (objects
10 and 17 were held for 3 frames) (figure 4).
Fig. 3. 3DS Max mock-up.
The virtual objects were recreated in polystyrene and
hung in place using fishing line. This was fiddly and
required each block to be tethered to the top, bottom
and side of the gazebo to achieve stability (figure 5).
Fig. 4. Viewer’s perspective in virtual 3D.
Fig. 5. Objects ‘floated’ in space on fishing line and
It was necessary to avoid hanging the three dimensional
objects perpendicular to the projection as only one
side would be illuminated. The optimal positioning was
where three sides are illuminated, emphasizing the
3d effect, but this was not possible for every block.
If we could have accommodated two projectors, we
could have used one as a ‘key’ light and the other
to act as a ‘fill’ light. When we were happy with the
block positions, we mapped the light projection using
a sequence of mattes animated in Adobe After FX.
This had to be done with much precision as each pixel
became crucial; too few and the definition of the object
was lost, too many and the light spilled off the object
and bled onto the black backcloth dividing attention.
Sound effects were added to the movement and relayed
via stereo speakers. The sound was designed to draw
attention and synchronize but not to provide a solid
identification for the objects as objects or characters.
The restrictive space meant we could only
accommodate one viewer at a time, and that viewer
was placed on a particular spot so that they might best
view the work’s 3D form. Each viewer was shown the
sequence three times, interviewed, and then asked to
turn around so they might experience the whole art
object at once (figure 6).
suspended on black sticks.
Design and semantics of form and movement
65
Fig. 6. Picture of the whole artwork.
For this first phase of the project, the participants were
drawn via open invitation to friends and colleagues.
The participants (whose ages ranged from 25 - 81) were:
Juliet - a Jazz singer (and tennis player),
Janet - a tennis player, Ruth - a retired doctor,
Clem - a graphic designer, Peter - a web designer,
Rob T - an actor and writer, Rob M - a film director,
Livia - a music therapy teacher, Joseph - a painter
(and tennis player), Lucie - a lecturer in digital media.
Individual responses to the artwork were recorded on
film at the time of viewing, then the participants were
asked some deliberately open questions such as: ‘What
are your first impressions?’ and ‘What was going on?’
This was followed by a filmed plenary discussion.
3 Findings
Generally the findings of audience response could be
divided into three areas: response to the animation;
response to the holistic artwork; and influence
of personal schema. This last proved particularly
interesting as it yielded the strongest sense of coauthored narrative, and was most satisfying for
both artists and participant. It also shed interesting
indications on the way personal schemas or memories
are laid down, offering a rich seam for further research.
3.1 Response to the Animation
The two-second animation was played three times
(http://vimeo.com/32451411gestalt/diasynchronoscopeanimation). Participant responses were recorded as
they watched the animation and they were then asked a
number of questions. The animation evoked a variety of
responses from ‘Is that it?’ to ‘wonderful!’ to laughter.
All participants read the movement as movement and
66
Design and semantics of form and movement
were content that it felt ‘natural’. The majority felt that
the action was that of a ball bouncing. The animation
had been designed to give the impression of a swing
and bounce, with sound that served both character
and object, so this was what we expected. Where
participants did not agree was whether the object was
an object or a character, i.e. what it was or whether it
was a single object or a multiple one. However this did
not seem to cause confusion in most. To some it felt
very like a digital animated experience and to others
it felt very different and ‘real’. This could be attributed
to the nature of the embodied experience as some
moved their heads more than others and so would have
achieved a sense of depth and a level of parallax. The
people who moved most were non-digital artists and
those who played tennis.
3.2 Response to the Holistic Artwork
Participant responses to the revealed whole artwork
were also recorded (figure 6). Responses to this were
extremely positive, eliciting comments such as: ‘Like a
dream’, ‘Tactile, I want to touch it’, ‘Magical!’, ‘Looks
larger than life’, ‘Fabulous!’, ‘Oh yeah!’. The unlikelihood
of the objects hanging in space without visible support,
and the new knowledge of the previous movement
made the holistic artwork a satisfying experience for all
of the participants. One question we asked everyone
while they viewed the holistic artwork was: ‘Would
you describe what you are seeing as a single object
or multiple objects?’. Responses were evenly divided
with four of the participants choosing it as clearly
representing a single entity, and four being equally
emphatic that the piece consisted of multiple objects.
Two participants were happy to conclude that it was
both multiple and single at the same time without
seeming to find this paradoxical.
3.3 Influence of Personal Schemas
The artwork was a sincere attempt to isolate movement
by suppressing the contexts of shape, mass and color.
Thus the shape was a block constructed of no immediate
contextual dimensions, and of no indicative material.
Although the mass had continuity, the size was indeterminate as there was no reference point to gauge size
against. The color was suppressed, deliberately neutral
in context and consistent. However it became rapidly
obvious that all participants imbued the experience with
interpretations drawn from their personal schemas;
often holding contradictory beliefs at the same time
with apparent ease.
Juliet and Livia, both from musical backgrounds were
primarily drawn to the audio, using their interpretations
of the sound as a springboard for their interpretation
of the whole artwork. Peter and Lucie, both working
in the digital domain, drew digital interpretations first
of all, before expanding their experience to other
areas. Joseph, Janet and Juliet who were all regular
tennis players interpreted the object clearly as a
tennis ball despite the contradictions of it being a) a
block and b) having the sound of a scrabbling creature.
The movement of the objects elicited a number of
ball responses from the participants, with Ruth (aged
81) interpreting the movement as being ‘just like the
Dambusters’ film. The synchronous nature of the
sound and the ‘cartoon’ quality of squash and stretch
elicited laughter from Rob T and Rob M (both used to
performance). The designer Clem was much drawn
to the holistic art object rather than the movement.
Several of the participants described the object as
bringing to mind a ‘kitchen sponge’. This could be
because of the ‘squashy’ nature of its mass or be
indicative of the prevalence of domestic associations in
their lives. Many described the final holistic object as
resembling a dinosaur spine, and there seemed to be
no problem for four participants to interpret the whole
artwork as ball, sponge and dinosaur spine within the
same contextual description.
These interpretations proved revealing of how varied
personal schemas are in individuals when they approach
stimuli and led to a deeper understanding of the real
nature of narrative co-authorship. Because of the
abstract nature of the artwork, it also shed some
interesting light on how memories may be laid down to
create our personal schemas. What is highly salient and
suggestive to one person may well be of no interest to
another, but what triggers saliency when the stimuli
is abstract?
4 Discussion
4.1 Animation: Gestalt, Phi, Beta and
Apparent Motion
The four separate parallel pathways of perception in
our visual cortex transmit signals devoted to form,
color, movement and depth and the animation was
designed to use Gestalt laws and animation principles
(such as arcs, and squash and stretch) to achieve
continuity of movement in all of these pathways.
Animators, magicians and film makers all exploit how an
audience will endow continuity of perception to objects
that displace within corresponding graded constraints of
shape, color, motion and mass, particularly if they move
in arcs and have verifying synchronous sound.
The Gestalt laws grew from experiments showing
that we group visual objects (in space) according to
similarity, pragnanz, proximity, continuity and closure.
It also demonstrated that auditory stimuli are grouped
(in time) according to similarity, proximity and closure.
A closer look at the Gestalt laws reveals how important
they were to underpinning the artwork in all of its
forms in time and space:
r5IFMBXPGTJNQMJDJUZTVHHFTUTUIBUTJNJMBSUIJOHT
tend to appear grouped together, the objects within
the artwork were of sufficiently similar nature.
r5IFMBXPGQSBHOBO[JTTPNFUJNFTSFGFSSFEUPBT
the law of simplicity. This law holds that objects in
the environment are seen in a way that makes them
appear as simple as possible.
r5IFMBXPGQSPYJNJUZIPMETUIBUUIJOHTUIBUBSF
near each other seem to be grouped together.
r5IFMBXPGDPOUJOVJUZIPMETUIBUQPJOUTUIBUBSF
connected by straight or curving lines are seen in
a way that follows the smoothest path.
r5IFMBXPGDMPTVSFSFGFSTUPUIFXBZPVSCSBJOT
often ignore contradictory information and fill in gaps
in information to group things together if they seem
to complete some entity. We do not find it easy to
attribute randomness to stimuli; we cannot help but
search for a pattern.
One question that occurred to the writers was: Could
dynamics enable or disable a viewer’s comprehension
and assimilation of data? In an earlier experiment,
Design and semantics of form and movement
67
the creators of the project tested the idea of graded
constraints by attributing varied colors to the objects,
thus disrupting the law of simplicity (figure 7). It was
immediately apparent that even though we were using
objects of great similarity, the color changes were too
great for their movement to be held as continuous.
The objects lost their grouping and became separate,
even though they held constant in terms of shape,
motion and mass. It would be true to say that any
large change in the constraints of each pathway would
also break the idea of Gestalt grouping. For instance
introducing random sizes would be similarly disruptive.
Fig. 7. Random adjacent color changes across a
wide range flouted the Gestalt law of simplicity
so that the impression of continuous motion
broke down.
This suggests that all four perceptual pathways of form,
color, movement and depth are equally important,
and raises the idea of further experimentation that
incorporates synchronized sound and vision.
The phi movement of Wertheimer has been proved
not to be a reasonable explanation for why we
interpret film as motion [10], [13]. This leaves only Beta
movement as a possible explanation. Beta movement
is apparent movement caused by luminous stationary
impulses. Multi-element or closely spaced displays may
be mediated by the same mechanisms as real motion,
while more widely spaced displays (such as the usual
two-flash displays used to demonstrate apparent
motion) involve a different type of processing. These
two types of processing are termed short-range and
long-range apparent motion, respectively. The four
perceptual pathways are processed in parallel, but
the movement pathway carries data more swiftly
68
Design and semantics of form and movement
to our visual cortex and according to the discoveries
of neurologists such as Livingstone and Hubel and
Zecki [13], [14] our visual systems employ two different
computational strategies for processing closely spaced
stimuli and widely spaced stimuli. This leads
to Anderson and Anderson’s intriguing conclusion:
If we viewers process the motion in a motion picture
the same way we process motion in the real world,
then we must ask how we process motion in the real
world. The short answer to this question is that we
process movement in active meaning-seeking ways.
We rapidly sample the world about us, noting the things
that change and the things that do not change. We turn
our heads for a better view; we move left or right to
gain additional information provided by a different angle.
We move closer or farther away. We actively seek more
information about things that interest us. [10]
It is for this reason that we regard the
Diasynchronoscope project as scientifically worthwhile.
4.2 The Satisfaction of Art
Is the Diasynchronoscope project a work of art or
a scientific experiment?
Margaret Boden defines creativity as ‘the ability to
come up with ideas or artefacts that are new, surprising
and valuable’ [15]. This clearly allows for scientific
experiment to be both creative and interpretable as Art.
Part of the fascination for participants in experiencing
the Diasynchronoscope was that it was viewed as an
embodied experience that allowed for individual opinion
and interaction. Because the artwork had different
points of access (as a time-based study and as a static art
object that held the time-based object inherently within
itself), the two forms informed each other. Paradoxically,
the holistic artwork could be conceived as though the
participant is a camera on a very slow shutter speed,
receiving two seconds of data in a single moment. This,
coupled with the embodied 3D element, is a new and
surprising sensation. The artists would also like to stake
modest claim for this sensation being potentially valuable
in exploring dynamic cognitive perception.
4.3 Attention, Saliency and Personal Schemas
Change can only be perceived through attention,
and attention can intrude on conscious perception
because attention is drawn to an unexpected stimulus
on an involuntary basis. Thus a person suffering from
arachnophobia will spot a spider in a room before
anyone else, as their awareness antenna for such a
stimulus is always near the surface, and it is most
pertinent to them. It is obvious that personal schemas
are highly influential in shaping our attention through
salience. Salience brings a stimulus to consciousness, and
immediately we set about selecting associations from
our personal schema to interpret the stimulus. Because
the stimulus was abstract and open to interpretation, it
seems likely that we store memories dynamically with
confirmatory sensory input increasing their likelihood of
truth. This is at odds with the frequently used analogy
of our minds as indexical filing cabinets; How likely is
a static mental model of anything if we can associate
so freely, riffing mentally between dinosaur bones,
sponges, balls and bombs in consecutive moments?
Memories are not stored indexically but dynamically,
i.e. when movement is interpreted in an abstracted
manner, we access mindful connotations that are not
drawn upon through taxonomy, but through metaphor
and metonym. Neurologically, we need to access events/
objects with a degree of intrinsic abstraction, so we
store memories dynamically. Hence art, poetry and
metaphor and metonym are simple manifestations of
the way our brains work. This tallies with the thesis of
Neuroaesthetics, namely that artistic models that echo
the psychophysical architecture of the mind are best
for depth of communication and qualia of experience
[14]. For art and poetry, metaphor and metonym are
not found in the concrete, but in the ‘gaps between’ the
concrete, and these gaps are filled by audiences acting as
exegetes. Exegetes who, despite being guided by artists
and sharing cognitive neural architectures tend to take
the same perceptual shortcuts, consulting our personal
schemas immediately and unconsciously, and bringing
forward contextual associations which have been stored
dynamically in our minds.
Because the work is performative yet entirely replicable
despite being unmediated by screen, it offers a new and
embodied way of exploring the physiological nuances
of Gestalt law and perceptual cognition.
The intention is to use this experimental artwork as a
springboard for a more ambitious artwork that removes
some of the restrictions of time and space inherent in
the piece.
References
[1] Zacks, J.M., Speer, N.K., & Reynolds, J.R. (2009). Situation
changes predict the perception of event boundaries, reading
time, and perceived predictability in narrative comprehension.
Journal of Experimental Psychology: General, 138, 307–327.
[2] Mack, A., & Rock I. (2000). Inattentional Blindness. London: MIT Press.
[3] Gregory, R.L. (1997). Eye and Brain: The Psychology of
Seeing. (pp. 224-5) Oxford: Oxford University Press.
[4] Hitchcock, A. (1958). Vertigo. USA: Paramount Pictures.
[5] Marchant, P., Raybould, D., Renshaw, T., & Stevens, R. (2009).
Are you seeing what I’m seeing? An eyetracking evaluation of
dynamic scenes. In Proceedings of Lansdown Centre Symposium,
Computer Arts Society. Completing the Circle. www.cea.mdx.ac.uk.
[6] Arnheim, R. (1964). Art and Visual Perception: A Psychology
of the Creative Eye. Berkeley and Los Angeles: University of
California Press.
[7] Hubbard, T.L., & Courtney, J.R. (2006). Evidence suggestive
of separate visual dynamics in perception and in memory. In L.
Albertazzi, (Ed.), Visual Thought: The Depictive Space of the
Mind (pp. 71-98). Amsterdam: Benjamins Publishing Company.
[8] Wertheimer, M. (1925). Über Gestalttheorie at an address
before the Kant Society, Berlin, December 7, 1924, Erlangen,
Reprinted (1997). New York: Gestalt Journal Press.
[9] Pizlo, F.J. (2000). Phi is not Beta. Retrieved November 21,
2011. www.psych.purdue.edu/Magniphi/PhiIsNotBeta/phi1.html
[10] Anderson, J., & Anderson, B. (1993). The myth of persistence
of vision revisited, Journal of Film and Video, 45(1) (pp. 3-12).
[11] Boden, M.A. (2010). Creativity & art: Three roads to
Carol MacGillivray,
Bruno Mathez
Goldsmiths
University,
London UK
4 Further Research
surprise. Oxford: Oxford University Press.
When principles of design replicate principles of
thought, the act of arranging information becomes an
act of insight. [16]
The first phase of the Diasynchronoscope project has
yielded results that appear to confirm some theories
of cognitive perception such as the design theory
of neuroaesthetics and the importance of personal
schemas in co-authored narratives. However in some
ways it is clear that the project needs to expand if
it is to answer other questions raised in this text.
[12] Sperber, D., & Wilson, D. (1986). Relevance:
Communication and Cognition. Oxford: Blackwell.
[13] Livingstone, M.S., & Hubel, D.H. (1988) Segregation of
form, colour, movement and depth: Anatomy, physiology and
perception. Science 240: 740–749.
[14] Zeki, S. (1993). A Vision of the Brain. London: Blackwell Science.
[15] Boden, M.A. (2010). Creativity and Art: Three Roads to
Surprise. Oxford: Oxford University Press.
[16] Tufte, E.R. (1997). Visual Explanations: Images and
Quantities, Evidence and Narrative, Ct. Graphics Press.
Design and semantics of form and movement
69
Edgar Rodríguez Ramírez
[email protected]
Semantics of surprise: Strategies
for eliciting surprise through design
Abstract
Relevance to Practice
This paper reports on the role of semantics in the
strategies industrial designers use when attempting
to elicit surprise. Surprise is the emotion people
experience when they appraise a stimulus as “novel”.
While “novelty” is one of the main factors that
designers can bring into a product, little has been
reported about what strategies designers use when they
intend to surprise.
Thirty senior representatives from influential design
organisations were interviewed with the intention
of uncovering strategies that designers use in their
process. The analysis of the responses suggests the
strategies are often connected to the semantics of the
product, and that a common factor in order to elicit
surprise is to challenge the expected semantics of an
object on different levels: social, cultural and emotional.
The suggested strategies are analysed in comparison
with current literature. The research suggests that the
strategies suggested in this paper represent explicit
ways in which designers attempt to elicit surprise. The
paper concludes suggesting that further research should
be carried out in a research through design approach to
uncover further strategies that designers use implicitly
and did not explicitly mention during the interviews.
The paper suggests a list of strategies that industrial
designers use when addressing the semantics of
products in order to elicit surprise. The strategies can
in turn be used by students and practicing designers.
Keywords
Industrial design, surprise, semantics.
70
Design and semantics of form and movement
1 Introduction
Semantics has been investigated and used to offer
meaning to complex to use objects: it helps designers
indicate how objects should be used [1], [2]. This has
been criticised [3] as producing objects that, while
offering an interaction in which people easily understand
what they need to do, it also produces objects that lack
novelty, and therefore a sense of surprise. On the other
hand, surprise breaks expectations, which implies that
sometimes it may go against usability, as people cannot
employ their expectations about how an object should
be used.
Surprise has often been used in design as an element
for eliciting an experience of amazement [4], to create
a sense of novelty [5], “to increase interest or prolong
the attention value of a product” [6], to “engage the
user” [7], and to elicit curiosity and further exploration
of the object [8]. Designers have often relied on their
intuition and informed experience to produce objects
that elicit experiences through the appearance and
interaction that their designs offer. However, little has
been reported about how designers approach projects
3 Results: Strategies to Elicit Surprise
The results from the situational analysis of the
interviews suggest strategies that designers explicitly
use when intending to surprise. They are divided into
two main sections: strategies used as a motivation
for designing; and strategies used during the act of
designing. The paragraphs below indicate whether the
projects address semantics at the cultural, social or
emotional levels, or a combination of the three.
Table 1.
Situational map
constructed from
when they intend to surprise and what strategies they
may follow and in particular, what the role of semantics
is in eliciting surprise.
the analysis of
interviews showing
2 Methodology
the constructed
There were 30 interviews with senior representatives
from design organisations. All interviewees were
senior designers, design managers or CEOs of design
companies. The interviews were semi-structured and
in-depth. The interviews were carried out according to
Victoria University of Wellington’s Ethics Committee
Approval number 16059. All interviewees agreed to
have their name published, as it would be impossible
to present their projects and maintain anonymity.
The interviews were semi-structured [9], in-depth [9],
[10] and lasted between 40-60 minutes.
The analysis of the interviews followed a Grounded
Theory approach with a postmodern turn, namely
situational analysis [11], [12]. Firstly, the interviews were
transcribed using a selective transcription approach
[13], [14]. The transcriptions were printed and used
for coding and memoing [12], [15]. There were several
stages of interviews following theoretical sampling and
opposite sampling approaches. The codes and memos
were used as raw data to build situational maps.
The questions necessary to build situational maps
are: “Who and what are in this situation? Who and
what matter in this situation? What elements make
a difference in this situation?” [12]. Once the maps
were constructed, they were used to find and describe
relations between the important elements in the
situation (Table 1). In particular, the analysis focused
on creating “positional maps”, which help develop
“strategies for plotting positions articulated and not
articulated in discourses” [12].
strategies.
3.1 Strategies Used for Observing the World:
Finding the Potential to Elicit Surprise as a
Motivation for Designing
The strategies found in this category refer to the
observations designers have of their world that inspire
design projects. There were several ways in which
designers found relevance in their worlds, and these
could be seen as “observations beyond my personal
experience”, “observations of my direct experience”,
and “issues in my world”. These observations have
the common denominator of having such a strong
impression on the designers that motivate them to
start and carry out full design projects, from concept
to manufacture, based on such observations alone.
Observations beyond my personal experience. This strategy
refers to observations that designers have of how other
people use objects, what they do in their daily activities,
what they need. There seems to be a difference between
observations of people close to the designers, such
as family and friends, and strangers, which means that
this strategy relates to semantics at the social level.
Observations of family and friends seemed to be more
meaningful for the designers. For instance, Matali Crasset
mentions how she designed a range of furniture based on
observations she had of her friends. The range is called
Les Amis de Matali (Matali’s friends). One of the projects
develops from the observation that some of her friends
often visit Paris, where apartments are small, and the
friends they stay with often lack appropriate furniture for
sleeping guests. In this case, Crasset designed furniture
for her friend Jim who often visits her in Paris. Quand
Jim monte à Paris (when Jim goes to Paris) is a structure
of everyday furniture that could become a small private
space and bed in a small apartment (Figure 1). It attempts
to elicit surprise by normally being a piece of furniture for
storage, and becoming bedroom furniture when Jim visits.
Design and semantics of form and movement
71
Fig. 1. Quand Jim monte à Paris by Crasset. Image
Fig. 2. Lamp On/Off by Santachiara, Meda and Raggi
courtesy of Matali Crasset.
for Luceplan. Images courtesy of Denis Santachiara.
Bill Webb mentions the example of designing an alarm
clock for Barbie based on the observations he had of
her nine year old daughter. He mentions:
I have a daughter, inevitably I start to see through her
eyes and what she might want and understand. Kids’
products or toys are challenging, trying to get into
their mindsets does not come as naturally. You want
to keep the energy alive. You want to be as passionate
about you doing [sic] a Barbie box as you are when
doing a Nike watch (B. Webb, personal communication,
October 6, 2008).
Webb talks about seeing the world through his
daughter’s eyes in order to keep a high level of interest
in the project as well as to better understand the
potential users. In order to achieve this, he turns to
people close to him, as Crasset turned to her friends.
Observations of my direct experience. Designers also
described finding relevance in their worlds through
“personal experience”. In this case, it is not an
observation of something happening to someone else.
It is something that happens directly to them. Personal
experience can be at the behavioural level “how I
use objects”, at the cognitive level “how I understand
objects” or at the emotional level “how I emotionally
experience objects and the world”.
Off by Santachiara (Figure 2). The lamp is turned on
or off by shifting its weight from one side to the other.
The lamp attempts to surprise people by not having a
standard switch. The design of the lamp originated as a
response to accidentally knocking over the bedside lamp
when falling asleep and trying to turn it off. The same
kind of gesture can in this design turn the lamp on or
off, without knocking it over.
Behavioural level: How I use objects. At the behavioural
level, designers mention that how they use objects
themselves is sometimes the deciding factor as to how
to approach the design project. This category refers
specifically to how they interact physically with objects.
This strategy refers to semantics at the cultural level, as
the designs attempt to elicit surprise by challenging the
ways in which people have culturally learned that the
physical interaction with objects should be like.
An example of “how I use objects” is the lamp On/
72
Design and semantics of form and movement
Emotional level: How I emotionally experience the world.
Bill Webb offers a description of “how I emotionally
experience the world”. He mentions that:
designers are pretty emotional people, there is a
lot of passion in the work that we do that hopefully
comes through the work that we do… Research can
be very instinctual to us as designers being part of the
culture, whether it’s sports, consumer electronics,
housing, gaming… Just because we live and breathe
as much as we can every day, we have our point of
view as consumers and designers (B. Webb, personal
communication, October 6, 2008).
This strategy is obviously within semantics at the
emotional level. Ingo Maurer offers a more specific
example of how an emotional experience of the world
can lead to a specific design. Back in 1975, Maurer was
in Lake Constance in Germany. He saw a fisherman
hauling in a net full of fish, and the drops of water falling
from the net looked like tears. There was a strong light
from the morning sun reflecting on the drops. He kept
this observation for 35 years and designed the Lacrime
del pescatore (Fisherman’s tears) installation (Figure
3). The design consists of a series of layered nylon nets
with 350 crystals illuminated by a halogen light bulb.
The nets move slowly and give a sense of breathing. The
illuminated crystals represent tears. Maurer admits that
the installation was very close to his own emotions,
Fig. 3. Lacrime del Pescatore by Ingo Maurer.
Fig. 4. The Euro Condom by Ingo Maurer.
Images courtesy of Ingo Maurer.
Image courtesy of Ingo Maurer.
and his intention was to express those emotions. He
even mentions that the design “was a bit on the edge,
it may be seen as kitsch, but I am comfortable with
it because the final result, including the movement
and composition, express what I felt 35 years ago” (I.
Maurer, personal communication, September 21, 2009).
Examples of projects that are the result of
experimenting and where the designers intend to elicit
surprise come from Front Design. For instance, in the
“Design by Animals” project, Front put different animals
to the task of affecting their material environment. In
one experiment, Front mapped the movement of a fly
around a light bulb in order to design a lamp (figure 5).
Issues in my world. “Issues in my world” involves
mainly current social issues that are important for the
designers and subsequently sits at the social semantic
level. Crasset mentions that she often develops
ideas from reading newspaper articles (M. Crasset,
personal communication, July 22, 2006). Maurer offers
an interesting example of this category. Maurer says
that for him the incandescent light bulb is “the most
wonderful object made by human beings” (I. Maurer,
personal communication, September 21, 2009). The law
introduced in Europe in 2009, which banned frosted
incandescent light bulbs, irritated Maurer. His team
designed the Euro Condom (figure 4) in response. The
design involves a silicon cover that diffuses light just as
the frosted incandescent bulbs do. The design intends to
make a statement about an important issue in the world
according to the designer.
Gaining design knowledge through surprising experiments.
A category beyond observations for intending to
elicit surprise as a motivation for designing is through
experimenting. The designers set up the experiments,
but they do not know what may come out of them. The
results of the experiments are then modified in order
to make them final functional products. This strategy
relates to semantics at the cultural level, in particular
as the designers are surprised themselves by producing
findings that challenge their cultural assumptions of how
objects ought to be.
The design process normally involves some degree
of experimentation, where the designer sets up
experiments with materials, textures, colours,
interactions or other media and cannot fully anticipate
the final result. For instance, the lamp Titania by
Alberto Meda and Paolo Rizzato incorporates
interchangeable coloured filters. The light from the
lamp changes colour when the user changes a filter
(figure 6). Meda mentions that the initial intention in
designing Titania was to explore the form of a plane’s
wing (A. Meda, personal communication, September
15, 2006). They found the property of changing the
colour of the light from experimenting with different
materials attempting to decide which colours of plastic
they should offer the lamp in. They found that including
only one coloured sheet of plastic was enough to
change the colour of the whole lamp, which was an
unexpected finding. The difference with projects such as
the ones described above by Front Design is that Meda
did have an intention to design a lamp with particular
characteristics and he found a particular feature –
changing colour through changing filters– during the
design process. On the other hand, Front Design did
not know what the actual object or formal result would
be at the start of setting up their experiments. The full
result of the experiment becomes an object, rather
than a feature of the object.
Design and semantics of form and movement
73
Fig. 5. Fly lamp by Front Design.
Fig. 6. Titania lamp by Alberto Meda and Paolo Rizzato.
Image courtesy of Front Design.
Images courtesy of Alberto Meda.
3.2 Strategies Used During the Designing Stage
of a Project: Applying my Informed Experience
As opposed to “strategies at the observation stage
of a design project”, this category does not focus
on observations by the designers; it represents the
synthesis of their research into a designed object. It
occurs during the design process, as opposed to being a
motivation for starting a project. The process has been
described as design synthesis [16] and the designers’
spark [17], [18].
It is important to note that some designers seem
unable to verbalise the properties of objects that will
surprise. For instance, Laura Polinoro from Alessi
states that what they look for in new projects is “to
create a resonance, an affinity”, and that “objects need a
personality, an identity” as well as “a semantic value to
create emotion” (L. Polinoro, personal communication,
September 25, 2004). It is impossible to tell how to
achieve this, as every object needs to be new, and
therefore there are no recipes to achieve such goals.
As creative director, Polinoro can only tell that such
goals have been achieved when she sees the finished
project, but she is unable to explicitly say what those
characteristics are.
The smart doubling of things. Morenstein asserts that
“it is ‘the smart doubling of things’ that I think people
discover, are surprised by, and enjoy” (personal
communication, November 10, 2008). For instance,
Fuseproject was involved in the branding and product
design for the Y water bottle (figure 7). Morenstein
states that “it is a very emotional product, and
that it had to appeal to children” (J. Morenstein,
personal communication, November 10, 2008). The
clients wanted to brand a healthy drink with special
ingredients for bone and muscle growth. They wanted
74
Design and semantics of form and movement
Fig. 7. Y water bottle by Fuseproject.
Image courtesy of Fuseproject.
Fig. 8. The way in which the Y water bottle needs to be
held implies a playful interaction. Image courtesy of
Fuseproject.
to elicit surprise through the aesthetics of appearance
reminiscent of the shape of bones through the form
of the bottle.
In the same project, Fuseproject approached aesthetics
of interaction through the form in which the bottle
is held, which offers a playful interaction for children
(figure 8). The design goes beyond an aesthetic
Fig. 9. The design of the Y water bottle becomes a toy
after consumption. Images courtesy of Fuseproject.
Fig. 10. Jawbone Bluetooth headset by Fuseproject.
Breaking expectations through challenging assumptions
of appearance. This strategy refers to projects in which
designers change what could be seen in the industry as
an assumption of how a specific object should look, and
refers to a culturally-learned visual-meaning. It seems
that the more established an assumption of appearance,
the more surprising a challenging design can be.
Morenstein mentions the Jawbone Bluetooth headset
(figure 10) as an example of a project attempting to
challenge assumptions of appearance. He states that
previous Bluetooth headsets looked like “blobs” as
they were designed with the premise that “if we are
going to design something for your face, let’s make it
organic, so that it’ll blend in” (J. Morenstein, personal
communication, November 10, 2008). Fuseproject’s way
to address this issue was to design beyond an expected
assumption: that a Bluetooth set needs to look organic.
By addressing such assumptions and expectations, the
designers attempted to elicit surprise. Morenstein adds:
“I don’t put a chicken wing on my face, that’s organic.
We said: things people wear on their face are jewellery.
They have to have details and materials selection…
that was [sic] complementary to the way that I look”.
Image courtesy of Fuseproject.
Fig. 11. Levitating lamp; Fig. 12. Separating drawers;
Surprising through magical interactions. There were at
least two studios that explicitly claimed to attempt
to elicit surprise through magic and used cultural
expectations of the objects’ meaning in order to
surprise. Front Design has incorporated magic in an
attempt to elicit surprise. For instance, they developed a
range of products in collaboration with magicians.
The designs are created with what would seem
“impossible characteristics that defy the laws of nature”
[19]. The results include a lamp that levitates (figure 11);
a chest of drawers that separates and floats away (figure
12); and another chest of drawers with fronts that
disappear, revealing an empty inside (figure 13).
Fig. 13. Disappearing drawer by Front Design.
Images courtesy of Front Design.
interaction during the consumption of the drink and
includes literal playfulness by using a biodegradable
rubber band that serves to join bottles together (figure
9). This feature incorporates an environmental factor
that is achieved through creating an appealing use
after consumption. Fuseproject claims that the use
after consumption is only possible because the object
produces an emotional connection with children.
Surprising through using archetypes in unexpected
contexts/objects. Under this category, designers used
an archetype for an object and translated it into
unexpected situations. Pezzini’s design of the moving
stool reflects the use of an archetype –a bucket– used
in a different context: as a stool (figure 14).
Designers have often used this strategy in the design of
lamps. The archetype used is the form of a trapezoid to
signify a lamp. For instance, Santachiara’s workstation
Design and semantics of form and movement
75
Fig. 14. Moving Stool by Pezzini.
Fig. 15. Workstation/lamp Angel by Denis Santachiara
Image courtesy of Pezzini.
for Naos. Image courtesy of Denis Santachiara.
desk “Angel” includes doors that when closed make
it look like a giant lamp (figure 15). This project also
connects to the strategy below “Surprising through
unexpected scale”. The design includes another identical
but smaller lamp inside the bigger one, as a medium to
emphasise the changes in scale.
plays with words to offer exactly that: a long chair
(figure 18). The design plays with the scale of the length
of the chair.
Sismo Design from Paris often uses archetypes in
different contexts. For instance, they used the form of
a wine glass to design the vase Verre (“glass”, figure 16).
They also used the shape of a key as an element to turn
switches on and off (figure 17).
The use of an archetype in a different context can
sometimes allow only a very literal interpretation:
once the new connection is understood, there may be
very little more about the object that is interesting. As
mentioned in the introduction, the emotion of surprise
is very short lived and may be a single experience.
Fox-Derwin suggests that some projects are “oneliners”, which means that they hold an interesting
feature, but once that feature is experienced there is
not much more to the design (E. Fox-Derwin, personal
communication, May 26, 2010). To express the idea
of a “one-liner”, Fox-Derwin refers to the “crushed”
ceramic cups by Rob Brandt that imitate what a crushed
plastic disposable cup would look like. In her research,
she explores the idea of incongruity and tries to find
ways to take designs beyond “one-liners” [20].
Surprising through unexpected scale. Designers referred
to the use of scale for intending to surprise people. For
instance, Santachiara’s workstation/lamp Angel uses
the form of an archetypal bedside table lamp on a much
bigger scale (figure 15). Sismo Design uses unexpected
scale in the form of changing the proportions of an
object. The design of their Chaise longue (long chair)
76
Design and semantics of form and movement
4 Conclusion
This paper serves as a snapshot of how designers use
semantics to intend to elicit surprise. This paper offers
specific strategies that are directly applicable to design
projects. The strategies are specific enough to be
workable, but general enough not to dictate recipes
that limit creativity.
4.1 Attempting to Elicit Surprise through
Interaction as a Motivation for Designing or
During the Design Process
The ways in which designers apply their informed
experience could be put into two categories: as a
motivation for designing or as strategies to use during
the design process. The first category refers to the
inspirational observations of the designers’ world.
These could include observations of other people’s
experiences, or observations of their own experience.
When the observations related to the designer’s own
experience, projects were discussed that came from
a behavioural level: how the designer uses objects;
from a cognitive level: how the designer understands
objects; or from an emotional level: how the designer
emotionally experiences the world.
The second category refers to strategies through which
designers intend to elicit surprise through interaction.
This section reported on five specific strategies:
smart doubling of things; breaking expectations
through challenging assumptions of appearance; use
of archetypes in unexpected contexts/objects; use of
magical interactions and use of unexpected scale. It
should be remembered that these categories relate only
to the findings from the Grounded Theory analysis of
Fig. 16. Vase Verre; Fig. 17. Switches Les reférents; Fig. 18. Chaise longue (long chair) by Sismo Design. Images courtesy of Sismo Design.
the interviews. There are many more strategies
that designers use. Ludden et al refer to at least six
strategies to elicit surprise specifically through visualtactual incongruity [21]. These include using a new
material with unknown characteristics; a new material
that looks like a familiar material; a new appearance
for a known product or material; combination with
transparent material; hidden material characteristics;
and visual illusion. Ludden also reports other strategies
to elicit surprise through visual-auditory and visualolfactory incongruities [8].
Two of the visual-tactual strategies offered by Ludden
et al [21] resemble strategies from this research. The
first one is “a new appearance for a known product or
material” [21]. It is similar to the strategy “breaking
expectations through challenging assumptions of
appearance” from this research. I specifically refer to
it as “challenging assumptions” as this implies that an
assumption has been formed about what the appearance
of an object should be like. This is important, as it
may seem that the more established the assumption
of an appearance is, the more surprising a challenging
appearance would be. Josh Morenstein reinforces this
idea when he says “we found that people thought that
all Bluetooth headsets needed to look organic… so
we challenged that notion” (J. Morenstein, personal
communication, November 10, 2008). Ludden refers
to this characteristic by mentioning that the strategy
refers to a “familiar” or “well-known” product [21].
The second strategy that resembles the findings from
this research is “visual illusion” [21]. The strategy
connects to “use of magical interactions” from this
research. It also relates to and involves “visual illusions”,
but expands such a notion by allowing more than just
visual illusions and allocating other potential ways to
use magic. An example of this can be seen in the use of
technology and touch in the Leaf lamp by Fuseproject,
which uses tactile sensors to turn on and off, dim the
light, and change colour through an RGB controller,
all without moving or visible switches.
The categories described above do not intend to
be exclusive or exhaustive. As previously discussed,
some designs served as an example for two or more
categories. For instance, the project for the On/Off
lamp started from a behavioural observation by the
designer – knocking over bedside lamps with one’s arm.
The solution relates to the specific way in which the
designer understood objects, and involved an intention
to elicit surprise through magic and unexpected
movement.
4.2 Further Research
The limitations of this research include that the
suggested strategies are based solely on the responses
from the interviews. Such an approach gives depth
to the analysis and assures that the intentions of the
designers are accounted for, which is the main goal of
this research. However, it should be pointed out that
there are other categories that could be found through
analysis of existing designs without directly talking to
the designers. Such categories were omitted in the
analysis, as I had no further information about the
original intention of the designers.
It was mentioned in the introduction of this paper that
surprise has the characteristics of drawing people’s
interest towards stimuli and that it relates to familiarity.
Designers either moved away from the familiar to
elicit surprise, or used familiar elements in a different
context to elicit surprise. However, further discussion
about familiarity is necessary. For instance, how do we
establish what, and for whom, something is familiar?
How far away can designs step from familiarity and still
Design and semantics of form and movement
77
elicit a pleasant surprise? Using unexpected contexts
seemed to be a common strategy for designers.
Nevertheless, an unexpected context could be anything
that is not the expected context. How do we know
that an unexpected context may work in a favourable
way for eliciting surprise? Such questions go beyond
the answers from the interviewees and the scope of
the research this paper draws on. The balance between
eliciting pleasant surprise and moving away from
familiarity should be further investigated.
The goal of this paper was to report on explicit ways
that designers use to surprise. Further research should
investigate implicit strategies designers use to surprise.
The research could follow a participatory research
approach, with researchers working as designers with
design studios, attempting to uncover strategies that
designers do use but were not able to verbalise through
the interviews.
[10] Rubin, H.J., & Rubin, I. (2005). Qualitative Interviewing:
The Art of Hearing Data. Thousand Oaks: Sage.
[11] Clarke, A.E. (2003). Situational analyses: Grounded theory
mapping after the postmodern turn. Symbolic Interaction, 26(4),
553–576.
[12] Clarke, A.E. (2005). Situational Analysis: Grounded Theory
After the Postmodern Turn. San Francisco: Sage: University
of California.
[13] Gilbert, N. (2001). Researching Social Life (Vol. 2nd).
London: SAGE Publications.
[14] Poland, B.D. (2003). Transcription quality. In J.F. Gubrium
(Ed.), Inside interviewing: New lenses, new concerns
(pp. 267–288). Thousand Oaks: SAGE Publications.
[15] Strauss, A., & Corbin, J. (1998). Basics of Qualitative
research: Techniques and Procedures for Developing Grounded
Theory (Vol. 2nd). Thousand Oaks: Sage Publications.
[16] Kolko, J. (2009). Abductive thinking and sensemaking:
The drivers of design synthesis. Design Issues, 26(1), 15–28.
[17] Robinson, R.E. (1994). The origin of cool things. Design
References
that Packs a Wallop: Understanding the Power of Strategic
[1] Demirbilek, O., & Sener, B. (2003). Product Design, Semantics,
Design (pp. 5–10). Presented at the American Center for
and Emotional Response. Ergonomics, 46(13), 1346–1360.
Design Conference, USA: American Center for Design.
[2] Norman, D. (1990). The Design of Everyday Things.
[18] Robinson, R.E., & Hackett, J. (1997). Creating the conditions
New York: Double Day.
of creativity. Design Management Journal, 8(4), 10–16.
[3] Rodríguez Ramírez, E.R. (2011). Elements of surprise:
[19] Front. (2010). Front projects: Magic. Retrieved May 25,
Industrial designers’ strategies for eliciting surprise through
2010, from http://www.frontdesign.se/newsupdate_MAGICAL.
interaction (Doctoral dissertation). Victoria University of
htm
Wellington, Wellington.
[20] Fox-Derwin, E. (2012). Honest deception: Skin, incongruity
[4] Desmet, P.A. (2003). A multilayered model of product
and surprise (Masters dissertation). Victoria University of
emotions. The Design Journal, 6(2), 4-13.
Wellington, School of Design, Wellington.
[5] Desmet, P.A. (2002). Designing emotions (Doctoral
[21] Ludden, G., Schifferstein, H., & Hekkert, P. (2008).
dissertation). Technical University of Delft, Industrial Design
Surprise as a design strategy. Design Issues, 24(2), 28–38.
Engineering, Delft.
[6] Hekkert, P. (2006). Design aesthetics: Principles of pleasure
in design. Psychology Science, 48(2), 157–172.
[7] Grimaldi, S. (2006). The Ta-Da series: Presentation of a
technique and its use in generating a series of surprising designs.
In M.A. Karlsson, P. Desmet, & J. van Erp (Eds.), 5th Conference
on Design and Emotion (p. CD). Presented at the Design and
Emotion, Göteborg, Sweden: Chalmers University
of Technology.
[8] Ludden, G. (2008). Sensory incongruity and surprise in
product design (Doctoral dissertation). Technical University of
Edgar Rodríguez
Delft, Faculty of Industrial Design Engineering, Delft. Retrieved
Ramírez
from http://repository.tudelft.nl/view/ir/uuid%3A8cafddd5-6a38-
School of Design,
Victoria University
of Wellington,
New Zealand
42e4-9ca7-62abb4bfcaab/
[9] Hesse-Biber, S.N., & Leavy, P.L. (2010). The Practice
of Qualitative Research (Vol. 2nd). Thousand Oaks: Sage.
78
Design and semantics of form and movement
Nazli Cila, Paul Hekkert, Valentijn Visch
{n.cila, p.p.m.hekkert, v.t.visch}@tudelft.nl
The role of designer expertise
in source selection during product
metaphor generation
Abstract
Metaphors have a communicative role in design that
entails a transfer of meaning from an entity (i.e. source)
to the designed product (i.e. target). In this paper, we
investigate the effect of the expertise of designer on
the accessibility of the sources that they employ in
metaphors. In the study conducted, novice and expert
designers were asked to generate metaphors and the
sources they selected were used for analysis. The
results indicated that, (1) novices tended to select easily
accessible sources whose similarity with the target
were obvious, and (2) experts tended to use sources
that were less similar with the target and more difficultto-access in the first place. These results were then
discussed in the light of metaphor theories and product
design knowledge.
Keywords
Product Metaphors, Metaphor Generation, Source
Selection, Expertise, Source Accessibility, Target-Source
Similarity.
1 Introduction
The flyswatter of Atypyk nicely illustrates the power
of metaphor use to tell stories through products
(Fig. 1). By associating a flyswatter with a magic wand,
the designer of this product wittily brings in a novel
approach to get rid of the unwelcomed flies and insects
in your house. Actually, killing flies is an activity that is
not quite pleasant for the most, and to be able to kill
them “magically” without the brutality is a tempting
idea in itself. Although the product works as a regular
flyswatter rather than with magic in this case, its
form leads the users to use it in a creative manner.
By means of this product, it is easy to picture people
running around in the room and getting rid of flies with
a magical “wave and flick”. This gentle movement is
also in contradiction with the harshness of the hitting
that you would do with a regular flyswatter. For these
reasons, the product has a strong humorous and playful
dimension that associates the mundane activity of killing
flies with the world of fairies and magic.
As a matter of fact, this is what the metaphors are
used for. They have a communicative role that entails
the transfer of meaning from one entity to another.
This quality extends the limits of metaphors beyond
language. Since metaphors associate two distinct entities
in order to attain new meanings, various disciplines
make use of them frequently (e.g., psychology,
linguistics, art, advertising, architecture, music,
mathematics, gesture studies, literature, film studies,
religion, etc.). In the design domain, metaphors are used
as tools to define the design problem, generate ideas
and enhance creativity [1], and/or as a communication
medium with the users to express meaning and achieve
cognitive and affective effects [2], [3].
Design and semantics of form and movement
79
In the sections that follow, we will first expand on
what is currently known about target-source similarity,
source accessibility and their relation to expertise in
the literature, and then present the study we conducted
together with the discussion of the results in the light of
metaphor theories and product design knowledge.
2 Similarity of Target and Source
Fig. 1. “Bzzzz... Vlan” flyswatter by Atypyk Design, 2005.
To go back to Fig. 1, in technical terms, the flyswatter
is referred to as the target of the metaphor, which is
the product that is designed for conveying a particular
meaning. The magic wand is called the source, namely
the entity that is associated with the target to convey
that meaning. The generation of product metaphors
involves a transfer of attribute(s) from the source to
the target. In a typical metaphor generation situation,
there are tens, if not hundreds, of possible sources to
associate with the target, yet the designer focuses on
only one of these. This selection is a highly important
decision that determines the comprehensibility and the
quality of the metaphor, but little is known about the
factors affecting this selection. Metaphor generation
and source selection are mostly neglected even in
the linguistics domain, where the wealth of research
is directed towards metaphor comprehension and
appreciation [4].
In the literature, it is widely accepted that target and
source domains in a metaphor share some attributes,
and this similarity is the reason why they are brought
together [5], [6], [7], [8], [9]. This similarity may be
rather obvious and easily accessible; or it can be based
on “deeper” relations and thus more difficult to notice
in the first place. In this paper, we address the ease
of accessibility of a source based on target-source
similarity. We argue that the expertise of designer
has an effect on the ability to access sources that
have different levels of similarity with the target. For
this reason, our aim in this study is to investigate if
novice and expert designers tend to use sources with
a particular level of similarity with the target while
generating metaphors.
80
Design and semantics of form and movement
Similarity refers to the likeness or resemblance of two
entities, and it increases as the number of common
features between the entities increases and as the
number of distinctive features decreases [10]. In
metaphor literature, it is acknowledged that the target
is seen through the lens of the source by construing
attributes that apply to target as corresponding to
those that apply to source [9]. This correspondence
is on account of the similarity of target and source,
which governs the association of these entities with
each other. To go back to our example, we see that
a flyswatter and a magic wand share many perceptual
and conceptual attributes. They somehow look like
each other since they have similar size, proportions and
construction (i.e. flat surface attached to a stick), and
they are held by the hand in the same way. They are also
similar to each other if we abstract their function – they
can both make things disappear. In that sense, we can
state that the association of the flyswatter with the magic
wand is considerably meaningful as they are both used
similarly on an abstract level. Still, this association is not
very obvious, which is why the connection is a clever one.
What the similarity between target and source entails
is often investigated in analogy and metaphor literature.
Although these descriptions are all given in relation
to the semantic qualities of “words” or “concepts”,
we maintain that they are also applicable to targetsource similarity in product metaphors because they
are generic enough to cover all dimensions of an entity,
whether it is abstract or tangible. For instance, some
scholars argued that perceptual similarity between
target and source is essential for metaphoric success
[11], whereas some others maintained that good
metaphors have targets and sources that are similar
in their functional qualities; especially in the way they
affect our emotional states. In another study, two
terms, which are called taxonomical and thematic
relations, were defined in order to explain the possible
relations between two concepts [13]. The former is
a relation that represents the physical resemblance
between two objects, and the latter represents the
relation between two concepts through a thematic
scene. Seitz [14] distinguished six types of metaphoric
matches that originate from target-source similarity:
color, shape, cross modal, physiognomic, psychologicalphysical and taxonomic. Color and shape matches refer
to pairing off shape or color of an entity with another
in a nonliteral way, like comparing an egg yolk to the
sun. Cross-modal matches build an association between
various sensory modalities like matching a sound to
a smell or visual experience to touch. Physiognomic
matches involve attributing emotional qualities to
inanimate objects. Psychological-physical matches
refer to relating a physical aspect of an object to a
psychological characteristic like “He is very tough”.
Taxonomic matches associate abstract qualities of two
distinct things when there is no physical resemblance,
like comparing a cheetah to a car. These examples
indicate that at the origin of a metaphorical association
lies a certain level of similarity between target and
source, and the content of this similarity can be based
on various aspects of these domains.
When it comes to analogy literature, the most common
distinction is made between surface similarity and
structural or deep similarity between target and
source [15], [10], [5], [16], [17]. The use of these terms
are also metaphorical: surface similarity is described
as the easily accessible similarity since the similar
attributes are on the “surface”, and deep similarity
refers to the similarity that is more difficult to notice
because the similar attributes are rooted in “deep”.
Most of the time, surface attributes are addressed by
physical descriptive attributes like color, form, sound,
etc. because perceiving tangible attributes are quicker
and easier than the others. On the other hand, deep
attributes are defined as non-physical abstract qualities.
For this reason, it is argued that the ease of accessibility
of the surface similarity comes from being based on
physical and tangible attributes, whereas deep similarity
is difficult to notice since some kind of an abstraction is
necessary [18], [10], [5], [17]. However, Vosniadou [17]
maintains that this argument fails to take into account
the status of the similar attributes in people’s underlying
representations. If the attributes in question are salient
with respect to underlying representations, the easily
accessible similarity can also be based on relational,
abstract or conceptual properties. For this reason,
Vosniadou does not differentiate surface from deep
similarity according to its content, and argue that any
kind of attribute can be in surface or in deep as long as
it is prominent.
Regardless of the discrepancies of the arguments related
to its content, we can state that the accessibility of a
source lies on a continuum that has surface similarity
at one end (i.e. easily accessible similarity) and deep
similarity at the other (i.e. difficult to access similarity).
Having a surface similarity with the target makes a
potential source to be easily identified when building
associations, whereas it is more difficult to find sources
that are based on deep similarity because the domains
may differ in many subtle ways [15]. This makes noticing
deep similarity relations a more sophisticated way of
building associations. Correspondingly, in the problem
solving literature poor analogies are ascribed to the
tendency to notice superficial attributes while failing
to see deep structural relations [19], [17]. Similarly,
designers may also find it difficult to notice the relevant
relationships if their problem at hand carries structural
similarity but no surface similarity [16]. For this reason,
it is argued that seeing deep similarity between two
distinct entities requires expertise [19], [20], which we
will investigate in the next section.
3 The Effect of Expertise on Seeing
Similarity
There is a substantial and long history of work on
understanding the differences between experts and
novices in various disciplines, including physics, chess,
music, sports, architecture, ballet and medicine.
However, it is interesting to see that all these studies
mainly show a similar trend regardless of the discipline:
experts are able to abstract the situation at hand and
see the underlying structure, whereas novices’ actions
tend to be limited with the surface attributes of the
situation. As an example, it is argued that there are
fundamental differences in the representations of
physics problems among experts and novices [20].
When asked to categorize problems according to
the similarity of their solutions, experts mainly use
underlying principles as the similarity criteria and
novices focus on surface features. For this reason,
novices tend to be misled in identifying similar problems
because their surface features are different.
Design and semantics of form and movement
81
These kinds of examples are given by many scholars
from different fields. It is argued that the acquisition of
skills to abstract the essence of the situation and see
underlying relations are related to the development of
expertise. As maintained in Casakin [19], “Experience
in a certain domain allows the generation of abstract
or conceptual problem representations and enhances
the probability of analyzing a problem more in-depth,
by focusing on structural features”. On the other hand,
the fields that are mentioned are domains where the
constraints of the problem are well defined, and a
limited number of rules apply (e.g. chess, physics, etc.)
[21]. However, the problems in the design domain are
described as ill-defined problems, meaning that there
exist many possible solutions and there are no rules
to obtain these solutions [21], [22]. For this reason,
the expertise in the design domain is an “adaptive
expertise”, which the experts adjust to situations [22].
Despite this difference, expert designers are also able
to see an underlying pattern and make a connection
with a solution, which can be from an entirely
different context. As Lawson [23] aptly puts forward,
“Experienced designers have simply seen more and
made more connections already than inexperienced
designers”. This brings us to the metaphors, in which
Casakin and Goldschmidt [19] found out both expert
and novice designers are able to employ visual analogies
and associations in solving design problems, however
using them in a better way requires expertise.
Based on this literature, the aim of this study is to
investigate the effect of designer’s expertise on the
accessibility of the sources that are employed in the
metaphors. In other words, we aim to find out if novice
and expert designers tend to select sources from
different levels of the accessibility continuum.
Our hypotheses are:
r/PWJDFTBSFMJLFMZUPTFMFDUFBTJMZBDDFTTJCMFTPVSDFT
whose similarity with the target is obvious.
r&YQFSUTBSFMJLFMZUPTFMFDUEJGàDVMUUPBDDFTTTPVSDFT
whose similarity with the target is less obvious than
the sources used by novices.
4 Methodology
In the sections that follow, the experimental setup designed for investigating the hypotheses will be
presented.
82
Design and semantics of form and movement
4.1 Participants
In this study, 24 expert designers (20 male and four
female) and 25 novice designers (12 male and 13
female) participated. The expert group comprised of
professional designers who had an average of six and a
half years of working experience within the field, varying
from two to 16 years. The novice group included second
year Bachelor students studying in the Industrial Design
Department of Delft University of Technology, who
took two design courses within the first year of their
education.
4.2 Experimental Task
In order to minimize the possible effect of the product
type on the experimental task, four different products
were used in the study: MP3 player, chair, nail clipper
and self-cleaning trash bin. Each participant generated
a metaphor for each one of these products. Two other
experimental conditions were also utilized – the time
allotted for completing the design (short-long) and
the intention of the designer to use the metaphor
(pragmatic-experiential) – however, the effect of these
conditions on the ease of accessibility of the source will
not be reported in this paper.
After getting an introduction to what a product
metaphor is and how it is used in the design domain,
the participants were given the design tasks one at a
time. These tasks were set up by systematically altering
the combination of the product type and intention, and
their order of presentation to the participants. The
design session always followed the sequence of long
task (40 min.) – short task (8 min.) – long task (40 min.)
– short task (8 min.), and a break was given between
the second and third tasks. The whole study took
around two hours and 15 minutes. Upon completing
the session, the participants were asked to explain their
designs orally and explicitly point out the sources they
associated with the target products together with its
reasons. Then, they were debriefed and compensated
for their participation.
4.3 Assessment of Similarity
Two naïve judges who were blind to the aims of
the study and the experimental conditions scored
independently the ease of accessibility of the sources
considering the targets they were associated with.
They were provided with a questionnaire that included
the sources that are selected by the participants and the
targets in random order. The targets and sources were
presented to the judges as words, and they were asked
to rate word pairs (e.g. nail clipper – scissors, chair –
cloud) regarding “how easy it is to see the similarity
between the items” on a 9-point-scale. The ratings were
then compared, and if there were minimum 3 points of
difference between the ratings for each item, the judges
reconciled their assessments by discussion and came to
a consensus. The remaining scores were averaged
to form an overall similarity index.
5 Results
The sources employed by participants can be seen in
Table 1. The mean ratings indicate that novices tended
to employ sources that were easier to access (M = 5.48,
s = 2.59), whereas experts selected sources that were
less accessible (M = 4.07, s = 2.71).
Table 1. The sources used by novice and expert
designers in our study.
Sources associated with target products*
Expertise Level
Nail clipper
MP3 player
Chair
Self-cleaning trash bin
Novice
Beaver
Biting nails
Caterpillar
Crocodile (2)
Door
Finger
Fingerprint
Guillotine
Ladybug
Nail polish
Nails (2)
Peg
Pencil sharpener
Scissors (3)
Shark
Teeth (5)
Battery
Cassette (2)
Cassette player
CD (2)
Clef
Disco ball
Ear (3)
Frog
Guitar amplifier
Headphones
LP player
Music note (5)
Music sheet (2)
Turn table
Samba balls
Walkman
Arrow
Bottom
Car seat
Cave
Clothing
Clouds (2)
Cushion
Dinnerware
Hammock
Mushroom
OK hand gesture
Sitting human body (6)
Stop sign
Trace of butt
Vertebrae
Broom
Canon
Cleaner
Drain
Droid
Elephant
Flushing toilet (2)
Garbage bag (2)
Maid
Paper recycle bin
Recycle sign
Red cross
Rug
Soap bar
Space pod
Trash car
Trash tube
Wiping hand (3)
Expert
Animal
Biting nails
Cutlery
Door
Finger
Guillotine
Heart
Monster
Paper cutter
Pencil sharpener
Perfume bottle (2)
Pincers (3)
Scissors
Snake
Stapler
Teeth (2)
Vending machine
Washing machine
Whale
Cap
CD rack (2)
Cloud
Comforting stone
Dragonfly
Ear
Headphones
iPod nano
LP player
Microphone
Music note
Navigating in car
Necklace
Pirate’s hook
Rubik’s cube
Roulette
Sea shell (2)
Storage box
Alarm clock
Arrow
Being in dark
Bird nest
Birthday decoration
Cake
Cloud (3)
Log
Nail bed
Pants
Piece of a pie
Shovel
Sitting human body (2)
Supporting hands
Tensegrity cons.
Thread ball
Toilet roll
Wheel
Automatic vac.
Cleaning wipes (2)
Dining plate
Dyson fan
Chlorine bottle
Hourglass
Hygienic smell
Jail
Kitchen paper roll
Lotus leaf
Paper bag
Shower
Transparency (2)
Soap bar
Supermarket belt
Toilet (3)
Tongue
Twisting bolt
Windshield wipers
* The numbers in brackets indicate the number of participants who used that same source.
Design and semantics of form and movement
83
In order to investigate the main effects, the data
was analyzed using ANOVA on the ratings for the
accessibility, with the “order of the presentation of
each product”, “product type” and “expertise level”
as between-subject factors. With regard to the first
two control factors, the results indicated no significant
effect on similarity and interaction effect with the
expertise. This means that the order we presented
the products to the participants and the four product
types did not affect the tendency of the participants to
employ a particular similarity degree, and the effect of
expertise is independent of these two factors. Although
not significant, the means of each product for similarity
ratings show an interesting trend. MP3 player and selfcleaning trash bin lead to the association of sources that
are easier to access, whereas chair and nail clipper lead
to the association of difficult-to-access ones (Fig. 2).
Fig. 2. The means for the accessibility of the sources
the target product was rather obvious when compared
with the sources selected by the experts. On the
contrary, experts’ preferences were more on the
“distant” sources, in which their similarity to target was
less obvious. Therefore, both of our hypotheses were
supported with the findings that novices used easily
accessible sources while experts used more difficultto-access ones. This means that the abilities of novices
are yet to be limited with noticing the sources that
are similar to target in terms of the attributes on the
surface only. This situation is also apparent when we
have a look at the sources that participants selected
(see Table 1). Let’s take the MP3 player task as an
example. Some of the examples of the sources that
the novices used in their metaphors are music notes,
music paper, CD, Walkman, record player, etc., which
are all highly related to a MP3 player but meanwhile this
relation is somewhat obvious since they are all in the
“music” and “music listening” domain. On the other
hand, experts associated sources like dragonfly, Rubik’s
cube, seashell, etc. with the MP3 player. The relationship
of these entities is in fact quite abstract: dragonfly was
chosen because it buzzes in your ear, Rubik’s cube was
chosen because it is a lovely way of entertainment,
and seashell was chosen because you can put it in your
ear and listen to the humming of the sea. Compared
to the music notes or CD’s, these entities are rather
meaningful and novel associations. As can be seen from
these examples, novices built more obvious relations
but experts used more distant sources in their designs
most of the time.
associated with each product type.
ANOVA analysis yielded a main effect of expertise
on source accessibility F(1, 31) = 13.39, p < .001, 2 =
.086. This finding indicates that expertise significantly
influences the accessibility of a source. When combined
with the mean scores, our results show that novices
tended to employ sources that were easier to access,
and experts selected sources that were more difficultto-access.
6 Discussion and Conclusions
The research findings indicated that expertise had
a significant effect on the selected sources to be
employed in the products. Novices were likely to
select easily accessible sources whose similarity with
84
Design and semantics of form and movement
Accessing sources that are based on deep common
attributes with the target and being able to build
associations in between might be a natural tendency
that is acquired with expertise because metaphors
or analogies that are based on a distant relationship
between target and source are deemed more creative
and original than those based on close relationships [1],
[15], [18]. Bringing two very similar entities together
may lead to hackneyed and uninteresting metaphors
[7], [8], and correspondingly, it has been shown in
the design literature that distant associations play a
significant role in the design of creative products [15].
In that sense, the tendency of the professional designers
to use more distant sources in our study might support
their desire to attain interesting metaphors. Our results
are preliminary to make generalizations and require
a follow-up study to investigate if the designs of the
experts were more creative, but we argue that with
the acquisition of expertise, the designers are likely
to prefer more distant and less obvious relationships
between target and source.
It should be noted that the source whose relationship
is obvious with the target does not necessarily mean
that it is highly similar to the target at the same time,
and vice versa. We acknowledge that they are parallel
mostly, but there might be conflicting cases. For
instance, in one of the cases the designer associated
a lotus flower leaf with a self-cleaning trash bin (see
Table 1). The reason was that the lotus leaf has a
special surface structure that repels dirt and water,
which eventually provides the plant to clean itself.
The transfer of this attribute to the trash bin fulfills the
self-cleaning function of the target product. For this
reason, a self-cleaning trash bin and a lotus leaf becomes
highly similar since they both do self-cleaning. However,
seeing this connection might be difficult in the first
place because these are two distant entities belonging
to different domains and self-cleaning is a rather hidden
attribute of a lotus leaf. Especially for a person who
is not knowledgeable about this attribute, accessing
this association might be difficult. For this reason, the
answer to the questions “how similar these two entities
are” is very similar, whereas “how easy it is to see the
similarity of these two entities” is not very easy. Thus,
we maintain there might be some cases like this that
overall similarity of a source with a target differs from
the accessibility of this similarity. In this study, we only
focused on the accessibility since we consider it one
of the main aspects affecting metaphoric quality. Two
entities may be quite similar, but as long as seeing this
similarity is not very easy, their association leads to
clever connections like in the magic wand –flyswatter
example (see Fig. 1).
In addition to this, we want to say a few words
regarding the type of similarity of the sources that are
associated with target products. As can be seen in Table
1, participants employed various sources that have
perceptual and/or conceptual similarity with the target
products. A summary of the content of these similarity
types can be found in Table 2, together with example
associations. We mentioned earlier that some scholars
use perceptual similarity and conceptual similarity
for referring to surface similarity and deep similarity,
respectively; and some others do not differentiate
the content of surface similarity from deep similarity.
As Vosniadou (1989) argues, conceptual and abstract
attributes can also be on the surface as long as they
are prominent, and vice versa. Within the context of
our study, the first argument leads to the assumption
that novices would select perceptually similar sources
while experts would go for conceptually similar ones.
However, our initial analysis on the sources used by
the participants shows no such tendency. The entities
associated with the target products by novices were
also somewhat conceptual as much as the ones
associated by experts, and the experts also used
perceptually similar sources commonly. For instance,
if we take the nail clipper as an example, novices
used sources like beaver, shark and crocodile that are
conceptually similar to a nail clipper since they all cut
something with their teeth in an abstract level. On
the contrary, experts associated sources like pincers,
scissors and stapler, which are similar to a nail clipper
perceptually. For this reason, there is no difference in
the tendency to use a particular type of similarity more
often among the novice and expert designers. We can
state that our sample supports Vosniadou’s view that
the surface attributes are not necessarily perceptual,
and deep attributes are not conceptual all the time.
Type of
similarity
Similarity in:
Example
Appearance
Form, color, texture,
material, size,
proportions
Chair – Sitting human
body
Movement
The way main body/
parts move (direction,
position change, force)
Nail clipper –
Guillotine
Sound
Sounds they make
MP3 player –
Dragonfly
Interaction
pattern
The way a person uses,
holds, etc.
Nail clipper – Stapler
Perceptual similarity
Conceptual similarity
Function
The main purpose
MP3 player – Walkman
Working
principle
How they work; the
mechanism/ structure
behind
Self-cleaning trash bin
– Lotus flower leaf
Emotions they
evoke
What they make
people feel
Chair – Bird nest
Environment/
context
The environment they
are in, the people who
use them, the time
they are used, etc.
Nail clipper – Perfume
bottle
Table 2. The types of similarity between target and source
Design and semantics of form and movement
85
A final remark is about the products used in this study.
We mentioned earlier that MP3 player and self-cleaning
trash bin were associated with sources that are easier
to access, and the case with the nail clipper and chair
was the reverse. For the former two, we observed that
participants aimed to convey the purpose and usage
of the product first, and therefore looked for sources
that “do the same”, i.e. play music or clean itself. In
particular, the self-cleaning trash bin is a rather novel
product concept. The participants aimed to make
this special function apparent and understandable.
For this reason, the similarity of the sources that
were associated with these products were obvious,
which aims to convey to users the function, usage and
interaction pattern as clear as possible. On the other
hand, a nail clipper and chair are products whose
function is fixed and easily understandable. Therefore,
we observed that participants were able to experiment
with associating various entities with these products
without worrying about the functionality. They tried
to communicate a specific message, improve the visual
look or associate it with an entity for fun purposes.
That is why we think the sources these products were
associated with had a more distant relationship. The
product type did not have any statistical effect on our
findings, however we acknowledge the fact that certain
product types would be more suitable for focusing on a
particular level of similarity.
To summarize, our results indicate how ease of access
affects the selection of a potential source and how
different groups of designers behave while selecting
a source. Novices used sources that are easier to
associate with targets in general, while the tendency
of the expert designers was the opposite. The present
study is one of the first attempts to investigate how
target and source of a metaphor are associated with
each other during metaphor generation, yet this issue
needs further attention. Metaphors are powerful tools
for conveying meaning through the product, and if we
are to understand how to make good and creative
decisions in this respect, the designers can create
comprehensible, aesthetic, interesting and pleasant
metaphors, and rich and meaningful user experiences.
Noyons, Van der Veer Designers, Waarmakers, We Are
Perspective, Wit Industrial Design, de Wolkenrijders,
and to our students.
References
[1] Casakin, H.P. (2007). Metaphors in design problem solving:
Implications for creativity. International Journal of Design, 1(2), 21-33.
[2] Forceville, C., Hekkert, P., & Tan, E. (2006). The adaptive
value of metaphors. In U. Klein, K. Mellman, & S. Metzger
(Eds.), Heuristiken der Literaturwissenschaft: Einladung zu
disziplinexternen Perspektiven auf Literatuur (pp. 85-109).
Paderborn: Mentis.
[3] Van Rompay, T.J.L. (2008). Product expression: Bridging
the gap between the symbolic and the concrete. In H.N.J.
Schifferstein, & P. Hekkert (Eds.), Product Experience
(pp. 333-352). Amsterdam: Elsevier.
[4] Flor, M., & Hadar, U. (2005). The production of metaphoric
expressions in spontaneous speech: A controlled-setting
experiment. Metaphor and Symbol, 20(1), 1-34.
[5] Holyoak, K.J., & Koh, K. (1987). Surface and structural similarity in analogical transfer. Memory & Cognition, 15(4), 332-340.
[6] Kelly, M.H., & Keil, F.C. (2009). Metaphor comprehension
and knowledge of semantic domains. Metaphor and Symbolic
Activity, 2(1), 33-51.
[7] McCabe, A. (1983). Isolated sentences versus extended
contexts. Journal of Psycholinguistic Research, 12(1), 41-68.
[8] Ortony, A., Vondruska, R.J., Foss, M.A., & Jones, L.E. (1985).
Salience, similes, and the asymmetry of similarity. Journal of
Memory and Language, 24, 569-594.
[9] Tourangeau, R., & Sternberg, R.J. (1982). Understanding and
appreciating metaphors. Cognition, 11, 203-244.
[10] Goldstone, R.L., Medin, D.L., & Gentner, D. (1991).
Relational similarity and the nonindependence of features in
similarity judgments. Cognitive Psychology, 23, 222-262.
[11] Johnson, M.G., & Malgady, R.G. (1979). Some cognitive
aspects of figurative language: Association and metaphor. Journal
of Psycholinguistic Research, 8(3), 249-265.
[12] Asch, S.E. (1958). The metaphor: A psychological inquiry.
In R. Taguiri & L. Petrullo (Eds.), Person Perception and
Interpersonal Behavior (pp. 324-333). Stanford: Stanford
University Press.
[13] Shoben, E.J., & Gagne, C.L. (1997). Thematic relation and
the creation of combined concepts. In T.B. Ward, & J. Vaid
(Eds.), Creative Thought (pp. 31-50). American Psychological
Acknowledgments
Association.
We would like to express our deepest gratitude to our
participants from Bleijh, Buro Raak, Fabrique, G-Star,
IPV Delft, MMID, Muzus, Natwerk, Scope, Studio Anna
[14] Seitz, J.A. (1997). The development of metaphoric
86
Design and semantics of form and movement
understanding: Implications for a theory of creativity.
Creativity Research Journal, 10(4), 347-353.
[15] Christensen, B.T., & Schunn, C.D. (2007). The relationship
of analogical distance to analogical function and preinventive
structure: The case of engineering design. Memory & Cognition,
35(1), 29-38.
[16] Tseng, I., Moss, J., Cagan, J., & Kotovsky, K. (2008). The
role of timing and analogical similarity in the stimulation of idea
generation in design. Design Studies, 29, 203-221.
[17] Vosniadou, S. (1989). Analogical reasoning as a mechanism
in knowledge acquisition: A developmental perspective. In S.
Vosniadou & A. Ortony (Eds.), Similarity and Analogical Reasoning
(pp. 413-437). Cambridge: Cambridge University Press.
[18] Dahl, D.W., & Moreau, P. (2002). The influence and value
of analogical thinking during new product ideation. Journal of
Marketing Research, 39(1), 47-60.
[19] Casakin, H.P., & Goldschmidt, G. (2000). Reasoning by
visual analogy in design problem-solving: The role of guidance.
Environment and Planning B: Planning and Design, 27, 105-119.
[20] Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981).
Categorization and representation of physics problems by
experts and novices. Cognitive Science, 5, 121-152.
[21] Ahmed, S., Wallace, K.M., & Blessing, L.T.M. (2003).
Understanding the differences between how novice and
experienced designers approach design tasks. Research in
Engineering Design, 14, 1-11.
[22] Popovic, V. (2004). Expertise development in product
design – strategic and domain –specific knowledge connections.
Design Studies, 25, 527-545.
[23] Lawson, B. (2004). Schemata, gambits and precedent: Some
factors in design expertise. Design Studies, 25, 443-457.
Nazli Cila,
Paul Hekkert,
Valentijn Visch
Delft University
of Technology,
Department
of Industrial
Design, Delft,
The Netherlands
Design and semantics of form and movement
87
Jorn Guldberg
[email protected]
Design and iconicity: Design, materiality
and the meaning of things
Abstract
The semiotic problematic to be addressed in this
paper is the construction of meaning in serial objects
or multiples. At least three ‘thing-identities’ are in play
during the lifetime of a merchandise, and the question
is how things actually attain meaning in their different
contexts of design, production and consumption and
how to differentiate methodologically between stages
within the commercial and cultural trajectories of one
particular item (a thing).
Since Jean Baudrillard posed the question of the modelseries relation as defining “a perpetual dynamic which
is in fact the very ideology of our society” [1], it has
been a permanent challenge to design studies to
address the question of how meaning is construed
in mass produced objects for use. There seems to
be three levels or stages of meaning production; in
relation to, respectively, (1) the model or prototype
for a series, (2) any produced item in the series, and
(3) any singular object in the series. New approaches
to the problematic have, by employing terms such
as ‘indication’, ‘affordance’ and ‘agency’, furthermore
emphasized the differentiations in regard to the
ambivalent statuses and meanings of mass products,
especially the relation of model to singular objects.
The status of the latter has been described as, for
example, ‘personalized objects’ (Baudrillard, Sophie
Woodward) and ‘singularized objects’ (Igor Kopytoff).
However, nor the processes neither the specific
88
Design and semantics of form and movement
‘sites’ of meaning production have not yet, in my view,
been addressed properly or convincingly.
This paper will consider the possibility of differentiation
between the three levels mentioned above, by
introducing the semiotics of Charles S. Peirce’s, and,
more specifically, his so-called first trichotomy and
the concepts of legisign, sinsign, and qualisign, (or, in
another terminology of Peirce’s, type, token, and ‘tone/
tuone’, and in a third listing, potisign, actisign, and
famisign). This modeling, so it will be claimed, provides
a theoretically satisfying conception of how and ‘where’
and by whom the multiple meanings in design objects
are being produced, as well as the consequences of
this for appropriate analytical strategies with regard to
the processes of design, production, distribution and
consumption.
Keywords
Model and series, design semiotics, Peirce and iconicity,
quality as sign, designing quality.
1 Introduction: Jeans as Art and Jeans
as Jeans
The German ‘fat and felt’ artist Joseph Beuys (192186) was highly visible and easily recognizable, due to
his personal dress code. From the early sixties he
used to appear in public in an outfit consisting of four
elements: trilby hat, fisherman’s vest, white shirt, and
blue jeans. As synecdoche, each of these elements has
been iconized as a sign of avant-garde attitudes and
of deviation from mainstream values. As indicated in
the photos from commercials found on the internet,
the Beuys name has in particular served as a label in
promoting new, fashionable clothing, and younger artists
have paid their kitschy tributes to the master by hinting
at the Beuys-fat-felt connection.
Beuys included jeans in some of his artistic statements,
not as ready mades, but modified in various ways. Two
years prior to his death, Beuys conceived a so-called
multiple with the title of The Orwell Leg – Trousers
for the 21st Century, 1984. The work was issued as
an unlimited series, and by now, approximately 35
exemplars, with variations within the edition, have been
produced. The series consists of new blue jeans the
modification of which is the stamped-out circular holes
on the legs, the one on the front right leg, the other
on the behind left leg It seems that Beuys originally
donated the Multiple concept and the right to sell the
whole series to the Korff Foundation in Germany,
a foundation which finances projects for children in
Germany and worldwide.. In this case, due to the
possibility of alternation, that is to say differentiation
within the series, one may speak of The Orwel Leg as
a conceptual object, and not as a prototype of a pair
of jeans. The latter, in contrast, is the case with a pair
of jeans, modified too, from 1957, with the mystifying
title, Aquarius, which is the astrological name for
the Water Carrier (not Beuys’ constellation). The
work is catalogued as an object belonging to a series
named Healing Objects 1951-75 in the context of
the first retrospective of Beuys’ works, the famous
exhibition organized by the Guggenheim Museum in
New York in 1979 [2, p.216]. The Aquarius-object is
a pair of the artist’s own, old and outworn blue jeans.
The modifications include the elongated slits in the
leg cloth and the small patch attached to the band.
Rather untypical in shape for worn holes, the slits are
very probably made by force. The patch exhibits the
imprinted circular logo or emblem of “Fluxus Zone
West” with a red cross and a symbol of something
that Beuys made use of in order to brand his actions,
installations and drawings while being associated with
the international Fluxus group during the late fifties and
early sixties. Indetectable from the photo are a number
fish bones stuck into the fabric. The Aquarius is in the
private collection of the German medical historian and
art collector Hinrich Murken who acquired the “jeans
as art”, as he said in an interview in which Murken
also stated that the imprinted patch, the slits and the
fish bones in particular gave the jeans a fascinating
expressive quality: “But the bones were what make the
work really mysterious and puzzling and gave it the aura
that it now has” [3].
Beuys’ jeans based art works pose questions of
authenticity and meaning, not only questions concerning
the art status of his objects and actions with things, but
also in relation to how the artist break into and disturbs
the ‘normal’ trajectory of serial products.
The Orwel Leg series is based on preexisting, mass
produced standard jeans, seemingly of the same model
he used as part of his outfit. Beuys is not the origin
or designer of these jeans, and in the first place, he or
his assistants bought a pair of standard jeans in a shop
somewhere, and the Korff Foundation or whoever has
the right for producing The Orwel Leg have had to buy
the jeans successively and subsequently modify them in
accordance with, for instance, Beuys’ written instruction
or his first pair of modified jeans. One may say that
Beuys is the designer of the first issued exemplar of The
Orwel Leg which functioned as a prototype for the rest
of the entire edition. His role as designer, rather than
artist, is emphasized by the fact, that most of the edition
until now has been produced by others than Beuys
himself, at the latest from 1986 and onwards. The post1986 exemplars of The Orwel Leg are not produced
as reconstructions of the originally modified pair of
jeans. On the contrary, the whole edition consists of
singular realizations of a Beuys concept like this (for
example): “a pair of standard blue jeans with circular
holes stamped out on the front right leg and the left
behind leg, both indicating the approximate position of
the knees”. All the jeans from the edition are authentic,
and if two exemplars are completely identical, they are
both still evenly original and authentic since they are
made individually as singulars and may deviate from the
prototype or, more correctly, the first original, without
losing their status as unique manifestations of that
particular work of art titled The Orwel Leg.
The original pair of standard jeans bought by Beuys or
someone else is a specific and individual manifestation
as is each pair in the series of ‘jeans as art’. The original
pair that was bought with the intention of turning them
Design and semantics of form and movement
89
into an art work by means of partial molestation, was
chosen in a shop from a pile of identical jeans, that is,
jeans of the same model, size and colour tone. The
point of departure for Beuys could have been any item
from that pile of jeans. It would not have affected the
character and material or design quality of the first
Orwel Leg. The subsequent manifestations of the
artistic concept may have been realized on the basis
of equally arbitrary choices from the right pile of jeans.
The fact that allows for this arbitrariness or freedom
of choice is, of course, that all the jeans are ‘the same’
only because they are all manufactured from one and
the same prototype design and additional specifications
and instructions with regard to materials and making. In
the case of the uncountable number of standard jeans
in shops worldwide, probably no original exists, and the
first pair manufactured may be considered a curiosity,
and maybe these jeans will gain a status as collectible.
However, the very first pair and pair number 1000
or 1 Million are identical and have the same material
and functional qualities, provided no alterations of the
design or revisions of the instruction to manufacturers
have taken place. Nevertheless, any single pair of jeans
from any model and of any mark will inevitably have the
potentiality of becoming one among a number of the
unique and personal belongings of a particular owner
and user, who has the occasion and right to handle the
jeans as he or she may wish. For instance, the owner
may hang them on the wall as ‘jeans as art’ in order
to decorate or memorialize.
In this way, they share a cultural destiny with Beuys’
jeans, though in different settings; in both cases, a
pair of jeans has been derived of its performativity as
garment and its practical function as protection of a
user’s lower body, and they are both attributed a new
status as one exemplar of a ‘jeans as art’’ edition, and,
say, ‘jeans (as decoration) as art’, respectively.
The Aquarius case is different in one respect. The pair
of jeans used here is the artist’s own jeans, showing
signs of wear and active elaboration. The change
performed by Beuys are the applications, if one may
say so of the elongated slits in the front of the legs,
the undetectable fish bones, and the imprinted patch
attached to the band above the left front pocket.
While the applications are intended, the traces of wear
are not, the latter being signs of both Beuys wearing
the jeans while working, sitting, sleeping etc. and the
90
Design and semantics of form and movement
gradual outwash of indigo pigments. In principle, what
happened to Beuys’ individual pair of jeans is what any
user-owner may experience in relation to his or her
own jeans. On the one hand, the gradual alteration of
colour and loosening of the twill structure of the fabric
is the inevitable effect of wear and maintenance. On the
other hand, from time to time user-owners add to their
favourite jeans symbols, marks, patches, embroidery,
tapes, coloured patterns or slogans in the same way as
they voluntarily cause marks of molestation by means
of scissors, knives, and other instruments. The aim
of this performance may be to indicate the owner’s
individuality or the owner belonging to some sub group.
Formally, there is no difference between Beuys’ actions
in relation to Aquarius and what the average userowner of jeans may do to them, since ownership grants
the privilege freely to create details not inherently
part of the jeans as once being bought in the shop.
Of course, the average user is also free to hang a
favorite pair of jeans on the walls of the living room
as a gesture of veneration or because they are seen
as a suitable decoration.
The only special thing about Beuys’ Aquarius is the art
status of this particular pair of jeans, a status that is
guaranteed not by their specific physical or functional
characteristics but by their status as being a part of
a celebrated, yet controversial artist’s oeuvre. This
transgression of the sphere of the ordinary by a pair
of regular jeans is due to a kind of institutionalized
mystique that eventually is of no concern to this paper.
It should be noted, however, that the mechanisms
hinted at above in relation to jeans are also active in the
case of Beuys’ general oeuvre. The prominence of one
of the most famed and discussed artist of the second
half of the 20th Century suffice to ensure any of his
objects the status of an interesting art work and to turn
this object into a collectible. What characterizes this
process is that the Beuys’ signature establishes a Beuystype, thus any of the pictorial, material, theatrical and
other kinds of rebuses he is famed of are Beuys-tokens,
while each of the works has individual characteristics, even
if, as in the case of The Orwel Leg, the work is a Multiple.
The legend of Beuys’ jeans may seem to be but a
metaphorical use of the mechanisms governing the
acquisition of status and meaning in serial products, but
the general point is that any serial product has a triple
status, and one and the same object refers to three
identity contexts, which is, at the same time, three
logics of meaning formation. Firstly, it is an individual
object characterized by its physical quality. Secondly, it
is a member of a series of identical objects, and, thirdly,
the identical qualities are determined by the design,
that is to say, the design process as concluded by the
final version of a prototype or a model which does not
have to exist, showing or comprising the same material
qualities as the serial object. In other words, the issue at
stake here is the relation of the individual serial product
to the model, hence the meaningful connection between
the singular object, the users legal property, and its
designer. In the end, an interconnected question may be
asked; by which means are the intentions of the designer
as revealed by the model transferred to the final user?
Has the designer a real opportunity of protecting
himself or herself, as well as the commodity against
completely arbitrary meaning attribution from the side
of actual user(s)? Is it the designer’s job more likely to
enable or service the user-owner in her or his efforts to
turn commodities, physically or mentally, into rarified,
unique signs of personal or collective identity?
2 Model and Series
1
This argument is very
similar to Roland
Barthes’ distinction
between ‘denotation’
and ‘connotation’
[14 p. 89].
The problem of the model and series as presented
above, has been addressed by a few scholars in a way
that has had an echo in subsequent research in design
semantics and semiotics. Two contributions stand
out, the first of which to be referred to in the present
context is Jean Baudrillard’s chapter on “Models and
Series” in part four of his early critical work, Le système
des objets published in 1968, English translation some
30 years later [1, pp. 137-155]. The chapter, as does
the book in general, focusses on the market as the
cultural matrix of late capitalism, that is to say the
cultural function of consumption. In his discussion of
the relationship between what he terms ‘model’ and the
serial product he makes some interesting observations
concerning the dialectics of uniqueness and multiplicity
in serial objects. Only two brief comments should
be made in this place on Baudrillard’s theorizing,
remembering also that the book represents a round off
to his occupation with design matters during the sixties,
and that in 1970 he served as a member of the jury of
the ‘Compasso d’oro’, the design prize presented by the
Milanese department store, La Rinacenta, the same year
he attended the legendary design congress in Aspen,
Colorado as a keynote speaker.
First, it is worth noting that his distinction between
model and series seems surprisingly insensitive to the
reality of industrial design. This means that a ‘model’,
as he defines it, always is superiour to a series; models
have ‘style’, that is to say, a specific chronological and
geographical identity, while series products have no
such characteristics. In his view, models are originals,
the series products are imitations or copies of the
original. This also means that the serial product is of
inferior cultural value in comparison to the model.
For example, he exemplifies the differences between
model and series by citing a prestigious car, the
Facel-Vega, as opposed to a Citoën 2CV, or between
haute couture in contradistinction to ready to wear
garments. One of Baudrillard’s most puzzling ideas
is expressed in his statement that at the level of
“pure function … there cannot be any models …”. In
general, he plays down the meaning of what he calls
the “primary function” of objects. And as he suggests
a process of “personalization” as part of the operation
of “the psycho-social dynamic of model and series”,
he only leaves room for this process at the level of
the “secondary function” of objects [1 p. 140].1 Finally,
it seems that to him the privileged instantiation of
personalization is the process during which art works
attain their metaphysical value. But this model applies to
industrial artifacts only on rare occasions, such as the
fictional one alluded to above where a user is hanging
jeans on the wall in a place usually reserved for posters,
prints or paintings.
However, the standard situations in everyday life with
their numerous, trivial examples of people making use
of objects in their doings, are no more than hinted at in
Baudrillard. In everyday life tools, articles of furniture
and clothing, etc. continuously afford their assistance to
their users and perform their function as expected and,
at the same time, the objects in question are personal
belongings that gradually receive traces of having been
used or soiled. The effect of this daily intercourse is that
the user-owner may be able to recognize the particular
object – his or her object - among hundreds of similar
objects. Eventually, in everyday life the process of
‘personalization’ is not least a material one.
Second, and in order to proceed from this point,
what is missing in Baudrillard’s essayistic reasoning
is a more focused discussion of actual processes of
Design and semantics of form and movement
91
personalization. The determinations and definitions
offered in the text are merely abstract semiological
references to t systems of differences as the cultural
system. Such a discussion could have been informed
by ethnographies of groups and individuals. In relation
to jeans, a number of such ethnographies have been
published recently along with cultural histories
of exemplary cases within the framework of the
international research project, Global Denim [4]. The
absence of a more elaborate strategy of how to get valid
knowledge of the processes in which things are being
personalized, has as its consequence that Baudrillard
does not really provide a methodological framework for
addressing how, where, when, and why these processes
of appropriation are accomplished.
Another keyword for such processes is ‘singularization’,
as suggested by anthropologist Igor Kopytoff in his
contribution to Appadurai’s seminal book on the
social and cultural trajectories of commodities [5,
pp. 64-91]. In his search for a symbolical economy
(or “moral economy” as he writes) behind the
monetary framework of exchange, Kopytoff discusses
the processes of commoditization and de- or
recommoditization, and his general point is that the
decommoditization is a precondition of singularization,
a process which implies that objects are “culturally
redefined”. 2 Not the change of status per se, nor
change as such is enough to allow for the singularization
to take place [5, p. 67]. The social position of
singularized objects is due to the fact of “being pulled
out of their usual commodity sphere” [5, p. 74], hence
the objects have become “priceless”, meaning either
“uniquely valuable” or “uniquely worthless” [5, p. 75].
There are several forms of decommoditization and
singularization, both public and private – from a state
enforcing singularization of places, buildings, and objects
– via change of status by social groups to the common
consumer using applications or other alterations to
single out, say, a pair of jeans.
According to Kopytoff’s analysis the logic behind this is
that in complex, capitalist and heterogenous societies
the in-build and dominant tendency of the market
economy is to subjugate and commodizise whatever can
be turned into marketable entities. On the other hand,
Kopytoff states:
92
Design and semantics of form and movement
There is clearly a yearning for singularization in complex
societies. Much of it is satisfied individually, by private
singularization, often on principles as mundane as the fate
of heirlooms and old slippers alike – the longevity of the
relation assimilates them in some sense to the person and
makes parting from them unthinkable [5, p. 80].
The ‘assimilation’ issue in relation to singularization is
of special interest in the present context. Singularization
is not necessarily the same as privatization. The intertwining of persons and objects may reflect general
cultural patterns, for instance when specific groups
or classes of objects for some reason or another are
conventionalized as signs of a social category, once
named status symbolism. Singularization concerns not
only the kind of assimilation Kopytoff points at in the
above quote, but includes also the social dynamics
of singling out objects as something special and of
invaluable importance to national heritage, cultural
identity, etc. In this case, singularization is effected by
use of various forms of power based on the possession
of necessary amounts of a relevant type of capital political, economic, social, cultural, rhetorical, etc.
All this is actually included in Kopytoff’s grounded
theorizing, but the core question of how, where,
when, by and for whom things are singularized is only
answered in general terms, while instantiations are what
they are, namely concrete ethnographies that do not
add up to a proper theoretical stand, nor do they outline
methodological strategies that may stimulate further
scrutiny of the double and triple statuses of objects.
What will be the concern in the remaining part of
this paper is, firstly, the theoretical issues of how to
identify what may be termed the status stratification
of serial objects; next, this theoretical framework will
be applied to the case of jeans, not in the shape of
concrete ethnographies, but of an analytical model. And
finally, the possible role of design (the design process)
will be addressed speculatively. In this connection it is
worth noting that neither Baudrillard, nor Kopytoff are
focusing the fact that the objects they talk about, the
consumer’s commodities in the market and the user’s
personal belongings, are not reflected as forms the
character of which has been suggested or determined
by a form-giver, a designer.
The theoretical framework to be introduced briefly
and applied more or less schematically to the issues
of model and series is Charles S. Peirce’s sign theory.
2
It is Kopytoff who
introduces the term
‘commoditization’
(with various prefixes).
This spelling will
not be questioned
here, though other
writers (including the
present writer) prefer
‘commodification’.
3 Things and Signs
In contradistinction to the various semiologies bases on
Saussure’s structural linguistics, in the case of Peirce’s
theory of a general semiotics, we do not have to
operate by means of metaphorization. While Saussure’s
is a theory of conventional signs, the privileged instance
of which is language signs and sign systems that function
as the system of language signs do, Peirce’s semiotic
theory is based on the assumption that anything may
function as a sign in so far as it is actually perceived as
a sign of something. This is the first general point to
be made in this context. The next is that in Peirce for
anything to be understood as a sign it must exist as
a perceivable entity. Both conditions are of immense
importance to any attempt to explore the functions
of things as signs.
To put these arguments more precisely, anything is not
a sign per se, and the implication of this is that meanings
do not simply reside in things, nor does it in designed
objects. At the very moment an observer (onlooker,
listener, etc.) decides to take something for a sign, a
meaning of that sign is inferred, and this inference is
grounded in the observer’s mobilization of relevant
quantities of acquired knowledge, learning, proficiency,
understanding, and mental disposition as well as a fund
of life experiences. What trigger off the inferential act
is the specific interests that urge the observer to try to
understand what exists or happens around him or her
in order to sense, conclude and act accordingly. This is
why things and actions may mean differently to different
observers and actors.
Another point is Peirce’s stipulation of the physical
character, the material quality, of a sign manifestation.
European semiology or semiotics is based on
metaphysics, a metaphysics that does not evaporate
when applied to meanings generated by material
substances. In Peirce a sign manifestation is defined as
(1) an interpretation of the materiality, hence sensorial
characteristics, of the sign; (2) The interpretation is
accomplished by either focusing exclusively on the
material features, or by relating the perceived material
features to something else. This other is what the
material features actually are (or have been) in direct
relation to, or what a convention has established as a
interpretative rule; (3) The interpretative act takes place
in the mind of someone who has an interest in knowing
the meaning of something that comes to the same
someone’s mind via the senses.
More formally, the above points are phrased like this
in a definition from 1897, at the time Peirce used the
term ‘representamen’ as substitute for ‘sign’, since the
meaning of the latter may be confused with his general
use of the same term:
A sign, or representamen, is something which stands to
somebody for something in some respect or capacity. It
addresses somebody, that is, creates in the mind of that
person an equivalent sign, or perhaps a more developed
sign. That sign which it creates I call the interpretant of the
first sign. The sign stands for something, its object. It stands
for that object, not in all respects, but in reference to a sort
of idea, which I have sometimes called the ground of the
representamen [6, 228].
The general point is that all three components in the
triadic structure of the sign – representamen, object
and interpretant - are in play in a sign process, a
semiosis. Peirce’s semiotic theory is notoriously difficult
to summarize since the theory was revised from time
to time without reaching a final or conclusive form. As
indicated above, he also employed various terminologies
for his many differentiations. However, without any
ambition of presenting an infallible representation, the
model below is intended as a cumulative overview of
Peirce semiotics.
In addition to the illustrative function of this model, it
also makes explicit the underlying phenomenological
structure of Peirce’s thinking. His phenomenology
(or phaneroscopy, as he also termed it) represents an
attempt to reduce radically the number of categories
to three. Instead of the 10-12 proposed in Aristotle’s
and Kant’s epistemologies, it is Peirce’s claim that all
that is needed is to reflect that phenomena may exist
as a first due to its inherent qualities; as a second
referring to the relation of a quality to something
else; and as a third since a phenomenon may exist
meaningfully as such because of a law, a convention or
a tradition. This categorization establishes the three
so-called trichotomies. The places of operation for the
trichotomies are marked by bold lines, arrows and dots
in the model which, by the way, happens to be an icon,
more specifically a diagram.
The icon is defined by Peirce as a sign, the meaning
of which is based on its own qualities; the implication
of this is that representamen share at least some
Design and semantics of form and movement
93
of the qualities the object has. The icon, then, is a
representation that represents the object because of
its own perceivable qualities. In other words, qualities
are what matters in relation to icons. And qualities as
analyzed in the first trichotomy are what are of interest
in the present context. This means that ‘the rest’ of
Peirce’s notoriously complicated sign theory is left out
here, with a few exceptions, in order to concentrate
on the qualities of things as signs that exist in reality as
design(ed objects), that is to say, the physical dimension
of meaning as opposed to the metaphysical. There exist
recommendable general introductions to the whole of
Peirce’s semiotic theory [7], [8], [9], while the following
as elucidation is but an emergency ration.
This means that the determination of the qualities of
signs has its own trichotomic structure: tuone is a first,
token is a second, while type is a third. And again, tuone
is a quality that functions as a sign, as when the very
experience of a particular colour tone of blue is taken
for a sign. Token refers to a blueness that functions as
sign qua its actual presence as a determining quality
in a specific case. Finally, it is a type if a specific blue
colour tone is a sign by convention, as is the case in
all instances where blueness is an all-embracing and
inescapable, ‘law like’ quality defining all manifestations
of that particular sign.
REPRESENTAMEN
Fig. 1. A cumulative model of Charles Sanders Peirce’s
semiotics.
As indicated in the box concerning the first trichotomy,
Peirce used various terminologies in different version
of his sign theory. Since the term prototype has been
used previously, the register of tone, token, and type
[6, 4: 537] is used here. In one place he radicalized the
idea of tone by terming its specific qualitative modality
tuone (a modification of the terms tone and tune). By
that he wanted to stress the very uniqueness of quality
[10, 339d: 533]. Following Liszka’s definitions, the
first trichotomy of qualitative modalities is about the
“presentative character” of signs [8 pp. 35ff.].
94
Design and semantics of form and movement
A very short version of this idea applied to design is,
for example: the mug I am using for coffee in my office
at the university has a number of traces of having been
used (the rim is chipped in one place) and only cleaned
occasionally. My colleague in the office next to mine
has a similar mug and uses it for drinking coffee and,
occasionally, tea. My mug is my mug, and I will never
mistake it for my colleague’s because of the chipped
rim. We have both received the mugs some years ago
as a gift from the university’s vice-chancellor. In fact,
all university staff have the same mug, and therefore I
immediately wrote my initials on the bottom side with
china painting in order to avoid future mistakes. The
whole series of mugs were produced on the basis of
an anonymous design, and the occasion that inspired
the vice-chancellor to act as a donor of china ware,
was a board decision on a branding strategy which
should bring public attention to the university, and at
the same time engage the staff in discussions and the
dissemination of the so-called values of the institution.
To this effect, three value indicating slogans were
printed in decorative bands on the exterior of the mug.
trousers, and the kind of meaning construction
draws on design, colour, texture, etc. of denim. 3
This means that once again the focus will be on the
firstness, secondness, and thirdness of the firstness
of the sign, i.e. the presentative capacity of jeans as
sign, while the secondness of the sign’s secondness –
the index sign - will be included briefly.
According to Peirce’s concepts, my mug is a toune,
in that chippings, cracks, and initials give it its unique
quality, while my and my colleague’s and the rest of the
staff’s mugs are all tokens, that is, singular examples
from the series of university mugs, and they have all
the potentiality of becoming singled out as tuones by
being used intensively and by application of letters,
figures, labels, etc. The shared quality of all the mugs is
determined by the vice-chancellor (or the management,
or the board) with a designer’s professional assistance,
hence the designer’s final model or drawing functions as
the proto-type of the series of mugs distributed around
at the university’s campuses.
One may now ask how the meanings referring to the
quality of the mugs are established, and how far the
designer’s decision about material, shape, decoration,
etc. affects the meaning of my mug considered a sign
– by me and people around me who may encounter
my ‘private’ coffee-mug. Suffice here to say that it
depends solely on what is found relevant or necessary
to know about it and – to be sure – which status is of
interest: my mug, the mug as multiple, or this particular
university mug as a particular type.
A few remarks should be made in order to pin down
an historical point of reference: The most famous jeans
design is undoubtedly the Levi’s 501. The first patented
design was introduced in 1873. At that time it had three
riveted pockets, two in front, and one behind. Later a
second back pocket was added, and still later the small,
so-called watch pocket was introduced. This five-pocket
model is the distinctive form of Levi’s 501 as far as the
number of fabric units is concerned. Other distinctive
features are the yellow stitching, double arcuade on
back pockets, visible out seam, felled inseam, copper
rivets, etc. Many of the design historical details of Levi’s
501 are not settled definitively, but the introduction
of copper rivets and the successive expansion of the
pocket system from three to five units are documented
in the archives of patent agencies. The first application
for the 501 patent was based on the inventions made
by a Reno tailor, Jacob Davis, whose supplier of fabric
was a San Francisco merchant named Levi-Strauss. Davis
offered Levi-Strauss a partnership in order to be able to
finance the application for the patent, and by May 1873
the rights were granted. Several versions exist of the
story of what gave Davis the impetus to invent this type
of wear-proof dungarees, but it seems safe to conclude
that he is the first to be credited the design of Levi’s 501.
The basic design has been modified several times. Its
cut, especially in relation to the width of the legs, has
changed gradually to keep pace with shifts in fashion.
The Davis patent of 1873 is the master prototype, and
in technical terms all subsequent modifications and
adaptations of the original to fashion trends establish
different prototypes, irrespective of how the modifying
process is named: design, redesign or styling. However,
the following arguments are based on the assumption
that the prototype simply is a generic pair of Levi’s
501 without considering their exact chronological and
sociological dates.
In one form or another, Levi’s 501 exists as a prototype,
in Peirce’s terminology: a type or legisign. The latter
term means literally law sign, that is, a sign which
4 Jeans as signs
3
For the time being,
the most reliable
accounts of the history
of jeans are Sullivan
[15] and Marsh &
Trynka [16], while
Miller & Woodward
[4] include a number
of fresh perspectives
on both the history
and cultural function
of jeans.
The exciting story of blue jeans will not be repeated
or summarized here, though the function of blue jeans
as cultural signs seems to be an inexhaustible source
of information in relation to the issues discussed in
Baudrillard and Kopytoff. And also, the function of jeans
as artistic outfit and as works of art in Beuys depends
to a certain degree on the histories and mythologies of
jeans. As a case, the history of jeans is perfectly suited
to serve as illustration of Peirce’s semiotic thinking,
both in general and in the particulars of, for example,
the trichotomies.
However interesting this story is from a design
historical perspective, the efforts in the present context
are concentrated on the material aspect of this type of
Design and semantics of form and movement
95
meaning in terms of quality is a convention or rule.
The function of the type is to stipulate the qualities of
any materialization of the specific features that make
an identification of type possible. Each instantiation of
the type is a token of the type, also named sinsign, as
registered in the schematic rendering above, meaning
the actual, singular manifestation which is, in fact, a
singular existing among identical singulars, each of
them owing collectively their specific qualities to the
type. As tokens, any exemplar in the series of tokens is
replaceable by another exemplar from the same series.
Tokens may gradually alter or suddenly change and
thereby begin to diverge from its siblings. Alterations
and changes produce a unique quality, a quality which
singles out one particular token as a tuone (or qualisign)
having now something of its own, and consequently
becoming irreplaceable.
It is the tuone quality that allows for and facilitates
processes of personalization and singularization. In
relation to jeans, this is a point stated in several studies
of jeans culture and in a number of ethnographies
in relation to the Global Denim project, referred to
above, in this and other contexts Sophie Woodward
also employs the term ‘personalization’ [11], [12]. It
should be emphasized that singularization also may be
the case if blue jeans are established as part of a dress
code for members of, say, a biker’s club. In this instance,
the singularization refers either to the token quality,
in so far as it is stipulated that mark and model is, say,
Levi’s 501, or to the type, if the conventionalization only
have a general reference to black jeans, for instance,
as a piece of apparel. In the latter case, any pair of
black jeans represents the required quality, blackness,
irrespective of design, cut, or trademark, etc.
The type Levi’s 501 warrants for the authenticity of
any pair from any pile of 501s in any shop around the
world. If a pair of 501s are lost or damaged during a
trip abroad, an unproblematic replacement is possible
as long as the wearer’s need is confined to having the
opportunity of wearing 501s. Any 501s will replace any
other pair of 501s as long as focus is on 501 tokens. The
type guarantees that the pair of jeans with 501 qualities
bought in a shop somewhere is the real deal and that
the price is acceptable, compared to cheaper jeans,
which very probably have similar, but not identical,
functional qualities. The wearing of 501s may secure
acceptance among the user’s peers. The opposite may
96
Design and semantics of form and movement
be the case if wearing of this particular mark and design
of jeans is politicized for one reason or another, to
the effect that wearing as well as not-wearing become
a manifestation of a political standing. Semiotically
speaking, for instance, non-wearing becomes a sign of
protest to those fellow citizens who are familiar with
the political code. The type may in this context be
subjected to a boycott, and this means that the very
appearance of 501s tokens on the lower body of a user
causes condemnation from the surroundings. The user’s
relation to his or her 501s tuones may in this case be
rather traumatic and something that has to be taken
care of only in privacy.
The only quality at stake in relation to political activism
is what establishes the type and not the functional
qualities. Loyalty and disloyalty are a matter of
attachment to or distancing from immaterial values
that are shaped by culture. Of course, loyalty may arise
from and grow in the course of the users long termed
use of and interaction with a pair of jeans. The effect
of this is that gradually the tuone quality develops and
the relationship will pivot around that quality: the jeans
eventually become representations of the user, and the
user becomes a representation of the jeans. In order
to unfold this argument, some ideas have been offered
by sociologist Tim Dant, who has described nicely the
appropriation process and what happens to jeans during
periods of wearing and maintenance [13 pp.102f.]. The
transformative potential of jeans with regard to their
function in introducing alternative garment norms for
both sexes in the 1950s is summed up by Dant as follows:
Jeans went against the grain of the dominant clothes culture
of western modernity and reversed many established clothing
signifiers. They were made of cotton (vegetable) instead of
wool (animal); fixed in shape instead of tailored; had visible
seams but no pressed creases; revealed the form of the body
rather than covering it [13, p.103].
In particular, Dant wants to explore the last point
concerning the body-clothing relation, while he only
cites a growing number of accounts of jeans as fashion
and of the design history of jeans.
Instead he focusses on “the nature of the material
form” and what he calls the “ambivalence of meaning”
[13, p.103]. The ambivalence hinted at relates to the
many shifts in jeans’ cultural meaning, from work
wear to high fashion, but in the following, the issue of
ambivalence or, better, polysemy will be related to the
nature of material and form, and this implies that Dant’s
observations are partly underlined, partly reworked.
Most interesting is Dant’s description of how the twilled
denim gradually takes up the shape of the wearer’s body.
This is due to the thickness and stiffness of the denim
which means that it is stretched against the skin, at
least in the beginning. One may add that being worn and
washed regularly, the fabric is gradually softened and
becomes more flexible, thus the jeans will be even more
‘susceptible’ to the imprints made by knees, thighs and
buttock. The result is that the jeans and their intimate
coexistence with the wearer eventually turn into an
icon for their user as they adopt the shape of the body.
Originally a neutral envelope, now a second ‘skin’ the
jeans function as a membrane reflecting the shape of
the lower body and having developed the same shape
as the wearer. The fact that the adopted jeans are an
iconic representation is due to another fact: the iconic
sign function is made possible only because the jeans
also are an index, since the form modifications of the
jeans is caused by the direct contact between them
and their wearer.
In addition to this, the jeans may also over time reveal
the wearer’s habits as to sitting position, the placement
of knees in relation to chairs, tables, etc., just as the
jeans receives additional imprints of use: spots caused
by tints of paint, acidiferous foods and liquids and the
like as well as traces of small and big accidents with
tools, climbing, falling, etc. All of this causes a process
during which singularization and personalization become
increasingly manifest. Such personalized jeans are indeed
irreplaceable. Though, from the perspective of both
type and token, the tuone quality is arbitrary.
The course in the direction of personalization may also
go off another way as immaterial values, experiences
and events, so to speak, are materialized by means of
projection. The presence of the jeans in the life of the
user implies that they may also function as a material
anchorage for memories, dreams and traumas. They
become monuments in and of the owner’s life, and
their presence is a constant reminder of events, people
and happy hours, thus they are difficult to get rid of
since they are a part of the owner’s self. On the other
hand, the scrapping and rejection of the jeans may be
furthered in situations in which it becomes evident
that the many visible and invisible traces are no longer
‘removable’. The jeans again function as a constant
reminder, but this time of past events, people or feelings
that the owner now wants to remove from the fund of
memories. The jeans are disposed of as abjects, and the
owner-user gains a new freedom of thought and action.
To others the traumatized jeans may have appeared as a
pair of ordinary, standard jeans, but to the owner-user
they had an ambivalent, not to say intolerable, tuone
quality that, however invisible, to a considerable degree
limited his or her freedom.
5 Designing Qualities – A Double Strategy?
The jeans case is not unique to the realities of design
and product culture. The observations made here
may be adapted to other cases with due revisions of
perspective if and when the model-series relationship
is in focus. For example, one may ask how to identify
the design issue in relation to the meaning function of
the material dimension of design objects and product
culture in general.
From time to time, the work of designers is selfevidently required as fashion trends demand changes,
unless such changes are solved on the factory floor
by the manufacturers’ technical staff. The designers’
contribution is certainly needed if jeans are planned
to be marketed in the upper end of the fashion system
as a creation by a designer with a high profile. And it
is known that designers once were asked to include
so-called slob lines in the fabric in order to facilitate
and accelerate the appearance of traces of wear and
tear. The ‘correct’ look of vintage jeans is considered
a sign of age, but in this case age is a material illusion,
while the tuone quality is genuine. The same may be
said in relation to other techniques of artificial aging,
such as stone wash, chemical de-dyeing, removal of the
weft in selected places and other forms of intended
decomposition of the fabric. In such cases aspects of
design are involved in defining the effective quality.
However, the material effects are developed arbitrarily
since whoever decided the catalyzing attack on the
fabric only initiated the process, not its actual direction.
Then, one may state that in cases like these, users
realize the designer’s intention, that is to say, they are
in fact completing the design process, while this is not
the case with the university mugs discussed previously.
In the latter case, partial destruction is caused by
inappropriate handling.
Design and semantics of form and movement
97
The example of the production of tuone qualities
in jeans makes some challenging dilemmas of design
evident. It is obvious that the designers’ job is best
understood in relation to the production of prototypes,
and the jeans case also indicates that the design may
have a long termed impact on alterations, though
the alterations in question are in fact also processes
of accelerated decomposition. And in principle, this
policy is not dissimilar to the strategy of planned
obsolescence introduced in the 1920s. Both types of
decision making are allegedly a service to consumers.
Without moral pretentions, the question could be
asked how far designers should be concerned with the
material destiny of their designs, or to what degree
the fate of materials has to be prefigured in and by the
qualities of the prototype. In some cases, consumers
would want, even need, things that are pre-demolished,
in other cases they would certainly not. The jeans case
obviously reveals that for body related designs to which
an intimate relationship may develop singularization
and personalization are at the core of how users
relate to their things. The effect of singularization and
personalization is the creation of a particular quality,
both physical and emotional – a tuone quality.
University of Massechusetts Press.
[11] Woodward, S. (2005). Looking good: Feeling right –
aesthetics of the self. In S. Küchler, & D. Miller (eds.), Clothing
as Material Culture. Oxford: Berg.
[12] Woodward, S. (2010). Jeanealogies; Materiality and the
(im)permanence of relationships and intimacy. D. Miller, & S.
Woodward (eds.), Global Denim (pp. 145-158). Oxford: Berg.
[13] Dant, T. (1999) Material Culture in the Social World.
Buckingham: Open University Press.
[14] Barthes, R. (1973). Elements of Semiology (French original,
1964). Transl. by Annette Lavers and Colin Smith. New York:
Hill and Vang.
[15] Sullivan, J. (2005). Jeans. A Cultural History of an American
Icon, New York: Gotham Books.
[16] Marsh, G., & Trynka, P. (2002). Denim: From Cowboys to
Catwalks. London: Aurum Press.
References
[1] Baudrillard, J. (1996). The Systems of Objects. Transl.
by James Benedict. London: Verso.
[2] Tisdal, C. (1979). Joseph Beuys. New York: The Solomon
R. Guggenheim Museum.
[3] http://www.dw-world.de/dw/article/0,,1866423,00.html
Retrieved November 28, 2011
[4] Miller, D., & Woodward, S. (Eds.) (2010). Global Denim.
Oxford: Berg.
[5] Kopytoff, I. (1986). The cultural biography of things:
Commoditization as process. In A. Appadurai (Ed.), The Social
Life of Things: Commodities in Cultural Perspective (pp. 64-91).
Cambridge: Cambridge University Press.
[6] Peirce, C.S. Collected Papers of Charles S. Peirce, 1-8.
Cambridge: Harvard University Press, 1931-1968.
[7] Johansen, J.D. (1993) Dialogic Semiosis: An Essay on Signs and
Meaning. Bloomington: Indiana University Press.
[8] Liszka, J.J. (1996). A General Introduction to the Semiotics of
Charles Sanders Peirce. Bloomington: Indiana University Press.
Jorn Guldberg
[9] Short, T.L. (2007). Peirce’s Theory of Signs. Cambridge:
Design Studies
Cambridge University Press.
Department,
[10] Peirce, C.S. (1967). MS, Annotated Catalogue of the Papers
University of
of Charles S. Peirce. Compilation by Richard S. Robin. Amherst:
Southern Denmark
98
Design and semantics of form and movement
Baki Kocaballi, Petra Gemeinboeck, Rob Saunders, Lian Loke, Andy Dong
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Embracing relational agency
in design process
Abstract
Our research investigates how the design process can
accommodate a relational view of agency. According to
the relational view, agency - or capacities of action - is
neither an attribute of subjects nor of objects. The
relational view of agency in design may allow designers
to recognize and support the diversity and richness
involved in human agency. To this end, we developed
six design qualities to embrace the relational view of
agency in design process. Using these qualities, we have
created design inscriptions in the forms of materials
and process constructs and applied them in a series
of participatory design workshops, focusing on the
notion of connectedness. We present how effective our
inscriptions were in supporting the ASD qualities
in each workshop.
Keywords
Design process, relational agency, participatory design,
Actor-Network Theory.
1 Introduction
The concept of agency is defined in its simplest sense
as the capacity for action or transformative capacity
[1]. Yet, there has been ongoing debate surrounding
definition, emergence and possession of agency in
artificial intelligence, cognitive science, philosophy and
many other fields. One particular point of controversy
is related to the attribution of agency to entities.
As opposed to the traditional humanist view of agency
as a property of individual entities, Barad [2] suggests
that agency is not an attribute of subjects nor of objects
or systems but is the ongoing reconfigurations of
the world, an enactment. Agency emerges out of the
dynamism between entities.
Our research aims to explore how design process can
embrace the relational nature of human agency. We
suggest six qualities to characterize a more relational
design approach referred to as Agency Sensitive Design
(ASD): relationality, visibility, multiplicity, accountability,
duality and configurability. We used qualities in a similar
way to those featured in Bardzell’s [3] study. Bardzell
developed a “constellation” of design qualities as part of
a feminist interaction design program focusing on values
like agency, empowerment, diversity and social justice.
The qualities we propose are similar to those developed
by Bardzell. However, somewhat different from
Bardzell’s approach, our qualities primarily focus on
ways of promoting relational agency: more in the nature
of process-oriented qualities characterizing how a design
process might embody a relational view of agency, our
qualities provide conceptual lenses through which to gain
a relational understanding of the situation. As well, they
aim to increase the designers’ awareness of relationality
of human agency, i.e., the relational, embodied and
situated characteristics of human action, allowing them
to tune their practices to recognize and support the
diversity and richness involved in human agency.
Design and semantics of form and movement
99
In the paper, when required, we used the term human
agency as a convenient way of highlighting the main
actor of interest in a situation. It is a term just to refer
to the human side of the relational agency. It is not
problematic to use ‘human’ in front of agency as long as
one is aware of the relational nature of agency. In this
study, we run two parallel streams of research: research
into ways of integrating ASD qualities into design
process; and research into various forms of connections
between humans and technologies. In parallel to this,
our evaluation has two streams: we evaluate the forms
of connections and, as well, our ways of exploring the
forms of connections.
2 Design and Agency
Design activities, in varying degrees, ultimately aim to
create, modify, enable and/or constrain some capacities
of action through designed artefacts. Designers inscribe
values, visions, programs of actions and modalities of
perception into technology design. Akrich [4] explains
the notion of inscriptions in technology design in the
following way:
Designers thus define actors with specific tastes,
competences, motives, aspirations, political prejudices, and
the rest, … A large part of the work of innovators is that of
“inscribing” this vision of (or prediction about) the world in
the technical content of the new object. To be sure, it may be
that no actors will come forward to play the roles envisaged
by the designer. Or users may define quite different roles of
their own [4, p.208]
The technical content of the objects embodies a script
similar to a film script, defining the actors, roles and
their settings [4]. A script involves, in varying strengths,
“programs of action” that are “translated” in practice
[5]. Translations are processes in which “the identity
of actors, the possibility of interaction and the margins
of manoeuvre are negotiated and delimited” [6, p.203].
However, should the translation processes vary, these
inscribed programs of action may not succeed; in
addition, actual interactions between entities may unfold
in unexpected ways.
The strength of an inscription may vary from very
strong, that is, imposing one particular inflexible
program of action, to very weak, offering many
flexible programs of action. Increasing the strength
of an inscription can also be considered as an attempt
100
Design and semantics of form and movement
to confine the relational character of human agency.
Strong inscriptions belong to a perspective of design
that aims to predict, prescribe and control the kind of
relations between humans and technologies and the
ways in which their interaction unfolds. Repeatability,
consistency and reliability are particular kinds of
qualities that characterize the human-technology
interactions shaped by strong inscriptions. Although
these are definitely desirable qualities for some
settings such as legal, medical and educational, they
may not be very suitable for some other situations
where appropriation, personalization, adaptation,
entertainment and exploration are needed [7]. In
practice, the human-technology interactions may
happen in unexpected ways or as Akrich [4] pointed
out users’ definitions of roles may deviate from the
intended roles. Thus, rather than assuming agency as
a predictable and fully controllable phenomenon, we
may acknowledge its relational character and develop
sensitivities to manage relationality in the design and
use of technologies. In this way, we can see relationality
with its ambiguities and contingencies as a resource for
design [7] [8] and formulate design solutions to deal
with unexpected situations that may happen during the
use of technologies.
3 Towards a Relational Approach to Design
A relational approach to design process might be
beneficial in many ways: i) it provides resources and
mechanisms to deal with unexpected situations
[7] [8]; ii) it supports responsible and ethical practices
by recognizing and supporting different and marginalized
actors, and their ways of knowing and doing things
[9]; iii) it supports the creative potential of users by
supporting user appropriation and what Aanestad [10]
refers to as design in use; and iv) it supports innovations
by making design process as open and as inclusive as
possible [11].
Our approach to developing a relational approach
to design is referred to as Agency Sensitive Design
(ASD). Our aim is not to replace existing design
methodologies but rather to complement them by
developing sensitivities in the form of design qualities.
The fundamental principle of ASD is recognizing and
supporting variety in the formation and exhibition
of agency in the design and use of technologies.
This principle includes a large range of aspects
of relational agency in design. In a design process,
while the formation of agency refers to the construction
of a heterogeneous network or assemblage of human
and non-human actors, exhibition of agency refers to
the effects of that network. We need to recognize the
influence of multiple sources on design problems and
then find ways to consider their concerns and effects.
Similarly, we need to support variations in the network’s
effects, i.e., the collective actions of actors. However,
it is important to note that variations in the network’s
effects may not be desirable for safety critical or high
reliability required situations. Thus, relationality should
be tailored very carefully in these cases.
Drawing upon the extant works and approaches in
Human-Computer Interaction, Participatory Design,
Actor-Network Theory and Science and Technology
Studies, we developed six design qualities: relationality,
visibility, multiplicity, configurability, accountability and
duality. These broad categorical qualities, which may
overlap and be further divided into a few other qualities,
provide a useful starting point from which to articulate
some of the implications of a relational view of agency
for the design process. As well, they aim to increase the
designers’ awareness of relationality, allow them to tune
their design practices to accommodate the diversity and
richness involved in human agency. In the next section,
we will explain these qualities briefly. For an extended
presentation of these qualities, please see [12].
3.1 Relationality
The quality of relationality refers to the connectedness
and relatedness of human and non-human actors
comprising heterogeneous networks [5] or sociomaterial arrangements [13] in which humans and
non-humans co-constitute each other through their
interactions. According to Suchman, relationality
emphasizes the “relational character of our capacities
for action, the constructed nature of subjects and
objects, resemblances and differences; and the
corporeal grounds of knowing and action” [14, p.3].
In design processes, the quality of relationality asks for
three sensitivities: (i) understanding of mutual influence,
shaping and co-constitution of actors and artefacts; (ii)
embracing and supporting emergent and improvised
action and (iii) consideration of the system as an
assemblage/network of actors, artefacts or collective
hybrids. In order to develop these sensitivities, we
first need to stop formulating design solutions based
upon the assumption of a well-defined individual with
fixed characteristics and capacities of action. Design
solutions should recognize and support the existence of
the multiple individuals embodied in one individual and
the possibility of multiple enactments of one individual
within a network of other human and non-human actors
interacting with each other and exhibiting different
capacities for action [11]. Rather than prescribe or
control, we may design for appropriation and designin-use, interactive systems do not impose a particular
pattern of action; rather, they provide a space of
negotiation in which individuals can exercise their
“multiple” capacities of action in creative ways.
3.2 Visibility
Visibility, one of the most essential qualities, facilitates
responsible design and the emergence of different
arrangements or couplings between humans and
technologies. The quality of visibility, which plays a key
role in developing other sensitivities such as multiplicity
and accountability in the design process, involves
variously making visible invisible work, human and
non-human actors, and infrastructure and interactions
in both design and use of technologies. Visibility not
only facilitates a heightening of the overall awareness
of human actors of themselves and of others, but also
helps the performance of more responsible design
practices [9] [15] and discovery of new opportunities,
constraints and matters of concern in design process [5].
Quality of visibility operates in both technology
design and use. Visibility in technology design refers
to recognizing every human and non-human actor
and their roles in the formulation of design problem
and the design process. This means that the different
values, views and concerns of the human actors - and
various affordances of non-human actors - need to be
explicated and considered. Moreover, the term ‘visibility
in technology use’ refers to keeping the boundaries
and interactions between all humans and technologies
distinct and observable. Seamful design [7] advocates
the use of (beautiful) seams in interactive systems:
seams can be basically gaps and breaks in functionality,
and boundaries between different components or
systems. Seamful design deliberately makes the seams
visible and encourages system users to appropriate
them as a resource for reflection and creative
engagement.
Design and semantics of form and movement
101
3.3 Multiplicity
The quality of multiplicity refers to multiplicity in ways
of knowing, performing and representing, which entail
participation of multiple and heterogeneous sources of
influence in the design process. Collaborative, generous
and flexible methods and tools such as sketches, low-fi
prototypes, rich pictures, and cartographic maps could
prove useful in obtaining multiplicity in representation.
These rich representations are particularly important
vis-à-vis keeping the concerns of the different
stakeholders or multiple sources of influence visible.
While the design process can embrace multiplicity by
supporting participatory, democratic and open practices
together with rich representations of multiple partial
forms of knowledge, design artefacts can embody
multiplicity by utilizing flexible, context-sensitive and
adaptive mechanisms.
3.4 Configurability
The design process does not stop after the technology
production phase but continues in the actual use of
technologies. In this broader view of design, the activity
of design continues in the sites of technology use and
is performed by users in the role of designers [10].
Aanestad describes this activity as ‘design in use’, a
process which mainly involves continuous organization
of activities and the re-configuration of relations
between human and technological actors [10]. Users
may opt to reconfigure or customize technologies and
tune their relationships with technologies. The quality
of configurability asks for developing mechanisms of
supporting design in use or tuning operations during the
use of technologies. This can be achieved by designing
open, modular and flexible technologies. Kahle defines
“openness of technology” as “the degree to which it
empowers users to take action, making technology
their own, rather than imposing its own foreign and
inflexible requirements and constraints” [16, p.35].
The quality of configurability, inline with other qualities,
supports variety in the formation of human capacity
of action. By virtue of their modular and flexible
structure, technologies may become less isolated and
take part in a network or ecology of other technologies
and humans [3] [11].
3.5 Accountability
The quality of accountability is applicable to both
humans and technologies. Button and Dourish [17]
102
Design and semantics of form and movement
define accountability as the property of action being
organised so as to be observable and reportable.
Whereas accountability of technological systems
entails the existence of accounts that systems provide
users with information about their own activities [17],
accountability of human actors requires them to be
aware of their own position relative to other actors
and taking responsibility for their own perspectives and
partial knowledge [18]. The quality of accountability
might be promoted by making visible the actors,
roles, their locations and system accounts. However,
an essential part of the designer’s task is to provide
other actors involved in the design with resources for
increasing critical awareness of the notion of located
accountability and its implications.
3.6 Duality
The quality of duality refers to consideration of the
dual characteristics of design decisions. Van der Velden
[9] maintains that technology is never neutral; neither
in use nor in non-use. Dual characteristics of design
decisions should be considered. Duality can manifest
itself in many forms, e.g., privileging/ignoring, inviting/
inhibiting and amplifying/diminishing. Our designs can
privilege the values of some actors while ignoring the
values of some other ones [15]. The inscription of
values into technologies is inevitable. However, the
problem is less about the inscription of particular kinds
of values and more about the invisible, unquestioned
and taken for granted values embedded in our thinking
and practices. Parallel to the quality of visibility, values
shaping our thinking and design decision should be
made visible and open to negotiation. Moreover, the
quality of duality involves consideration of both kinds
of invited and inhibited actions and accounting for their
implications.
4 Inscribing ASD Qualities
In our approach to developing ASD, we employed
two important concepts of ANT: inscriptions and
translations. We aimed to inscribe ASD qualities
into design process and assess the capacities of the
inscriptions to support ASD qualities in translation
processes taking place in design activities. To this
end, we conducted participatory design workshops
consisting of various activities in which we employed
at least one of the six ASD qualities in each session.
The workshops were situated in an early exploratory
phase of design process, in which a consideration of
relational view of agency can contribute more. In early
phases of design, actors or stakeholders of a design
problem are not aligned yet, and it is valuable to explore
various ways in which these actors might be aligned. As
designers usually aim to get a broader view of a design
problem, reveal concerns of stakeholders, and explore
the alternatives, ASD qualities with their emphasis on
variety and multiplicity of agency might particularly
prove useful for conducting exploratory activities in
early phases of design. Therefore, we decided to situate
our workshop in early phases of the design process.
Participatory design workshops, with their emphasis on
negotiation, diversity and co-construction of meaning,
provided us with a suitable play platform. Muller
explains the key characteristics of workshops as follows:
[W]orkshops usually introduce novel procedures that...
take people outside of their familiar knowledges and
activities, and must be negotiated and collectively defined
by the participants. Workshops are thus a kind of hybrid
or third space, in which diverse parties communicate
in a mutuality of unfamiliarity, and must create shared
knowledges and even the procedures for developing those
shared knowledges [19, p.9].
We imagined a system utilizing full body interaction and
ambient feedback. Full body interaction was selected
because of its capacity to support a large variety
of interactions between human body and machine
compared to more conventional ways of interaction
based on screen, mouse and keyboard. In order to
not limit the ways of interacting with the system, we
decided to not use a screen-based visual feedback,
which may confine the range of interactions between
body and screen. Instead, we preferred to use sound
feedback, which provides more ambient feedback and
does not limit the direction of body in space. In a very
broad sense, the system will capture the movements of
people, process it and provide some audio feedback.
Two researchers with backgrounds in interaction design
conducted three workshops each lasting approximately
four hours. We worked with two female dancers, aged
22 and 23, in the first workshop, two female interaction
designers, aged 22 and 23, in the second, one female and
one male musician, aged 34 and 36, in the third. As our
study involves design of human full body movements and
audio feedback, dancers with expertise in movement
improvisation and choreography, musicians with their
expertise in music improvisation and composition, and
interaction designers with their expertise in bringing
together different aspects of interactive systems
provided us with a suitable set of participants. They
allowed us to view the design problem from different
angles and reveal different concerns about the system
to be designed. The main theme of the workshops was
connectedness, which is a suitable concept to explore
various relations between humans and non-humans.
There were four sessions involving exploratory
activities: a silence session, a physical sensitivity session,
a rich-poster session and finally a machine-mediated
performance session. The activities in the sessions were
selected according to their potential of supporting ASD
qualities. However, the important point is less about this
particular set of activities than about bringing together
a diverse set of activities supporting ASD qualities and
facilitating multiple ways of knowing, performing and
relating. Thus, other kinds of activities can be added
or some extant activities might be removed. What is
important is to inscribe ASD qualities into the design
process.
Our focus was upon the quality of multiplicity as the
quality of multiplicity plays a key role in supporting the
fundamental principle of ASD, but we also supported
other qualities at varying degrees. The quality of
multiplicity was inscribed in the entire workshop
containing different kinds of activities and, as well, in
roles, representations, and mediums. Each workshop
session provided opportunities and resources for the
participants to engage in the design concept in various
ways. In addition, we, as researchers, tried to be as
flexible as possible: this was important as it eschewed
any possibility of subjecting multiplicity to obstacles. In
addition to the multiplicity as an overall quality of the
entire workshop, each workshop session embodied
at least one of the six ASD qualities. The qualities
of accountability and duality were not considered at
this stage. We conducted a pilot workshop and found
that the accountability was not very relevant in such
exploratory design activities and considering the duality
at this stage complicated the workshop activities.
In the silence session, participants are asked to close
their eyes, concentrate on and listen to the existence of
their own and their partner’s body and space. They are
also asked to sit down on the floor in a back-to-back
position. This session facilitates a connection between
Design and semantics of form and movement
103
participants through silence and breaks the dominance
of vision as a main modality of connecting with other
entities. In the silence activity, we aim to inscribe the
quality of visibility by increasing the sensations of other
modes of perception.
In the physical sensitivity session, participants perform
physical exercises encouraging interaction through body
movements. There are three short activities in the
session; in the first, one of the participants was asked
to touch the crown of her partner’s head and lead her
to and fro using pressure changes. The two are asked
to keep the contact between hand and crown. In the
second activity, the participants change roles, then
pursue the same activity. In the third, they are asked to
simultaneously touch each other’s crown and to repeat
the same to and fro movement. The first activity is
called palm-crown exchange, the second, reverse palmcrown exchange, and the third, simultaneous palmcrown exchange. This activity enables participants to
create a touch-based connection between two bodies
and experience giving and receiving roles in a humanhuman connection. In the physical sensitivity session,
we aim to inscribe the quality of relationality by
introducing a physical touch-based exercise in which
the sensations and movements of one body is tightly
coupled with another.
In the rich-poster session, the participants make a
collage of pictures provided on a sheet of A0-paper,
then annotate them according to the particular kind
of connection that each represents. In addition,
participants talk about three objects that they felt a
connection. In rich poster session, we aimed to inscribe
the quality of multiplicity and visibility by making
visible the various forms of connections illustrated by
participants’ selection of pictures and their annotations.
In the final machine-mediated performance session,
participants perform five short activities using three
technological devices: two wearable devices - one with
tilt and another with distance sensing capabilities and one webcam with motion sensing capability. The
task of participants is to explore different forms of
connection with other bodies through technologies that
allowed participants to create various sound effects
through their body movements. In other words, they
are invited to improvise movements and sounds by
using different technological devices. When needed, we
provide additional instructions for participants, which
could guide their performance. These instructions
104
Design and semantics of form and movement
include selecting a theme for activity or using some
constraints on movements such as making slow/fast
movements or being mobile/stationary in space. In this
session, we aim to inscribe the quality of multiplicity
and configurability by providing participants with three
technological devices enabling three different ways of
coupling between human, technology, and space, and
various straps that allow them to attach the devices to
any parts of their bodies. Tilt devices generate sound
effects based on movement in vertical and horizontal
dimensions: the rangefinder devices produce sounds
based on the changes in the distance within a 70cm
range. Finally, the webcam detects motion and triggers
musical notes according to the place of motion in space.
While the Wii-motes and the range-finder devices need
to be attached to the body, the webcam can be placed
somewhere in the space detached from the bodies. The
multiplicity in capacities of technological tools and ways
of coupling with the technologies allow participants to
explore and perform various connections through their
movements and sound effects.
5 Analysis Methodology
We identified some indicators that show the
effectiveness of session inscriptions on supporting the
Session
ASD Quality
Indicators for Effectiveness
Silence
Session
Visibility
r"NQMJàFETFOTBUJPOPGPUIFSNPEBMJUJFT
r&NFSHFODFPGBOZLJOETPGDPOOFDUJPOTUISPVHI
amplified modes of sensation
r"QQFBSBODFPGQSFWJPVTMZJOWJTJCMFBDUPST
Physical
Sensitivity
Session
Relationality
r'PMMPXJOHUIFJOTDSJQUJPOPGLFFQJOHUIFDPOUBDUBOE
maintaining the roles
r$PNNFOUTPGQBSUJDJQBOUTPOUIFNVUVBMOBUVSF
of connection, strategies of coordination, and
reciprocity
r4NPPUIDPPSEJOBUJPOPGNPWFNFOUT
Rich Poster
Session
Multiplicity,
Visibility
r7BSJFUZJOUIFUIFNFTPGQJDUVSFTBOESFMBUFE
associations.
r. BOJGFTUBUJPOPGEJGGFSFOUQFSTPOBMVOEFSTUBOEJOHT
of connection
r&NFSHFODFPGOFXDPOOFDUJPOT
MachineMediated
Performance
Session
Configurability,
Multiplicity
r7BSJFUZJOXBZTJOXIJDIIVNBOTBOEUFDIOPMPHJDBM
devices are coupled
r7BSJFUZJOXBZTJOXIJDIIVNBOTDPOOFDUXJUIPUIFS
humans and environment through technological
devices
Table 1. Indicators for effectiveness of session inscriptions in supporting
ASD qualities.
relevant ASD qualities (Table 1). These indicators were
translated to the context of the design activity from the
original definition of each ASD quality after watching all
video sequences and reading all the transcriptions.
Our analysis is based on our in-situ notes, interview
transcriptions, video sequences and posters. In the final
session, we developed an extended version of Laban’s
[20] effort categories to characterise the different
forms of movement-based connections. Laban’s effort
categories are useful for describing the temporal and
dynamic qualities of human movement. There are
four categories, each of which has two polar values:
i) Space: Direct/Indirect; ii) Weight: Strong/Light; iii)
Time: Sudden/Sustained; and (iv) Flow: Bound/Free.
We segmented the video sequences according to the
different body-technology-space arrangements. There
were various arrangements during a session but not all
of them allowed participants to create a connection, in
which they were able to coordinate their movements
and co-compose sound effects. After watching the
video sequences multiple times, we concluded that
the arrangements that lasted less than three seconds
did not involve a connection between participants
and could be considered as connection attempts only.
Thus, our video segments included the arrangements
that lasted three or more seconds. We analysed video
segments by using a coding scheme, which included nine
codes: form of body-technology-space arrangement,
connection strategy, duration of connection, mobility
of participants, proximity of participants, movement
qualities of two participants, technologies, mapping
strategy and finally the sound effect. When coding the
segments, we also consulted our transcriptions of the
reflection sessions of activities.
6 Results
In this section, we present the results of the first three
sessions briefly and allocate more space to results of the
final machine-mediated session in which we observed
large differences between the workshops. In the next
sections, we usually refer to workshop participants
according to their area of profession as dancers,
interaction designers and musicians. Our aim is not
to make generalized claims on professions or casual
connections between professions and the workshop
outcomes; rather, it is just a convenient way we chose
to refer to the participants of three workshops.
6.1 Silence Session
In fact, the silence session was not designed to be a
fully featured session like the other three sessions in
which we tested the effectiveness of our inscriptions
to support the related ASD quality: it was more like a
prelude session preparing participants for the following
more demanding activities. Our inscriptions simply aimed
to support the visibility of other modes of perception.
According to participants, the temperature of their
partners’ body was the most dominant sensation. The
participants stated that they also noticed the sound
of air conditioner, noises from outside and the sound
of their watch. The participants said that they did not
feel any strong connection with their partners. The
temperature of bodies and previously insensible sounds
were new actors that emerged out of the activity. The
participants sensed the previously insensible things
in the space. However, they were not able to create
connections through the amplified modes of sensations.
6.2 Physical Sensitivity Session
Dancers and musicians performed the activities in a
similar way. They maintained the contact between palm
and crown during all three exercises and demonstrated
a very fine control of their movements. They were also
very attentive to the leader and receiver roles assigned
to them. In contrast, interaction designers were not
able to coordinate their movements smoothly and
frequently lost their connection. Their movements
were usually very fast and sudden. While the comments
of dancers and musicians described the subtleties of
the connection: “to what degree I’m sure that we are
connected or not?... It doesn’t feel like it is organic,
particularly in the simultaneous giving-receiving one”;
“There are lots of questions around to what degree to
keep a straight line”; “it was not until I was receiving I
really understood how it was to be at the other side of
the conversation... I realized that I didn’t have to push...
a little touch was sufficient to give a signal and initiate
the movement”, the interaction designers’ comments
described more like a playful exploration: “To see how
far she can go! It was funny... It was just to see what
happens, and she jumped. It was like she was a toypuppet”; “I was pre-determining what she was going
to do”. All participants developed some strategies to
explore the dynamics of the connection such as making
the movements at the same/opposite direction or in
different speeds.
Design and semantics of form and movement
105
6.3 Rich Poster Session
The objects that participants brought to the workshop
included books, pictures, drawings, quotes, a compass,
a pebble and a CD cover. The objects revealed many
different types of connections: connection as a
memento, connection as a feeling of absence,
connection as a shared interest, and connection as
transformation. The objects enabled participants to
make visible their personal understandings of what
makes a connection significant for them. Dancers and
interaction designers created similar posters involving
various pictures and themes. The diversity in the kinds
of images facilitated the participants to share many
stories and reflect on them. In contrast to the posters
in the previous workshops, musicians’ poster did not
demonstrate a large variety in the selected images
and associated connections. There were no stories or
experiences accompanying the images. The selected
images were either abstract patterns or images as
colourful geometric shapes. Their meanings were
abstracted away. In fact, in the final poster, collage
of all the pictures represented a single manifestation
of a visually balanced composition.
6.4 Machine-Mediated Performance Session
In this session, we needed to revise the software
algorithm mapping movements to sound effects after
the first workshop in which dancer participants could
not complete all five activities because of the complexity
of the mapping algorithm. In the first workshop, the
tilt and rangefinder devices were coupled. When two
devices are coupled, the sound producing system gets
sensing data from each device and combines them to
produce a single sound effect. Thus, participants using
the coupled devices do not have a total control over the
generated sound effects. The reason of using coupled
devices was to evaluate whether a preset coupling
between devices facilitates more collaboration and
creative engagements between participants. However,
the participants in Workshop 1 found the control
of the coupled devices complex and felt frustrated
and could not complete the session. Thus, for the
workshops 2 and 3, we decided to have two versions of
the devices: coupled and decoupled. When the devices
are decoupled, each device produces a separate sound
effect independent from the other device. A different
sound effect is assigned to each device and participants
have total control over the creation of sound effects.
106
Design and semantics of form and movement
Table 2 shows the number of connections constructed
in each workshop session using coupled devices,
decoupled devices and a webcam.
D-Tilt
C-Tilt
D-Range- C-Range- Webcam
finder
finder
Workshop 1
N/A
5
N/A
N/A
5
Workshop 2
7
3
14
4
12
Workshop 3
6
4
10
2
5
Table 2. Number of connections in each workshop
using five devices: D-Tilt (Decoupled-Tilt device),
C-Tilt (Coupled-Tilt device), D-Rangefinder
(Decoupled-Rangefinder device), C-Rangefinder
(Coupled-Rangefinder device) and Webcam.
Workshop 1. In this session, the participants were
only able to perform the first two activities and could
not complete the session because of some perceived
technological deficiencies. The participants considered
the devices incapable of doing what they were supposed
to do. In the first activity, the camera did not capture
one participant’s large movements as required, and
the sound effects seemed to be arbitrary. Therefore,
the participant could not understand the relation
between the sound feedback and her movements. As
a result, the participant got frustrated because of not
being able to get the feedback properly. In this activity,
connections were created through movements rather
than sound. The participants employed strategies of
stopping, making similar movements and varying the
tactile sensation to create connections. In the second
activity using coupled tilt devices, the participants
found the mapping between the sound and movements
complicated, and again, they felt frustrated. Thus, we
decided to stop the activities and continued with the
participants’ reflections and suggestions.
In the next workshops, participants used decoupled tilt
devices, coupled tilt devices, decoupled rangefinders,
coupled rangefinders and webcam respectively.
Workshop 2. While the highest number of connections,
14, was observed in the third activity using decoupled
rangefinders, the lowest number of connections, three,
was in the second activity using coupled tilt devices.
Although there was a large variety in human-human
and human-space couplings through technological
devices, individual human-device couplings did not
demonstrate such variety. Participants preferred to use
their devices with their hands although it was possible
to attach them to many parts of body. They did not use
the straps provided to them. Therefore, we observed
a single form of coupling, device-at-hand. However,
in the case of webcam, we observed a large variety in
human-device couplings as well: participants performed
ten connections by variously using their arms, hands,
legs, torso or full body to create sound effects. The
participants preferred to express some themes or
phenomena that they decided upon at the beginning
of the activities when using the range-finder devices
and the Wii-motes. When using the camera, however,
they chose to make free movements and express their
emergent ideas and feelings.
Workshop 3. Similar to the previous workshop, the
highest number of connections, ten, occurred in
the third activity using decoupled range finders.
Different from the previous one, the lowest number of
connections, two, was observed in the fourth activity
using coupled range finders. Again, participants did
not use straps and hold the devices with their hands.
Apart from the activity using a webcam, participants
only used their hands and arms to create sound effects.
Similar to the previous workshop, only the deviceat-hand coupling was observed. Participants’ bodies
were stationary in space and facing each other all
the time. Although they used their full bodies in the
final activity with the webcam, they could create five
connections. Different from the other workshops,
they developed a vocabulary of expression involving
various movement-sound pairs. When they discovered
a good movement-sound pair, they included it into their
vocabulary and then, used it again later to compose
melodies in the activity. However, this was not the case
for the interaction designers, who tried to find as many
interesting movement-sound pairs as possible.
7 Discussion
7.1 On Visibility
We aimed to support visibility in silence and poster
sessions. In silence session, our aim was to increase
the visibility of other modes of sensation. This was
achieved according to the participants’ statements.
The participants sensed the previously insensible things
in the space: the temperature of another body, sound
of watch and A/C. However, in order to be able to
create connections through these different modalities,
more time is needed. In the poster session, our aim
was to make visible the various forms of connections
between humans and other entities. Apart from the
third workshop in which the poster was a single
manifestation of visual aesthetics, the posters in the first
two workshops exhibited various forms of connections
like connection as memories, criticism and culture.
7.2 On Relationality
We aimed to increase participants’ awareness of
relationality of their capacities for action. The three
exercises in physical sensitivity session were scripted
activities amplifying the sensation of reciprocity of our
actions through sensing the effects of our movements
both visually and in a tactile way. The proximity of the
bodies amplified the sensation. The exercises were
extreme cases of connectedness where one body was
strongly connected to another in a physical way. The
sensations and movements of one body were tightly
coupled with another. There was a very high degree
of influence between the bodies, which increased the
visibility of the relationality of our bodies. In all three
workshops, participants’ comments demonstrated
sensitiveness to the shared capacity of their actions and
co-construction of their performance. Apart from the
second workshop in which the participants lost their
connection frequently, the exercises, in general, were
effective in emphasizing the quality of relationality.
7.3 On Multiplicity
We aimed to support multiple ways of engaging with the
design concept, multiple roles for, and multiple mediums
of expression.
Multiple ways of engaging with design concept: Different
kinds of activities allowed us to understand different
forms and dimensions of the design concept. In general,
rich poster and machine-mediated performance
sessions were effective in producing various forms
of connections. Many different forms of connections
were revealed: connection as movement, connection as
sound, connection as criticism, connection as memories
and so on. Many different strategies to construct and
maintain connections were observed: making similar
movements, making opposite movements, combining
stops and repetitive movements, combining stops
and varying movements and a-synching movements.
Design and semantics of form and movement
107
However, our inscriptions could not achieve their
goal in the rich poster session with musicians and the
machine-mediated performance session with dancers.
While the musicians preferred to create a single visual
form of expression, the dancers found the technology
insensitive to perform together and could not complete
all activities in the session.
Multiple roles. In the Physical Sensitivity Session, the
participant performed the same activity by switching the
roles of leader and receiver. It was effective in enabling
participants to develop a relational understanding of
their movements. Therefore, the quality of multiplicity
served to support the quality of relationality.
Multiple mediums of expression. The participants used
different mediums such as paper and technological
devices for expression. For example, the rich poster
session allowed the participants to express their views
of the design concept on a 2D shared medium, i.e., on
paper, in the form of a collage of pictures and texts.
They created representations or proxies of the previous
connections they had made in their lives. In addition,
the totality of pictures and texts revealed forgotten
or unknown connections between places, people and
memories. Moreover, each technological device invited
different patterns of action through different levels
of connectivity. The connectivity can be defined as an
entity’s ability to make connections. The webcam and
rangefinders with their high degrees of connectivity
facilitated the creation of many different connections
whereas the tilt devices with the low degrees of
connectivity could only support the construction of a
few connections.
Although we advocate the inclusion of multiplicity as a
quality in design, there might be some undesired effects
of multiplicity on the design process. One participant
from the first workshop vocalized her concern about
engaging with multiple activities and multiple media in
the activities:
Ultimately, we are transferring, transferring and transferring
through different media. But, in that transference, we are
getting further and further away from proximity to actual
sensitivity and composition.
This is an important criticism on using multiple activities
and multiple mediums in a single half-day workshop.
108
Design and semantics of form and movement
The participants could only spend short periods of
time in each activity, and this limited participants’
capacity to obtain a deeper understanding about each
of their relations with other participants, materials,
and technologies. Multiple activities might enable
researchers and designers to get a broader perspective
on many dimensions of a design concept or problem,
but the knowledge obtained from these short-lasting
activities might be imprecise, shallow and scattered. This
might be a disadvantage for design projects with a more
specific focus. However, it might be advantageous for
the design projects at an early explorative stage in which
getting a broader perspective on many dimensions of
a design concept or problem is very valuable.
7.4 On Configurability
We aimed to support configurability by making wearable
devices compact and portable. They were attachable
to different parts of the body by using various straps.
However, the participants did not use the straps and
hold the devices with their hands. The inscriptions
of straps involving the quality of configurability were
not translated in the practice in the expected ways as
the particular characteristics of sensing technology
and mapping algorithm, i.e. their inscriptions did not
invite use of many possible configurations between
human body and the wearable devices. The lack of
expressive capacity in many of the configurations
rendered those configurations useless or not preferable.
Here, configuring the ways in which the device and the
human body are coupled became less desirable for the
participants since a particular configuration, tilt-at-hand,
provided participants with the opportunity to exploit
the expressive capacity of the device at maximum.
The six qualities introduced here are a starting point
towards developing ASD. Our aim is not to replace
existing design approaches but rather to complement
them by relativising how we think and go about design.
What is needed is not to take these qualities as
prescriptions or strict guidelines for action but to use
them as lenses through which to see design problems
and processes from a relational point of view.
References
Cambridge, MA: MIT Press.
[1] Giddens, A. (1984). The Constitution of Society. Cambridge:
[17] Button, G., & Dourish, P. (1996). Technomethodology:
Polity Press.
Paradoxes and possibilities. Proc. of Conference on Human
[2] Barad, K. (2003). Posthumanist Performativity: Toward an
Factors in Computing Systems: Common Ground. ACM
understanding of how matter comes to matter. Signs: Journal
[18] Suchman, L. (2002). Located accountabilities in technology
of Women in Culture and Society, 28(3), 801-831.
production. Scand. J. Inf. Syst., 14(2), 91-105.
[3] Bardzell, S. (2010) Feminist HCI: Taking stock and outlining
[19] Muller, M.J. (2003). Participatory design: The third space
an agenda for design. In Proc. of CHI'10, ACM.
in HCI. In A.J. Julie & S. Andrew (Eds.), The Human-computer
[4] Akrich, M. (1991). The de-scription of a technical object.
Interaction Handbook (pp. 1051-1068). L. Erlbaum Associates Inc.
In Bijker et al (Eds.), Shaping Technology/Building Society: Studies
[20] Laban, R., (1971). The Mastery of Movement. Boston: Play Inc.
in Sociotechnical Change. Shaping Technology/Building Society,
MIT Press, 205-224.
[5] Latour, B. (2005). Reassembling the Social: An Introduction
to Actor-network Theory. Oxford: University Press.
[6] Callon, M. (1986). Some elements of a sociology of
translation: Domestication of the scallops and the fishermen
of St Brieuc Bay. In J. Law (Ed.), Power, Action and Belief:
A New Sociology of Knowledge?, 196–223. London: Routledge.
[7] Chalmers, M., & Galani, A. (2004). Seamful interweaving:
Heterogeneity in the theory and design of interactive systems.
Proc. of Designing interactive systems: processes, practices,
methods, and techniques. ACM.
[8] Gaver, W.W., Beaver, J., & Benford, S. (2003). Ambiguity
as a resource for design. Proc. CHI. ACM.
[9] Van der Velden, M. (2009). Design for a common world:
On ethical agency and cognitive justice. Ethics and Inf. Technol.,
11(1), 37-47.
[10] Aanestad, M. (2003). The camera as an actor: Design-in-use
of Telemedicine Infrastructure in Surgery. Computer-Supported
Cooperative Work (CSCW), 12 1-20.
[11] Callon, M. (2004). The role of hybrid communities and
socio-technical arrangements in the participatory design. Journal
of the Centre for Information Studies, 5(3), 3 -10.
[12] Kocaballi, A.B., Gemeinboeck, P., Saunders, R., & Dong, A.
(2011). Towards a Relational Approach to Design Process.
ANZAScA.
[13] Suchman, L.A. (2007). Human-Machine Reconfigurations:
Plans and Situated Actions (2nd ed.). Cambridge University Press.
Baki Kocaballi1,
[14] Suchman, L. (2005). Agencies in Technology Design:
Petra Gemeinboeck , Feminist Reconfigurations, Workshop on Gendered Innovations
2
Rob Saunders1,
in Science and Engineering, Stanford University, UK.
Lian Loke ,
[15] Friedman, B., & Kahn, P. (1992). Human agency and
Andy Dong1
responsible computing: Implications for computer system design.
The University of
Journal of Systems Software, 17, 7-14.
Sydney; University
[16] Kahle, D. (2008). Designing Open Educational Technology.
of New South Wales;
In T. Iiyoshi & M.S.V. Kumar (Eds.), Opening up Education:
3
1
2
3
University of
Technology Sydney
The Collective Advancement of Education Through Open
Technology, Open Content, and Open Knowledge, 27-45.
Design and semantics of form and movement
109
Thomas Markussen, Elif Özcan, Nazli Cila
[email protected], {e.ozcan, n.cila}@tudelft.nl
Beyond metaphor in product use
and interaction
Abstract
Keywords
The overall aim of this paper is to demonstrate that
there is a need for supplementing the theory of product
metaphor with a more elaborate theory of product
meaning. More specifically, we argue that the notion of
product metaphor neglects three critically important
aspects of meaning making in product use. First, the
notion of product metaphor usually accounts for how
the visual form and appearance of a product might
cue people to conceive of the product in terms of
another conceptual source (e.g. a coffee maker as a
butler), while leaving the role of cross-modal sensory
experience in product meaning out of consideration.
Secondly, like other theoretical frameworks in design
semantics, the notion of product metaphor primarily
accounts for the semantic operations that are involved
in the first initial phase of product categorization and
interpretation, while eschewing the question as to how
product interpretation might evolve over time as people
interact with and use the product. Finally, in product use
there often emerge more complex and even ambiguous
forms of meaning, which fall outside the explanatory
scope of the source-target construal principle – the
key semantic principle of product metaphor. In order
to remedy these limitations inherent in the theory
of product metaphor we introduce a new semantic
framework based upon Fauconnier and Turner’s theory
of conceptual blends.
Sensory engagement, conceptual blends versus
metaphor in product design, emergent forms of
meaning, product blends.
110
Design and semantics of form and movement
1 Introduction
The overall aim of this paper is to demonstrate that
there is a need for supplementing the theory of product
metaphor with a more elaborate theory of product
meaning. More specifically, we argue that the notion
of product metaphor neglects three critically important
aspects of meaning making in product use. First, the
notion of product metaphor usually accounts for how
the visual form and appearance of a product might cue
people to conceive of the product in terms of another
conceptual source (e.g. a coffee maker as a butler), while
leaving the role of cross-modal sensory experience in
product meaning out of consideration. Other fields
of study have uncovered the role of cross-sensory
experience in metaphor in language [1] as well as in film
and advertising [2], [3], but design semantics still awaits
further inquiry in respect to product metaphor.
Secondly, like other theoretical frameworks in design
semantics, the notion of product metaphor primarily
accounts for the semantic operations that are involved
in the first initial phase of product categorization and
interpretation, while eschewing the question as to how
product interpretation might evolve over time as people
interact with and use the product. Karapanos et al. [4]
have developed an initial framework for understanding
how a user’s relationship to a product changes over
time, but they do not offer any insight concerning
product metaphor.
Finally, in product use there often emerge more
complex and even ambiguous forms of meaning, which
fall outside the explanatory scope of the source-target
construal principle – the key semantic principle of
product metaphor [5].
These inherent limitations of product metaphor theory
will be laid bare through a detailed product analysis of
Anna G, Alessandro Mendini’s design of a corkscrew for
Alessi. At the mere sight of it, Anna G seems to be a
paramount example of product metaphor. However, we
will demonstrate that the actual use and multi-sensory
interaction with Anna G may cue people to construct
new ambiguous forms of product meaning that fall
outside the explanatory scope of product metaphor.
More specifically, we shall refer to such ambiguous
forms of product meaning as “product blends”.
Unlike product metaphors, product blends demand
of the user that he or she conceptually unifies two
divergent or even contradictory meanings. This kind
of meaning construction has been largely overlooked
in design semantics (for exceptions, see [6], [7], [8],
[9]. Closely related studies have dealt with how, for
instance, interaction designers can explore ambiguous
forms of meaning as a resource for spurring users’
imaginative interpretations of technology [10]. In a
similar vein, [11] present an interesting study of how
contradictory meanings often underlie experiences of
product novelty and surprise. Yet, there is a lack of a
coherent semantic account of how products cue users
to construct ambiguous forms of product meaning.
In this paper, we introduce a new semantic framework
for understanding ambiguous meaning in the form
of product blends and how it differs from product
metaphors. In so doing we wish to shed light on
processes of meaning attribution that emerge from
multi-sensory interactions with products over time. By
drawing upon recent research in cognitive semantics,
notably Fauconnier & Turner [12], [13] we furthermore
offer a basic diagram enabling design researchers to give
an accurate formal description of the semantic principles
of product blends. Finally, we compare weaknesses and
strengths inherent in our framework with related work
within design research.
2 Product Analysis of Anna G
The creation of product meaning is a dynamic process
relying on the product-user relation. Product meaning is
not an invariable entity, but is a result of a hermeneutic
act of user interpretation. In this sense product
meaning may vary according to people’s socio-cultural
background, past product experiences, gender, selfidentity, and so forth. In fact, one and the same product
may prompt users to construct multiple meanings, even
meanings that contrast or contradict one another.
Anna G provides us with a good example of such a
product. In this section, we offer a product analysis
showing how each author of this paper interprets Anna
G differently. None of these interpretations should be
seen as the authoritative one. Rather, what we want
to demonstrate is that products like Anna G are able
to prompt two alternative forms of product meaning:
product metaphor or product blend. While research
literature on product metaphor is vast, product blends
have received little attention so far. The theoretical
explanation of the difference between these two forms
will be given in the following section.
Fig. 1. Anna G, corkscrew by Alessi.
2.1 Cila’s Interpretation
Anna G is a tall and thin figure with a tall neck, big
eyes and long dress call to mind the “gothic look” of
the 1990s (Fig. 2a). Especially, the one with the black
dress has almost a Tim Burton-esque image. The visual
association grows stronger with the notches around
the shoulders and neck, and the thin arms screwed to
the body. This can be seen as more of a visual analogy
Design and semantics of form and movement
111
that is based solely on Anna G’s appearance, which
does not make a contribution to the functionality of the
product. Still, its proportions and stylistic features make
the product a beautiful and iconic object, which make it
pleasurable to simply look at it.
2.2 Özcan’s Interpretation
As a first impression Anna G looks inviting with her soft
lines and colors, she looks like she would love opening
wine bottles. After placing her on top of the bottle
and turning her head to push the corkscrew into the
cork, her arms start to rise in the air slowly (Fig. 2b).
This movement is interpreted as the first sign that she
gets happy. As she gets closer to opening the bottle,
her arms rise higher and higher; consequently she gets
happier and happier. The smile on her face strengthens
this meaning attribution. Her happiness corresponds
with the emotions of the user after opening a wine
bottle; the user’s feeling of accomplishment is reflected
in the behavior of the product.
2.3 Markussen’s Interpretation
In its visual form Anna G evokes associations to a holy,
almost saint-like female figure with a glory surrounding
her head. This metaphor is further entailed by the fact
that, when standing on a shelf or table, the “arms” of
the corkscrew can be placed in a position similar to
those of praying figures found in so many Renaissance
paintings or catholic visual culture (Fig. 2a). However,
when the corkscrew is not simply looked at, but used
to open a bottle of wine, the user is cued – through
haptic interaction – to construct a thought-provoking
counter-image. After opening a wine bottle, the cork is
hidden under the skirts of the female figure. Because it
is hidden, the user has to look up under the skirts and
reach out after the cork to remove it (Fig. 2c). This is
a rather rude gesture that actually conflicts with the
product metaphor elicited through visual experience.
Thus, it draws attention to the effects of wine drinking
being associated with lust and desire, male-female
relationships, gender issues, perhaps even sexual
harassment, etc. This second product meaning is also
motivated by the product’s name – Anna G – which
is a name a stripper may take on when dancing and
performing in a nightclub.
112
Design and semantics of form and movement
Fig. 2. Anna G in use (from left to right, a, b, c).
2.4 Anna G as a Challenge for Design Semantics
While Cila’s product interpretation is primarily
grounded in the perception of the visual form and
appearance of the product, Özcan’s and Markussen’s
stories also include meaning attribution as they evolves
from using the product. However, there is a central
difference between the two last interpretations.
While in Özcan’s interpretation the image of the woman
celebrating the opening of the wine bottle is consistent
with the image evoked through her visual experience
of Anna G, in Markussen’s account the use of the
product resulted in a counter-image of a woman that
is contradictory to and violates the first image evoked
through his visual product experience.
For design semantics it is a theoretical challenge to account
for how one and the same product is able to evoke such
ambiguous forms of meaning. Taken together the three
product interpretations are useful for delving into (i) the
interplay between multi-sensory product interaction and
meaning attribution; (ii) the process of meaning attribution
as it evolves over time through product usage; and (iii)
how a product can cue people to construct a variety of
different, and even conflicting meanings.
In the next section, we will show that while the theory
of product metaphor indeed enables us to go a long
way in addressing these questions, there is also a need
for incorporating a new concept into the vocabulary
of design semantics, namely “product blends”. The
notion of product blend is derived from the theory
of conceptual blends as developed recently within
cognitive semantics as a remedy for some limitations
inherent in metaphor theory.
3 From Product Metaphors to Product
Blends
Taken in its most basic sense, a metaphor is defined
as “understanding and experiencing one kind of thing
in terms of another” [14]. For example, in the phrase
“love is a burning fire” an emotional state such as ‘love’
is understood in terms of a fire, which is uncontrollable
and potentially destructive [15].
Until the 1980s it was usually believed that metaphors
were primarily whimsical products of the poet’s mind,
but the groundbreaking study by Lakoff and Johnson
showed that metaphors are better understood as a
fundamental cognitive operation of the human mind
allowing us to make sense and understand the world.
This assumption has been supported by countless
examples in everyday language, which is permeated
with metaphors (some of which we are hardly aware
of anymore). For example, “she is unable to defend her
claims” is an instantiation of the metaphor ‘Argument
is War’; “you’re wasting my time” instantiates the
metaphor ‘Time is Money’, and so forth.
On the basis of their study of metaphors in everyday
language, Lakoff & Johnson [14] developed a cognitive
linguistic account of the underlying semantic principle
involved in the creation of metaphors. More precisely
they proposed that metaphor creation is governed by
a source-target construal principle, which could be
illustrated as in Fig. 3:
meaning structures from source to target cannot take
place unless there is a structural isomorphy between
source and target (also known as the invariance
principle [16]). Hence, the metaphorical mapping can
only be processed if love is malleable in some way
according to the structural qualities of fire.
In their work, Lakoff and Johnson have succeeded
in uncovering how the source-target principle is
responsible for organizing meaning on a wide range of
levels ranging from abstract thought in mathematics
to gestalt laws of perception as well as neurobiological
processes [17], [18], [19]. The cognitive semantic theory
of metaphor has also proven useful for understanding
how metaphors work in product design. In product
metaphors, the target is the product in question and the
source is the entity that is associated with it in order
to modify the target to convey particular meanings.
This modification is provided by projecting some
attributes of the source onto compatible attributes of
the target. The metaphorical mapping is quite tangible
in this case; the mapped qualities are visible to users in
the appearance, movement or interaction pattern of
the target [20]. In this way, the target is seen through
the lens of the source, which leads to cognitive and
emotional effects in the users [21], [22].
By applying the notion of metaphor, it is possible to
uncover semantic principles underlying Cila’s and
Özcan’s product interpretations. In both cases, the
corkscrew is understood and experienced as a womenlike figure: for Cila a Tim-Burtonesque figure and for
Özcan a celebrating woman. Despite their differences
each of these two product interpretations are governed
by the source-target principle as depicted in Fig. 4:
Fig. 3. The rule of metaphor (adapted from Wulff et al.
1990).
In “Love is a burning fire”, fire is the source domain, the
qualities and structures from which are metaphorically
projected onto love being the target. When an
argument is compared to war, ‘war’ becomes the
source domain that lends its action structures to a
metaphorical mapping onto the phenomena of ‘verbal
dispute’ being the target, and so on. Further, it is
forfeited by Lakoff that the metaphorical mapping of
Fig. 4. Anna G as a product metaphor.
However, in Markussen’s product interpretation, a new
meaning emerges that cannot be adequately captured
by this diagram. As Anna G is used to open a bottle
of wine a counter-image is evoked which is laden with
entirely opposite connotations of a woman than that
Design and semantics of form and movement
113
of a saint. This counter-image is contradictory to the
initial product interpretation, but nevertheless calls for
integrating the elements and meaning structures into
the same overall product interpretation making it highly
ambiguous. Now, for design semantics, what is central
to ask is, what are the underlying semantic principles of
this type of product meaning?
3.1 Semantic Principles of Product Blends
In order to answer the question in the previous
paragraph, it is valuable to take a brief look at some
of the more recent developments within cognitive
semantics. In particular, those made around the mid
1990s, when Gilles Fauconnier and Mark Turner
discovered some forms of meaning making that
appears to be instances of metaphor, but which, under
closer scrutiny, turns out to be governed by semantic
principles violating the source-target and invariance
principles. Moreover, Fauconnier and Turner suggested
that these new forms of meaning should be termed
‘conceptual blends’ [12].
A conceptual blend is not the result of meaning
structures from a source being projected onto a
target. The creation of a blend relies on conceptually
integrating structures from two so-called input spaces
into a third space: the conceptual blend. This third
space is not the sum total of meaning structures form
the two input spaces. On the contrary, the third space
represents emergent new meaning, which cannot be
explained by refuge to any of the inputs. In order to
illustrate the semantics principles behind this conceptual
blending process, Fauconnier and Turner [13] suggest
the following basic diagram (Fig. 5):
In Fauconnier and Turner’s diagram, the dashed lines
from the inputs into the blend represent conceptual
projections of structures. Since structures from both
inputs get projected and integrated into new hybrid
meaning structures in the blend there is a violation
of the source-target construal principle. Moreover,
the full lines in the diagram between input elements
represent one of Faunnier and Turner’s key insights,
namely that conceptual blends “can operate along
strong clashes between the inputs” [13]. Let us try to
somehow substantiate these theoretical arguments by
applying the basic diagram as an analytical framework
for understanding Anna G (Fig. 6):
114
Design and semantics of form and movement
Fig. 5. Basic diagram illustrating the semantic
principles of conceptual blending [13].
Fig. 6. The semantics of product blends.
In Fig. 6 we have elaborated on Fauconnier and Turner’s
basic diagram making it more fit for describing the
construal principles of conceptual blends in product
design. For instance, we have added visual experience
and touch in product use in order to depict how
meaning is attributed differently according to how the
product is experienced. The saint-like woman arising
from visually perceiving Anna G is represented by a
dashed circle in input 1. The stripper being evoked from
touching and using Anna G is represented by a full circle
in input 2. Now, what makes Anna G a prompt for a
conceptual blend is that it opens up for a process where
the saint-like woman and the stripper get mixed into one
ambiguous female blend associated with the product.
All of this takes place through time (the arrow below
the diagram, where time is indicated as tn). By adding
a temporal axis to the diagram, we wish to mark time
as an important factor for product interpretation. This
should be seen as broadening product metaphor theory,
which is usually focused primarily on the first initial
stages of visually perceiving a product.
Further, we propose the notion of “product blends”
rather than conceptual blends as a new concept for
working with ambiguity and contradictory meanings
in product design. As a start, we predict that product
blends can be found on various levels of product
experience, most notably on a sensory, a conceptual and
an emotional level.
3.2 Conceptual Structure of Product Blends
As we demonstrated, meaning attribution is a complex
process that is facilitated by sensorimotor and semantic
systems. Although the verbal correspondents of a
meaning may be stored in the semantic system, the
attribution of the meaning is primarily triggered
by multi-sensory interactions with objects. During
meaning attribution, these two distinct types of mental
representations (i.e., sensorimotor and semantic)
do not interact with each other directly. Paivio [23]
suggests an amodal system, i.e., a conceptual system
that bridges the representational gap between the
semantic and the sensorimotor systems. Thus, a
concept is the melting pot of mental representations.
A conceptual representation of an object can be
activated earlier than its semantic representation [24].
That is, as soon as people are confronted with objects
(e.g., an apple), a network of mental representations
will be automatically activated (e.g., the image, taste,
sound and smell of an apple, and other meanings stored
in the semantic system). Because, sensorimotor and
semantic representations are connected via conceptual
representations, a concept can be activated either by
the semantic system or the sensorimotor system.
Barsalou [25] presents a further explanation for
conceptual systems. He suggests that a conceptual
representation is an embodied representation
construed by prior experiences taking place not only in
the sensorimotor and semantic systems but also in the
emotional systems. According to Barsalou’s account,
object representations can also be variant resulting from
infinite arrangements of experiences. Furthermore,
Bar [26] suggests that all objects belong to a context
frame, which a network conceptual associations
again construed by exposure to real world events.
Thus, conceptual representations of an object cannot
be disconnected from its context of existence, and
thereby from other objects within the same context.
Furthermore, because everybody experiences objects
and events in their own settings, conceptual associations
also become idiosyncratic to people. To summarize, the
word or the image ‘apple’ activates a network of mental
representations referring to sensorimotor, semantic and
emotional systems, and other conceptual associations
(i.e., sub-concepts) resulting from a contingent activation
(a farmer, an orchard, grandmother’s apple-pie,
happiness, Adam & Eve, guilt, sin, iMac, Steve Jobs, etc.).
Thus, when a product such as Anna G is experienced,
it is very likely that Anna G evokes all kinds of
associations intrinsic to the product and idiosyncratic
to the user (see Section 2 for interpretations the
authors provided from their perspectives). Therefore,
it is more logical to explain such rich interpretations
and the underlying mental processes with conceptual
systems in consideration. As opposed to the semantic
system, which provides a narrow linguistic account,
the conceptual system tackles first the sensorimotor
representations and relates them to several meaning
structures occurring in the semantic system [27]. As
a result, a blending occurs between the conceptual
associations (rich in multi-sensory, emotional and action
content) activated by one single product.
As we mentioned before, product blends tackle
human-product interactions over time. Interactions
with products occurring at different stages may
evoke different conceptual associations. Barsalou’s
[28] account with ad-hoc categories may explain this
phenomenon. That is, people depending on the context
and situation may activate a network of items associated
by one common goal and/or function. Moreover,
concept structures are not fixed and can be updated
depending on the new inputs and new situations [29].
Thus, in the course of product use, the activated
conceptual associations can take different forms and
may prime the following associations to be activated.
Therefore, in the essence of product blends, a
continuous negotiation for meaning attribution occurs
between the primary concept that is activated (e.g.,
saint-like woman in Figure 4) and sub-concepts that
are subsequently activated. For example, visually
experiencing Anna G as ‘a woman’ (i.e. the primary
concept) activates woman-like knowledge in the
beginning of product interaction. Later haptic and tactile
interactions with the product (e.g., holding Anna G
from the waist, turning it around looking under her
skirt) activate another concept, which is the ‘stripper’
Design and semantics of form and movement
115
(sub-concept). Accordingly, the primary concept of
Anna G gets updated starting from a saint-like woman
figure and becoming a stripper. In this example, it is
the ambiguous form of Anna G and the conflict in
the conceptual associations that mainly underlie the
occurring product blend.
Product blends can be constructed as a result of
product experiences on different levels (i.e., sensory,
cognitive and emotional). In Markussen’s account of
Anna G, both the primary concept and the sub-concept
are activated by sensory product experiences (i.e., the
visual and tactile product experiences, respectively).
However, the activations of primary and sub-concepts
may also be a result of emotional and cognitive product
experiences. In Özcan’s account of Anna G, an emotion
(i.e., happiness) is identified in the course of the product
use, which serves as a sub-concept that updates the
‘woman’ concept. However, in this case, the occurring
concepts are not contradictory but complementary to
the overall product experience. In Cila’s account, the
visual experience refers primarily to ‘a gothic woman
image’ and a further cognitive elaboration into the primary
concept activates the sub-concept of ‘Tim Burton’s
heroines’. Consequently, the primary concept gets
stronger making the product more pleasant to the user.
Although product blends can occur as a result of
different types of product experiences, what makes
its effect strong lies in the (in)congruency between
the activated primary concept and the sub-concept.
We suggest the notion of product blends as a new
concept for understanding the semantics of ambiguity
and contradictory meaning in product design. Over the
years, there has been an increasing interest in design
research in ambiguity and contradictory meaning.
In the following section, we review some of the
contribution and relate and discuss them in relation
to our own framework.
4 Contradictory Meaning and Ambiguity
in Design
Without doubt, we can consider Anna G as a
rather atypical product within the product category
of corkscrews. For some atypical products, the
functionality is not the primary concern from the
perspectives of both designers and users. For example,
Phillippe Starck states that he mainly designed the
Juicy Salif (the famous Alessi lemon squeezer) as a
conversation starter in awkward situations [30].
116
Design and semantics of form and movement
Products such as Anna G and Juicy Salif are rather
ambiguous in their physical constitution – physical
constitution of a product normally gives rise to a ‘lexical
level’ meaning attribution (i.e., corkscrew or lemon
squeezer). These products rather refer to complex
meaning structures beyond a lexical association, which
makes them good examples for product blends. Thus,
ambiguity in product categorization is an essential factor
for product blends to take place in the first place.
Furthermore, contradiction in occurring concepts is
what makes the product experience intriguing.
If the designers of ambiguous products have similar
concerns as Starck’s, then facilitating the activation of
contradictory concepts in human-product interaction
can be seen as a deliberate design decision for creating
richer product experiences (that is, experiences that
go beyond the functionality of the product).
In the following paragraphs, an overview will be given
on how ambiguity and contradictory meanings
have been studied within the design literature and
shortcomings of these studies will be discussed.
4.1 Sensory Incongruity
Sensory incongruity has been studied in the context
of surprise in product design [31]. Ludden studied
three types of sensory incongruities (i.e., visual –
tactual, visual – auditory and visual – olfactory). For
evoking surprise reactions, visual-tactual incongruity
was found to be the most effective. Furthermore, it
was shown that the surprise reaction had a long-term
effect on other emotions such as interest, fascination
and confusion. Ludden’s main interest was to observe
the emotional effect of the synthesized sensory
incongruities in product use in order to understand
whether sensory incongruity could be used as a design
strategy. However, Ludden’s doctoral dissertation
has not covered the semantic impact of sensory
incongruities.
Sensory incongruities by eliciting conflicting emotions
may further influence the meaning attribution process
to the product experience. Our framework on product
blends assumes emotions as part of the embodied
experience [c.f. 25] and therefore part of the activated
concept frame. Thus, we assume that emotional
responses are involuntary and play an important role
especially in assessing conflict between the emotional
responses towards the primary concept and the
sub-concepts.
4.2 Contradictory Meanings
Hong and Chen [11] offer a study of simultaneous use
of contradictory meanings in product design. More
specifically, they focus on how meaning attribution
triggered by a product may manifest itself in the
semantic continuum between two opposite adjectives:
typical-unique, traditional-modern, simple-complex,
rational-emotional. As an example of how products can
be designed according to the typical-unique polarity,
they mention a chair that makes double use of one of
the legs as an umbrella stand and a flowerpot. In this
instance, contradictory meaning occurs as a result
of adding functions from other product categories
(umbrella, flowerpot) to the existing product category
of a chair. By merging typical product genres in a unique
way in its design, the chair plays humorously with the
typical-unique polarity. We argue that, by introducing
the notion of product blends, we have added a new
perspective on contradictory meaning, which is absent
from Hong and Chen’s study.
First of all, Hong and Chen’s framework is too coarsely
grained to capture contradictory meanings like the
one we find in Anna G. Anna G does simply not
fit into any of the four categories of contradictory
meaning. Secondly, many of Hong and Chen’s examples
such as the chair provide insight into lexical level
meaning attribution, i.e., the process of associating
primary concepts with a product. In this paper we
have broadened the perspective so as to include the
interplay between primary concepts and sub-concepts.
Thirdly, for Hong and Chen meaning attribution
is primarily a matter of how people react towards
visual images of products, whereas we see meaning
attribution as emerging from multi-sensory product
use, which is a time-based interactional process. Finally,
as Hong and Chen base their study on the so-called
differential semantic method, they end up conceiving of
contradictory meaning as being manifested in between
two opposite adjectives. However, product blends
are contradictory forms of meaning, which cannot be
adequately accounted for in bipolar terms. Thus, a
product blend is not the sum total of two adjectives
or something in between, but a new third structure.
The basic diagram enables us accurately to account
for the semantic principles behind this structure.
4.3 Ambiguity as a Resource for Design
Gaver, Beaver and Benford [10] have developed an
analytical account of how ambiguity in design can be
roughly grouped into three categories: (i) ambiguity
of information, an example of which would be Mona
Lisa’s smile where the technique known as sfumato is
used to create a certain sense of indeterminacy in the
viewer; (ii) ambiguity of context which is exemplified by
Marcel Duchamp’s Fountain from 1917, a urinal brought
into the context of an art gallery; and (iii) ambiguity of
relationship is a third type of ambiguity illustrated by
Lieshout’s Bais-ô-Drôme which is a trailer designed
to become a love caravan that in a strange way mixes
utility (a trailer) and sexual debauchery.
While Gaver, Beaver and Benford [10] offer many
central insights into the nature of ambiguity and how
it can be used by designers in various ways to increase
user creativity and interpretations, they do not offer a
semantic explanation of this phenomena. We argue that
the basic diagram of conceptual blends can be valuable
as an explanatory tool for providing a more fine-grained
understanding of the semantics of ambiguity in design.
Thus, the diagram can be used to account for the
semantic principles underlying ambiguity of information,
ambiguity of context and ambiguity of relationship.
5 Conclusion
Alessandro Mendini who designed Anna G for Alessi,
once said that as a designer he wishes to communicate
through his objects and work in general, “trying to say
things that encourage people to deepen meditation and
spirituality”. As a product, Anna G surely encourages
people to meditate upon a series of topics related to
celebration, wine drinking, gender issues and so on.
In this paper, we have demonstrated that the product’s
ability to trigger this rich variety of interpretations
has to do with a certain ambiguity of product meaning.
More specifically, we have shown that this ambiguity
can be explained in terms of two divergent semantic
principles: product metaphor and product blend. While
the existing research in design semantics has a lot to say
about product metaphor, little has been written so far
about product blends. To fill in this lack of knowledge,
we have found it necessary to introduce conceptual
blending theory as a supplement to metaphor theory.
In so doing we have remedied the inherent limitations
of metaphor theory that has to do with its inability
to account for:
Design and semantics of form and movement
117
r$POUSBEJDUPSZGPSNTPGNFBOJOHUIBUWJPMBUFUIF
source-target principle;
r)PXNVMUJTFOTPSZQSPEVDUFYQFSJFODFTHJWFSJTF
to different interpretations of the product;
r)PXNFBOJOHBUUSJCVUJPOFWPMWFTPWFSUJNFUISPVHI
product use and interaction.
on the cross disciplinary import of «embodied meaning» into
interaction design. In Proceedings of the Design Research
Society Conference 2008. Hallam University, Sheffield: Hallam
University, Sheffield.
[8] Markussen, T. (2010). A cognitive semiotic approach to
the aesthetic interplay between form and meaning in responsive
environments. In Vihma. S. & T.M. Karjalainen, T.M. (Eds), Design
Needless to say, our contribution must be further
elaborated by future work as it is built up from only one
single product analysis. In particular there is a need to
study the many forms that product blends can take, and
to support such studies with empirical findings and user
tests. For example, we could empirically study the shift
in product semantics over time through experiences
deriving from a range of non-rich (mono-sensory) and
rich product interactions (multi-sensory), that is, from
visual only to multisensory or auditory only to multisensory. Despite this missing body of work, we argue
nonetheless that the notion of product blends marks
a new interesting research topic for design semantics.
As they increase user creativity and interpretations,
product blends could serve as a resource for deriving
new design ideals for designing richer and meaningful
user experiences.
Semiotics in Use. Helsinki: University of Art and Design Helsinki.
[9] Markussen, T., & Krogh, P.G. (2008). Mapping cultural frame
shifting in interaction design with blending theory. International
Journal of Design, 2(2), 5-17.
[10] Gaver, W.W., Beaver, J., & Benford, S. (2003). Ambiguity as
a resource for design. In Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems (s. 233-240). ACM.
[11] Hong, W.K., & Chen, L.L. (2010). Exploring contradictory
meanings in product semantics. In Proceedings of DESFORM
2010. Lucerne, Switzerland.
[12] Fauconnier, G., & Turner, M. (1998). Conceptual integration
networks. Cognitive Science, 22(2), 133-187.
[13] Fauconnier, G., & Turner, M. (2002). The Way We Think.
New York: Basic Books.
[14] Lakoff, G. & Johnson, M. (1980). Metapahors We Live By.
University of Chicago Press.
[15] Cila, N., &. Hekkert, P. (2009). Product Metaphors:
A Framework for Metaphor Generation and Experience in
References
Products, Proceeding of International Association of Societies
[1] Ricoeur, P. (1975). The Rule of Metaphor: Multi-disciplinary
of Design Research (IASDR) 2009 Conference, (pp. 316-320).
studies of the creation of meaning in language. Toronto:
Sungnam, South Korea: Korea Society of Design Science.
University of Toronto Press.
[16] Lakoff, G. (1990). The invariance hypothesis: is abstract reason
[2] Forceville, C. (2006). Non-verbal and multimodal metaphor
based on image-schemas? Cognitive Linguistics, 1(1), 39-74.
in a cognitivist framework: Agendas for research. In
[17] Johnson, M. (1987). The Body in the Mind: The Bodily
G. Kristiansen, M. Achard, R. Dirven, & R. Ibanez (Eds.),
Basis of Meaning, Imagination, and Reason. Chicago: University
Cogntive Linguistics: Current Applications and Future Perspectives
of Chicago Press.
(pp. 379-402). Berlin/New York: Mouton de Gruyter.
[18] Lakoff, G. (1987). Women, Fire, and Dangerous Things:
[3] Forceville, C. (2008). Metaphor in pictures and multimodal
What Categories Reveal About the Mind. Chicago: University
representations. In R.W. Gibbs, Jr. (Ed.), The Cambridge
of Chicago Press.
Handbook of Metaphor and Thought (pp. 462-482). New York:
[19] Lakoff, G., & Johnson, M. (1999). Philosophy in the Flesh:
Cambridge University Press.
The Embodied Mind and its Challenge to Western Thought.
[4] Karapanos, E., Hassenzahl, M. & Martens, J. B. (2009).
New York: Basic Books.
User experience over time. Proceedings of CHI09. Boston,
[20] Cila, N., Hekkert, P., & Visch, V. (in press). Source selection
Massachussetts.
in metaphor generation: The effect of salience and relatedness.
[5] Cila, N., Hekkert, P., & Visch, V. (2010). As light as a leaf:
[21] Cupchik, G.C. (2003). The ‘interanimation’ of worlds:
Product metaphor generation for experience-driven design.
Creative metaphors in art and design. The Design Journal,
7th International Conference on Design & Emotion. Chicago.
6(2), 14-28.
[6] Imaz, M., & Benyon, D. (2007). Designing with Blends:
[22] Van Rompay, T.J.L. (2008). Product expression: Bridging
Conceptual Foundations of Human-Computer Interaction
the gap between the symbolic and the concrete. In H.N.J.
and Software Engineering. Boston: The MIT Press.
Schifferstein & P. Hekkert (Eds.), Product experience (pp. 333-
[7] Markussen, T. (2008). Disciplining the body? Reflections
352). Amsterdam: Elsevier.
118
Design and semantics of form and movement
[23] Paivio, A. (1991). Dual coding theory: Retrospect and
current status. Canadian Journal of Psychology, 45(3), 255-287.
[24] Orgs, G., Lange, K., Dombrowski, J., & Heil, M. (2007).
Is conceptual priming for environmental sounds obligatory?
International Journal of Psychophysiology, 65(2), 162-166.
[25] Barsalou, L.W. (1999). Perceptual symbol systems.
Behavioral and Brain Sciences, 22, 577-660.
[26] Bar, M. (2004). Visual objects in context. Nature Reviews
Neuroscience, 5(8), 617-629.
[27] Özcan, E., & Sonneveld, M. (2009). Embodied explorations
of sound and touch in conceptual design. In Design and
Semantics of Form and Movement (DeSForM), Taipei, Taiwan.
[28] Barsalou, L.W. (1983). Ad hoc categories. Memory &
Cognition, 11(3), 211-227.
[29] Solomon, K.O., Medin, D.L., & Lynch, E. (1999). Concepts
do more than categorize. Trends in Cognitive Sciences, 3(3),
99-105.
[30] Lloyd, P., & Snelders, D. (2003). What was Philippe Starck
thinking of? Design Studies, 24, 237-253.
[31] Ludden, G.D.S., Schifferstein, H.N.J., & Hekkert, P. (2007)
Effects of visual - auditory incongruity on product expression
and surprise. International Journal of Design 1(3), pp 29-39.
[32] Ludden, G.D.S., Schifferstein, H.N.J., & Hekkert, P. (2009)
Visual - tactual incongruities in products as sources of surprise.
Empirical Studies of the Arts, in press.
Thomas
Markussen1,
Elif Özcan2 , and
Nazli Cila 2
1
Kolding School of
Design, Department
of Communication
Design, Denmark
2
Delft University
of Technology,
Department of
Industrial Design,
Delft, The Netherlands
Design and semantics of form and movement
119
Alan Young
[email protected]
Graphic design and the tyranny
of connoisseurship: An argument
for a semiotic approach to graphic
design pedagogy
Abstract
1 Introduction
In design pedagogy, a connoisseurship model prevails,
while semiotics has struggled to become a significant
pedagogical tool outside of the theory class. Yet whilst
it has much to offer, connoisseurship is a limited
approach that struggles with current philosophical and
political concerns. This paper describes its historical
trajectory and influence on design discourse and
then suggests how semiotics can be an important
addition to this system. Foregrounding the relationship
between forms, perception and social communication,
semiotics can provide more effective approaches
to design instruction. More importantly, its robust
theoretical framework allows for in-depth enquiry into
the social and cultural meanings that spring from and
traverse graphic media. I argue that semiotics should
be approached in design pedagogy, not as a minor
area of theory, but as a larger system through which
many other design issues can be explored – in effect to
generate a ‘semiotic sensibility’. This can help to impart
a richer awareness and deeper understanding of how
design works, resulting in more informed, responsible
and professionally capable design practitioners.
This paper is in two parts. The first section deals with
the emergence of what I refer to as the connoisseurship
model, and its historical influence in setting a particular
understanding of design practice and pedagogy. The
second section describes how semiotics can work
alongside this approach to create a more effective
pedagogical process, the result of which can be more
informed and professionally capable graphic designers.
Although I use the term graphic designer, everything
here applies equally to the communication designer
and will have resonance across a range of other design
and design related disciplines. By semiotics, I mean the
broad notion that originated with de Saussure [1] and
was developed by Roland Barthes in the following sense:
Semiology aims to take in any system of signs, whatever
their substance and limits; images, gestures, musical
sounds, objects, and the complex associations of all of
these, which form the content of ritual, convention or public
entertainment: these constitute, if not languages, at least
systems of signification [2].
Elements of semiotics of particular import for graphic
design are signification and its component elements
(sign, signifier, signified and referent) as well as second
order signification and the notions of denotation,
connotation, myth and ideology. In this regard, it aims
to present a model, or way of seeing, that foregrounds
the relationship between communication, culture and
the making of meaning.
Keywords
Semiotics, connoisseurship, graphic design, pedagogy,
discourse, design history, design theory.
120
Design and semantics of form and movement
2 Graphic Design Discourse
It is useful to regard design as a discourse in the
Foucauldian sense [3], whereby certain practices are
brought together as a unity. Through this process,
previously disparate components of the discourse are
linked and officially sanctified through a professional
language with concurrent systems of accreditation which
specify what may be practiced and who may practice
it. An important part of the process of gathering
together these disparate practices is the defining of
various sub-categories such as graphic design, industrial
design, interior design and the like. In graphic design,
the traditional practices of aesthetic construction of
type elements on a page and the ‘laying out’ of images
and type together, as well as many other traditional
and modern practices, are drawn together under the
‘graphic design’ banner, whilst other previously related
practices, such as much of the preparation work for
printed media are excluded. Precisely what is chosen
to be included and what remains excluded from the
category ‘graphic design’ and thus what is and is not to
be considered within the boundaries of design discourse
are seen here in terms of the historical power struggles
of various social and institutional groups.
It can be argued that graphic design emerged out of a
social, economic and a political shift which designated
the legitimate creative aspects of printing as separate
from, and beyond the grasp of, the traditional printing
industries [4]. As these practices became the practices
of a newly emerging professional – the graphic designer
– hierarchies, new historical trajectories and structures
of legitimacy of this profession were found from various
fields, but I argue much of the language took its form
of expression from the discourse of fine art. Fine art,
after all, had been at work separating itself from the
uneducated and uninitiated, for 150 years.
2.1 Connoisseurship and Graphic Design History
The Oxford dictionary defines connoisseur as an ‘expert
judge in matters of taste’ [5], yet it is much more than this.
Connoisseurship involves membership to a certain group
with a certain specialized knowledge, access to a special
language, a keen awareness of the hierarchical relations
of power and processes for the conferring of legitimacy.
Indeed, all of these things fit well with Foucault’s notion
of discourse, and we might regard the connoisseur as a
stakeholder, or speaking position, within a discourse
which positions ‘taste’ as a key discursive element.
Beatrice Warde was an influential figure in the early
emergences of what was to become known as graphic
design, particularly through her role working for the
Monotype Corporation to promote their typefaces to
the printing industries. She worked alongside other key
figures in printing history in Stanley Morrison and Eric
Gill. Belanger gives the following account of her work
for British Monotype:
Beatrice Warde gave a talk pretty much everywhere; for
many years she was the British Monotype Corporations’s
[sic] promotion director, and part of her job was going
around talking up good typography in general and
Monotype typefaces in particular [6].
Warde is perhaps best known for her book published
in 1955 and the title paper, originally published in 1932,
The Crystal Goblet, in which she presents what is often
considered the quintessential modernist perspective of
typography and also in which she is considered to have
coined the term ‘transparent-‘ or ‘invisible-‘ type. The
text is still used in classes today, and most students
of design would be familiar with it. Through this text,
typography and a relation to social distinction becomes
apparent.
Imagine that you have before you a flagon of wine. You
may choose your own favourite vintage for this imaginary
demonstration, so that it be a deep shimmering crimson in
colour. You have two goblets before you.
...if you have no feelings about wine one way or the other,
you will want the sensation of drinking the stuff out of a
vessel that may have cost thousands of pounds; but if you
are a member of that vanishing tribe, the amateurs of fine
vintages, you will choose the crystal, because everything
about it is calculated to reveal rather than hide the beautiful
thing which it was meant to contain [7].
Ironically, although Warde’s job was to speak to the tradeskilled printer, her metaphor of wine connoisseurship
was hardly likely to ‘speak’ to them in a language they
could relate to. In fact it was the growing numbers of
college educated ‘book artists’ and graphic designers
that proved to be Warde’s real audience. Indeed, at one
point she remarked that ‘real’ book readers constituted:
‘a relatively small élite... distinguished by their ability
to concentrate continuously on one fairly long piece
of reading matter’ [qtd. in 8].
Design and semantics of form and movement
121
To understand the shift exemplified by Warde’s
approach, it is necessary to see the historical lineage
that Warde comes from. This is a historical trajectory
that emphasises the creative as connected directly to
art discourse. This lineage tends to follow the same
historical progression as that of printing history up
until about the time of the Arts & Crafts period of the
late 1800s, when fine art and printing reach a climactic
embrace. In 1891 William Morris founded the Kelmscott
press, one of the key highlights of typographic histories
in design literature [9]. Morris’s press was to spawn
a generation of artistic printing but more importantly
was to set a language for typography as it could emerge
in the discourse of design. Shortly after the Kelmscott
Press opened, Charles Ricketts founded the Vale
Press (1896-1903) and Kinross makes the following
observations:
He may stand as one of the clearest representatives of a
new figure who appears in printing and publishing at this
time: the book designers... Before the appearance of the
book designer, ‘designer’ had, in the context of publishing,
meant essentially ‘illustrator’. The work of Ricketts, and
other designers for commercial publishers of the late
nineteenth century, represents the incursion of art into
machine production [10, p. 38].
Kinross notes that “artistic printing occurred at the
moment when a change in taste became apparent:
between highbrow and lowbrow, between a minority
and a mass-market” [10, p. 40]. Although Warde herself
states that “it is mischievous to call any printed piece a
work of art, especially fine art” [7] her positioning of
typography as a creative endeavor steeped in a historical
tradition firmly aligned with connoisseurship values has
done much to dictate the constitution of the discipline.
2.2 Connoisseurship and Contemporary
Graphic Design
In 2002, Satellite interviewed a noted typographical
designer and the prelude to this interview is revealing:
He walks into the basement of Eshleman, our interview
spot, and homes straight in on a framed poster leaning
against a corner. “THIS IS A PRINTING OFFICE” the
elegantly printed and not entirely subtle sign says, “CROSSROADS OF CIVILIZATIONS, REFUGE OF ALL THE ARTS”.
Andy Crewdson stands next to the sign, his neck craned
122
Design and semantics of form and movement
down at it, and says practically without introductions:
“That’s a Beatrice Ward from 1932. That’s really cool.”
...I look at the sign and he’s right (the sign is a reprint), but
he’s right on both counts. Andy, a senior who has more or
less devoted his free time to typography and reviving long
out-of-use typefaces, goes on for about a minute about
the history of the sign (Ward created several important
typefaces), the methods of printing it (Letterpress), and its
rarity (“You don’t see many of those”). He has all this to say
about a sign hanging in my own office, a sign that I'd only
given half a look at about three years ago, and it occurs to
me that this is what graphic design is all about: recognizing
type. [11, n.p.]
We can see here how the author recognises the
language of typography as the language of graphic
design, but it is equally important to understand the
nature of this language – that is, one which has all of the
mystification of the discourse of fine art. Aside from
the misspelling of Warde’s name (a common mistake),
there is an unmistakable quality of traditional fine art
connoisseurship within this article. “That’s a Beatrice
Ward from 1932” smacks of the elevation of the artist's
significance over the work, similar to how one might say
“The Picasso is in the hall” or “We’ve just purchased a
Rembrandt”. The author is quick to point out the work
is not an original but a reproduction, even though the
work is a piece of printed copy. The language of fine art
connoisseurship is unmistakable.
2.3 Problems with Connoisseurship in Design
Pedagogy
I argue that in most design departments of universities
and colleges, graphic and communication design is taught
largely through a connoisseurship model. This is especially
so, when it comes to typography. One of the most popular
typography textbooks, Stop Stealing Sheep describes the
difference between the expert and the amateur:
It is a bit like having been to a concert, thoroughly enjoying
it, then reading in the paper the next morning the conductor
had been incompetent, the orchestra out of tune, and that
the whole piece of music is not worth performing in the first
place... The same thing happens when you have a glass of
wine. While you might be perfectly happy with whatever
you’re drinking, someone at the table will make a face and
go on at length why this particular bottle is too warm, how
that year was a lousy one anyway... [12, p. 17].
There are a number of problems with the
connoisseurship system. The fascination with what
is, that is, with the tangible, can work against the
recognition of what is not; of what is missing from the
picture. The connoisseurship model privileges object at
the expense of context; detail at the expense of wider
philosophical or sociological perspective. The question
must surely be asked here: why should any model have
to address any wider enquiry than its specific area of
interest? My answer takes up Freire’s [13] perspective
that everything is political – that, in fact, one acts
politically whether one recognises it or does not. If
there is social imbalance and one acts within the social
sphere, then one acts to maintain that imbalance or to
critique it. The connoisseurship model acts to maintain
and propagate class distinction, not only in the producer
of type, but also in the consumer. For the model to
work, a correlative degree of mystification around
aesthetic values is supported – a form of distancing
of the language of type from the uninitiated. Bourdieu
reminds us that the language of art is a language of
exclusion which privileges certain social groups over
others, noting that “the ‘naïve’ spectator cannot attain a
specific grasp of works of art which only have meaning –
or value – in relation to the specific history of an artistic
tradition”. [14, p. 4] This is as much the case in the art
of typography, as in any other art form.
There is a more immediate and vocational difficulty
with the connoisseurship approach. Students who have
learned their art through this approach are well-suited
at designing logotypes and the like for banks, lawyers,
and other elite clients, but often tend to see what they
constitute as ‘good design’, as universally appropriate.
Thus, anything that does not fit with the aesthetic to
which they are inculcated, is seen a ‘bad design’. This
means that, as communication designers, they are
limited in their abilities to communicate to anyone
outside of the ‘high design’ aesthetic. If the need arises,
as it occasionally does, the approach tends to be one
of maintaining this aesthetic, as a form of educating
those of lesser taste. This is not always a successful
approach. The emphasis of the connoisseurship model
is very much more on the producer, rather than on the
consumer, and although it may be recognised that what
is deemed good design changes over time, there is still
a certain essentialism in the value system. Good design
is seen as good, no matter the make up of the audience.
2.4 The Benefits of Connoisseurship in Design
Pedagogy
For all its possible failings, however, there are many
benefits offered by the connoisseurship approach. In
the first place it offers individuals a sense of belonging.
Well-respected Australian designer Alistair Morrison
had this to say about typography:
I like the discipline of it. I’ve always had the feeling of
satisfaction of belonging to an international and almost
timeless brotherhood which included people like Bodoni and
Aldus Manutius, a feeling that I am just another link in the
chain [qtd in 15].
For students, this sense can be a powerful incentive
to learn, and staff have commented that students
have ‘found themselves’ after discovering the world
of typography. These students are given constant
reminders of their newfound abilities, skills in type
recognition and aesthetic discrimination by the world
around them. Every menu, road sign, and website offers
up delicacies to be savoured, or crimes to be abhorred.
Every unrecognised font becomes a challenge and each
challenge becomes a reminder of the club to which they
now belong. As Garfield notes: “Identifying a particular
font can be the most infuriating task, and designers
can spoil their whole day by walking past a shop and
seeing something they can’t name.” [16, p. 174] Students
become more immersed in their art than any teacher
could otherwise hope for, and soon know as much,
if not more about the subject, than their professors.
Genuine engagement is the best of tutors. The joy
felt by the student is a combination of deep aesthetic
appreciation and the kind of excitement one has at
demonstrating (if even only to oneself) a finely tuned
skill. For those students who perhaps never excelled
at sports, or were not socially integral members of
any clubs in their younger years, typography can open
up a whole new aspect of their personality. This can
be an immensely rewarding experience that sees them
not only develop an interest, but often, sets them on
a career path, with a fascination and deep sense of
enquiry that lasts a lifetime. The flow on effects in
terms of self-confidence and personal sense of
direction are immeasurable.
Design and semantics of form and movement
123
3 Semiotics in Design Pedagogy
I argue that semiotics taught in conjunction with the
connoisseurship approach can not only address the
failings of the latter, but also contribute to a world-view
that can benefit design students immensely. Semiotics
is particularly useful in terms of relational values. The
object is always understood within, and with respect to,
its context. This context highlights that design is always
political, and although graphic design is visual in nature,
at its core there exists an element of the invisible.
As Fry notes:
Whatever design exposes, it also conceals. For instance,
architecture, product and industrial design aesthetically hide
the nature, quality and assembly of structural components
and operative functions, this most overtly via styling brought
to facades, casing, cladding, mouldings, etc. Packaging also
acts as more than just a protective wrapping that conceals
what it contains – its design often obscures the difference
between the projected image of the object packaged and the
experience of the object itself. Graphic design likewise both
exposes and masks what is seen... [17]
At the same time, a poststructuralist approach to
semiotics regards the language and value systems
as fluid. In this semiotic model, ‘good design’ has no
meaning outside of its particular context. If the model is
one of communication, then no particular aesthetic can
be placed ‘above’ another, and ‘good design’ becomes
that which is the most effective solution.
3.1 Combining Semiotics and Connoisseurship
in Design Pedagogy
There is an initial resistance of some students to
exploring the realm of semiotics in design. It seems they
are already familiar with Paddy Whannel’s statement:
“Semiotics tells us things we already know in a language
we will never understand.” [Whannel, qtd. in 18]. At
first, certainly, it does seem an unnecessarily complex
set of concepts and language to discuss sometimes the
most mundane of things. However, once students grasp
the concepts involved, they often relish the knowledge.
In fact, the discourse around semiotics works in a
way similar to connoisseurship – it provides a deeper,
or at least, different knowledge to which they are
familiar, along with its own historical lineage, language,
hierarchies and the like. It is in this sense, than van
124
Design and semantics of form and movement
Leeuwen’s notion of social semiotics, which “requires
immersing oneself not just in semiotic concepts and
methods as such but also in some other field” fits well
[19].
An important aspect that semiotics brings to graphic
design is an emphasis on the connotation, both in a
direct and immediate sense, but also in a wider political
sense. In the local sense, we become aware of a clear
differentiation in the way design writers articulate
how type, for example, works. In one section Garfield
discusses type from the traditional connoisseurship
perspective:
Will the ear be level (Jenson) or droopy and tear-shaped
(Century Schoolbook)? Will the upper bowl be more
voluminous than the bottom one (Century Old Style) or viceversa (Walbaum)? And what about the bowls? These are not
arbitrary decisions, but are tied to the pedigree of the type
[16, pp.175-6].
In another, the effect through connotation is clear:
Was the TIGER WOODS scandal a little too grubby for the
glossy magazines? Not if his first name was set in a huge
capitalized version of BODONI on the cover of Vanity Fair.
Then the story would look sophisticated, classy and refined
[16, p. 205].
That values such as class, sophistication and refinement
can be imparted by a choice of type may seem an issue
of connoisseurship, but semiotics can also be put to
work here and in this way students begin to see the
value in both. Similarly, both class and connotation are
suggested when Spiekermann points out that ‘wine
menus look different from snack menus... because wine
is more valuable – so that’s serif. The snack menus are
sans serif’ [qtd. in 16].
What I am advocating here then is not a rejection of
traditional connoisseurship approaches, but the addition
of a deeper sensitivity to connotative meanings and
possibilities. A thorough knowledge based in historical
knowledge and awareness of detail can be combined
with a sensitivity to connotation and meaning. As an
example, we can consider this passage discussing the
typeface Avant Garde, from Heller:
Avant Garde was adopted as symbolic of the raucous sixties
and me-generation seventies. While the face had roots in
modernism, it was also eclectic enough so as not to be too
clean or cold. As a headline face it said ‘new and improved’,
and as a text face it added quirkiness to the printed page.
It came alive on advertisements, and was appropriate for
editorial design too. Eventually, after excessive overuse
and rampant abuse, its quirkiness became simply tiresome
– something like the paisley of type fonts – no longer
fashionable, but not entirely obsolete either [20].
Here we can see that connotation can emerge out
of historical knowledge and experience. It can be
tempered by fashion, and artifacts that strongly connote
a particular historical era are highly susceptible to
the fortunes of fashion. The notion of the ‘classic’ can
escape this by a call to a particular elitist aesthetic (it is
a classic because it has a beauty, elegance, etc. above the
whims of fashion). This is a traditional connoisseurship
approach enhanced by an appreciation of semiotic
analysis whereby each informs the other.
3.2 Impediments to Semiotic Approaches
in Design Pedagogy
It may seem obvious that semiotics has a place in design
pedagogy, and yet currently it struggles to make any
impact on design practice and indeed, as Storkerson
notes, across design disciplines, semiotics lacks any
“broad visibility” [21, p. 7]. In fact, it is found in many
courses where graphic design is taught and it is promoted
by a number of interlocuters [19, pp. 21-28]. However,
it often tends to sit firmly within the theory area and
competes for attention with the history of art, the
history of design and a range of other important design
concerns. I would argue that semiotics can provide
language, which allows for a much richer discussion of
many of these other design issues, and has the potential
to add significantly to the design creative process.
One of the impediments for an uptake of semiotics
is that it removes the emphasis from the producer.
It appears to give no direction in terms of what should
be done. In fact, it de-emphasises the authorial role
of the designer, and introduces aspects of marketing,
a prospect disparaged by many a professional designer.
There is a politics to overcome here. Designers have
long seen themselves as quite distinct from advertisers,
and have often been keen to maintain the distinction.
There are many reasons for this [4], but a significant one
is that advertising is seen to have less of a connection to
art, whilst designers have in many ways tended to
maintain their historical connections to commercial art.
This connection marks a sense of authorial power over
the end product, and reifies the artistic skill of the
design professional. Marketing has been a key element
of advertising since the 1950s [29] and has often been
seen as a threat to the autonomy and expertise of
the designer [4]. Currently the boundaries between
advertising and design have become blurred and an
appreciation of what each area has to offer is beginning
to filter through to both discourses. Some would still
argue, however, that a knowledge of semiotics even
interferes with the creative process. Van Leeuwin makes
a salient point here: “Just as the ‘rules’ of spelling and
grammar did not destroy the possibility of linguistic
creativity, so an explicit semiotics of typography will not
destroy typographic creativity.”. [27]
Another important aspect of semiotics that militates
against its development in design education is its
emphasis on context. Whilst this, as I have noted, is
what makes it so attractive for those wishing to see
designers with a deeper awareness of their designs and
their social communicative impact and implications,
this is seen by some as having negative repercussions
for pedagogy. Professional design emerged out of the
industrial revolution, as a fundamental process of the
capitalist economic system. It is of course implicated
in the production of goods and in the selling of those
goods, and since the 1950s in the production of markets
to buy those goods [30]. In bringing to attention the
political context that surrounds design production
and promotion, there is an aspect that emerges that
can be seen as possibly vocationally discouraging to
students. Design often works to maintain and propagate
hegemonic ideological positions that can be seen as
restrictive to individuals and globally destructive.
Semiotics draws attention to the ideological import
of design as we can see from Kress and van Leeuwen:
Pictorial structures do not simply reproduce the structures
of ‘reality’. On the contrary, they produce images of
reality which are bound up with the interests of the
social institutions within which the pictures are produced,
circulated and read. They are ideological. Pictorial structures
are never merely formal: they have a deeply important
semantic dimension [28, p. 45].
Illuminating this aspect of design could be viewed
as having a negative impact on design courses and
Design and semantics of form and movement
125
the argument for globally responsible and ethical
designers needs to be strenuously upheld when semiotic
approaches are part of design pedagogy. As it is, I argue
that these fears are unfounded. Not only has the course
I teach in semiotics had continuously excellent student
feedback, but I have had numerous students impart to
me that the course has “changed their life”, and added
immeasurably to their appreciation of design. Even many
years later, ex-students have told of the lasting positive
effects from their course in semiotics and design, which
positively affect their work still.
3.3 Understanding the Design Process and
the Significance of the ‘World-view’
One of the major hurdles for the introduction of
semiotics in design discourse is functional in nature.
Numerous authors [31-35] have written on the subject
of the creative design process, conveying a number of
differently nuanced versions, yet all reject the notion
of design thinking as a simple linear process. Influences
and inspiration come from many directions, and at
different, sometimes unexpected points in the process.
There is often backtracking, restarting, learning from
errors, finding inspiration in mistakes, and importantly,
risk taking. General leanings can be as important as
specific intentions. It can be directed along a certain
path, but the requirement for creativity militates
against the formulaic.
Visual Rhetoric has been one approach to bringing
semiotics into design with a connection to the creative
rather than the analytical in design. As Buchanan notes:
“rhetoric is traditionally characterized from antiquity
by many of its leading theorists and practitioners
as an art of invention and discovery” [36] However,
visual rhetoric is often applied as a visual ‘version’ of
linguistic rhetoric, with a recipe book approach listing
the various tropes. Whilst it is relatively easy to find
visuals demonstrating figures of addition, subtraction
and the like; and examples of visual tautology, ellipsis,
or asyndeton, students are left with little understanding
of how these can be used in any creative process.
Used to a design process that is more organic than this
approach allows for, this tends to reinforce the notion
that rhetoric is an exercise in analysis and categorization
rather than a creative tool. For this reason, both
semiotics and visual rhetoric have been regarded
unsuccessful as creative tools for the designer. Instead,
terms like ‘intuitive’ are employed when discussing how
126
Design and semantics of form and movement
designers come up with their concepts or treatments.
I argue, however, that the perception that semiotics
cannot add to this process is a simple misconception
about how designers work.
In describing designers in the process of practicing
their work, Schön [37] introduces the concept
‘knowing-in-action’ whereby observed practitioners
drew on ‘prestructures’, in his terms “familiar models
and prototypes that shaped their views of materials,
their interpretations of the task, and their images of
a completed product, in order to design”.[37] I argue
that prestructures are not necessarily as ‘structured’ as
Schön describes. Instead, they are also informed by what
‘world-views’ – models through which we make sense
of, that is, make usable, our world. It is these worldviews that allow for, or make necessary certain specific
models to surface at certain times. To use a simple
example, if we give students a word such as ‘angry’ and
ask them to draw it – the images will be different for
each student. Some may seem familiar, but for others,
when we observe what they produce, we are left with
the sense: “I would never have thought to do that”.
Sometimes this is judged as good, and sometime the
reverse, but the important thing here is that each brings
to bear, whether recognised or not, their own personal
lived experience. This is the foundation for art therapy,
which sees what a person draws as directly connected
to their psyche [38], [39]. This can be seen as a direct
reflection of ‘who we are’. But who we are cannot be
separated from how we see the world, our world-view
– they are two sides of the same coin. Someone whose
world-view includes an awareness of semiotic theory
cannot but have the impact of that theory in the work
they do. Schön notes further that: “Paradoxically, it
was their subjective appreciations that enabled them
to make sense of what would otherwise be a mess and
to construct design worlds in which the problems they
had framed could be said, objectively, to be solved.” [37]
Semiotics gives designers a model for how the world
works – not every aspect of the world but certainly of
how their world works. Hopefully too, it is an expanded
version of their world as it was before they undertook
semiotic enquiry. Thus, their world-view incorporates
both what they know from the connoisseurship model
and the semiotic model. These models do not live
separately in the designer’s mind but coalesce, with
sometimes one having more resonance for a task and
sometimes the other, and yet there will be times where
both will have an influence – depending on the many
other experiences that have shaped the individual.
To use a somewhat pedestrian example, a student who
has a fear of dogs and who is asked to visually represent
the word ‘happiness’ is unlikely to come up with an
image of puppies. Similarly, a student who has developed
an awareness of gender politics through their semiotic
analysis of advertising, may be less likely to jump first
to a design solution that is inherently sexist in its
connotation and ideological effect.
3.4 Benefits of the Semiotic Sensibility in Design
In terms of its vocational advantage, a semiotics
perspective puts greater emphasis on the consumer, and
on the act of communication, than the connoisseurship
model does. As early as 1916, advertising executive and
critic Charles Higham had voiced the recognition of the
persuasive powers of type, stating:
A clever manipulator of type can make it serve the purpose
of publicity with a forcefulness that the uninitiated always
feel but seldom understand [qtd in 40].
This is an interesting perspective as it raises the
connoisseurship notion (initiated–uninitiated), but not
in the disinterested way of fine art – indeed the term
‘manipulator’ here seems to act not only on type but
also on an unwitting audience. Although this certainly
sounds morally dubious, it nevertheless makes clear
that type has a connotative effect beyond any notion
of aesthetics. It refers to effective communication.
My pedagogical approach to type works from the
following notion:
A piece of type ‘speaks’ to you, even before you read
what it has to say.
This may seem paradoxical, but of course I am using
‘speak’ in the semiotic sense, and ‘what it has to say’
in terms of the written content of the words. By
drawing attention to how typography ‘speaks’ or hails
you in the Althusserian sense [41], we foreground
effective communication, over any aesthetic dimension,
as we might find in a purely connoisseurship approach.
3.5 Teaching Semiotics in Design
My approach to semiotics in graphic design pedagogy
is largely through the work of Roland Barthes [2], [42],
primarily because of the strong connection here of
signification to culture. As noted above, connotation
is perhaps the most important aspect of semiotics for
design, and Barthes allows for the political and social
implications of design to become visible. The notion
of myth is also particularly useful for students and we
employ the description ‘little stories that help us make
sense of the world’. Students are taught Marxist and
post-Marxist conceptions of class and ideology, and
Freudian and Behaviorist psychology, as well as race and
gender politics; but importantly all of these are explored
within a framework of Barthesian semiotics. This
means a common language becomes available through
which these different theories, or metanarratives can
be approached. Although this growing knowledge is
applied at various stages to design artifacts, semiotics
as a recipe to develop the creative process of designing
is avoided.
Semiotics in design works best as a sensibility rather
than as a set of rules or structures. The linguistic
emphasis on structure is problematic for the design
process, which requires a fluidity and often a freedom
from strictly directive or formulaic methodologies.
The term ‘sensibility’ is used here to mean a familiarity
with, and working knowledge of, semiotics. In a sense, it
involves more of a sense of ‘background’ from which the
designer operates. It might be considered a ‘world-view’
and something that operates on an almost subconscious
level. Van Leeuwin uses the notion of semiotic resource
from Halliday:
The term ‘semiotic resource’ is therefore a key term in social
semiotics. It originated in the work of Halliday who argued
that the grammar of a language is not a code, not a set of
rules for producing correct sentences, but a ‘resource for
making meanings’ [19, p. 3].
The means by which a semiotic sensibility is achieved
is through an inculcation of the notion of semiotic
connotation across a wide range of different media, such
that students see the semiotic implications of an artifact
without necessarily having to consciously ‘switch’ into
an analytical perspective. This involves at first a very
directive and thoroughly researched semiotic analysis
of film, to be followed by one of advertisements,
then of product labels and packaging. Throughout the
process, there is constant reinforcement of the language
of semiotics with discussion of how it can be applied
to other systems, like architecture or even of how
Design and semantics of form and movement
127
someone dresses. Students also undertake a project
in which they swap the signifiers, but not the actual
words, of two different common supermarket product
packages. This highlights the powerful effects that design
signifiers like colour, typography and layout can have in
supporting and propagating social myths about class,
gender, and the like.
Although I have largely limited my discussion here to
typography, semiotics is a powerful tool in approaches to
all aspects of graphic design. What are the connotations
of lots of different typefaces and a multitude of photographs on the cover of one magazine, for example,
compared to say a single typeface with sparing use of
text and a single full-bleed photograph? How can one
discuss the ‘meaning’ of a photograph that bleeds off
the page, as opposed to one where the edges of the
image are contained within the layout? [43] What is
the connotation of white space? [44] These questions
are opened to a much richer investigation through the
language of semiotics, than through the limited and
directive approaches offered by a connoisseurship model.
The end result is a language which students can apply
to a vast range of artefacts and systems, allowing them
to compare and contrast design decisions and effects
in ways previously inaccessible to them. This is the
practical application in design of Kress & van Leeuwen’s
[28] multimodal approach:
We were driven, however, by the intention to support and
empower designers rather than to unravel the mystery of
design expertise and craftsmanship. Or, in the words of
Green: “Our objective is not to provide recipes for nondesigners to become designers, but to provide tools for
the design profession to do what they do better. [45]
Teachers who employ what I am calling here the
connoisseurship model often place much stake in the
‘intuitive’ process. Far from opposing this approach,
I see semiotics as providing students with a richer
historical and philosophical world-view, out of which
their ‘intuitive’ processes of creativity can flourish.
References
[1] Saussure, F.D. (1974). (orig. 1916). Course in General
Linguistics. London: Fontana.
[2] Barthes, R. (1995), (orig. 1964). Elements of Semiology.
New York: Hill and Wang.
[3] Foucault, M. (1972). The Archaeology of Knowledge.
London: Tavistock.
[4] Young, A. (2005). A Genealogy of Graphic Design in
Victoria. PhD Doctoral Thesis. Melbourne: RMIT.
[5] (2008). Concise Oxford English Dictionary. Oxford:
Oxford University Press.
[6] Belanger, T. (1995). Beatrice Warde at Columbia,
24 October, http://palimpsest.stanford.edu/byform/mailing-lists/
We seek to break down the disciplinary boundaries between
the study of language and the study of images, and we
seek, as much as possible, to use compatible language,
and compatible terminology in speaking about both, for in
actual communication the two and indeed many others come
together to form integrated texts [28, p. 183].
bookarts/1995/04/msg00093.html
[7] Warde, B. (1932) The Crystal Goble. July 25, 2008.
http://www.typographia.org/1999/graphion/crystal-goblet.html
[8] Stiff, P. (1996). Instructing the printer: What specification tells
about typographic designing. Typography Papers, 1, (pp. 27-74).
[9] Meggs, P.B. (1998). A History of Graphic Design. New York:
John Wiley & Sons, Inc.
4 Conclusion
[10] Kinross, R. (1992). Modern Typography: An Essay in Critical
This paper is not a call for the replacement of the
connoisseurship model with one of semiotics – indeed,
it sees many profound positive effects of both models
and regards that they can work together to produce
more informed, more impassioned, creative designers.
Desmet and Hekkert [45] quote Green, stating in his
introduction to the Proceedings of the 1st International
Conference on Design and Emotion, “there remains
a strong element within the design profession which
regards all analysis of design activity as merely attempts
to produce ‘recipes’ for those who can’t emulate those
who can!” Desmet and Hekkert’s response is that:
History. London: Hyphen Press.
128
Design and semantics of form and movement
[11] Chen, K. (2002). Man of Letters, October 24,
http://www.readsatellite.com/culture/2.4/crewdson.chen.2.4.1.htm
[12] Spiekermann, E., & Ginger, E.M. (1993). Stop Stealing Sheep
and Find out How Type Works. Mountain View, California:
Adobe Press.
[13] Freire, P. (1985). The Politics of Education. New York:
Bergin & Garvey.
[14] Bourdieu, P. (1984). Distinction: A Social Critique of the
Judgement of Taste. Cambridge, Massachusetts: Harvard
University Press,
[15] Caban, G. (1983). A Fine Line: A History of Australian
Commercial Art. Marrickville, NSW: Hale & Ironmonger Pty Ltd.
Theory. London: H. Karnac.
[16] Garfield, S. (2010). Just My Type. London: Profile Books.
[39] Rubin, J.A. (1999). Art Therapy: An Introduction.
[17] Fry, T. (2003). Televisual Designing. Design Philosophy
Philadelphia: Brunner/Mazel.
Papers, 3.
[40] Craig, R. (1990). Ideological aspects of publication design.
[18] Seiter, E. (1992). Channels of Discourse, Reassembled:
Design Issues, VI(2), pp. 18-27.
Television and Contemporary Criticism. North Carolina:
[41] Althusser, L. (1971). Lenin and Philosophy and Other Essays.
University of North Carolina Press.
New York and London: NLB..
[19] Van Leeuwen, T. (2005). Introducing Social Semiotics.
[42] Barthes, R. (1980). (orig. 1957) Mythologies. London: Granada.
New York: Routledge.
[43] Vanderlans, R. (2006). (orig. 2003). Looking Closer Five:
[20] Heller, S. (2006), Looking Closer Five: Critical Writings
Critical Writings on Graphic Design. New York: Allworth Press.
on Graphic Design. New York: Allworth Press.
[44] Robertson, K. (1994). Looking Closer: Critical Writings on
[21] Storkerson, P. (2010). Antinomies of semiotics in graphic
Graphic Design. New York: Allworth Press.
design. Visible Language, 44(1), pp. 5.
[45] Desmet, P.M.A., & Hekkert, P. (2009). Special issue editorial:
[22] Krippendorff, K. (2006). The Semantic Turn:
Design and emotion. International Journal of Design, 3(2), pp. 1-6.
A New Foundation for Design, London: Taylor & Francis.
[23] Hjelm, S.I. (2002). Semiotics in product design. Technical
Report No. CID-175, CID, NADA, KTH. Stockholm: Royal
Institute of Technology.
[24] Triggs, T. (2004). The Education of a Typographer.
New York: Allworth Press.
[25] Addison, N. (2005). Critical Studies in Art & Design
Education. Bristol: Intellect. Chap. 13.
[26] Ockerse, T., & Van Dijk, H. (1979), Semiotics and graphic
design education. Visible Language, 13, no. 4, pp. 358-378.
[27] Van Leeuwen, T. (2005). Typographic meaning. Visual
Communication, 4(2), pp. 137-143.
[28] Kress, G., & Van Leeuwen, T. (1996). Reading Images: The
Grammar of Visual Design. London and New York: Routledge.
[29] Hewat, T., & Rankin, P. (1996). Clemenger: First Fifty Years
1946-1996. Melbourne: Clemenger.
[30] Boihem, H., & Emmanouilides, C. (1997). The Ad and the
Ego. California: Newsreel.
[31] Potter, E. (2010). Ducks, Decorators, and the Dialogical:
An Examination of Approaches to Information Design. Masters
Thesis, AUT, Auckland.
[32] Von Engelhardt, J. (2002). The Language of Graphics.
PhD Dissertation, Universiteitt van Amsterdam, Holland.
[33] Tufte, E. (1998). Visual Explanations. Connecticut: Graphic Press.
[34] Jones, C.J. (1992). Design Methods. New York:
Van Nostrand Reinhold.
[35] De Bono, E. (1977). Lateral Thinking. London: Penguin.
[36] Buchanan, R. (2001). Design and the New Rhetoric:
Productive Arts in the Philosophy of Culture. Philosophy
and Rhetoric, 34(3), pp. 183-206.
Alan Young
[37] Schön, D.A. (1992). The theory of inquiry: Dewey’s
Auckland University
legacy to education. Curriculum Inquiry, 22(2), pp. 119-39.
of Technology,
[38] Schaverien, J. (2001). Where Analysis Meets the Art:
New Zealand
The Integration of the Art Therapies with Psychoanalytic
Design and semantics of form and movement
129
Tom Djajadiningrat, Luc Geurts, Jeanne de Bont, Pei-Yin Chao
jpdjajadining[email protected]
Grace: A gesture-controlled
wake-up light
Abstract
We investigated the use of a deviceless gesture control
for a wake-up light, a type of alarm clock which wakes
the user through both light and sound. We explain the
interaction design challenges for the wake-up light and
discuss the drawbacks of deviceless gesture control.
These challenges and issues were explored through
an experience prototype which we call Grace. We
argue that deviceless gesture controls fit the sleepy
interaction associated with the wake-up light and help
in realizing a calm product appearance. Our key finding
is that gesture control needs continuous guidance: realtime, augmented feedforward and feedback, which helps
to increase the user’s confidence during interaction and
to improve gesture recognition.
1 Introduction
In this project, we focused on an interaction style
called deviceless gesture control. Deviceless gesture
control allows users to operate devices from a distance
without the need for physical remote controls, bodyworn electronics or markers [1], [2].
One of our use cases was a wake-up light, a type of
alarm clock which wakes the user through both light
and sound. A wake-up light fades in the light over a
set period of time (default 30 minutes) before the
alarm time. By the time the alarm sounds, the lamp has
reached full strength. This slowly increasing brightness
causes the user to gently wake up. The intended
130
Design and semantics of form and movement
experience is captured in the product’s tagline
of “a natural, gradual way of waking up”.
In this paper, we first describe the wake-up light’s
interaction design challenges. Then we discuss the
interaction issues with deviceless gesture control.
We describe two participatory innovation events,
one with developers, the other with users. We then
argue the benefit of gesture control for a wake-up light.
Finally, we present a concept called Grace in which we
address the wake-up light’s interaction challenges and
discuss the interaction principles we used to mitigate
the drawbacks of deviceless gesture control.
2 Wake-up Light Interaction Design
Challenges
Designing the interaction for the wake-up light
is challenging for four reasons: multi-functionality,
programmability, sleepy users and conflicting interaction and product design requirements.
2.1 Multi-functionality
A wake-up light is a multi-functional product which
combines an alarm clock, a dimmable light, a radio
and a digital music player, operated through a single
user interface. Many of these functionalities are intertwined. For example, the brightness of the light is
controlled by the alarm time of the clock. As a result,
a modal approach to the interface is not possible as
many functions ‘straddle’ two or more modes. It would
also require users to switch back and forth between
modes to access functions which experientially
belong together.
2.2 Programmability
Many of the parameters are programmable, including
the wake-up time, the fade-in duration, wake-up brightness and the wake-up sound. Both this programmability
and the aforementioned multi-functionality stretch
the interface of the current generation wake-up light
which consists of an alpha-numeric display and push
buttons. Such a ‘display + push button’ interaction style
is known to put a heavy burden on users’ cognitive skills
and to be slow and cumbersome for adjusting analog
parameters [3].
appearance which fits the bedroom environment.
To create such a calm appearance, the design should
avoid the overt use of control panels and displays which
are likely to create visual clutter. This then is where
product design and interaction design potentially clash.
Product design on the one hand calls for minimizing
the number of controls and for repeated use of the
same type of control to realize a calm appearance
which makes it difficult to realize intuitive interaction.
Interaction design on the other hand calls for dedicated,
differentiated and analog controls such as sliders and
rotary controls, all of which results in the visual codes
which typify the consumer electronics ‘gadget’ genre.
Part of our challenge was to investigate how deviceless
gesture control could contribute to solving this conflict.
2.3 Sleepy Users
Another reason why users may have more difficulties
with operating a wake-up light than with other products
is simply because of sleepiness. Immediately after waking
up people suffer from sleep inertia, a physiological state
characterized by diminished motor dexterity and a
feeling of grogginess [4], [5]. This impaired alertness may
interfere with our ability to perform mental or physical
tasks. Similarly, sleep homeostatis–the physiological
process whereby our need for sleep increases the
longer we stay awake – results in sleep pressure which
leads to diminished cognitive performance [6]. Clearly,
these are the very moments – just after having woken
up and just before going to bed – when users interact
with their wake-up light. At these moments, tasks which
are trivial when fully awake suddenly require effort.
3 Gesture Control Challenges
2.4 Interaction vs Product Design Requirements
A Wake-up Light has many analog parameters (e.g.
brightness, volume, wake-up time) and long-list
parameters (e.g. radio station presets, MP3 tracks)
which in current models are operated through binary,
up-down controls. This requires the user to press
buttons multiple times when making large adjustments,
leading to a staccato style of interaction.
From an interaction design perspective, one approach
to improve the interface would be to add dedicated,
analog controls so that key analog parameters may be
controlled directly instead of via menus and up-down
buttons, resulting in a more fluent interaction style.
However, one of the starting points for the product
design is that a wake-up light should have a calm
3.1 The Need for Initiation
If gesture control were continuously enabled, users
may unintentionally trigger a function. For example,
if the brightness of a wake-up light is controlled through
an up-down gesture, any vertical movement in the active
zone such as picking up a glass of water may influence
the lighting. Clearly, such false positives are frustrating
as they break the user’s sense of control. To prevent
such unintentional triggering, users need to switch on
gesture recognition before they can make a functional
gesture. Our term for such an action is the initiation
action. The challenge in choosing an initiation action
is that if we opt for a physical movement, this initiation
gesture must not occur in daily ritual and is therefore
unnatural by definition. Here then lies the contradiction
Gesture control is often perceived to offer the
ultimate in interaction simplicity, a popular view which
is strengthened by science-fiction movies such as
Minority Report [7]. According to this view, gesture
control may add cost and technical complexity but will
definitely result in a superior end-user experience.
Considering how quickly the cost of the required
technology decreases, some even envisage a day when
we may interact with all our products through gesture
control.
However, for the end-user, gesture control may have
drawbacks more fundamental than a higher purchase
price. Here we describe two user experience issues
with gesture control: the need for an initiation method
and the lack of inherent feedforward and feedback.
Design and semantics of form and movement
131
within gesture control: whilst gesture control is often
described as offering natural interaction, the required
initiation action can make it feel contrived.
3.2 Lack of Inherent Feedforward and Feedback
Physical controls such as mechanical buttons, switches
and sliders have many interaction qualities, which we
take for granted. For example, when using a light switch,
we can see and feel the position of the switch on the
wall as well as see and feel what action is required (e.g.
toggling, pressing, rotating or pulling). When we switch
on a light, we can see and feel the switch move and hear
the click. Such inherent feedforward and feedback [8], [9]
is not self-evident when it comes to gesture control.
It may not be clear how to gesture or even where to
gesture. In some gestural interfaces, the only feedback
the user gets is when the functionality is triggered after
completion of the gesture.
Though this may seem sufficient, it becomes troublesome if the system fails to recognize a gesture because
it is not sufficiently well-defined, too quick or out of
range. Users then seek to understand what went wrong
but without feedforward and feedback it is unclear why
things fail and they cannot correct their actions.
As a consequence the user can feel lost, not quite
knowing what is happening in the interaction dialog.
We may draw a comparison with human conversation,
in which we judge from non-verbal body language
cues whether we are being understood. It is this body
language which is lacking from gesture-controlled
products: we gesture to the product but the product
fails to gesture back.
Karam [10] proposes a framework for gestural
interaction with three types of feedback: reflexive,
recognition and response feedback. Response feedback
occurs when the user is notified that the intended task
has been completed. Recognition feedback is given
when the system has successfully recognized the user’s
gesture. These two types of feedback are known from
interaction design for traditional user interfaces. For
example, the light switching on in reaction to the user
flicking a switch is a form of response feedback.
A beep as a reaction to a button having been pressed
is a form of recognition feedback. Of particular interest
to gesture control, however, is reflexive feedback which
provides the user with feedback on the state of the
input during performance of a gesture. Karam uses a
132
Design and semantics of form and movement
Fig. 1. Ideation through bodystorming in context.
Fig. 2. Participant acting out gestures with WoZ experience demonstrator.
window in a graphical user interface to provide reflexive
feedback. In this reflexive window, users can see themselves performing their gestures and get feedback on what
the computer sees in terms of objects being tracked.
4 Participatory Innovation Events
To explore the UX we staged two events: a multidisciplinary ideation session during a multiple
stakeholder workshop with developers and a multicultural Wizard of Oz test in which users were
confronted with an early experience prototype.
4.1 Multiple Stakeholder Ideation
Twenty developers, including technologists, marketeers
and designers, ideated in multi-disciplinary teams
through bodystorming in context [11], [12]. In a staged
bedroom, one team member acted out the gestures to
control the wake-up light, while others simulated the
reaction of the product (Figure 1). A rating session at
the end of the day showed that participants particularly
appreciated three features: brightness control through
vertical hand movement, stumble light activation – to
find one’s way to the bathroom at night – through a
quick flick of the hand, and direct manipulation of analog
clock and alarm hands through touch gestures.
Fig. 3. Vertical hand movement controls brightness.
Fig. 4. Lazy snooze. In the morning, waving in front of the Wake-up Light
suffices to activate snooze.
4.2 Multi-cultural Wizard of Oz Test
We turned the preliminary concept from the multiple
stakeholder workshop into a low-fi experience
prototype by back-projecting an interactive animation
onto the casing of an existing wake-up light (Figure 2).
This experience prototype was then evaluated in
Wizard of Oz style with nine users of different ethnic
origin. First, we explained the functions to be offered
by the wake-up light. We then asked participants to act
out the gestures they would choose to control these
functions and simulated the wake-up light’s response
from behind the scene. After having shared their own
preferred gestures, we explained to them the gesturefunctions combinations which had come out of the
multiple stakeholder ideation session. They were then
asked to also act out these gestures and again we
simulated the wake-up light’s response in Wizard of
Oz style. Finally we asked participants to reflect upon
the differences between the two sets of gestures.
5 Benefits of Gesture Control for
Wake-up Light
Our experiences with the multiple stakeholder
workshop and the Wizard of Oz user test sharpened
our understanding of how gesture control can be of
benefit for the wake-up light. Gesture control may help
realise the wake-up light’s promise of ‘a natural gradual
way of waking up’, by complementing the functional light
effect with a low cognitive load interaction style which
fits the user’s sleepy physiological state. Gestures can
offer control over the wake-up light’s key functions
without the user having to interpret labels, icons or
displays and without having to reach and fumble for
buttons. Finally, gestures allow the user to stay snug and
comfortable while operating the device from a distance.
We chose the following gesture-function combinations:
r#SJHIUOFTTWFSUJDBMIBOENPWFNFOU
r7PMVNFIPSJ[POUBMIBOENPWFNFOU
r4OPP[FXBWFPOBMBSN
r4IPXUIFUJNFMFGUUPTMFFQXBWFEVSJOHUIFOJHIU
r4UVNCMFMJHIUXBWFVOUJMBDUJWF
The gesture vocabulary was intentionally kept small
with only three gestures (horizontal, vertical and wave)
to avoid it becoming like a modern day DOS language
which requires users to learn and remember a large
collection of arbitrary gesture-function combinations
[13].
6 Concept Design
We designed and built an experience demonstrator of
a gesture controlled wake-up light which we call Grace.
Our aim was to address both the aforementioned wakeup light interaction design challenges and issues with
gestural interaction.
6.1 A Sleepy Interaction Style
We designed Grace to allow sleepy, low cognitive load
interaction.
6.2 Adjust Brightness/Volume Whilst Staying
Comfortable
When reading in bed or listening to music, the user
can simply adjust the light level through a vertical hand
movement or sound volume with a horizontal hand
movement without having to reach. Whilst adjusting the
light level, the light ring reacts with a ‘peacock’ effect:
the line between the dark and light part of the ring
follows the user’s hand movement (Figure 3).
6.3 Lazy Snooze
In the morning, waving in front of the Wake-up Light
suffices to activate snooze (Figure 4).
Design and semantics of form and movement
133
6.4 Ambient Sleep Time Indication
A wave during the night suffices to make the Wake-up
Light show the “time left to sleep”. It will light up an arc
from the current time to the wake up time. The length
of the arc will give the user an approximate indication
of the time left to sleep without having to interpret the
hour and alarm hands (Figure 5).
Fig. 5. Ambient sleep time indication.
6.5 Calm Appearance
Grace uses a combination of deviceless gesture control
for the key functions (Figures 3-5) and a touchwheel for
detailed settings (Figure 6). The touch wheel allows easy
adjustment of analog parameters such as the wake-up
time and the dusk duration. It also allows easy selection
from long-lists such as radio stations and MP3 playlists.
The combination of deviceless and touch-based gesture
control allowed us to eliminate all but one physical
control and to keep the product’s appearance very clean
and restrained. The final design is based on a round
form factor with three concentric circles (Figure 7),
the semantics of which suit an analogue clock, support
the sun metaphor and refer to the traditional alarm
clock product category. The inner circle is formed by a
circular colour OLED display, the middle ring is formed
by a touchwheel and the outer ring is formed by the light
surface. To further strengthen the reference to traditional
wind-up alarm clocks, the product stands on three legs,
two in front and one at the back (Figures 8 and 9).
6.6 Initiation by Delay
One way to make initiation faster, less convoluted
and less contrived is to use a multi-modal approach.
For example, a physical movement can be combined
with sound (e.g. handclap, finger snap) or voice [14].
However, when interacting with the wake-up light the
user’s partner may be asleep and initiation must be
silent. Therefore we opted for initiation by delay: the
user needs to hold her hand still for 1.5 seconds within
the active zone before gesture control becomes active.
6.7 Continuous Guidance
As previously mentioned, inherent feedback is missing
from deviceless gesture control. We therefore need to
add augmented feedforward and feedback to provide
users with information on what is going on [9]. With
gesture control this augmented feedforward and feedback needs to change in real-time in response to users’
gestures so that they may adjust their actions while
134
Design and semantics of form and movement
Fig. 6. A touchwheel is used to enter detailed settings.
gesturing. For this we use the term continuous guidance
(Figure 10).
With Grace, we provide continuous guidance by means
of an animated point cloud effect on the circular OLED
display to elegantly bridge the delay during initiation.
There are two possibilities when the user’s hand enters
the active zone. The first option is that the user has
no intention of operating the wake-up light and that
the hand passes through the active zone incidentally,
for example to pick up a glass of water, a set of keys or
a mobile phone (Figure 11). In this case, the clockface
which is shown in the idle state changes into a point
cloud which follows the user’s hand movement, to
return to a clockface when the user’s hand leaves the
active zone. The second option is that user does intend
to operate the wake-up light (Figure 12). In this case,
the clockface changes into a point cloud which moves
with the hand to then morph into a cross-shaped UI
during the initiation delay. The cross-shaped UI indicates
that horizontal (i.e. volume) and vertical (i.e. brightness)
gestures are possible.
Grace’s continuous guidance serves a purpose similar
to Karam’s [10] reflexive feedback. However, in the case
of Grace the display shows an abstract representation
of the user’s gestures rather than a camera image.
7 Reflection
Fig. 7. The design of Grace is based on three concentric circles.
Gesture control’s main drawbacks are the need for an
initiation action and the lack of inherent feedforward
and feedback. In our concept we tackled these issues
together: Grace makes use of initiation by delay while an
animated point cloud provides continuous guidance by
mirroring the user’s hand movements and bridging the
delay through a morphing effect. This implementation
shows that continuous guidance need not take the form
of a literal camera image of the user but can also take
a more abstract and impressionistic form.
Through this project, we discovered that gesture
control has a number of lesser known user experience
advantages. One is that gesture control requires less
focus than traditional, physical controls and therefore is
a comfortable form of control when the user is sleepy.
Another advantage is that gesture control offers can
offer direct, analog control over key functions without
the need for traditional analog controls such as sliders
or rotaries which clutter the product design.
In any case, gestural interaction raises many semantic
issues. In contrast to physical controls, gestures can
be coupled to form, feedback and function in any way
desired. A gesture has no fixed meaning that is only
suited to a particular function. Instead, the meaning
of gestures is pliable and can be influenced through
a product’s form, its physical and screen based UI
and light and sound. Gesture control puts much
responsibility on designers to prevent these couplings
from coming across as completely arbitrary.
Acknowledgments
Fig. 8. The front legs are reminiscent of traditional alarm clocks.
We gratefully acknowledge Jacqueline Janssen for advice
on Materials & Finishing, Henk Lamers of Loftmatic for
his work on video and sound, and our project manager,
Anja Janssen, for her continuous guidance and keeping
the project on track. We also thank Henk van der Weij
of Bigcamp Multimedia and Marc van Schijndel and Marc
Kemkens of KEMO for their work on the prototype.
References
[1] Flikkert, W. (2010). Gesture Interaction at a Distance.
PhD dissertation. Media Interaction (HMI) research group
of the University of Twente, The Netherlands.
[2] Wigdor, D., & Wixon, D. (2011). Brave NUI World:
Fig. 10. Continuous guidance is about repeating steps 2 and 3
Designing Natural User Interfaces for Touch and Gesture.
until the intended function (4) has been triggered.
Morgan Kaufmann.
Design and semantics of form and movement
135
Fig. 11. Presence detected but the user does not activate gesture control.
Fig. 12. User initiates gesture control.
[3] Buur, J., & Stienstra, M. (2007) Towards Generic Interaction
Styles for Product Design, HCI International 2007. Beijing:
Springer Verlag.
[4] Tassi, P., & Muzet, A. (2000). Sleep inertia. Sleep Medicine
Reviews, 4(4): 341. doi:10.1053/smrv.2000.0098. PMID 12531174.
[5] Wertz, A.T., Ronda, J.M., Czeisler, C.A., & Wright Jr, K.P.
(2006). Effects of sleep inertia on cognition. JAMA: The Journal
of the American Medical Association, 295 (2): 163.
[6] Alhola, P., & Kantola, P.P. (2007). Sleep deprivation: Impact on
cognitive performance. Neuropsychiatry Disease and Treatment,
3 (5), 553–567.
[7] Spielberg, S. (2002). Minority Report [Motion picture].
Twentieth Century Fox Film Corporation.
[8] Djajadiningrat, J.P., Wensveen, S.A.G., Frens, J.W., & Over beeke,
C.J. (2004). Tangible products: Redressing the balance between
appearance and action. Special Issue on Tangible Interaction of
the Journal for Personal and Ubiquitous Computing, 8, 294-309.
[9] Wensveen, S.A.G., Djajadiningrat, J.P., & Overbeeke,
C.J. (2004). Interaction frogger: A design framework to couple
action and function through feedback and feedforward.
DIS2004, pp.177-184.
[10] Karam, M. (2006). A framework for research and design
of gesture-based human computer interactions. Masters
thesis, Faculty of Engineering, Science and Mathematics of the
University of Southampton, United Kingdom.
[11] Burns, C., Dishman, E., Verplank, W., & Lassiter, B. (1994).
Actors, hairdos & videotape - informance design. In Proceedings
of the CHI '94 Conference, 119-120.
[12] Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003).
Understanding contexts by being there: Case studies in bodystorming. Personal Ubiquitous Computing, 7(2), 125-134.
[13] Cassell, J. (1998) A framework for gesture generation and
interpretation. In R. Cipolla, & A. Pentland (eds) Computer
Fig. 9. Grace looks lighter than
Tom Djajadiningrat,
vision in human-machine interaction. Cambridge University.
existing Wake-up Light models,
Luc Geurts,
[14] Djajadiningrat, J.P., Geurts, L., & De Bont, J. (2012).
and leaves space underneath
Jeanne Bont,
Table manners: The influence of context on gestural meaning.
for small objects such as mobile
Pei-Yin Chao
In Proceedings of DeSForM2012. Submitted.
phones and keys.
Philips Design Global
Headquarters
Research, Development
& Innovation Group
High Tech Campus,
Building HTC-33
5656 AE Eindhoven,
The Netherlands
136
Design and semantics of form and movement
Mads Folkmann
[email protected]
The aesthetics of immateriality in design:
Smartphones as digital design artifacts
Abstract
1 Introduction
The paper is a philosophical-theoretical contribution to
the conceptualization of the span of material extension
and immaterial impact in artifacts employing digital
technology. Using the smartphone as an example of a
widely distributed type of material artifact that operates
with immaterial structures of information, the paper
offers a theoretical discussion of how immateriality
can be conceptualized as a matter of aesthetics in the
face of the challenge that digital artifacts pose to the
role and understanding of materiality in design objects.
The paper proposes a framework of aesthetics that
describes sensual, conceptual, and cultural levels of
meaning in and through the object. Further, the paper
discusses how this connects to a notion of possibility
in design. Thus, the paper contributes to a discussion
of the sensuous character and impact of artifacts that
are on the verge of immateriality. The relevance to
design practice is motivated through the discussion of
central concepts of design ontology and the proposal
of a framework of aesthetics that in its discussion and
structuring of levels of meaning in design can inform the
process of developing design.
Seen within the context of the material culture of design,
artifacts employing digital technology, for example in
ubiquitous computing and wireless communication,
create a new culture of design where the elements of
the immaterial, the invisible and the non-sensual in the
design gain in importance. Increasingly, objects and
products contain or are enabled by digital technology
which, in turn, fundamentally structures the functions,
usability and character of the object. This ranges from
the interactive interfaces of computers and PDAs to
cell phones and “plainer” low-tech products such as
toasters and coffee machines which also incorporate
electronics. Even if the world still has plenty of fairly
low-tech material objects, we can still state that a
revolution, a digital turn, has taken place in the way
that objects are designed, constructed, and used. In
a historical reflection on shifts in types of design, the
German design theorist Gert Selle has described how
the microchip can be seen as the founding design for
our current era (following a line from the prehistoric
monolith to the industrial-age paperclip), which is
characterized, among other things, by a miniaturization
of technology and a new culture of invisible design
[1]: While the dictum of “form follows function” in
industrial Modernism should make objects display their
function, the hidden operation of the microchip in the
digital age has lead to “black box” design with object
surfaces that only hint at the function of the object.
Keywords
Aesthetics, materiality, immateriality, design ontology,
digital artifacts, smartphones.
Design and semantics of form and movement
137
We simply cannot directly detect the purpose of the
digitally operating black box objects; as a basic condition
today, we cannot assume that objects are easily decoded
based on their outer appearance. Further, objects that
rely on the wireless exchange of data can even create
a network of meaning that operates invisibly, as they
do not necessarily display their interface of interaction.
This phenomenon, which is often related to the concept
of ubiquitous computing, entails a change in perception,
as our interaction with our surroundings is increasingly
mediated by digitally operating artifacts and, to take
this reflection further, even creates objects engaged
in networks with other objects without any human
interference [2]. In terms of programming, the media
theorist Lev Manovich has claimed (mirroring Siegfried
Giedion’s classic 1948 book on anonymous design
culture Mechanization Takes Command) that ‘software
has taken command’ in “forming contemporary culture”
whereby, furthermore, “cultural, social, and economic
forces [...] are shaping development of software itself”
[3]. The hidden parts of design are taking command.
As a consequence, design objects are not only to
be regarded as objects with a material extension
but increasingly as objects of immaterial effect,
communication, and information. In turn, the ontology
of design objects is challenged, when their being and
constitution is not only marked by materiality and
physical extension but also by levels of immaterially
operating digital technology. This new ontology
of design needs to be conceptually described and
circumscribed. The thesis of the paper is twofold:
(i) Digital artifacts operate in the intersection of
material and immaterial dimensions of design; (ii) the
dimensions of the material and the immaterial in design
can be productively described within a framework of
aesthetics. Aesthetics in relation to design often deals
with the sensual and material aspects of design, but
the proposal here is that an aesthetics of immateriality
in design can be formulated that also regards the
conceptual and cultural aspects of the design. In relation
to aesthetics, the level of the immaterial in design
evokes and stimulates possibilities.
As an example for reflection throughout the paper, the
smartphone will be discussed as a contemporary digital
mass product that has entered the sphere of use and
consumption in a seemingly frictionless manner, but
which in its constitution manifests a ground-breaking
138
Design and semantics of form and movement
combination of material and immaterial design strategies.
Thus, the smartphone will be discussed as a type of
digital design that engages and widens the span of
sensual, conceptual, and cultural possibilities in and
through the design.
2 The Immaterial Through the Material:
The Smartphone
In this section, I will discuss how the material and
immaterial dimensions in design serve to structure
digital artifacts; in this discussion, I will use the
smartphone as an example, as the smartphone may be
seen as a design object that exists at the intersection
of material extension and immaterial implication.
To introduce the product type, a smartphone is
a combination of a cell phone and a pocket-sized
computer. Another example of this kind of portable
computer device for the consumer market would
be the tablet computer, invented as a product type
by Microsoft and considered a failure [4] until Apple
reinvented the product type with the 2010 iPad
and reopened the market for the benefit of other
manufacturers as well. With the iPhone from 2007,
Apple similarly contributed to defining a cultural
trend and enabling a market for the smartphone,
where RIM’s BlackBerry models have offered mobile
email devices ever since 1999, and where other
manufacturers, e.g. HTC, Sony Ericsson and Samsung
are currently contributing to product diversity. The
smartphone employs the wireless cell phone network,
an accessible and visible interface based on a touch
screen and the processing power of a small computer.
Thus, the smartphone is not only a phone; it is more
like a computer device that enables internet browsing
and multimedia interaction. Most of the difference lies
in the operating system (OS); in April 2011, Google’s
Linux-based Android system represented 33% of the US
market, RIM’s Blackberry OS accounted for 29%, and
Apple’s iOS held 25% [5]. A special feature to be noted
in this context is the app, short for “application”, which
is a small program that can be downloaded and used to
enhance the spectrum of functions and possibilities on
the device: games, maps, search functions (best places
to eat, etc.), fixed browser platforms with specific
targets (e.g. the nearest recreational space, vending
places for train tickets). A special smartphone ‘metaapp’ directs the user to the app store or the app market
(depending on the OS), where a variety of apps
can be purchased or downloaded for free.
As a contemporary digital mass product, the
smartphone has entered the sphere of use and
consumption in a seemingly frictionless manner, even
if, by virtue of its constitution, it operates on the basis
of a ground-breaking combination of material and
immaterial design strategies. It is defined as a product at
the intersection of material and immaterial properties,
which are mutually dependent. In this context, material
properties include the tangible, material, physical
aspects of the object or device, that is, the object
as thing or the thingness of the object. Immaterial
properties, on the other hand, include all the aspects
of the device that exceed or transcend the physical
extension of the object, such as its communicative
effects and its ways of structuring and affording patterns
of action. Often, the value that people attach to the
smartphone derives from its connection to other
phones through the network infrastructure. The apps
are particularly interesting in this context, as they
serve as a pathway for conveying wirelessly transmitted
content into the material frame of the smartphone
device. Thus, the apps contribute to the creation
of the smartphone as an open-structured design that
integrates streams of immaterial information and as
a design that widens the span of sensual, conceptual,
and cultural possibilities.
As a concept, immateriality entered the design debate
through communication and media studies when
computer technology emerged in the 1970s and
1980s, even if the new kind of immateriality carried
by digital technology and structured as streams of
information was, and still is, dependent on material
hardware [6] in the often “artificial channels” of
modern communication [7]. As a consequence of the
increase in the circulation of immaterial information,
the media philosopher Vilém Flusser has argued that
a shift has taken place in existential concerns “from
things to information,” where people are “less and
less concerned with possessing things and more and
more concerned with consuming information” [8].
Similarly, design can be seen increasingly as a matter of
information; thus, as early as 1973, Gert Selle pointed
out that design objects not only carry “functions
but also, always, information” [9]. Expanding on the
argument of the content of information in design and
focusing on its social impact, with the dictum “design is
invisible”, design theorist Lucius Burckhardt stated in
1980 that design is not only visible but also invisible in
its social effects and in the implications for the creation
of systemic organization via design [10]. This is a point
that Selle addresses in his recent discussion of the
dialectics of the visible and tangible aspects of design on
the one hand and its invisible conditions and effects on
the other. On the level of the invisible, Selle identifies a
phase before the design – all the aspects in the phase of
designing that condition the design – and a phase after
the design, which deals with the effects of design with
regard to creating and supporting culture, structuring
and guiding social behavior, and regulating action and
corporeal-emotional responses [1]. On this point, Selle’s
discussion is very brief – he merely mentions these
aspects – but illuminating and worth exploring: What
are the immaterial effects of design, and how can they
be divided into levels in the analysis?
The smartphone exemplifies how material and
immaterial features apply to different logics of design
but are nevertheless inextricably interlocked. On
the one hand, what makes the smartphone relevant
to users is its ability to communicate and serve as an
aggregate for the wireless exchange of information,
that is, the dimension of the immaterial; and, on the
other hand, it is the tangible product, the actual device,
that enables the immaterial effects through its material
being. Thus, the design of smartphones is comprised of
two elements that can be employed both individually
and in combination to achieve differentiation between
manufacturers and models: The design of the interface
(and of the underlying OS) facilitates the user’s
interaction with information, and the material design
of the device generates sensual and cultural appeal for
the consumer, who can see him/herself reflected in
the product and use it to create and flash an image of
personal identity. In general terms, this type of design
can be said to have two dimensions; although they point
in different directions, in combination they form the
complexity of the design:
(i) The design has an outer, finite, materially limited
extension (the device) where shape, the choice of
materials, and sensual interactions are crucial;
(ii) The design contains an infinite dimension of, on the
one hand, inward expandability in the form of software
Design and semantics of form and movement
139
structures and information processing and, on the other
hand, network links to information and other devices. In
its handling of information, the interface is an outwardpointing gateway that structures the flow of information
and makes it accessible for the user.
In a phenomenological reflection of meaning incorporated
in sensually appealing materiality, all design objects have
an immaterial dimension in the sense that they incarnate
ideas in their process of signification (e.g. of use,
relevance, or ideology) which, conversely, can only
come into being by means of the tangible product: To
summarize, we cannot conceive design as exclusively
material or immaterial; the immaterial is nothing
without a physical manifestation [11].
Of course, this phenomenological premise also applies
to the smartphone. However, its employment of
digital technology specifies the immaterial dimension
as a structural openness of meaning where different
kinds of information can be drawn into and mediated
through the device. The immaterial dimension of the
smartphone can be described not only in general
terms as ideational content but even more aptly as
a technologically enabled structure that offers new
meanings which, at the same time, must be contracted
in the interface of the smartphone and made accessible
by means of the physical design. Thus, the immaterial
dimension is a key defining element of the smartphone
as a digitally operating artifact; yet still, its immaterial
operation is confined and circumscribed by the material
extension of the design.
3 Aesthetics
The role and function of immaterial effects in design
and their challenge to the dominant material design
ontology can be investigated within an aesthetic
framework. In this context, aesthetics will not be
understood in a classic sense, as issues of beauty
and taste, but as a general framework for examining
how design objects mange to frame modes of
experience and processes of understanding. I will
point to three dimensions of aesthetics and aesthetic
theory: (i) a sensual-phenomenological dimension,
(ii) a conceptual-hermeneutical dimension, and (iii)
a cultural-contextual dimension which I have also,
in part, described elsewhere [11]. With particular
emphasis on the immaterial aspects of the design,
140
Design and semantics of form and movement
the three dimensions of aesthetics can be employed to
study how immateriality affects design ontology; how
the ‘being’ and ‘essence’ of design are conceived. With
reference to the three dimensions of aesthetic theory,
we can ask how the immaterial element in design
aesthetically operates (i) on the level of sensual meaning
and the material object in evoking emotional responses
through the appearance and “feel” of the object, as (ii)
a construction of paradigms of understanding which
is expressed, for example, in the interface and in the
aesthetics of use, and (iii) on a cultural level as part of
the social impact and symbolic communication engaged
by the use and consumption of objects. See an overview
in Table 1, which anticipates the following discussion.
Dimension of
aesthetics:
Sensualphenomenological
Conceptualhermeneutical
Culturalcontextual
Appeal to:
Senses
Understanding
Identity
Emphasis in
design:
Material device
Use
Social impact
Material
extension:
Appearance, texture
Interface
Appearance, texture,
function
(effect on)
Immaterial
impact:
Emotional response
(effect on)
(constructing)
Construction of
meaning
Identity, framing of
experience, symbolic
meaning
Table 1. Dimensions in aesthetics in relation to
material and immaterial aspects of design.
Essentially, the aesthetic tradition has developed along
two lines. Since Romanticism and German Classicism,
there has been a tendency to connect aesthetics with
artistic creation and works of art; this began with
the philosopher Friedrich W.G. Schelling’s RomanticIdealistic celebration of the work of art in Philosophie
der Kunst (1802) and reached its modern peak in
cultural theorist Theodor W. Adorno’s Ästhetische
Theorie (1970). Second, aesthetics has been connected
to questions of sensual impact; this line of reflection
points back to the work Aesthetica (1750-58), written
by the philosopher Alexander Baumgarten with the aim
of investigating the “lower” sensual aspects of human
experience as opposed to the “higher” realm of logic.
Baumgarten is an important figure, as he invented the
modern usage of “aesthetics” as a term that designates
a specific area of human knowledge and awareness.
3.1 The Sensual-Phenomenological Dimension of
Aesthetics
In recent years there has been a tendency to loosen
the connection between art and aesthetic theory and
to revisit, from various perspectives, Baumgarten’s
original idea of applying aesthetics to sensual matter
(in Old Greek, aisthetá, “that which can be sensed”);
in terminology there has been a similar tendency toward
a shift from aesthetics to aisthesis [12]. Notably, this
has been developed in works by the philosophers Martin
Seel [13], [14], Gernot Böhme [15], [16], and, drawing
on John Dewey’s Pragmatist aesthetics [17], Richard
Shusterman [18]. These inquiries deal with aspects of
sensual appeal and the question of appearance, that
is, on the one hand, how people respond to certain
kinds and structures of appearance, and, on the other
hand, how these are constituted in order to evoke
response. I use the term “phenomenological”, as this
view on aesthetics deals with the experience of things
and, more importantly, with the ways in which objects
influence and condition experience. Key concepts here
are “pure presence”, a term that was introduced by
Seel, and which designates special attention to and
attraction from appearance, and “ambience”, introduced
by Böhme as a term for the influence from the
surroundings on perception. Even if these concepts are
only briefly related to design by the authors, they are
nevertheless powerful means of examining how design
objects specifically have a sensual impact, operate as
appearance, and contribute to the creation of ambience.
Seen in relation to the span of the material and the
immaterial in digital artifacts, this aspect of aesthetics
mainly concerns the sensual communication of the
indispensable material dimension of the artifact, how
it is staged as appearance and a presence to be noticed
or to slip more anonymously into the mainstream of
products. With regard to the smartphone, this aspect
of aesthetics deals with the sensual meaning of the
artifact: the look, the feel of the texture, the application
of materials, the execution of the detailing, for example
in the assembly and seamless fittings, as well as the
sensual appeal of the overall physical presence and the
immediate emotional stimulation of the user. Here,
Apple’s iPhone 4 can be viewed as a paradigmatic
example of a device that is characterized by a high
degree of detailing and material-sensual aesthetics
intended to be noticed and to evoke a kind of material
presence; consequently, and probably as part of a
planned media strategy aimed at media specialized in
mobile devices, Apple designer Jonathan Ive has been
associated with an obsession with the physical materials
and the sensual “feel” of devices.
Within the realm of the sensual meaning of the material
device, the immaterial aspect can be described as the
different kinds of effects that are triggered by the
material dimension of the design. These may involve
the various kinds of emotional pleasure that the design
evokes (cf. [19], [20], [21]) or an appeal to the operation
of the device, i.e. how it affords its use (cf. [22]) or
the way in which it enables different kinds of functions
(cf. [23]). Importantly, these effects are stated on the
level of immediate, sensually engaging interaction with
the digital artifact in its material extension. Addressing
the sensual-phenomenological dimension of aesthetics
in this context means acknowledging the material
element of digital artifacts, i.e. the fact that digital
technology is bound to a particular form of mediation
or a representation in a sensually appealing material,
and that these material manifestations constitute our
immediate access points to the artifact. Without these
access points, we would not be able to interact with the
artifact: They condition our basic sensory experience
with the artifact. Going beyond material representation
would result in pure (fictitious) speculations on cyber
space (still, however, mediated through the material
device of implants as in William Gibson’s 1984 science
fiction novel Neuromancer, or through new devices of
visual representation, see [24]), or virtual worlds (still,
however, only accessible through the material medium
of the computer screen).
3.2 The Conceptual-Hermeneutical
Dimension of Aesthetics
While the emphasis on sensual issues can be traced
back to Baumgarten, the conceptual-hermeneutical
dimension of aesthetics has mainly been developed
in relation to art. The key concern here has not been
to celebrate artistic creation but rather to analyze art
as a medium for the construction of meaning and new
patterns of understanding. This line of investigation
was initiated in Kant’s Kritik der Urtheilskraft (1790),
where aesthetics (albeit not in relation to art but to
our experience of nature) was conceived as a basic
aspect of epistemology: Kant’s idea was to employ
Design and semantics of form and movement
141
aesthetics in bridging sensual appearance and
conceptually formulated meaning. Following this line
of reflection, much of so-called philosophical aesthetics
has been guided mainly by philosophical concerns
about epistemology and the construction of meaning
and – sometimes – less by an interest in the specific
communicative abilities and conceptual strategies of
the medium in question.
Design research has given little consideration to this
tradition of aesthetics, perhaps because of its focus
on works on art and the often abstract philosophical
vocabulary (an exception is [25]). My reason for
integrating it into the present discussion is its focus on
the enabling of meaning: the construction, articulation,
conveyance and understanding of meaning. Hence, when
dealing with the enabling of meaning, hermeneutical
questions gain in importance with regard to the staging
of processes of understanding (on behalf of the aesthetic
medium) and with regard to their operation (in the
perceiver). A crucial aspect of this line of aesthetic
theory has been an immense interest in the extra- and
trans-communicative effects of the work of art, that is,
its ability to contain or communicate something ‘more’,
an ‘added value’, or an aspect of meaning other than
or beyond normal communicative abilities. This latter
point of impossible communication that is only made
possible in the closed space of the work of art has,
for example, been the driving factor behind Adorno’s
aesthetic theory [26] and the line of aesthetic inquiry
that he inspired (e.g. [27]; [28]; [13]). In this context, the
questions to be asked of design objects in general and
digital artifacts in particular are, on the one hand, how
and by what means the aesthetic mediums enables and
constructs meaning, and, on the other hand, what is the
nature of this meaning, i.e. whether it transcends any
limitations, and what its implications are.
Further, this line of aesthetic inquiry places itself at
the intersection of immaterial concerns and material
execution. On the one hand, issues concerning
understanding as well as the construction, organization,
and distribution of meaning relate to the immaterial
dimension and the inner expandability of the design.
On the other hand, the staging of understanding and
meaning requires a material manifestation; without
some sort of representation in a concrete medium,
issues of meaning and understanding would remain
142
Design and semantics of form and movement
an abstract theoretical statement. The discipline of
philosophical aesthetics acknowledged this aspect
with its interest in the work of art, i.e. in the role and
importance of materially based representation (although
it has often been more interested in the philosophical
points than in the actual works of art). When focusing
on design objects with their prevalent material ontology
and often direct sensual appeal, again, the material
dimension of the representation of meaning is an
unavoidable premise for reflection.
In relation to digital artifacts and the smartphone, I will
point to the interface as a point of reflection for this
dimension of aesthetics and its material and immaterial
elements. In comparison to a “classic” cell phone where
the user navigates the screen and the menus through a
panel of buttons, such as arrows that allow navigation
in different directions, the most striking characteristic
of the interface of the smartphone is the large touch
screen. The interaction is more direct than it is via the
buttons on the cell phone; it is, however, still, mediated
through the interaction of the fingers (or a pen) and
the screen – and, hence, indirect. Thus, the operation
of the screen is aimed at direct interaction with the
interface and a limited use of buttons; beside the volume
buttons on the side of many models, the number of
buttons varies from one (e.g. the iPhone’s on/off button)
to several (e.g. the buttons on many HTC models:
“home”, “menu”, “back”, and “search”). The interface
is the connecting touch point between the underlying
operating software and the user’s experience. Just as it
is with regard to the sensual dimension of aesthetics,
the interface establishes a two-sided relationship that
involves the design of the interface itself and the user’s
experience of it. The dialectics between these two
sides in the transmission of the interface is the source
of meaning in the design. Taking the interface as the
starting point, it is true that “software takes command”
(cf. above), and that the design of the interface begins
with the conception of the overall system, which
requires a high level of cognitive skills in the designer (cf.
[29]). Through the software, the interface establishes
the horizon of our perception of the object; still,
however, the software-generated opportunities that
the interface affords must be met and actualized by
an actual user interacting with it. Thus, the interface
is nothing without the user. In contrast to the sensual
dimension, which deals with the immediate sensual
meaning and stimulus (e.g. in effects of pleasure) of a
given device, the design interface relates, on behalf of
the user, to complex cognitive operations and matters
of understanding. The connecting and transmitting role
of the interface is already a stated fact in research in
interaction design and HCI and in New Media Studies;
in addition to proposing an overall framework for
conceptualizing immaterial meaning in digital artifacts,
my point in introducing a framework of aesthetic theory
in relation to digital artifacts is, more specifically, to
employ a conceptual-hermeneutical dimension of
aesthetic theory to focus on matters of understanding
and construction of meaning as these are articulated in
the actual medium.
In relation to the questions about understanding and the
enabling of meaning, the apps are of particular interest,
as, on the one hand, they operate as part of the
interface, while on the other hand they designate the
element of inner expandability that I consider crucial to
the immaterial element of the design of smartphones.
As a vehicle for drawing external information into the
smartphone, the app creates an open structure within
the barrier of the material device; thus, the smartphone
interface is not only a carrier of the often well-designed
pictograms of the apps, placed in grids on a vertically
or horizontally moveable menu – more importantly, it
should also be seen as a crucial transmitter between
the device and the functions of the apps, the material
presence and immaterial structures of information. Of
course, the functions of the apps vary; some of them
simply point to pre-programmed functions in the device
(such as a calculator, programs for opening documents
in various formats, and the clock), others have been
downloaded as small programs from the app market and
operate enclosed within the device without an internet
connection (e.g. small games, music devices such as a
piano, or small art installations such as the interactive
and generative music and picture app Bloom by Brian
Eno and Peter Chilver for iPhone, 2008), and yet others
rely on a wireless internet connection (and/or GPS) in
order to work (e.g. maps, stock market information,
shopping guides, social platforms, guides for good places
to go kayaking, weather forecasts, etc.).
With their open structure, apps can be seen as an
important vehicle for a new type of operation in and
through a material device. They offer easy access to
the use of the device of the smartphone as a portable
mini-computer with constant wireless internet access.
By allowing various types of information to flow through
the device, the apps not only enable a new notion of
design ontology that gives material objects a limitless
inner extension; often they also challenge paradigms
of understanding and meaning formation. Through the
apps, the smartphone lets meaning be constructed in
an interactive dialogue with the user; that is, through
the open structure of the apps, meaning is staged as
unfixed and as something that evolves in a continuous
construction that is related to the user’s specific needs
and preferences. Of course, most apps are rationally
oriented with regard to their function (how to locate
or become aware of desirable features in the local area)
or mode of operation (they are designed to work well
and should not lead the user astray). But because they
are, by nature, dynamic and open-structured, the apps
also create a paradigm for the user’s interaction and
engagement with them that requires flexible responses
and an acceptance of new and sometimes unexpected
meaning content, as for example when a search for a
certain kind of shop leads to new discoveries, or when
a slightly modified search profile points the user in new
directions. Of course, search results rely on the search
profile and on how the search elements have been
tagged in relation to the search engine in question; the
key point in this context is that searching for something,
for example within a geographic area, is a dynamic
process that relies on multiple factors such as tagging,
the search engine and the user’s expectations.
In a hermeneutical reflection, understanding is always
a dynamic process; parts always relate to an ongoing
construction of a whole [30], although the conditions
for understanding may vary and change and even lead
to non-understanding, the flip-side of understanding (cf.
[31]). In relation to the smartphone, it is a basic premise
that the user who carries the phone changes position,
that meaning changes, and that, accordingly, the
whole can constantly alter its basic constituents: The
geographical position that limits the restaurant search
changes, the search criteria can be modified, the point
of interest can be displaced, etc. Hence, the smartphone
is not a device aimed at understanding existing wholes
of meaning but rather of constructing perspectives on
and entries to engaging with meaning. Further, the apps
enable a plurality of possible meanings that challenges
the extension and limitation of meaning, even if does
Design and semantics of form and movement
143
not take place as an Adornian reversal or otherness of
meaning. However, Adorno’s reflection on the role of
the specific medium as the only locus for communicating
an otherwise impossible and incommunicable meaning is
relevant, insofar as the processes of meaning discussed
here can only take place within the medium of the digital
artifact of the smartphone (or the tablet computer). It
is an essential characteristic of the smartphone and its
mode of operation that the integration of various types
of meaning is catalyzed through the device interface
and apps; they cannot take on any shape outside the
medium of the smartphone. Thus, as a medium, the
smartphone co-initiates a paradigm of understanding
where the processes of understanding unfold in an
interaction with the device itself and its mediumspecific interaction platform (interface, apps). Thus,
the processes of understanding is both (i) structured
around an exchange of immaterial communication and
information and (ii) bound to the specific device and
its specific internal expandability of meaning. Thus, the
smartphone condenses and keeps the meaning within
its limitless boundaries – hence the fascination with this
type of device, people’s increasing dependence on and
addiction to it, and the prevalent attribution of basic
human functions such as memory and orientation to it.
3.3 The Cultural-Contextual Dimension
of Aesthetics
While the two previous dimensions of aesthetics deal
with aspects of meaning integrated in the given design
object, the cultural-contextual dimension focuses on
the wider implications of the circulation of objects on
a cultural, social, and political level. According to the
cultural theorist Jacques Rancière, who has influenced
this recent trend in aesthetic theory, aesthetics can
be investigated as a political power issue in relation to
the distribution of sensual material and the ability to
determine “what presents itself to sensory experience”:
Thus, aesthetics can be seen as “a delimitation of spaces
and times, of the visible and the invisible, of speech
and noise, that simultaneously determines the place
and the stakes of politics as a form of experience”
[32]. Rancière seeks to discuss possibilities that are
determined by aesthetic media; these not only behave as
transmitters of new possible meaning but also produce
possibilities by defining and conditioning domains of
the sensual experience. By pointing to new modes
of experience, with acts of “distinguishing a sensual
144
Design and semantics of form and movement
mode of being specific to artistic products” and of
disseminating experiential knowledge, aesthetic media
can promote a “recomposition of the landscape of the
visible, a recomposition of the relationship between
doing, making, being, seeing and saying”. This means
that aesthetic media have the capacity to radically reconfigure and transfigure the territories of “the visible,
the conceivable, and the possible”: They can propose
possible models for accessing the world in new ways.
In relation to the digital artifact of smartphones, this
approach to aesthetics as a delimitation of visible and
invisible and of what can be experienced and sensed is
expressed in a variety of ways.
First, smartphones are designed with sensually
appealing smooth surfaces and thus contribute to
the aestheticization of the modern world. Today,
aestheticization can be seen as an ubiquitous process
of distributing sensual meaning (cf. [33]); not only
are we surrounded by things that strive for sensual
appeal, but as suggested above, even the immaterially
operating knowledge society depends on materially
present objects. But immateriality affects the state of
the material. Thus, it can be stated at that we live in a
time of light materials, as “things lose their weight and
materiality through modern production technology. [...]
[I]t is as if all matter has changed into a sovereign play
of forms” [34]. In this perspective, smartphones can be
seen as light and flashy objects that contribute to the
overall ambience of e.g. an urban cityscape; they play
a part in defining what it looks like when people speak
and interact with each other through wireless devices.
Secondly, smartphones may have a social and cultural
effect with regard to how people use and consume
artifacts and attribute symbolic meaning to them and
the ways in which they construct and reflect their
personal identities in their own and others people’s
use and consumption of the artifacts. On this level,
the artifacts form part of an exchange of value and
symbolic meaning between people. The question of how
people develop an attachment to objects is especially
related to studies in consumption, to sociology, and
to anthropologically informed material culture studies,
but it should be acknowledged as part of the overall
complex of various aesthetic effects of design objects.
In particular, the role of symbolic meaning in design
is important for the discussion of the immaterial in
design, as it can be conceptualized as an aspect of
meaning that transcends purely functional, denotative
elements of design and thus opens its space of meaning
(cf. [35]). The potentiality of symbolic meaning operates
as an immaterial asset of design objects and enables
the attribution of personal and cultural meaning to
material objects. Smartphones can enter the cultural
circuit of symbolic meaning not only through their
outer ‘face value’ where certain models, for example,
bestow a certain status on the user. Remarkably, one
of the problems for people wanting to obtain social
differentiation through the use of smartphones is that
even advanced models have become a mass product
that are available to large market segments; thus,
superficial and expensive styling in the outer casing
(gold, diamonds) is the only remaining option. But even
more importantly, smartphones can actualize symbolic
meaning through the vast inner expandability that
connects to many different contexts of meaning and lets
people communicate, share values, and engage in acts
of participation in new ways: Through the interaction
with apps related to e.g. games, music, or sub-cultural
content, the user’s preference of choice and, in a wider
context, value systems and preferences are actualized
and, in a larger context that includes other users, put
into social interaction.
And thirdly, smartphones are also subject to the
distribution of sensual material and to the reconfiguration of, in Rancière’s words, “the visible, the
conceivable, and the possible”. On this level, smartphones can be conceptualized as media for determining
and conditioning perception and experience; they can
contribute to the way we assign meaning to the world.
Through their material sensual appeal and their interaction with modes of understanding, they can serve
as media for staging not only immediate and direct
sensual effects but also cognitive impact that affects
how experience is framed, that is, the boundaries and
possibilities of experience. Partaking in the framing of
the conditions of experience, digital artifacts such as
smartphones have a potentially massive impact: They
function as devices for organizing memory, keeping
track of meetings and social events, and, in a radical
sense, serve as an extension of human cognition and
an interface for social interaction (through texting;
apps for social platforms such as Facebook or Twitter).
As digital artifacts, smartphones operate on the
complex immaterial level of organizing experience.
4 Possibility
The immaterial effects of design objects employing
digital technology activate the sensual, conceptual, and
cultural possibilities that are evoked by the objects.
Thus, the element of immateriality in the objects
makes them open-structured and widens the span and
number of possibilities. The philosopher Peter Sloterdijk
connects the expansion of possibilities through design
with the paradoxical ability of design of enabling “the
capacity of incapacity” [36]. On the level of use, design
objects, according to Sloterdijk, have a ritual quality
in creating possibility by simulating a sovereignty that
emerges when we are able to conceive of otherwise
inconceivable objects. For example, the hermetic
“black box” of a seemingly incomprehensible digital
product can become “useful” and develop an “unlocked
exterior” through devices of interface design. I will,
however, propose to connect this “making capable”
of the otherwise not capable not only to the material
extension of the design, for example in physical artifacts
that create new functions and uses, but even more to its
immaterial dimension of operating invisibly on the level
of information and its ability to configure new modes
of experience. The immaterial dimension of design
evokes possibility in the sense that its open structure
lets the immaterial be potentially actualized in many
ways and directions. Further, by introducing the role of
possibility in relation to the aesthetics of immateriality
in digital artifacts, the ontology of immaterially operating
objects can be stated more precisely: They are objects
of sensual, conceptual, and cultural stimulation of
possibility.
In an exchange of material constraint and possible
meaning, the design of smartphones should serve a
dual purpose: In an outward, centrifugal movement it
should act through its open interface structure and
the apps, or rather its structure of opening up new
meanings; and in an inward, centripetal movement, it
should contract and contain the centrifugal aspect of
the device within its design. Thus, we can assess the
design from the perspective of the specific design with
an emphasis on the ambition of the design to reach out
for the space of possibility that it potentially engages.
It may, of course, be poorly designed, and in that case
it does not actually serve to frame possibility in a way
Design and semantics of form and movement
145
that makes the possible a potentiality for the actual.
And from the perspective of possibility, we can focus
on the ability of the design to condense and transmit
possibility: How the smartphone, in its concrete being,
conveys the open-ended structure of the possible
while simultaneously defining a direction and a scope
of operation for the possible which in turn becomes
concrete, potential meaning. Thus, the possible partakes
in a process of detaching meaning from the static
closure of material objects: The object, the smartphone,
is made transparent to its open-ended constitution.
And at the same time, this ‘abstract’ aspect of its
operation is drawn into the actual operating system,
the interface design, and the interaction and navigation
structures of the design.
The more the possible is made tangible and present in
the design, the easier is the user’s access to the world
of possibilities and, further, the more intuitive in use
is the design. The manufacturers of smartphones are,
of course, aware of this aspect of the design, and the
competitive and differentiating factor rests not only
on the advanced technology (in a continuous flow of
constantly upgraded software and operating systems).
It also, and to an even greater degree, relies on the
aspect of intuitive interaction that makes technology
accessible (as in Sloterdijk’s concept of design as giving
way to the capacity of incapacity) and performs a task
on the level of symbolic communication: Through their
design of intuitive interaction – signaled through the
often discreet cabinets of the devices with only a few
bottoms outside the screen and marketed, for example,
as “HTC Sense” by HTC – the smartphones enter the
circuit of attribution and consumption of symbolic
meaning, as it is the easy adaptation that enables the
users to ascribe symbolic meaning to the product.
Plainly put, a smartphone not only has the potential
of looking good but also of becoming an integrated
part of the user’s basic way of structuring experience
and cognition. By creating a horizon of perception, it has
the potential of becoming an extension of the primary
sensory, tactile and mental interface through which the
user meets the world. Thus, on a phenomenological
level, smartphones have the potential capacity to define
new ways of experiencing as well as “living everyday
mobility and relating to information” [37].
Further, on a symbolic level, this experiential aspect
146
Design and semantics of form and movement
together with its integration in the open-structured
search for the new and unknown – as every new app
widens the scope of possibility – facilitates a type of
product design that is easily attributed with symbolic
meaning.
5 Conclusion
Concealing their technological secrets inside, digital
artifacts need to be investigated in order to reveal
how they create, stage, and articulate cultural meaning.
Hence, the dual ontological premise of the material
extension and the immaterial effects needs to be
acknowledged. In essence, design objects have always
incorporated this dual constitution (no objects without
effects of information, and no design solutions without
material extension), but with the digital turn in design,
the immaterial aspect has come directly into focus, as
the performance of objects far exceeds their outer,
often discrete and miniaturized appearance. In this
sense, digital artifacts, such as the smartphone, are
limitless in their inner expansion of what they are
capable of (there are, of course, limitations due to
processor power and memory capacity which, in turn,
are constantly increasing). The proposal of the paper
has been to relate the discussion of materiality and
immateriality in design to various aspects of aesthetics,
ranging from directly addressing sensual questions in
the sensual-phenomenological dimension of aesthetics,
over the construction of meaning in and through the
design in the conceptual-hermeneutical dimension of
aesthetics, to the cultural-contextual creation of social
meaning and structures of experience. The discussion
has revolved around the device of the smartphone,
but hopefully, the proposal can help to define and
circumscribe how immaterial meaning evolves on a
more general level, and how digitally operating artifacts
act in constructing meaning and creating paradigms that
frame our perception and understanding.
References
[1] Selle, G. (2007). Design im Alltag. Vom Thonetstuhl zum
Mikrochip. Frankfurt a.M.: Campus.
[2] Wershler, D. (2010). The locative, the ambient, and the hallucinatory in the internet of things. Design and Culture 2 (2), 199-216.
[3] Manovich, L. (2008). Software Takes Command. Retrieved
November 15, 2011, from http://lab.softwarestudies.
com/2008/11/softbook.html
[4] Atkinson, P. (2008). A bitter pill to swallow: The rise
and fall of the tablet computer. Design Issues 24 (4), 3-25.
of culture. In M. Rampley (Ed.). Exploring Visual Culture.
[5] Blodget, H. (2011). Android Is Destroying Everyone,
Definitions, Concepts, Contexts (pp. 1-17). Edinburgh:
Especially RIM – iPhone Dead In Water. Retrieved November
Edinburgh UP.
15, 2011, from http://articles.businessinsider.com/2011-04-
[25] Steinbrenner, J., & Nida-Rümelin, J. (Eds.) (2010).
02/tech/30089528_1_android-phones-google-s-android-
Ästhetische Werte und Design. Ostfildern: Hatje Cantz.
smartphone-market#ixzz1dmZBhGWr.
[26] Adorno, T.W. (1970). Ästhetische Theorie.
[6] Moles, A. (1995). Design and immateriality. What of it in
Frankfurt a.M.: Suhrkamp.
a post industrial society. In R. Buchanan, & W. Margolin (Eds.),
[27] Bubner, R. (1989). Ästhetische Erfahrung.
The Idea of Design (pp. 268-274). London: MIT Press.
Frankfurt a.M.: Suhrkamp.
[7] Moles, A. (1972). Théorie de l’information et perception
[28] Menke, C. (1991). Die Souveränität der Kunst.
esthétique. Paris: Denoël/Gonthier.
Frankfurt a.M.: Suhrkamp.
[8] Flusser, V. (1999). The Shape of Things. A Philosophy
[29] Romero-Tejedor, F. (2007). Der denkende Designer.
of Design. London: Reaktion Book.
Hildesheim: Georg Olms Verlag.
[9] Selle, G. (1997). Ideologie und Utopie des Design.
[30] Gadamer, H. G. (1960). Wahrheit und Methode.
Zur gesellschaftlichen Theorie der industriellen Formgebung.
Tübingen: Mohr.
Vienna: Pumhösl.
[31] Hörisch, J. (1998). Die Wut des Verstehens (2nd ed.).
[10] Burckhardt, L. (2010). Design ist unsichtbar. In K. T.
Frankfurt a.M.: Suhrkamp.
Edelmann, & G. Terstiege (Eds.), Gestaltung Denken. Basel:
[32] Rancière, J. (2004). The Politics of Aesthetics.
Birkhäuser, 211-17.
London & New York: Continuum.
[11] Folkmann, M.N. (2010). Evaluating aesthetics in design.
[33] Oldemeyer, E. (2008). Alltagsästhetisierung. Vom Wandel
A phenomenological approach. Design Issues, 26 (1), 40-53.
ästhetischer Erfahrung. Würzburg: Königshausen & Neumann.
[12] Engholm, I. (2010). The good enough revolution –
[34] Liessmann, K. P. (2010). Das Universum der Dinge.
The role of aesthetics in user experiences with artifacts.
Zur Ästhetik des Alltäglichen. Vienna: Zsolnay.
Digital Creativity 21 (3), 141-154.
[35] Folkmann, M.N. (2011). Encoding symbolism: Immateriality
[13] Seel, M. (2000). Ästhetik des Erscheinens. München: Hanser.
and possibility in design. Design and Culture 3 (1), 51-74.
[14] Seel, M. (2007). Die Macht der Erscheinung. Frankfurt a.M.:
[36] Sloterdijk, P. (2010). Der Welt über die Straße helfen:
Suhrkamp.
Designstudien im Anschluss an eine philosophische Überlegung.
[15] Böhme, G. (1995). Atmosphäre. Essays zur neuen Ästhetik.
München: Fink.
Frankfurt a.M.: Suhrkamp.
[37] Vial, S. (2010). Court traité du design. Paris: PUF.
[16] Böhme, G. (2001). Aisthetik. Vorlesungen über Ästhetik als
Allgemeine Wahrnehmungslehre. Munich: Wilhelm Fink Verlag.
[17] Dewey, J. (2005). Art as Experience. New York: Penguin.
[18] Shusterman, R. (2000). Pragmatist Aesthetics. Living
Beauty, Rethinking Art (2nd ed.). Lanham & Oxford: Rowman
& Littlefield Publishers.
[19] Jordan, P. (2000). Designing Pleasurable Products.
London: Taylor & Francis.
[20] Norman. D.A. (2004). Emotional Design. New York:
Basic Books.
[21] Desmet, P.M.A. (2010). Three Levels of Product Emotion.
In Proceedings to KEER 2010, International Conference
on Kansei Engeneering and Emotion Research, Paris,
Mads Folkmann
March 2-4, 2010.
Institute for Literature,
[22] Norman, D.A. (2002). The Design of Everyday Things.
Culture and Media
New York: Basic Books.
Studies, Design,
[23] Crilly, N. (2010). The roles that artefacts play: Technical,
University of
social and aesthetic functions. Design Studies 31 (4), 311-344.
Southern Denmark
[24] Rampley, M. (2007). Visual culture and the meanings
Design and semantics of form and movement
147
Cassie Hester
[email protected]
Elevating the everyday:
Designed objects as companions
Abstract
From fleeting interactions to long affairs, our
relationships with things are diverse and complex.
Like people, designed objects and environments are
close companions in our lives. A constant source of
comfort and delight, frustration and anxiety, their
personalities inform our everyday dialogues and experiences. Creating objects with congenial personalities
requires careful consideration of semiotics: syntax,
semantics, and pragmatics within specific con-texts.
Designed artifacts assume engaging and pleasurable
personalities when they transcend materials, challenge
perceptions, and garner active, reflective interactions.
By constructing friendly and intriguing personalities
through de-familiarization/recontextualization and
human-centered designs, everyday interactions and
experiences are elevated as users invest in pleasurable
forays and relationships.
Keywords
Experiential design, interactive design, semiotics,
human-centered
The Companionship of Designed Objects
The way we understand and engage with artifacts
is parallel to the way we interpret and interact
with each other. Both humans and designed objects
are interpreted through formal and contextual
information. The means of understanding this infor148
Design and semantics of form and movement
mation is semiotics; the triadic relationship of syntax,
semantics, and pragmatics in a given context. From
form, content, function, and context, we find meaning.
Perhaps, more significantly, we find personality – the
conversational nature of a person or object. There are
personalities who fade to the background and don’t
entice interaction – wallflowers, if you will. There are
the strangely familiar, those who possess a certain
familiarity and allure. And there are true companions
and confidants, personalities that are supportive and
compatible. The various relationships range from
fleeting and ephemeral to enduring and intimate. We
think with, invest in, and express ourselves through
design; therefore, unconventional and human-centered
artifacts are empowering, resonant, and key to creating
extraordinary experiences.
Dig Deep (Fig. 1) is installed on the exterior wall of
the Pollak Building, one of Virginia Commonwealth
University’s design facilities. This 16 foot by 4 foot
typographic installation is made of treated plywood,
white paint, galvanized nails, clear plastic straws, and
pink and white outdoor sequins. Each nail on the one
inch grid contains at least one sequin and a clear plastic
buffer between the sequin and the board. The nails
that comprise the message feature two sequins, a white
sequin facing the viewer and a pink sequin facing the
board. Reflected light creates the sprawling message.
The discs flutter and form a living surface on windy days.
Cassie Hester
Virginia Commonwealth
University
Fig. 1. Dig Deep.
Fig. 2. Alphabet Shift.
Alphabet Shift (Fig. 2) is an interface that facilitates the
creation of messages in a public space – the third floor
of the Pollak Building at VCU. The structure consists of
birch strips supported on a frame of birch and aluminum
braces. Each strip contains a full alphabet and one
blank space. The strips can be pulled to the left or the
right to highlight one letterform in the frame, creating
a message. Many inhabitants of the third floor create
messages alone or with the support of their friends.
From “MID-TERM REVIEWS SUCK” to “FORM LIKE
VOLTRON ON” and “ILLUMINATE MY COCK,” the
messages provide a brief ethnographic glimpse of the
third floor.
Fig. 3. Make More/Worry Less, twenty minutes after installation.
Make More/Worry Less (Fig. 3 & 4) is a piece I installed
on the third floor of Pollak. The installation seemed
like a blank sheet of paper. With the markers provided,
passersby doodled and wrote on the paper – starting
conversations and responding to the contributions
of others. As the paper filled, my message, “Make
More/Worry Less” was revealed as marks made in
the letterforms of my message did not appear. Like
Alphabet Shift, the sociable personality of Make More/
Worry Less promotes intrapersonal exploration while
also allowing individuals to invest in interpersonal
relationships by participating in collective conversations.
Fig. 4. Make More/Worry Less, four weeks after installation.
Please visit www.cassiehester.com for more information on
these projects as well as the pieces exhibited at DeSForM.
Design and semantics of form and movement
149
Ozge Merzali Celikoglu
[email protected]
Tradition and innovation: A study on
reconfiguration of product language
through innovative approaches
Abstract
Based on the subject – object relationship which
is reflected as user and product, this paper deals
with the concepts of tradition and innovation where
craftsmanship and redesign are included. Thereby, the
reconfiguration of product language is examined within
innovative design approaches to traditional cultural
products as sample cases: Iznik tiles, nargile and coffee
pot, where different layers of this reconfiguration can
be observed.
Keywords
Tradition, material culture, craftsmanship, innovation,
product language.
1 Introduction
Innovation in design can be made in various ways:
new technologies lead to new production techniques
and materials; new marketing strategies lead to new
organization models, and new product languages lead
to new socio-cultural formations. Thus, considering the
concept of innovation within its basic reference to the
‘new’, to define new ways of life is possible by defining
new products and environments of use. So, the whole
process actually generates new subject-object, or more
concrete, user-product relationships in terms of
product language.
Within new products that are pioneers of their
own type, it is comprehensible that the whole
150
Design and semantics of form and movement
communication should be built in a brand new way.
However, it is also possible to approach the most
traditional artefacts with new perspectives and reconfigure the existing product language, which can
sometimes be harder than to configure a language for
a brand new product. Developing technologies and their
implementations on traditional products that allow
mass-production rather than craftsmanship techniques;
discovery of traditional cultural values; global marketing
strategies and changing socio-cultural lifestyles are the
initial triggers of such an approach.
Based on the changing user-product relationship,
this paper deals with the concepts of tradition and
innovation where it questions whether an innovation
in design of traditional cultural artefacts is possible.
Thus, technological and socio-cultural innovation are
taken as reasons and results on changing user-product
relationships and examined in sample cases of three
Turkish traditional artefacts’ novel designs; the nargile
(waterpipe), Iznik tiles and Turkish coffee pot, where
the product language and consequently the userproduct relationship are re-configured.
2 On Culture, Tradition and Craftsmanship
The diversity in definitions of culture arises from the
fact that it can be approached from several perspectives
and disciplines; moreover, there is no individual who is
not involved with culture and can formulate a definition
with an external perspective to it. As Lotman states
We live in a world of culture. Moreover, we are in the thick of
it, inside of it, and that is the only way we are able to continue
our existence. The importance of the notion of culture, the
Thus, culture can be defined in that it refers to the
cumulative deposit of knowledge, experiences, beliefs,
values, attitudes, meanings, hierarchies, religion, notions
of time, roles, spatial relations, concepts of the universe,
material objects and possessions acquired by a group of
people in the course of generations through individual
and group striving or that it is the systems of knowledge
shared by a relatively large group of people, or that
it is a way of life of a group of people the behaviors,
beliefs, values, and symbols that they accept, generally
without thinking about them, and that are passed along
by communication and imitation from one generation to
the next, or that it is symbolic communication etc.
Considering that culture consists of explicit and
implicit patterns that shape social behavior acquired
and transmitted by symbols, it is possible to speak of a
certain coding with it. The term cultural code includes
both meanings of culture and code. Cultural codes are
defined as symbols and systems of meaning that are
relevant to members of a particular culture [2]. They
provide that individuals get a similar perception and
understanding of certain facts and phenomena so that
they go through a similar process of interpretation
and come to similar conclusions [3]. Thus, common
meanings and symbols serve as constituents of a
society’s paradigm since without a common perception
frame, each sent message can be interpreted totally
differently and a common behavior cannot exist. These
characteristics help the individuals of the society
to build a language through which visions, voices,
colours, words, gestures and facial expressions can be
interpreted in a common way [4]. Furthermore, Eco
(1968) asserts that communication is based on codes,
agreed upon in society. Understanding the codes and
their meanings is necessary, as they are determinant in
communication. He also claims that the general use of
products is not only functional but also cultural, and the
use also determines the meaning [5]. Since iteration and
continuity are important for a social structure according
to Durkheim [6] that can only be built with common
values and behavior of a certain society, rituals stand
out as basic carriers of both concepts [7]. Rituals are
repeated acts reinforcing the sense of identity for the
individual in society, building up the links with a shared
past [8], and their support to social construction can
thus be explained as the formality and repetitiousness
that give much of the power to ritual and generate
a sense of belonging and continuity between the
individual and the group, as well as between present
and past. Additionally, all differing approaches of
culture emphasize the “affinity about transmission
of sociological, physical and mental cultures to next
generations which is called as tradition” [9].
Continuing with the definition of ‘tradition’, it is the
transmission of customs or beliefs from generation
to generation, as well as, an artistic or literary method
or style established by an artist, writer, or movement,
and subsequently followed by others [10]. Tradition
contains every kind of belief, images of individuals and
events, techniques and institutions [11]. Thus, cultural
traditions are a continuation of the past, as well as a
projection into the future, actually being elements of
the continuity of a society’s history [12]. At this point,
craft products can be taken as one of the transmitting
elements of culture and tradition. Craftsmanship and
so, craft products, are material beings in the intersection of culture, tradition and society where historical
production techniques and rituals generate the culture
of craftsmanship. It is important to see craft products
with their cultural depth because they are not only
products of traditional hand-labor but also carriers of
the tradition itself. A craft product offers the user to
get informed about its cultural background visualizing
the traditional usage of the object. Due to this kind of
communication, the relation between the product and
the user becomes a cultural interaction happening on a
personal experience for each individual in the society.
Also, this situation allows the user or the subject
to share the traces of the unique object’s or craft
product’s moments of creation [13].
Various interpretations of culture and tradition show
that these two concepts are strongly related to each
other; they have the power to define and transform
each other. Craftsmanship, in addition, is a ‘product’
of them both; a more concrete outcome of their
reflections on the society. Thus, the concept of craft
products includes histories, cultures and traditions of
societies; like myths which transform a meaning into
a form [14]. Craft products, like all other products,
Design and semantics of form and movement
151
have their own language, however their languages
are configured over a longer period of time and in a
cumulative way. Thus the relationship between the
user and the product is defined by a cultural and
historical involvement with the related tradition, i.e.
an internalization of the tradition by every individual
since the craftsman’s cumulative work is actually the
cumulative work of every individual by means of society.
product design, several approaches occur focusing
on new production technologies, materials, product
languages, usage, markets etc. For instance, Verganti
(2008) mentions:
3 Towards the Relation between Innovation
and Product Language
be more or less radical. In particular, innovation of meanings
is incremental when a product adopts a design language and
delivers a message that is in line with the current evolution
Innovation stands for “a new method, idea, product,
etc.” (Oxford, 2011). Rather than its most common
connotation as “technological innovation”, this study
is grounded on its basic definition which emphasizes
novelty. Since Steffen (2007) also claims:
Science and technology are not the only sources of innovative
product design. Alterations of basic societal conditions, sociocultural upheaval, and the arrival of new viewpoints and
values are just as important for the creation of innovative
designs or new product languages. Artefacts that are not
innovative with regard to technology might be epochal with
Keeping in mind that all products move from a technical
system into a cultural system, having both (objective)
denotations and (semantic) connotations, there is
a constant interference between a practical and a
technical system [13]. Design helps to explore the
nature of the modern world since it is a reflection
of the economic system showing the effect of the
technology that is used. Thus, it is a kind of language,
which reflects emotional and cultural values that stand
beyond the function of products. Sudjic [16] emphasizes
the importance of design language for the user-product
communication:
It is the language of design that serves to suggest an object’s
gender, often through the most unsubtle of means, through
color, shape, size and visual reference. It is design that
cynical salesmanship. And it is design that can serve to signal
Design can react to different cultural conditions, in
which designed products can accept the rules of existing
conditions or go against them where it can offer new
culture practices [17]. Combining innovation with
152
Design and semantics of form and movement
Innovation may concern a product’s functional utility, its
meaning, or both. In addition, functional innovation may
imply an incremental or radical improvement of technical
A certain innovation in technology can lead to an
innovation in a particular product, and this furthermore,
can lead to a socio-cultural innovation in the society.
The important thing here is to notice that the innovative
approach should also be conducted within product
language since the communication of the product with
its user is actualized by means of its language. Since
language is a code with human history built in, it carries
traces of cultures, perceptions and societies throughout
time [1]. For products that went through technological
and socio-cultural innovation, relationships between
user and product should be re-defined and thus product
language should be re-configured. The innovative
approach towards traditional cultural products does
not only lead to a change in these products’ forms,
usage or production techniques, but it also changes the
perception of the user so that the physical and social
environment of use is affected. Thus, such innovative
approaches can serve as proposals to keep the
traditional cultural products alive in today’s society, to
define new lifestyles with old traditions etc. Whether
these proposals will be accepted or rejected depends on
the communication between those products and their
users, beyond the intentions of their designers.
4 Configuration of Product Language in the
Intersection of Tradition and Innovation
The aforementioned theories and approaches related to
culture, tradition and innovation in design and product
language are profited to examine three cases, such
as Iznik tiles, nargile and Turkish coffee pot. With
innovative design approaches to these products, the
forms, environments and socio-cultural contexts of
use are redefined. Considering the characteristics
of these artefacts, nargile is for outdoor public use;
Iznik tiles belong to traditional palace and mosque
ornament tradition; coffee pots are for indoor private
use. However, with innovative design approaches to
these traditional artefacts, nargile is considered to be
a product for indoor private use; Iznik tiles become a
product for private use, and a Turkish coffee pot can
be used in offices by the public.
4.1 On Iznik Tiles
The history of Iznik tiles goes back to the second half of
the 16th century which is named as the classical age of
Turkish art during Ottoman rule. Following the recent
excavations of İznik kilns, it is stated that the Ottoman
ceramics in İznik had a Seljuk background [19]. Tiles
became such a predominant feature of architectural
decoration that they took on an important function in
the overall aesthetics of Middle Eastern cities. Mosques,
mausoleums, madrasahs, libraries, public baths,
fountains and palaces were all decorated with panels
of tiles that covered walls, domes, shafts of columns,
tympana and prayer niches (mihrab), in a profuse
decoration that expressed the great wealth and power
of Ottoman Empire. From the 15th to the 18th century,
the glazed tile did in fact represent the main decorative
element in Ottoman architecture. In this period, Iznik
ceramic production was at its height, being almost
exclusively devoted to meeting the orders placed for
thousands of glazed tiles for civil and religious buildings
in Istanbul and other cities of the Empire, planned by
Sinan, the great architect of the court of Suleiman the
Magnificent (ruled 1520-1566) [20].
In the geometrical design of the Seljuk inheritance, the
palmettes and leaves were used. The plant motifs of
the classical age were drawn on the white undercoats.
A superficial abstraction is dominant in the naturalistic
plant designs. The main examples of nature motifs
were carnations, tulips, plum blossoms and branches
in full blossom, pomegranates, peonies, broken leaves,
rosettes, roses, bunch of grapes, acanthus leaves, vases
and birds with black, thin countermines. There is always
an effort to keep motion and dynamism in balance and
symmetry both in the designs and the colours. Each
motif is a whole in itself whereas it is also an inseparable
part of the eternal whole (Ministry of Culture, 2010).
Considering the characteristics of colours, at first, blue
and white were the prevailing colors in the pots and
wall tiles. During the 16th century, the turquoise was
introduced. The embossed red of the wall tiles of the
mihrab of Süleymaniye Mosque (1555) marks the peak of
Ottoman tiles and ceramics. Seven colors were used in
various combinations (though there are many effective
İznik designs using only two, three or four colors):
blue, purple, red, green, turquoise, gray and black.
The recession in Iznik and the decadence of the workshops started in the beginning of the 17th century. The
colors lost their vividness, the coral and tomato blues
darkened. Quality deficits and cracks on the glazes
began. The net lines of the contours were dispersed.
The political regression was felt most at the Iznik
tile workshops among all the handcrafts. Since the
production technique details were kept secret, and the
technical development knowledge was not mentioned
in written documents, an important gap of information
was formed for the following generations. The attempts
for revival required thoroughly new efforts and these
efforts could not be a substitution for the traditional
training passing from one generation to the next.
After 1995, TÜBİTAK (The Scientific and Technological
Research Council of Turkey) decided to produce İznik
tiles again. However, it had been 400 years and it was a
hard task. As a result of research and development, tiles
started to be produced again. Today, İznik tiles are used
as an architectural element in old and modern buildings
by the discriminating decorator and art-lover alike.
The production of handmade tiles of the desired quality
in this era of speed and automation is a particularly
difficult task.
Traditional Iznik tiles are reinterpreted by designer
Defne Koz for Vitra so that they will be used in
bathrooms as wall tiles. The process of this redesign is
basically realized by abstraction of existing traditional
figures and patterns. The designer reinterpreted
characteristic İznik patterns and cypress motifs,
adapting them to the contemporary bathroom. The
stylized patterns, colors and texture of the hi-tech tiles
aim to exhibit the same sensitivity and refinement as
the handcrafted originals and the traditional İznik quartz
glazing process. Contemporary techniques of tile, glass
and glass ceramics manufacturing are used in order to
mass produce these tiles. The aim was not to lose the
sensitivity dedicated to handwork and bring a feeling of
Design and semantics of form and movement
153
depth and softness. The designer, basically, formated
three lines of patterns: Line 1 consists of shades of
azure, which heralds a return of the traditional İznik
blue. It aims to display the taste reminiscent of timeless
handcrafts (Fig. 1).
with flowers that are positioned at certain angles and
within a symmetry. These characteristics belong to
a craft history, tradition and taste in Turkish culture.
Fig. 1. A sample wall tile from Line 1 of Vitra.
Fig. 4. Iznik tiles in Selimiye Mosque.
Line 2 includes diagonal stripes, hexagons and trefoils.
New patterns are built into the quintessential geometric
patterns of İznik tiles. This line aims to achieve a
contemporary look that is as powerful as the İznik
originals (Fig. 2).
In Figure 1, there is an innovative product that is
manufactured with new technologies in order to be
used in bathrooms as wall tiles. The product kept the
main elements of the traditional Iznik tiles, however, it
configured a new language, as patterns, that is adapted
to innovation rather than tradition. Still it is possible to
say that the cultural core of the product is somehow kept.
The designer does indeed have a particular approach to
design, “I see design as part of everyday life. I don’t like
it when design is presented as an elitist style, or even
worse as the vulgarisation of an elitist style”.
Fig. 2. A sample wall tile from Line 2 of Vitra.
And Line 3 focuses on the stylized cypress tree,
lends its elegant form to the contemporary design.
A traditional İznik tile motif, the cypress is painted in
various green tones (Fig. 3).
Fig. 3. A sample wall tile from Line 3 of Vitra.
As seen in Figure 4, the original hand-painted tile
pattern consists of blue, red, green and white as
dominant colors. The pattern is basically constituted
154
Design and semantics of form and movement
Design is about what people do everyday. I hate to think that
something I designed may remain on a shelf as a decoration.
The ideal object is one that you adopt and use every day
and one that possibly continues to surprise and excite you
every time you use it.” Additionally, she claims that she is
fascinated by the history and tradition of Turkey’s material
To start a formal analysis of novel design for Iznik tiles,
the designer abstracted traditional patterns of Iznik tiles.
Figures of nature do not exist anymore, however, main
colours are kept. Production technique of the tiles has
changed since it became a product of mass-production
rather than of craftsmanship. Thus the shades of
colors and painting do not require any craftsmanship
competence. Additionally, materials of which dyes and
ceramics are produced have also changed. From the
socio-cultural perspective, it is necessary to remember
that Iznik tiles belonged to palaces and mosques during
the Ottoman period. Since painting of human figures
was forbidden according to Islamic view, craftsmen
focused on natural figures. Therefore the patterns
of traditional Iznik tiles have such a religious-cultural
background. The novel design for Iznik tiles is loyal
to this understanding, and keeps its figures close to
nature and geometrical shapes although it proposes
an everyday use in bathrooms rather than being an
element of a prayer environment. Considering the fact
that traditional Iznik tiles were placed on inner walls of
public buildings, such as mosques, they became a public
possession that is shared by the society, mentally. By
changing the production techniques and actualizing a
mass-production, the designer allows the individuals of
the society to keep a ‘material sample’ of this ‘common
possession’. So, a product that does not actually belong
to individual use at home, changes its socio-cultural
position, from being of everybody’s mentally and nobody’s
materially, it becomes a commodity serving individuals
with its materiality and thus, with its new language. The
novel wall tiles have been manufactured and attracted
attention. They are sold in wide numbers and bathroom
walls of many people are decorated with them. The
product has also had a Red Dot design award in 2006.
Although there are essential parts that a nargile should
contain, in time however, nargiles became real objects
of art. The bottles come in different colors and graceful
shapes, the flexible hoses are often embellished with
embroidery or woven handicraft art (holding the
owner’s initials), and the plastic mouthpiece may be
of fine porcelain or the more popular precious amber.
The bottles were usually manufactured in Beykoz.
In this bottle, the bowl filled with water is placed to
cleanse the filth made by burning tobacco. It also served
as a filter to absorb the nicotine and to cool the smoke.
The body was made of glass, porcelaine, silver or crystal
and very elegantly embellished with floral motifs. Bowls
were generally made in Tophane by bowl (lule) makers.
This is the portion where the tobacco is placed and lit.
A plug of the tobacco is placed in the bowl and covered
with a conical cap pierced on top. A piece of oak
charcoal was placed on top of the tömbeki or tobacco
to generate the smoke [22].
Fig. 5. Traditional nargile and its redesigns.
As mentioned above each element of the nargile is
given a particular form with particular materials,
ornaments and even colors. The choice of these
elements is made according to traditional usage and
craftsmanship methods. In its varying forms and colors,
nargile however, carries a continuity of a culture and
a tradition. Even only considering its formal aspects
gives some clues about the lifestyle to which it belongs:
During smoking the nargile, it is possible to see how the
water boils and bubbles in the bowl. The material, the
glass, of which the bowl is made, serves for this show.
The length of the tube decides how close the user should
be to the nargile and the height of the nargile itself defines
a lower sitting position for the user. Even these three
aspects refer to the Eastern lifestyle and joy culture.
The minimum time interval to smoke nargile is given as
one hour. It is actually the “tömbeki” (a special tobacco)
that is smoked. Nargile, thereby serves as a signifier of
the joy during smoking this tobacco since the joy and
the ritual do not only consist of smoking. It is a time
for peace and relaxation; a time that should be spent
slowly and quietly where only the voice of boiling water
accompanies a tranquil conversation. To watch the
movements of the boiling water, additionally, increases
degrees of joy and tranquility [22].
Nargile, belonging to Eastern culture and tradition,
carries the traces of Eastern lifestyle, as a craft product.
Such an object that realizes itself as a craftsman’s
labour is totally integrated with cultural references and
4.2 On Nargile
Nargile entered the Turkish culture in the 16th century
during Ottoman period. It became popular in the 1700s,
and even became a status symbol. Nargile, basically
consists of four main parts: the body (gövde), the bowl
(lüle), the flexible hose (marpuç) and the metal tube
(ser). There are craftsmen who are specialized in the
production of each of these parts, and even there are
neighborhoods that are called with these parts’ names
[22]. All of these parts have their own function in this
specialized way of smoking. Thus, in all three types of
nargile designs (Fig. 5) it is possible to find these main parts.
Design and semantics of form and movement
155
symbols which overcome its functional use. To have
a deeper understanding of it a deeper involvement
with the related culture is required [14]. However, an
innovative approach to this traditional product can change
many things related to it from production methods to its
relationship with the user. Thus, it brings out the question
whether an innovation in tradition is possible.
signals lead to a partial elimination of the traditional
perception and use. So, the product becomes an
industrial product which fulfills its functional duty but
does not really belong to a certain culture. Moreover,
this nargile signifies to the innovation of manufacturing
technologies, lifestyles, maybe new rituals, in short,
to a non-traditional, to an ‘innovative’ way.
While designing nargile, Kunter Șekercioğlu started
with the idea of ‘local product for global market’. He
remembered Sotsass’s proposal to designers ‘not to
loose one’s cultural DNA’ [23]. The first nargile he
designed, is manufactured by him with traditional
craftsmanship techniques. He used a glass-blowing
technique for producing the bowl of the nargile.
The aim of the designer is that this nargile should be
appropriate for mass-production but it should also keep
its connection to craftsmanship since he believes that
craftsmanship products should keep their ‘traditional
essence’. According to this concept, the bowl would
be produced by craftsmen, and the rest would be
mass-produced. The second nargile design has another
concept so that it can totally be mass-produced. So,
the glass bowl in the first design is replaced by some
plastic material (policarbonat) which eliminates the risk
of breaking. Although all problems related to massproduction for this nargile design are solved, it is not
produced. However, there are several reasons for it.
Analyzing formal and socio-cultural characteristics
of both products, the traditional nargile and its novel
design, the first thing to notice is that the traditional
parts of the nargile and their relation to each other
has been redefined which lead to a certain change in
its form. However the dominant element in generation
of the form actually had to follow again its special way
of functioning. Thus the novel approach resulted in a
product language which is more simplified, in terms
of the designer’s aim, more ‘modern’. Since it will be
mass produced its whole manufacturing process needed
to be redefined where methods of craftsmanship are
abandoned. Consequently, materials that are used by
nargile production needed to be redefined, too, such as
the replacement of glass that is used in the production
of the bowl with polycarbonate. So, the designer
with the aim of ‘local product for global market’ reconfigured the product language for nargile. At this
point, a discussion on the socio-cultural meanings
of nargile should be included. First of all, nargile is
traditionally for public use. It is an object of joy and
is smoked in coffeehouses accompanied by company,
conversation and table games. The culture of nargile is
connected to the culture of the eastern coffeehouse
[24]. The novel design for nargile proposes that it will
be used at home. Thus, it changes the environment
of use and also the traditional culture of use. So, it
does not only change the traditional manufacturing
technique of nargile, which is craftsmanship, but it also
changes the tradition of use where a socio-cultural
heritage is integrated. It is the adaptation of technology
to a traditional cultural artefact and the adaptation
of a local product to a global perception. However,
it should be considered whether the novel nargile
design has become a ‘local product for global market’
by losing its cultural references or whether this also
caused the loss of locality. Moreover, the distinction
from the traditional form may cause an alienation of
the local society since the novel designed nargile may
not reflect the common cultural code of this society
According to the designer, this nargile was designed
with contemporary aesthetic perspective to be world
widely. In the first example (Fig. 5) the novel design
of the nargile, it is possible to notice a certain kind of
simplification in the form and ornaments which refer
to the mass production of the nargile rather than being
a craft product. However, it also signifies some elements
of traditional nargile. Especially the form and color of
the bowl as well its relationship with the ground allows
a closer perception to the traditional version.
On the second nargile design (Fig. 5), a totally new
product language is configured. Traditional forms, colors
and ornaments are avoided so that the four main parts
of the nargile have lost their characteristics. Thus, they
are hard to differentiate and became a main body like an
outer shell. All traces of craftsmanship are eliminated
and replaced by geometrical forms so that it can be
produced in mass numbers. The elimination of crafts
156
Design and semantics of form and movement
anymore. Thus, whether it became a local product for
global market or a product which is neither local nor
global is debatable. It is possible to say that some formal
re-codings are not enough to reconfigure the product
language in this case. Coming to the situation in practice
today, the novel nargile design is not mass-produced,
although all problems according to mass-production
have been solved, because it could not find a ‘definition’
in the society’s perception. Furthermore, it is stated that:
An old Turkish company, Arçelik, producing and selling
domestic appliances came out with Telve (2005),
a Turkish coffee machine. This was the first proposal
for making Turkish coffee with a machine (Fig. 6) and
was positioned for the market serving to working
environments such as offices. It changed the whole way
of making Turkish coffee and took it from the manual
way to a machinic way for office environment.
The product has won the IF Design Award in 2005.
new versions or designs of it are being used in these
coffeehouses. The archetypal object, with its complicated
silhouette and basic working principle, is accepted as it is.
Furthermore, the more the object reminds us of an historical
entity, with references to the oriental or Ottoman contours
and lines the closer it gets to the idea of preservation of
an essence about it. This objective resistance to change
also becomes the symbol of resistance to modernity itself.
It becomes an aesthetic as well as a cultural response. It
positioning towards life, with its own distinctive style, whose
Fig. 6. Telve – The Turkish coffee machine.
As the designer also claims, companies that produce
household electrical appliances have been interested in
it at first, however, they could not position it into one
of their categories. In addition, there are of course legal
restrictions related to smoking, and whether it would be
an encouragement for smoking is another ethical issue.
4.3 On the Coffee Pot
There is a certain ritual for making Turkish coffee which
requires time and talent. The most traditional way is to
heat coffee in a copper coffee pot slowly over charcoal
embers for 15 to 20 minutes, and taking it frequently
away from the fire to prevent overheating. Generally,
a medium-size coffee pot is used to make two cups of
coffee at each shot. The most important thing while
making Turkish coffee is to decide when to take the
pot away from the fire since it can suddenly overflow
after boiling. In addition, the foam of the coffee is a
signifier of how well it is made and taste which should
be kept while serving the coffee in cups. The ritual
of making coffee is usually followed by the ritual of
fortune-telling. Coffee cup reading is a widespread
and popular fortune telling method, which speaks
of both the past and the future.
A formal analysis of the novel design for Turkish coffee
pot shows that it proposes a whole new product
with a novel language rather than being a redesign of
traditional Turkish coffee pot. Besides, it does not
only propose a new form but also a new way of making
Turkish coffee which replaces the traditional process
which basically depends on the ‘talent’. The new design
is called ‘Turkish coffee machine’ rather than Turkish
coffee pot, which can easily be read on the product.
Other essential differences are that the coffee machine
works with electricity rather than being in direct
contact with the fire; can make more than 2-3 cups of
coffee; does not allow to mix the coffee manually, and
actually does not leave any field for manual reach or
interference during cooking.
This new machine proposes a new way of coffee making
which is faster, easier and more ‘serial’, however, it
demolishes the ritual of coffee making since it works
exactly in an opposite way of traditional Turkish coffee
making ritual. The traditional way depends on the talent
of the coffee maker, and needs a certain time so that
the coffee will taste good: It should be braised slowly
and the person who is making the coffee should wait
Design and semantics of form and movement
157
watching on it because there is a risk that the coffee
overflows when it boils. Still, it should be emphasized
that this whole ritual actually belongs to a ‘home
environment’ where everything can be done without
a rush. So, the new Turkish coffee making product
with its closed machine form and metallic color actually
‘says’ that it does not belong to this kind of home
environment; it is rather supposed to be used out of
home, mostly referring to working environments, i.e.
offices. Basically it provides an answer for the need
to consume Turkish coffee in offices. Thus, instead
of trying to re-form the ritual, it completely removes
the ritual and focuses on functionality. Telve, with its
novel product language which does not refer to any
traditional background of Turkish coffee making is the
first product that makes Turkish coffee in the form of
a coffee machine, and it has been widely used and still
being used in offices as it aimed to. The public acclaim of
Telve is based on the new defined way of coffee making,
on the new defined environment of coffee making and
the new language of the product which provides the user
a perception of both novelties.
take a public heritage as a starting point and allow
to individualize and materially possess it; the novel
design of nargile aims to ‘dress’ the traditional artefact
with modernity and still hopes to keep the ritual as
well as the traditional essence, and finally, the Turkish
coffee machine design changes the whole experience
of coffee making where it decides for function rather
than tradition. The public acclaim of these products can
depend on their functionality, honesty, integrity into
daily lifestyle, etc., which actually all depend on
product language.
To conclude, it is open to debate whether an innovation
in tradition is possible or required, since conceptually,
one refers to novelty while the other one refers
to antiquity. However, it is a fact that every kind of
innovation in a product leads to a new product language
which generates new relationships between the user
and the product, consequently building new sociocultural lifestyles.
References
[1] Andrews, E. (2003). Conversations with Lotman: Cultural
Semiotics in Language, Literature, and Cognition. Canada:
5 Conclusion
University of Toronto Press.
Interpreting the concept of innovation in a sociocultural context allows one to notice new relationships
between subject and object, within this case, user
and product. Changes in society lead to novelties in
products and, likewise, novelties in products lead to
changes in society. The product language as the basic
carrier of communication between user and product
needs to be re-configured with every novelty. Thus,
it not only defines new relationships between user,
product, environment etc., but it also serves as a
signifier of novelty where it combines or differentiates
the ‘traditional’.
In all three sample cases, examined in this paper,
innovation takes place where new ways of production,
use and socio-cultural meanings are described in
product design. Thus, different layers of the change
in product language are discussed. The innovation in
production processes of Iznik tiles, nargile and Turkish
coffee pot lead to new product languages which try
to combine a traditional essence and a functional daily
use, since it becomes hard to find the required time
to complete a whole ritual. Thus, they speak of new
social lives and the ‘new’ position of tradition in these
lives. While designs related to traditional Iznik tiles
[2] Hyatt, J., & Simons, H. (1999). Cultural codes – Who holds
158
Design and semantics of form and movement
the key? The concept and conduct of evaluation in Central and
Eastern Europe. Evaluation, 5(23). Retrieved July 12, 2011, from
http://evi.sagepub.com/content/5/1/23
[3] Turner, G. (1974). British Cultural Studies: An Introduction.
London & New York: Taylor&Francis Group.
[4] Giddens, A. (1979). Central Problems in Social Theory:
Action, Structure, and Contradiction in Social Analysis. Berkeley:
University of California Press.
[5] Eco, U. (1989). The Open Work (A. Concogni Trans.).
Cambridge: Harvard University Press.
[6] Alexander, J.C. (1992). Durkheimian Sociology: Cultural
Studies. Melksham, Wiltshire: Redwood Press Limited.
[7] Cohen, A.P. (1985). The Symbolic Construction of
Community. London & New York: Taylor&Francis Group.
[8] Timur Ogut, S. (2009). Material culture of tea in Turkey:
Transformations of design through tradition, modernity and
identity. The Design Journal, 12(3), 339-364.
[9] Posner, R. (2001). What is culture? Semiological interpretation of main concepts. In E. Onat, & S.Ö. Yıldırım (Eds.),
Anthropology. Göstergebilim Tartışmaları. İstanbul: Multilingual.
[10] Oxford Dictionary, 2011.
[11] Shils, E. (2003). Tradition. Doğu Batı, 25, 101-135. İstanbul:
Cantekin Press.
[12] Batca, M., & Fulga, L. (2002). Globalization and Cultural
Tradition. VI. Milletler Arası Türk Halk Kültürü Kongresi,
Küreselleşme ve Geleneksel Kültür, Seksiyon Bildirileri. Ankara:
T.C. Kültür Bakanlığı.
[13] Baudrillard, J. (2005). The System of Objects (J. Benedict,
Trans.). London & New York: Verso Books. (Original work
published 1968).
[14] Barthes, R. (2000). Mythologies (A. Lavers, Trans.). London:
Vintage Books (Original work published 1957).
[15] Steffen, D. (2007). Design semantics of innovation: Product
language as a reflection on technical innovation and sociocultural change. Design Semiotics in Use Conference 2007,
Helsinki.
[16] Sudjic, D. (2009). The Language of Things. London: Penguin
Books.
[17] Krippendorff, K. (2006). The Semantic Turn. New York:
Taylor&Francis Group.
[18] Verganti, R. (2008). Design, meanings, and radical
innovation: A metamodel and a research agenda. The Journal
of Product Innovation Management, (25), 436-456.
[19] Aslanapa, O., & Yetkin, S. (1989). İznik çini fırınları kazısı II.
Dönem 1981-1988. [Excavation for Iznik tiles II. period] Ankara:
Tarihi Araştırmalar Ve Dokümantasyon Merkezleri Kurma ve
Geliştirme Vakfı.
[20] Pereira, J.C. (2009). Iznik Pottery and Tiles. Lisbon:
Fundaçao Calouste Gulbenkian
[21] Sheppard, S. (2009). Turkish delight. Designer, July.
[22] Gürsoy, D. (2007). Nargile: Bir nefes keyif. [Nargile:
A Breath of Joy] İstanbul: Oğlak Yayıncılık.
[23] Demircan, D. (2005). Interview with Kunter Sekercioglu,
Unpublished.
[24] Hatton, R.S. (1988). Coffee and Coffeehouses: The Origins
of a Social Beverage in the Medieval Near East. Washington:
University of Washington Press.
[25] Timur, S. (2006). The eastern way of timekeeping: The
object and ritual of nargile. Design Issues, 22(2), 19-26.
[26] http://www.thedesignermagazine.com/interviews-profiles/
turkish-delight/>, accessed 20 August 2011.
[27] http://www.defnekoz.com/, accessed 27 August 2011.
[28] http://www.kilittasi.com/>, accessed 3 September 2011.
[29] http://www.arcelik.com.tr/>, accessed 3 September 2011.
[30] http://www.vitra.com.tr/, accessed 3 September 2011.
Ozge Merzali
Celikoglu
Technical University of
Istanbul, Department
of Industrial Product
Design, Turkey
Design and semantics of form and movement
159
Bram van der Vlist, Gerrit Niezen, Stefan Rapp, Jun Hu, Loe Feijs
{b.j.j.v.d.vlist, g.niezen, j.hu, l.m.g.feijs}@tue.nl, [email protected]
Controlling smart home environments
with semantic connections: A Tangible
and an AR approach
Abstract
1 Introduction
In the transition from a device-oriented paradigm
towards a more task-oriented paradigm with
increased interoperability, people are struggling with
inappropriate user interfaces, competing standards,
technical incompatibilities and other difficulties. The
current handles for users to explore, make and break
connections between devices seem to disappear in
overly complex menu structures displayed on small
screens. This paper tackles the problem of establishing
connections between devices in a smart home
environment, by introducing an interaction model
that we call semantic connections. Two prototypes
are demonstrated that introduce both a tangible and
an augmented reality approach towards exploring,
making and breaking connections. In the augmented
reality approach, connections between real-world
objects are visualised by displaying visible lines and
icons from a mobile device containing a pico-projector.
In the tangible approach, objects are tagged and can
be scanned to explore connection possibilities and
manipulate the connections.
When Weiser wrote his vision of ubiquitous computing
about 20 years ago [1], he postulated that we will be
surrounded by networked displays of various sizes,
and that we will use them to explore and access our
information and computerized infrastructure. They
would simply be there, around us, like a piece of scrap
paper or a blackboard, their use woven into the fabric
of everyday life. It would be easy to switch between
actively using them and barely noticing their mere
existence. People would concentrate on their everyday
activities, unaware that they are using possibly more
than a hundred computers within their vicinity to
carry out these activities.
In today’s reality, although there are rooms accumulating
almost comparable amounts of computers in the form
of smart phones, web tablets, TV screens, netbooks,
personal computers and so on, we have not yet
achieved seamless operation among them. Each and
every one of these devices demands our attention,
uses a different user interface and allows access to
none of the other components (or only to very few
other components within the room). While many of
the devices are, or can be networked, the process
of making the actual connections and exchanging the
information between them is painful without extensive
networking knowledge. Configuration details and
connectivity settings are hidden, deeply nested within
menu structures. Even with the connections in place,
Keywords
Product semantics, interaction design, smart home.
160
Design and semantics of form and movement
exchanging the actual information is cumbersome, and
users have to dig into the file structures to find the files
to be exchanged. In contrast, from a user’s perspective,
the devices should be easy to connect since they are
physically close to each other (and can thus be touched
or pointed at). The information to be shared might have
been on the screens moments ago and could form part
of the interaction, depending on the user’s intention.
Consider a seemingly simple task, like listening to
your music stored on your PC or home stereo system
from your mobile phone’s headphones in the kitchen.
It is practically impossible for many users, despite the
principal technical ability of the involved devices and
available network technologies. Part of the problem
may be attributed to the fact that user interfaces are
still highly focused on device-oriented operation.
Competing standards and technical incompatibilities
exist at the service-level, contributing to the problem
and making it impossible for non-experts to take full
advantage of today’s technology.
Some of the irritations that users face today are a
consequence of the mechanisms of the market, that
imply different goals for the stakeholders. Developers
of devices need to have a strong device-oriented
view, whereas users’ goals are often more easily
resolved within a system-oriented view. Developers
are concerned about the functionality and usability
of the device at hand, possibly harmonizing its usage
over the range of products provided by this specific
manufacturer. Users, on the other hand, find themselves
with a set of devices and services from different
manufacturers, or even different industries. As an
example: users still have to set the integrated clocks
of many devices, even if they are all connected to each
other. Although a scanner and a printer make up a
nice copier, only selected models offer this combined
functionality. If you would want to directly print the
image that the video camera is currently sampling,
you need a PC and install specific software to do so.
Seemingly easy tasks (for an unbiased observer) are not
possible, because at development time, nobody thought
about it, and only minimal cross-device capabilities have
been implemented.
1
http://www.sofia-
project.eu/
One possible solution to solving the interoperability
problem at the infrastructure-level is a software
platform developed within the SOFIA1 (Smart
Objects For Intelligent Applications) project. SOFIA
is a European research project within the ARTEMIS
framework that attempts to make information in the
physical world available for smart services – connecting
the physical world with the information world.
We aim to enable users to explore and make
configurations on a high semantic level without
bothering them with low-level details. We believe
this can be achieved by making use of Semantic Web
technologies and ontologies in an interoperability
platform as proposed by the SOFIA project. Such a
platform may be used to support semantic interaction
in a smart home environment, as is described in [2].
Building on the SOFIA software platform, we propose
a user interaction model and two interface solutions.
One user interface solution we propose uses a projected
augmented reality approach, based on a concept called
Spotlight Navigation [3], [4]. Here, a mobile device
containing a pico-projector visualizes connection
possibilities between devices in the environment.
By using direct pointing gestures with the device in
the user’s hand, users can intuitively explore and
manipulate the virtual network connections as if they
are part of the user’s real world environment. The
second user interface solution is a tangible interaction
approach, enabling users to physically select devices in
their environment and directly view and manipulate the
connections in a simple, universal way. In this paper we
illustrate both interaction approaches to manipulate
semantic connections in a smart home setting, where
the tangible UI solution and Spotlight Navigation can be
used interchangeably.
2 Semantic connections
We defined the term semantic connections [5] to refer
to meaningful connections and relationships between
entities in an ubiquitous computing environment. These
connections are invisible by default, but can be viewed
and manipulated on demand, using a special-purpose
device or application. We envision these connections
as both real “physical” connections (e.g. wired or
wireless connections that exist in the real world) and
“mental” conceptual connections that seem to be
there from a user’s perspective. Their context (what
things they connect) is pivotal for their meaning. The
term “semantics” refers to the meaningfulness of the
connections. We consider the type of connection,
which often has the emphasis now (e.g. WiFi, Bluetooth
Design and semantics of form and movement
161
or USB) not to be the most relevant, but that what
the connection can do for someone – its functionality
– even more. Semantic connections exist in both the
physical and the digital world and can exist between
objects, people and places.
Semantic connections have properties like directionality,
transitivity and modality (i.e. what things they carry).
The rationale behind semantic connections is to rely on:
- the meaning of existing objects to provide meaning for
the relationships between the objects and the resulting
meaning of the networked objects.
- the power of natural mapping and locality, using real
objects and locations to provide meaning for the
connections that are created between the objects and
(object) locations.
- inherent, augmented and functional feedback and
feed-forward (using terminology as proposed in [6])
to strengthen the meaning of the connections and the
emerging functionality.
Crucial to our approach is to make the gap between
user goal and action smaller. If we consider streaming
music from one device to another, “streaming” now
consists of multiple actions that do not necessarily make
sense. In our view, this single high-level goal, should
have one (or at least as few as possible) single high-level
action(s). That single action should carry the meaning
of its goal. By using the physical world as interaction
space and using the real location of the objects, we are
reducing the need to identify the devices from a list with
names or rely on other forms of representation.
3 The Connector: A Tangible Approach
As a portal to the semantic connections, we introduce
our tangible user interface approach which we called
the connector. The Connector can be used to explore
and manipulate semantic connections between different
devices in the home environment. It is a handheld
device that identifies devices, by scanning RFID tags
that are located on the devices themselves. By holding
the Connector on top of the tag, users can explore the
connection possibilities that are visualized with lights
on top of the Connector. After holding the device in
the RFID field for a moment, the device-ID is locked
and the other device to be connected can be selected
in a similar fashion. With a push-to-click action a
connection between two devices can be established.
For removing an existing connection, the ring on the
lower part of the device should be pulled until it clicks.
162
Design and semantics of form and movement
Fig. 1. The Connector prototype and a smart phone
used as a media player.
3.1 Design
The cylindrical shape of the connector (figure 1) is
loosely inspired on that of a loupe or hand lens. By
moving the connector over a tag, the connection
possibilities can be “read” from the top of the cylinder.
The display consists of two rings (made up of LEDs),
each divided into four segments. The connector supports
several actions. You can move it over an object or tag
to see whether it is active. A device or object can be
selected by holding the connector close to or on a tag
until the selection sequence is completed. The connector
can be compressed by pushing the top and the lower part
together, and it can be pulled, by pulling the lower part
and the top part away from one another until it clicks.
When the tag is in the range of the Connector’s
RFID field, it reads the tag and the first (yellow) light
segment on top of the Connector will light up, serving
as feedback that the Connector recognises the device.
After holding the Connector over a device tag for a
moment, a sequence starts, lighting up the second, third
and fourth segment of the inner ring. This can be seen
as feedforward to hold the Connector over the tag until
it has been selected and all four segments are lit. After
the device is recognized and selected, another device
may be selected in a similar fashion. Now, the second
ring of lights will start lighting up in sequence and one
should wait until both rings are fully lit. Removing
the Connector from the tag prematurely cancels the
selection process.
When a connection between the selected devices
is possible, both rings start flashing green. When
no connection is possible, they will turn red. When
a connection between the devices you scanned
already exists, the rings will turn green. To make the
connection, the Connector is compressed by pushing
the top and lower part together, or by pushing the
Connector down on the device it is touching, until
it clicks. To remove an existing connection between
two scanned devices, the ring on the lower part of the
Connector should be pulled until it clicks. The rings
will show a red light to indicate that the connection
has been broken. The segments will turn off once the
Connector is moved away from the device. Performing
the opposite action of what is required to make or
break a connection, cancels the procedure.
3.2 Prototype
The Connector prototype is made out of four separate
pieces that are 3D printed. The lower part and the
top part of the Connector can be moved inward and
outward serving as a two-way spring-loaded switch.
The prototype packages all the necessary components
into one integrated device, which is wirelessly connected
to a computer using a Bluetooth connection.
The Connector contains the following main components:
- Arduino Stamp 02
- Innovations ID-12 125kHz RFID reader
- SparkFun BluetoothMateGold
- 8 bi-colour LEDs
- Switches
- 3.3v LiPo battery (850 mAh)
4 Spotlight Navigation: An Augmented
Reality Approach
Spotlight Navigation can be used to explore and
manipulate connections between smart devices.
With Spotlight Navigation, connection information
contained in the smart space is projected into the
real world, augmenting the real environment with
virtual information, making it intuitively perceivable
for users. Spotlight Navigation projects icons close to
the actual devices in physical space. It allows for the
creation of new connections simply by drawing lines
between these icons, using a “pick-and-drop” action
with a push-button on the prototype (press and hold
the button when pointing at one device, move over the
second device and release the button). Additionally the
connection possibilities are projected between devices
that allow for a connection, by changing the colour of
the projected line (while the connection is being drawn)
from yellow to green when the line’s end is moved over
the frame of the targeted device. When a connection
is impossible, the connecting line will turn red and
disappear as soon as the button is released.
4.1 Design
Spotlight Navigation was invented as an intuitive way
of accessing large data spaces through handheld digital
projection devices. Rather than directly projecting the
equivalent of a small LCD display, Spotlight Navigation
continuously projects a small portion of a much larger
virtual pane or data space. It is the device’s orientation
that defines which part of the larger pane is selected
for display. This is done in such a way that the virtual
data appears to have a fixed location in the real world.
By moving the projector’s light spot over the wall,
users make portions of the data space visible through
intuitive, direct pointing gestures. This intuitiveness
stems from the fact that the projected content always
stays roughly at the same physical place, regardless
of the orientation of the device. It becomes visible
depending on whether it is in the projector’s light cone
or not. In other words, users have the impression that
they are illuminating a part of a conceptually unbounded
virtual data space, just as if they would be looking at
hieroglyphs on a huge wall in a tomb with a flashlight.
As people are familiar with operating flashlights, the
operation needs no or little training. When accessing
a data space with the device, users can zoom in and
out of the data space by using a scroll wheel control,
resulting in a pan-and-zoom user interface. To visualise
the semantic connections in physical space, we rely on
the symbolic meaning of colour, where green colour
means “proceed” and red means the opposite. Using
green, yellow and red lines we aim at referring to
the “existence” of a connection, the “possibility” of
a connection or to indicate that a connection is not
possible. Figure 2 shows the projection when connecting
two devices together.
With Spotlight Navigation, devices are identified by
their physical location, relying strongly on natural
mapping. Connections are created simply by drawing
lines between the devices. An erasing gesture with
the Spotlight Navigation device pointed at an existing
connection, breaks the connection.
Design and semantics of form and movement
163
Fig. 2. Projection when connecting two devices
together.
4.2 Prototype
On a technical level, the operation is achieved through
continuously measuring the orientation, and optionally
also the position, of the device. Our prototype is using
an inertial navigation module, also called an inertial
measurement unit (IMU), that directly measures the
orientation by means of accelerometers, gyroscopes
and an electronic compass.
Fig. 3. Spotlight Navigation prototype.
The Spotlight Navigation prototype is a fully embedded
setup integrated into a 3D printed casing. The design of
the casing was targeted at getting the smallest possible
setup that could run on the integrated batteries.
A dummy ring was added to the prototype to
strengthen the semantics of a mobile projector. Figure 3
shows the prototype. Our current setup consists of the
following components:
- OMAP3530 board (IGEP module)
- Pico projector (Microvision SHOWWX)
- Orientation sensor (Sparkfun 9DOF Razor IMU)
- scroll wheel (with button press functionality)
- two additional buttons
- two 3.7v li-ion batteries (Nokia BL5J)
164
Design and semantics of form and movement
The OMAP3530 processor contains a 3D-graphics core
(PowerVR) that is capable of rendering the connection
visualizations and device icons in real-time. Our current
prototype still requires the object positions to be
manually configured in space, as it did not contain a
camera. By using a camera, as is planned for future
versions, our intention is to recognize the identity and
physical location of each device, so that it is no longer
necessary to align the projected object icon with the
location of its associated device.
5 Pilot Evaluation
Both our prototypes were evaluated in a pilot user
study. This pilot was composed of demonstrators made
by the different partners in the SOFIA project and
was conducted with users in a setting that resembles
a real home. In order to get enough insights to
improve the system, seven groups consisting of three
people each were asked to interact with the system,
during which their experiences were recorded. The
two interaction prototypes presented in this article
were part of a larger test setup, which was evaluated
during a full week of experiments. In this paper we
focus on the results, which are relevant for evaluating
the interaction concepts. During the pilot, users
experienced a smart space with various automated and
interactive appliances and devices, which we refer to
as smart objects. The appliances in the smart space are
interoperable, sensitive to changes in their environment
and exchange information with one another. There
exist several explicit and implicit relations between the
smart objects, of which some can be explicitly viewed
or manipulated with the Spotlight Navigation device
(available in the study room of the pilot setup) or the
Connector device (available in the living room of the
pilot setup).
5.1 Participants
Twenty-one participants were recruited in seven
groups of three friends. Selection was based on age
(between 20 and 35), availability during the week of the
pilot and their mutual friendships. Of the recruited 21
participants that successfully completed the trials, 13
were male and eight were female. Their age ranged from
23 to 34, with an average age of 28.5. Nine participants
were living alone and 11 were cohabiting. The median
score of self-report familiarity with interactive systems
was 6 on a 1-through-7 scale.
5.2 Materials
Figure 4 shows a brief overview of the different parts
of the system. The experiment took place in two rooms,
the study and the living room of the Experience Lab on
the High Tech Campus in Eindhoven. The facilities and
infrastructure of the Experience Lab were used to set
up the demonstrator system and to collect observation
data (video and audio recordings).
Functional Lighting (FL)
Presence
Sensor (PS)
Presence
Sensor (PS)
Presence; Control;
Non-Functional Lighting (NFL)
Lighting
Device (LD)
Lighting; Control;
Lighting;
Preset; Permanent
Lighting; Control;
Media; Content; Music;
Lighting; Temporary
Music
Player (MP)
Spotlight
Navigation (SN)
Media; Content; Music;
Lighting; Temporary
Stereo System (SS)
(UPSTAIRS) STUDY ROOM
Lighting
Device (LD)
Connector (CN)
(DOWNSTAIRS) LIVING ROOM
Fig. 4. The devices and their connections as used in
the system.
5.3 Measurements
During the pilot, several measurement instruments
were employed. Participants were asked to rate the
pilot setup on three different scales; the HED/UT scale
[7], the Perceived Control scale [8] and a questionnaire
developed by the SOFIA project for internal use.
The mental models that users developed during their
interaction with the system were recorded using the
Teach-Back protocol [9], and the participants’ attitudes
towards the system were recorded with a semistructured interview. Because the HED/UT scale and
the Perceived Control scale were targeted at the entire
system, we do not discuss their results in this paper.
Mental models were extracted using the teach-back
protocol. Because users’ mental models consist of both
semantic and procedural knowledge about the system
they were interacting with, teach-back questions can
be subdivided into “what is?” questions focusing on
semantic knowledge, and “how to?” questions focusing
on procedural knowledge [9]. Using such questions,
adjusted to our specific situation and research goal, we
aimed to extract the semantic and procedural concepts
that were relevant for our users. Participants were
asked to explain to the researcher what they thought
the system was and was for, including listing all the
components and the relationships and connections
between the components they thought made up the
system. By asking for the perceived connections and
relationships between the components, we aim to gain
a better understanding of how users conceptualize the
cause-and-effect relationships between their actions
and the responses of the various devices in the smart
home. This includes the information that is exchanged
between these devices. By asking the participants to
explain to the researcher how to perform a specific task
with the system, we aim to get insights into how well
the participants understood the necessary steps and
devices involved to achieve their goal. To support and
communicate their answers to both types of questions
to the researchers (and for recording purposes),
participants were asked to make drawings, schematics
or use a textual representation.
Interview. In order to gain a deeper insight into the
things that occurred during the experiment sessions and
record the users’ general opinions, a semi-structured
interview was conducted. Using a list of open questions
as a structure, participants were evoked to share
their experiences with the test setup and think along
for possible improvements. During the interview, the
researchers also asked questions based on specific
behaviour or actions of the participants that they
observed during the trial.
5.4 Procedure
Participants had already received written information
about the experiment together with an official invitation
by email. After the participants were welcomed in the
Experience Lab and were briefed, they received and
signed an informed consent form and were asked to
fill out a pre-experiment survey. This survey included
demographic questions and a self-report scale of
familiarity with interactive systems like (tablet) PCs
and smart phones.
The groups of 3 participants were split up into two
groups of which two participants were led to the
livingroom area to perform the role of Mark and Dries,
Design and semantics of form and movement
165
and one participant was taken to the study to perform
the role of Sofia (these names will be used later to
identify the different treatment groups). All participants
were introduced to the devices, which they had to
interact with before the experiment started.
During the experiment, the participants were asked to
perform a series of predefined tasks that revealed the
functionality of the system. Every participant received
these tasks on paper and was asked to think aloud,
or for the participants in the living room (Mark and
Dries), to share and discuss their thoughts during the
whole experiment. After they performed the tasks,
they were asked to freely explore the system to deepen
their understanding and check their assumptions of its
operation. They could continue this free exploration
until they thought they understood the system’s
operation and would notify the researcher that they
had finished. The researchers (one in the living room
and one in the study) sat down in the back of the room
during the entire session and were available in case
anything went wrong.
After interacting with the demo, the participants
were asked questions to elicit their mental models
and were interviewed. The Mark and Dries characters
were interviewed together, and they could openly
discuss their opinions and mental models. Some
of the participants agreed on their answers and
agreed on one drawn representation of their shared
mental model. Others disagreed, and created their
own representation. The duration of each trial was
approximately 50-60 minutes, including briefing,
instructions, filling out the questionnaires and the
closing interview.
6 Results
6.1 Mental models
Of the 21 participants who participated in the pilot, we
collected 18 mental models. The teach-back protocol
with the Sofia characters (n=7) resulted in seven unique
mental models, while for the Mark (n=7) and Dries
(n=7) characters we obtained 11 mental models, of
which three were shared. We will first give an overview
of the overall results of the mental models, followed
by a more detailed description of the mental models
recorded from Sofia characters and the Mark and
Dries characters (which we treated as one group).
166
Design and semantics of form and movement
Completeness. Out of all the mental models, 15 did not
note that presence detection was used; seven out of
seven for the Sofia characters and eight out of 11 for the
Mark and Dries characters. Of the three that included
presence detection in their drawings, one was a shared
model and the other two were from the same session.
A few other components of the system that were in
the study and the living room were occasionally not
included in the mental models. This includes the nonfunctional lighting (NFL) in the living room, the relation
between the NFL and the Living Colour (LC) light (the
NFL would dim down when the LC was active), the
functional lighting (FL) in the study upstairs, and the
dimming of the FL when the Spotlight Navigation was in
use. The NFL was missing in two mental models, as was
the connection between the NFL and the LC. These
two mental models were from the same session. For the
mental models of the Sofia characters, one out of seven
missed the FL and two were missing the connection
between the FL and the Spotlight Navigation.
Semantic Connections Concepts. During the user
experiments some of the participants noticed and
discussed interesting networking concepts like
transitivity and directionality. These concepts were
also considered in the semantic connections interaction model, but were not implemented in the pilot.
Despite the absence of these concepts, participants did
intentionally (or sometimes perhaps unintentionally)
draw them in their mental models or discuss them.
Among the concepts of our interest are directionality,
transitivity, priority and the temporary or persistent
nature of the connections.
Transitivity was noted in three of the mental models
and directionality in nine of them. Two mental models
indicated a notion of priority in their mental models,
concerning one out of multiple conflicting connections
to have priority over the others. Two persons discussed
the persistence of connections, wondering when connections
would stop existing (for instance when the person would
take a mobile device out of the house) or indicating,
what they described as a permanent connection, distinctively from the other non-permanent connections.
Organisational Layout. We identified three types of
organisational layouts in the way people draw their
mental models. The majority used a physical/spatial
way of describing their mental model, of which we
identified eight as being fully spatial (the main structure
of the network is based on the physical location of
the components) and another eight mental models
have what seems to be an arbitrary mapping, using the
physical appearance of the components to identify them
in the drawing. Some of these representations include
spatial information but it is not used as their main
structure. We label these hybrid layouts. There are two
mental models that show a logical way of representing
the network and its components using blocks and labels
to identify the components. Similar ways of organising
mental models were found in [10].
Network Structure. For the mental models of the Mark
and Dries characters, we observed three main trends
in the structure of the networks they drew. We
distinguished between network structures that define
a central entity (which is close to the actual network
architecture), network structures that have a mainly
peer-to-peer structure, and a mixed infrastructure
which both have peer-to-peer connections and
connections going through a central entity (the
Connector object). All of these mental models of the
network are compatible with the actual situation in the
pilot. We observed five mental models with a central
entity, four with mainly peer-to-peer connections and
two with a mixed structure.
For the Sofia characters we mainly observed two
different network structures: A daisy-chained one
(every component connects to one or two others
in a serial manner) found in five mental models, and
a parallel structure (where connections had a more
parallel nature), which occurred two times. What is
interesting to note was that the Spotlight Navigation
device was often seen as an entity that was not
connected to the network, while the Connector
object was in all cases considered part of the network
(and in some cases even as being the central entity).
despite the fact that there were many more devices
and appliances available (especially in the livingroom
e.g. TV, stereo set, other light sources and luminaries).
Most participants were enthusiastic about the “simple
way” of making connections. However, they did indicate
that they wanted to be more in control of what would
actually happen when the connection was made. Some
participants indicated that this lack of control was
not crucial, because they figured that the connections
could be undone in the same fashion when they did not
like the effects of the connection. With regard to the
overall functionality participants also indicated that they
would like to see more “practical applications” that
would make their daily life easier. These remarks were
mostly in the direction of concepts known from the
home automation domain.
Several remarks were made concerning the user interaction with the Spotlight Navigation and Connector
device. For the spotlight navigation, remarks were often
made about the icons that were projected. The icon for
the Living Colour lamp was not always clear to users.
Remarks were also made about the (mis)alignment of
the icons and the physical devices, and many participants
indicated that the icons could be omitted or be replaced
by boxes around the physical objects. Additionally,
remarks were made about the inaccuracy of the
pointing gesture and difficulties in operating the button
on the device. For the connector device, the low speed
of the selection procedure was often mentioned. The
effort required to physically select a device was often
mentioned as a downside, while others mentioned it
as a positive point as it was considered playful. Remarks
were also made about the limited pairwise selection
-participants indicated that they would want to have
the possibility to select and connect more devices
at the same time.
7 Discussion
6.2 Interviews
From the interviews we observed a few trends. Some
of which were to be expected, while others were
more surprising. Many participants were disappointed
in the limited functionality of the current setup.
Although the participants were enthusiastic about
the ease of which the connections between devices
could be made, they were disappointed that they could
only control the connections between two devices,
Spotlight Navigation and the Connector are two
alternative user interface approaches to configuring
ubiquitous computing infrastructure. Although we
cannot directly compare the mental models elicited
during the user experiment, which would have asked
for a more controlled setting (e.g. having the same
setup and having an equal number of participants
for both treatments), we did make some interesting
observations.
Design and semantics of form and movement
167
The most striking difference between the way users
described the setup was the perception of the users
that the Connector was part (if not the central part)
of the system, while the Spotlight Navigation was often
considered outside of the system. We hypothesise
that this is due to the different roles that the Spotlight
Navigation and the Connector have in the interaction
with the connections. The Connector is used to
conceptually “carry” the content between the two
devices and in itself represents the relation between
these two artefacts. The Spotlight Navigation is, in
contrast, perceived as a “remote control” that visualises
the connections in physical space. This might lead the
users to conclude that the projected lines are the
connections, directly between the devices, and leave the
Spotlight Navigation itself outside of this network.
The results show that devices and appliances that
automatically act and react to people’s behaviour are
often not considered in the mental models, compared
to the devices and relations that users interact with
explicitly. However, the results also show that some
participants noticed these relations, and incorporated
them in their mental models. More interestingly, some
of the participants expected that they could manipulate
these relationships (e.g. between sensor and light)
in the same way as they could manipulate the other
connections. This result is promising and might indicate
that people project their experience with one part of
the system to the rest of it. This also became apparent
when participants started looking for tags on other
devices they thought could also be connected.
An interesting observation is the rather direct impact
of the interaction device’s design on the mental models.
For instance, the design and interaction of the Spotlight
Navigation reminded them of a remote control, and
consequently they used and described it as such. One
participant even thought it was connected to the
speaker set because it controlled the music (i.e. making
or breaking a connection between the Living Colours
lamp and the speaker set started and stopped the music
playback).
Another observation was the complicated conceptions
the participants had about the connections and their
properties. Although there was no explicit directionality
on the interactions or the connections, participants
conceived the connections that for instance carried
168
Design and semantics of form and movement
music to have a direction, travelling from its source to a
destination. Directionality was also indicated where one
device seemed to control the behaviour of the other.
By allowing users to use this sense of directionality in
their interaction to establish the connection, we could
easily give them more control over the connections.
Transitivity was another, less obvious, concept that
emerged from some of the mental models. Transitivity
of a connection is a logical property that emerges when
a network node A is connected to B, and in turn B is
connected to node C. Transitivity then defines A to
be connected to C as well. We observed participants
to erase connections they indicated to exist before
because they “were no longer needed” because of
transitivity. And, in another case, worried about
(hypothetically) removing a device from the network
when it was in a chain of multiple connected devices,
because it would lead to removing the transitive
connections as well.
8 Conclusion
The SOFIA project provides a platform and therewith
the possibilities to improve the interoperability among
devices. In this context, two prototypes were developed
to experiment with tangible and augmented reality
approaches to manage semantic connections. Both show
their potential in moving the interaction with devices
from a device-oriented paradigm towards a more
task-oriented paradigm with increased interoperability.
Although we are still exploring the possibilities of our
approach, promising results and insights have been
achieved already. The results obtained during this
evaluation will be used to further define our semantic
connections interaction model, and may hopefully
help other interaction designers to deal with design
opportunities and challenges that emerge when
designing for interoperability.
Acknowledgment. We would like to thank Aly Syed,
Riccardo Trevisan, Sriram Srinivasan, Hans van
Amstel, Jettie Hoonhout and Jolijn Teunisse for their
contributions to the smart home pilot. This work is
supported by Smart Objects For Intelligent Applications
(SOFIA), funded by the European Artemis programme
under the subprogramme SP3 Smart environments and
scalable digital service.
References
[1] Weiser, M. (1991). The computer for the 21st century.
Scientific American, September.
[2] Niezen, G., van der Vlist, B., Hu, J., & Feijs, L. (2010). From
events to goals: Supporting semantic interaction in smart
environments. In Computers and Communications (ISCC),
2010 IEEE Symposium, pp. 1029 –1034.
[3] Rapp, S., Michelitsch, G., Osen, M., Williams, J., Barbisch,
M., Bohan, R., et al. (2004). Spotlight navigation: Interaction
with a handheld projection de-vice. In Advances in Pervasive
Computing: A Collection of Contributions Presented at
PERVASIVE 2004, pp. 397–400, Oesterreichische Computer
Gesellschaft, April 2004.
[4] Rapp, S. (2010). Spotlight navigation: A pioneering user
interface for mobile projection. In Proceedings Ubiprojection.
Helsinki.
[5] Vlist, B.J.J. van der, Niezen, G., Hu, J., & Feijs, L. (2010).
Design semantics of connections in a smart home environment.
In L.L. Chen, J.P. Djajadiningrat, L. Feijs, S. Kyffin, D. Steffen,
& B. Young (Eds.), Proceedings of Design and Semantics of
Form and Movement (desform) 2010 (pp. 48-56). Lucerne,
Switzerland: Koninklijke Philips Electronics N.V.
[6] Wensveen, S.A.G., Djajadiningrat, J.P., & Overbeeke, C.J.
(2004). Interaction frogger: a design framework to couple action
and function through feedback and feedforward. In Proceedings
of the 5th conference on Designing interactive systems:
processes, practices, methods, and techniques, (pp. 177–184).
[7] Heijden, H. van der, & Sorensen, L.S. (2003) Measuring
attitudes towards mobile information services: An empirical
validation of the HED/UT scale. In European Conference on
Information Systems (ECIS), Naples, Italy.
[8] Hinds, P. (1998). User control and its many facets: A study
of perceived control in human-computer interaction. tech. rep.,
Hewlett Packard Laboratories.
[9] Veer, G.C. van der, & Carmen Puerta Melguizo,
M. del. (2003). The Human-Computer Interaction Handbook.
Bram van der Vlist ,
1
(pp. 52-80). Hillsdale, New Jersey: L. Erlbaum Associates Inc.
Gerrit Niezen1 ,
[10] Poole, E.S., Chetty, M., Grinter, R.E., & Edwards, W.K.
Stefan Rapp , Jun
(2008). More than meets the eye: Transforming the user
Hu1 and Loe Feijs1
experience of home network management. In Proceedings
1
of the 7th ACM Conference on Designing Interactive Systems,
2
Department of
Industrial Design
DIS ’08, (New York), (pp. 455–464).
Eindhoven University
of Technology
Eindhoven,
Netherlands
2
Conante, Eindhoven,
Netherlands
Design and semantics of form and movement
169
Vicktoria Teinaki, Bruce Montgomery, Nicholas Spencer, Gibert Cockton
[email protected], [email protected], [email protected], [email protected]
An aesthetics of touch: Investigating
the language of design relating to form
Abstract
Keywords
How well can designers communicate qualities of touch?
This paper presents evidence that they have some
capability to do so, much of which appears to have
been learned, but at present make limited use of such
language. Interviews with graduate designer-makers
suggest that they are aware of and value the importance
of touch and materiality in their work, but lack a
vocabulary to fully relate to their detailed explanations
of other aspects such as their intent or selection of
materials. We believe that more attention should be
paid to the verbal dialogue that happens in the design
process, particularly as other researchers show that
even making-based learning also has a strong verbal
element to it. However, verbal language alone does not
appear to be adequate for a comprehensive language
of touch. Graduate designers-makers’ descriptive
practices combined non-verbal manipulation within
verbal accounts. We thus argue that haptic vocabularies
do not simply describe material qualities, but rather
are situated competences that physically demonstrate
the presence of haptic qualities. Such competencies are
more important than groups of verbal vocabularies in
isolation. Design support for developing and extending
haptic competences must take this wide range of
considerations into account to comprehensively
improve designers’ capabilities.
Design, language, touch, haptics, product semantics,
sensory design.
170
Design and semantics of form and movement
1 Introduction
Are talking and making related? Is it important for
designers to be articulate about the haptic qualities of
their work? This paper tackles two relevant issues: the
role of language in design, and within this, whether and
how designers are able to articulate haptic qualities in
relation to their work. To investigate this, it reviews
relevant literature on design discourse and haptics,
cognitive apprenticeship, aesthetic frameworks, and
means-end approaches. These theoretical resources
are used to critique language use in interviews with
graduate designers.
1.1 Haptics and Aesthetics
The term haptics [from the Greek haptikos ‘pertaining
to the sense of touch’] specifically relates to touch
and the cutaneous senses. While touch was deemed
the most animalistic of the senses by Aristotle, and
for long after was deemed mere carnal sensation,
more recent philosophers such as Merleau-Ponty have
sought to reunite the body and mind, and in the case
of Wyschograd, highlight the difference between touch
(which can be affective, emotional and metaphorical)
and sensation [1].
The term ‘haptic qualities’ is used here to explain the
qualities specific to touch. Unlike fixed features such as
form elements, qualities are open to interpretation. In
design, there are strong precedents for discussing visual
qualities based on the Bauhaus and gestalt psychology
[2], but we have not yet made a similarly thorough
exploration of haptic qualities and their origins and
consequences.
2 Relevant Literature
Relevant literature spans language in design, touch
in design, intersections of the two through cognitive
apprenticeships and criticism, and the role of qualities
in designs’ consequences.
2.1 Languaging Design
While there has been a sustained interest in designers
and sketching e.g. [3], what is largely unnoticed is that
language underpins much sketching and other design
decisions.
A strong provocative comment on language comes from
Hodges [4], who believes that the so-called soulless
modern buildings of the 20th century are a result of
designers being unable to communicate their vision to
clients. Lawson [5, p.85] notes that while designers may
claim most of their thinking is captured in their sketches
“100% is held in the design discussions”. He also
highlights that the concept of the designer-as-sketcher
is a modern one, as masons and other craftsmen
as recently as 100 years ago dispensed instructions
verbally.
A number of other studies have noted the relevance
and importance of language in relation to design.
For example, Krippendorf [6] draws attention to the
multiple stakeholders involved in design discourse.
Also, Cross [7] reported on how words could change
the direction of design discussions. Language is also key
to design placements [8], and Schön’s ‘appreciations’ [9].
The practical use of language for meaning-making
is highlighted in Verganti’s study of design-driven
innovation in companies [10] He suggests that the
designers and the other stakeholders involved in the
design process create meanings through language, which
are shared amongst the various stakeholders.
If verbal language is so important, why has it received
relatively limited attention in design research? In part,
this could be due to the challenges in studying verbal
language within design processes, relative to the ease
with which sketches can be collected and analysed.
Design language is often ephemeral: written notes may
be regarded as merely aids to back up sketches and
prototypes; and critically important design dialogues
regularly go unrecorded.
2.2 Touch and Language
So how does one talk about touch? We can understand
the means with which people understand touch:
Lederman and Klatsky’s Methods of Haptic Exploration
[11] identify six aspects through which we approach
objects (lateral motion/texture, pressure/hardness,
static control/temperature, unsupported holding/
weight, enclosure/volume, contour following/shape).
Akerman et al. [12] suggest three global dimensions
of touch (hardness, roughness and weight) with
implications such as weight implying qualities such
as trustworthiness [1].
Much current research on touch and design language
focuses on consumer responses rather than the
designers’ expertise [13], often through use of simple
Semantic Differentials [14]. While this can be useful in
terms of understanding the language of experts, e.g.
skiing [15], it presumes that users are the best sources
for both evaluating and articulating haptic language.
Such assumptions need to be tested by comparing
consumers’ language use with designers’ language use.
Those that do look at how designers approach touch
often collapse it into being able to categorise a product
to fit a specific emotion, e.g. a glamorous kettle. This
rationalising of the senses can be seen in an extreme
form in Kansei Engineering, where emotions are
condensed into single words that are next used in
matrixes for comparing sensory responses [16]. While
this does serve some use in allowing designers to
consider some consequences of their actions, it also has
the unfortunate effect of ignoring designers’ discourse
related to the qualities of the product, turning our use
of our senses and emotions into something that is to be
measured once, rather than explored regularly during
the design process.
So why is touch marginalized in relation to design
conversations? Semiotics – a school of thought
enthusiastically adopted by design – has tended to
concentrate on visual language and its associated verbal
interpretations [17].
One possible reason is that visual qualities can be
critiqued via inspection of images with designers
Design and semantics of form and movement
171
absent. In contrast, critiques of haptic qualities are
more effective when both designers and their designed
artefacts are present within the research process.
2.3 Apprenticeships: Connecting Language to
Making
A notable exception to the lack of attention given to
touch and design language can be found in research
into communities of practice, notably the concept of
‘cognitive apprenticeships’. Collins and Seeley-Brown
[18] have studied accounts from practices such as
dressmaking, and attempted to both identify the
methods with which students are taught to improve
their skills. This has been transferred to non-manual
methods such as mathematics and reading, but as
Collins and Seeley-Brown note, there are gaps in
traditional apprenticeship teaching (namely that skills
are taught to students to meet business needs, rather
than in a way that best supports learning) [18].
The importance of understanding language – and
the means with which it can be carefully combined
with making – is well described in research on
cognitive apprenticeships. Collins and Seeley-Brown’s
investigation into why traditional apprenticeships work
revealed that discourse played a key part in learning:
novices picked up their cues for critique through
engaging with experts and picking up the language.
Ackerman et. al [12, p.1713] suggest that “touch acts as an
ontological scaffold, which is reflected in language”. This
has been similarly noted in wine tasting: when matching
words to a specific taste or scent, novices became far
more accurate choosing from the words created by
experts to choose from [19].
We can conclude from this that novice designers who
engage with experts will be exposed to language that
will help them to become more advanced at critique
of haptic qualities. Such exposure could be direct,
via traditional atelier relationships, or indirect, via
interactive audiovisual resources. Indirect exposure
allows the introduction of innovative approaches to the
critiquing of haptic qualities in design, taking novice and
experienced designers beyond existing craft practices
and expertise.
While designers are not always taught in the way that
apprentices used to be, it is worth reconsidering the
nature of how materials are used in design education,
e.g. making sure that materials are not merely selected
because they are close to hand (as noted in Jacucci and
172
Design and semantics of form and movement
Wagner’s research on architecture students and
material events) [17]. Innovative and imaginative
selection of materials could focus more on exploration
and understanding of haptic qualities.
2.4 The Importance of an Aesthetic Language
Aesthetic sensitivity through use of language (to
reference the Greek aesthetikos), can also be linked
to critique. Bardzell [20] has noted that criticism
is valuable in design because it not only heightens
perception and allows for alternating between the
whole and cultural particulars, but also spurs the critic
to worthwhile action. Drawing on cultural and literary
theory, he suggests that criticism be categorised in
terms of the author (creator), text (artifact), context,
and audience. Haptic vocabularies can reflect sociocultural context, direct perception towards qualities
of artifacts, and empower both creators and consumers
through critical vocabularies that reveal opportunities,
meanings and interpretations. A comprehensive haptic
vocabulary would thus extend beyond the qualities of
artifacts to cover design intents, audience responses,
and relations to broader social and cultural contexts.
2.5 Qualities, Value Theory and Worth Mapping
Cockton [21] provides a basis for relating product and
service qualities to usage outcomes (via consumer
experiences) as well as to concrete product attributes
(features, materials). Qualities are abstract product
or service attributes with a specific role with the
means-end chains of product and service consumption.
Consumption motivators (the ends of means-end
chains) result from the cumulative effects of interactive
means (the materials, features, and qualities of products
and services, and the user experiences of their
consumption). Qualities are therefore the interface
between the material product or service and the
phenomenology of use and consumption.
Means-end chains can be combined into graphical
representations as Worth Maps [22]. Each chain is a
path through a network (box and arrow diagram) from
a product or service’s material attributes, through its
features and qualities, via user experiences, to the
outcomes of usage and consumption. Such means-ends
chains express comprehensiveness and coherence in
design reasoning. The extent to which haptic qualities
are understood and leveraged in a design is reflected
in their explicit roles in means-end chains. Articulate
designers with a strong understanding of haptic qualities
can relate these to both the materiality of products
and services (features, materials) and phenomenology
of usage and goal achievement. For example, as already
noted, a material’s weight may imply qualities such as
trustworthiness [1]. Trustworthiness in turn may lead to
confident use of an artifact, leading to valuable outcomes
that could not be achieved without confident use.
Designers who lack explicit and/or thorough understanding of haptic qualities are less likely to both name
and demonstrate haptic qualities, and also less likely to
explain how these qualities relate to material causes and
phenomenological consequences.
A comprehensive haptic vocabulary would thus be
anchored in material qualities, but would associate
those with their origins and features on the one hand,
and with their consequences for consumer experiences
and value realisation on the other. There are
structural similarities between Bardzell’s [20] aesthetic
frameworks and the elements of Cockton’s [21] meansend chains. Authored intent [20] relates to intended
qualities and actual experiences and outcomes [21].
Audience response [20] relates to perceived qualities
and actual experiences and outcomes [21]. Means-end
chains as a whole [21] owe their dynamics to the sociocultural contexts within which they are situated [20].
Such contexts make possible the meanings and motives
that transform qualities into outcomes via experiences.
2.6 Literature Summary
The existing literature on haptic qualities in design
and their relation to verbal and non-verbal expression
provides a range of bases for interpreting designers’
accounts of the nature and role of haptic qualities in
their design work.
The combined perspectives provided a basis for coding
and interpreting graduate designers’ accounts of
their work during interviews at a UK national design
exhibition. By comparing graduate designers’ accounts
with theoretical potentials for haptic vocabularies,
we can reveal opportunities for improving designers’
capabilities for exploring haptic qualities during design.
3 Interviews
In order to investigate the levels of fluency that
designers have, in both articulating the qualities of their
work, and specifically the haptic qualities, the author
conducted short interviews with ten design graduates
at the 2011 NewDesigners Exhibition. The students,
from schools throughout Britain, were exhibiting work
ranging from furniture to product design to jewellery,
but were all designer-makers in that they had all
designed and fabricated their work. The designers were
also chosen on the basis of work that relied extensively
on materials and physical form.
The short interviews were carried out alongside their
designed products and audiotaped (videotaping was
deemed inappropriate due to the spontaneous nature
of the talks). Interviewees were asked to describe:
‡WKHLUSURGXFWVDQGWKHSURFHVVWKDWKDGJXLGHGWKHP
to the final product,
‡WKHTXDOLWLHVWKDWWKH\OLNHGDERXWWKHLUILQDOSURGXFW
‡WKHLUWKRXJKWVDERXWPDWHULDOVDQGSK\VLFDOIRUP
The interviews were conducted on the third and final
day of the show. This is relevant for two reasons: firstly
the students had had time to get used to talking about
their work (or ‘refine their pitch’), and secondly as
they’d also been able to get audience feedback on their
work (some noted that they had only just got the work
ready in time for the show).
3.1 Results
As in all qualitative research, interview data was coded
to reveal emergent themes within it. Initially the analysis
focused on specific use of haptic terms, but this proved
to be too narrow a focus, due to the broader extent
of graduate designers’ accounts of their work.
Interview data was primarily coded through a phenomenological set of codes that generalized over designers’
foci on, and subjective understandings of, their work.
Theoretical analyses were secondary, and were focused
on themes and evidence within the primary coding.
Once a broader view was taken, the following primary
codes emerged from analysis of the data:
‡intent: what the designers intended to achieve;
the intended outcome;
‡evaluation: allowing for reflection, also taking into
account the feedback they’d been able to pick up
during the show;
‡references: notes of specific trade and physical terms
related to touch (taken from Lawson’s observation
of design language as being compact and full of
references) [5];
‡qualities: language usages which specifically related
to aesthetics and haptics.
Design and semantics of form and movement
173
While many of the designers initially protested that
they weren’t able to discuss their products, this did not
prove to be the case. The only problematic case came
from a student with English as a second language, and
even in this situation, the particular student proved to
be more nuanced than might be expected. This is in
accordance with Krippendorf’s [23] and Lawson’s [5]
assertions that designers are in fact highly articulate
and speak in a highly evocative and compact language.
However, haptic qualities played only a limited role in
the graduate designers’ accounts of their work.
Through basic coding of the interviews, two themes
emerged:
‡&KDOOHQJLQJ0DWHULDOVDQG
‡/LPLWHG5HFRJQLWLRQRI+DSWLF4XDOLWLHV
Key observations and evidence for each of the themes
is now presented.
Challenging Materials. The graduate designers talked
more about materials in general than about the qualities
that resulted from both materials and features (form).
The designers who were interviewed had played with
materiality in some respect, and so their work often
included some sense of challenging expectations in
regards to a material:
Y (wooden steam bent table): I’ve had a lot of people
think that the underneath [the wooden table] is all
was really soft and flowing, and they’ve had to come
along and like, touch it, and make sure it’s all solid.
But yeah, everyone loves it and making sure like,
feeling all the curves and everything.
A (metal necklace): I’ve actually had someone come
from up there [the top floor of the exhibition] and
go, “is it [the metal necklace] rubber? It looks like
rubber”. and lots of people have said, what is it
actually made of? It’s not plastic, but it’s usually the
sound it creates, you can tell it’s metal.
K (acrylic jewellery): If people have knowledge already
about the material and the processes, they come in
and question more, where’s the join, or how it is
worked? If it’s people who don’t and have no idea,
they come and approach and question: is it glass?
In Bardzell’s [20] terms, the graduate designers’
intentions sought to disrupt existing audience
expectations about specific materials. However not all
174
Design and semantics of form and movement
consequences of materials and forms were intentional.
One student found that he had created an “accidental
illusion” with a stool and table that looked as if both
were made of the same materials but were not:
M (stool with metal strap): It wasn’t really intentional
to bring the strapping through and keep it the same
aesthetic, it just sort of ended up that way, without
me realizing … I was er, surprised at the number
of people saying, “I thought it was fabric, how is it
standing up?” which I kinda like really.
These comments emphasise the element of verification
prevalent in haptics [24], and are a reminder that the
“seeing is believing” is short for “seeing is believing, but
feeling’s the truth” [1]. Designers’ intentions may not
be realised until constructed artefacts are critiqued
or used, but more importantly, physical realisations
and human encounters may reveal unintended
consequences. In Bardzell’s [20] terms, the designer
here extended his understanding of his work through
an audience’s encounters with it. Within Cockton’s [21]
means-end chains for design, this graduate designer
was mostly focused on relations between materials,
features (form) and qualities. There was little focus on
experiences, and none on outcomes, which in Bardzell’s
terms, indicated a limited contextualisation of the
design work [20].
Haptic Qualities: Recognition, to an Extent. Several
students identified materials and their handling during
making as a key influence what they did, through their
process of making and/or use:
Y I let the wood talk to me, to let it bend the way it
wanted to, and then I just wanted to force it a bit more.
E (glass bowl): I’m just so happy that I’ve found a
product, a technique, which really suits me...
I’ve found, my language as it were, with the glass.
K My work doesn’t necessarily have some in-depth deep
concept; it’s all about the material and the handling of it.
Once again, graduate designers were focused on the
initial elements of Cockton’s [21] means-end chains,
but some could articulate qualities and even outcomes.
Interestingly (but perhaps to be expected) the most
interesting comparisons in terms of the value of specific
materials emerged when designers justified why they’d
chosen one material or process over another (Table 1).
One designer who had created a laser cut clock in
both acrylic and wood, attempted to describe why the
former was more popular:
Acrylic
Wood
Weighty, solid, finished (outcome: sold
better), lego-like
Lighter, more flimsy (outcome: not as
popular)
Table 1. Terms comparing wood to acrylic in a laser
clock and related success.
Another graduate designer had tried a number of
materials before setting on glass, noting the following
properties of all of them (Table 2):
Glass
Metal
Plastic
“water-like, touchable”
harsh, “though beautiful
polished”, didn’t flow
“not a material I enjoy
working with, it’s all wrong”
[see below]
Table 2. Material comparison for selection.
The contrasts here were mostly in terms of material
qualities, including the way in which glass invited touch.
One response was visceral (which is how Aristotle
considered touch, i.e. as animalistic): this designer
simply did not like working with plastic.
Another graduate designer contrasted the two different
materials that met in his work. While the haptic and visual
qualities of wood and pewter were directly referenced,
form dominated his account of their jointing (Table 3):
Pewter (cast)
Two Meeting
Wood
“soft, metal”
Controlling the wood,
joints being snug or not
strong enough
warm, nice qualities
Table 3. Pewter and wood chair.
Several students had worked with metals (Table 4):
Metal (vs glass)
Aluminium
Pewter
“Harsh, didn’t flow”
“cool … not warm, but
comfortable”
Soft when in thin sheets …
but difficult to compress
when in large blocks [meant
as positive]
Table 4. Comparison of terms and materials.
Several haptic qualities were in evidence here, with
the occasional reference to experiential qualities
(‘comfortable’).
Even within single products, they were aware that the
value of a haptic quality could change: a necklace was
wonderfully ‘fragile’ and ‘delicate’, but not so good when
‘broken’.
When asked about material qualities most felt that they
had made successful choices (sometimes rationalised
through why they had not worked with various other
materials), there were some terms that attempted to
capture haptic aesthetic qualities.
‡Flow “it just didn’t have flow” (E). This was used in
a way similar to the concept of harmony in graphic
design, in that it represented everything working
together (or in this case, not). It could be speculated
that this has come from interior or landscape design,
as Lehrer [19] notes in wine tasting language that
terms can easily move from one domain to another
(she relates the concept of a wine having body to the
80s popularity of bodybuilding).
‡Finish: this was noted by six of the graduate designers
as being something they were proud of. This was
generally described as sheen, but in one case as patina.
These are standard words used in product design
language, so it is not surprising to see them used.
‡Comfort was noted, but used in different ways from
furniture to jewellery: in the former, an uncomfortable
chair was too angular (M), whereas in jewellery
weight, softness, and (to some degree) temperature
played a part.
‡Thrown was used by Y “it’s all very organic and
thrown” to describe her hand/steam bent wooden
table. This was one of the few times a word was
clearly taken from another field (here clay working)
and shifted into another.
However, it is telling that many other accounts had to
rely purely on material and feature (form) references
without exploring their consequences, and mostly
viscerally, for qualities or experiences:
E plastic was just, was just for me, it’s all wrong...
not a material I enjoy working with.
J too angular... soft wood, which is really warm and has
some nice qualities about it … fluid qualities that were
left in it as well, like you can actually see the process
Design and semantics of form and movement
175
going on, like a little snapshot of what was going on...
process, it like, held within the metal.
Other popular terms that came up when explaining
design decisions were:
‡Tactility: “you want to touch and feel it” (E), a rare
articulation of an explicit experiential consequence.
‡What it did not have: “no joins”, “no glue” was also
common, focused on features.
‡Natural: there were a number of mentions of this,
through the use of both terracotta and wood. In
contrast to some graduate designers’ challenges
to existing contexts of audience interpretations,
here others were articulating established semiotic
associations and their consumer valuations.
3.2 Comparisons with a Previous Study
In a previous study by the authors of an experienced
fashion expert’s understanding of textiles’ haptic
qualities, video recordings revealed how swatches
were manipulated to ascertain and demonstrate their
tactile qualities. The use of audio recording alone
in the current study reported above highlighted the
combination of ostension (pointing at), manipulation,
and verbal descriptions and interpretations when
communicating haptic qualities. When transcribing the
recorded interviews, memories of graduates’ specific
manipulations and ostensions had to be recalled to
make sense of some of their language use.
A key insight here is that improved understandings of
haptic qualities require more than a simple vocabulary,
Ostension and manipulation are integral aspects of
haptic understandings. This is also seen in areas such
as wine tasting where the glass has to manipulated in
a specific way to reveal the ‘legs’ or ‘tears’ of a wine,
which indicate its alcohol content.
4 Discussion
The primary aim of this research is to help tease out
the verbal and non-verbal language that designers use
to help make meanings out of forming, and be able to
present it back to them in a way that they can both
be made aware of it, and also be able to improve on
it. Such improved vocabularies have to be productive,
relating both to the assumed material causes of qualities
and also to their expected effects e.g. through situating
haptic qualities in means-end chains [21].
176
Design and semantics of form and movement
Given the broad range of uses for vocabularies of haptic
qualities, the interviewed graduates demonstrated only
a limited ability to critique the haptic aspects of their
designs. Much of the references to haptic qualities
were related to the processes of designing and making,
and not to their consequences for human experience
and outcomes from ownership and/or use of their
designs. Their accounts often related viscerally to
materials or focused on features (form), with limited
extensions beyond these to qualities and experiential
consequences. In these graduates’ accounts of their
designs, few existing theoretical understandings showed
through.
The importance of language in design, as established by
a range of design researchers [4], [5], [6], [7], [8], [9],
& [10], is not well reflected in the graduates’ accounts,
although most of Lederman and Klatsky’s six ways
to approach objects haptically were demonstrated in
product interactions [11]. This may indicate that the
graduate designers’ vocabularies had largely been picked
up through a cognitive apprenticeship that drew little
on existing relevant design research. Existing design
education may thus not be preparing students well for
articulating their intentions for qualities, experiences,
and usage outcomes.
What emerged from student discussions was a
disconnect between the language they were able to
use to describe their thinking process in concrete
terms (references) on the one hand, and their
aesthetic decisions relating to touch on the other.
While they showed awareness of the haptic qualities
and their importance, the language that they used to
communicate it was far less specific, except where this
was related to processes of making. Few theoretical
resources were in evidence.
Graduates’ accounts of haptic qualities had a narrow
critical range, making limited use of the range of critical
possibilities outlined by Bardzell [20]. Similarly, while
some graduates occasionally demonstrated an ability to
relate haptic qualities to materials or design features,
they rarely related qualities to likely or intended
experiences or consequences, as made explicit in
Cockton’s [21] worth maps.
Making designers more aware of the language that they
use and their means of communicating and reflecting
through it should make designers more aware and more
successful when it comes to both making decisions
in relation to touch and therefore being able to
communicate it.
There are other disciplines that do not have a strongly
established language for discussing aesthetics, but they
are anchored by a useful means of replicability. A prime
example of this is cooking: Fine notes that the bluecollar staff have to learn to develop an eye and taste
for dishes that they’ve never eaten before, but do so
through repetition and not having to be able to discuss
it [24]. However, it is not clear that design can rely
exclusively on such tacit non-verbal knowledge.
5 Future Work
There is an opportunity to investigate the language of
designers who have a highly developed appreciation of
touch and how their language could be used to improve
the design work of novice designers, and perhaps
experienced designers too.
Beyond this, design education could be reconsidered
in relation to both the cognitive apprenticeship model
and design criticism in order to encourage more design
dialogue in relation to touch. Similar work has been
done by Sonneveld [25] in encouraging design sensitivity
in relation to touch, but did not investigate the role of
expert discourse.
The current apprenticeship elements of design
education could be improved. While some of the
aspects of the process are already well formed for
learning (e.g. dressmaking students having their
knowledge staggered), one of the limitations of
the current system is that it often works from
practicalities rather than ideals. This is noted with
architecture students using materials that are to hand
to make prototypes, rather than considering their
appropriateness [17].
Understandings of visual qualities are well grounded
in gestalt psychology as explotied by Paul Klee and
others in the Bauhaus. Such imports from other fields
are not uncommon. For example, language used for
understanding wine was developed by chemists (and
interestingly, later extended by a linguist to be more
of a nuanced study [19].
There are clear advantages in developing haptic
vocabularies, with accompanying audio-visual
demonstrations, augmented where relevant by
interactive capabilities. We need to consider allowing
students to find ways to enrich their vocabularies, in
particular in relation to touch. The reasons for this
include:
‡'HVLJQHUVZLOOEHHPSRZHUHGWRFUHDWHDZLGHUUDQJH
of potential meanings through improved ways of
critiquing and interrogating their material choices.
‡,IGHVLJQHUVZDQWWREHDEOHWRFUHDWHEUDQGRU
product systems, they similarly need to be given the
language to be able to articulate what constitutes the
product system. At least one student noted that they
were attempting to extend their singular product
to a range, but needed to figure out how to do so.
Without an understanding of the key ways to manage
this, change, any translations risk being diluted in the
way that the brown Microsoft Zune changed from a
concept model of two-tone brown and green metal
to brown plastic described by Gizmodo magazine as
“swamp water Jell-O” [26].
‡'HVLJQHUVLQFUHDVLQJO\ZRUNLQPXOWLGLVFLSOLQDU\
team settings, where traditional tacit knowledge must
be made explicit to communicate with other team
members, and to allow discussions of design options
[27]. Designers need to be able to articulate and
demonstrate haptic qualities.
‡'HVLJQHUVQHHGWREHDEOHWRUHDVRQDERXWWKHRULJLQV
and consequences of haptic qualities, drawing on
structures such as the means-end chains of Cockton’s
Worth Maps [21]. Making the connections within
design thinking explicit requires confident broad
vocabularies for all elements in design’s means-end
chains.
We plan to further explore expert, graduate and
student use of haptic vocabularies in their accounts
of design intentions and opportunities. We will pay
particular attention to non-verbal behaviours when
accounting for haptic qualities, especially the role of
ostension and manipulation. We will use relevant design
research as a basis for understanding expert behaviours.
Examples and understandings will provide content for
an interactive tool for developing designers’ haptic
vocabularies and understandings. The tool will combine
physical and digital resources. Use of this tool by
designers will support assessment of the relative value
of existing bases for understanding haptic language use
and critical behaviours from design research.
Design and semantics of form and movement
177
6 Conclusions
Graduate designers are aware of the importance of
touch in their work (both through their work and its
importance to their audiences), and are able to point to
some decisions that they have made in relation to it, but
appear to lack a broad enough lexicon to be able to fully
communicate this, and to relate haptic qualities to both
their origins and their consequences.
Designers at all stages of development should be
empowered by an improved vocabulary, with supporting
practices, for the aesthetics of touch. Such a vocabulary
is not only key to enable them to communicate, but
also to be able to make complex and demanding design
decisions such as translating a single product into a
product line.
We have identified relevant theoretical perspectives
from across a range of design research and have used
this to augment bottom-up analyses of graduates’
accounts of their design work. Through this, we have
identified opportunities for improving the extent,
depth and effectiveness of haptic vocabularies and their
associated non-verbal resources. We plan to extend
the above studies to expand our corpus of examples
and understandings. With a suitable corpus in place,
we will then transfer examples and understandings into
an interactive tool that integrates physical and digital
resources we will then evaluate the effectiveness of
this tool.
[8] Buchanan, R. (1992). In design thinking wicked problems.
Design Issues, 8(2), 5-21.
[9] Winograd, T. (1996). Reflective Conversation with Materials
An interview with Donald Schön by John Bennett. Bringing
Design to Software (1st ed.) Harlow: Addison Wesley. 171-189.
[10] Verganti, R. (2008) Design, meanings and radical innovation:
A meta-model and a research agenda. Journal of Product
Innovation Management, 25(5), 436-456.
[11] Lederman, S.J., & Klatzky, R.L. (1987). Hand movements:
a window into haptic object recognition. Cognitive Psychology,
19(3), 342-68.
[12] Ackerman, J.M., Nocera, C.C., & Bargh, J.A. (2010).
Incidental haptic sensations influence social judgments and
decisions. Science, (328), 1712-5.
[13] Dagman, J., Karlsson, M., & Wikström, L. (2010).
Investigating the haptic aspects of verbalised product
experiences. Design, 1-15.
[14] Hsu, S.H., Chuang, M.C., & Chang, C.C. (2000). A semantic
differential study of designers’ and users' product form perception.
International Journal of Industrial Ergonomics, 25, 375-391.
[15] Dore, R., Pailhes, J., Fischer, X., & Nadeau, J. (2007).
Identification of sensory variables towards the integration of
user requirements into preliminary design. International Journal
of Industrial Ergonomics, 37(1), 1-11.
[16] Schütte, S. (2005). Engineering Emotional Values in
Product Design. PhD Thesis, Institute of Technology, Linköping,
Dissertations No. 951, last accessed 15/5/09 at liu.diva-portal.
org/smash/get/diva2:20839/FULLTEXT01
[17] Jacucci, G., & Wagner, I. (2007). Performative roles of
References
materiality for collective creativity. Proceedings of the 6th ACM
[1] Paterson, M. (2007). The Senses of Touch: Haptics,
SIGCHI Conference on Creativity and Cognition - C&C ’07,
Affects and Technologies (Senses and Sensibilities). Oxford:
73-83. New York: ACM Press.
Berg Publishers.
[18] Seely Brown, J., Collins, A., & Holum, A. (1991). Cognitive
[2] Kepes, G. (1995). Language of Vision. New York: Dover
apprenticeship: Making thinking visible. American Educator, 6, 38-46.
Publications.
[19] Lehrer, A. (2009). Wine and Conversation (2nd ed., p. 336).
[3] Buxton, B. (2007). Sketching User Experiences: Getting
USA: Oxford University Press.
the Design Right and the Right Design (Interactive Technologies).
[20] Bardzell, J. (2011). Interaction Criticism : An Introduction
Boston: Morgan Kaufmann.
to the Practice. Interacting with Computers.
[4] Hodges, R.M. (1991). Opening the designers’ spatial
[21] Cockton, G. (2008). Designing Worth – Connecting
dictionary: The power of a professional vocabulary.
Preferred Means to Desired Ends. Interactions, (4), 54-57.
The Journal of Architecture and Planning Research, 8(1), 39-47.
[22] Cockton, G. Kirk, D., Sellen, A. & Banks, R. (2009).
[5] Lawson, B. (2005). How Designers Think, Fourth Edition:
Evolving and augmenting worth mapping for family archives.
The Design Process Demystified (4th ed.). Oxford: Architectural
In Proceedings of HCI 2009 – People and Computers XXIII –
Press.
Celebrating people and technology, ed. A.F. Blackwell, 329-338,
[6] Krippendorff, K. (2005). The Semantic Turn: A New
BCS eWIC, available at http://www.bcs.org//upload/pdf/ewic_
Foundation for Design. CRC Press.
hci09_paper42.pdf
[7] Cross, N. (2006). Designerly Ways of Knowing (1st ed.).
[23] Krippendorff, K. (1995). Redesigning Design: An Invitation
Springer.
to a Responsible Future University of Pennsylvania.
178
Design and semantics of form and movement
[24] Fine, G.A. (2008). Kitchens: The Culture of Restaurant
Work, Updated with a New Preface (p. 328). Berkeley:
University of California Press.
[25] Sonneveld, M. (2004), Close encounters of the first kind:
Meet the material world. In D. McDonagh, P. Hekkert, van Erp,
J., & D. Gyi, (Eds). Design and Emotion: The Experience
of Everyday Things. (pp. 436-437). London: Taylor & Francis.
[26] Ashlock, J. (2007). What can brown do for you? I.D.
Magazine. Retrieved June 7, 2007, from http://www.idonline.
com/features/feature.asp?id=1575
[27] van Halen, C., Vezzoli, C., & Wimmer, R. (2005).
Methodology for Product Service System Innovation.
Assen: Koninklijke van Gorcum.
Vicky Teinaki,
Gilbert Cockton,
Nick Spencer,
Bruce Montgomery
School of Design,
Northumbria
University
Design and semantics of form and movement
179
Camille Moussette
[email protected]
Learn to make, make to learn:
Reflections from sketching
haptics workshops
Abstract
1 Introduction
This paper presents results, observations and insights
from four workshops on the design of haptic interfaces.
The workshop series was called Sketching Haptics,
and the primary objective was to explore how the
fields of Haptics and Design can come together for
educational purposes during 4-5 days. The current
haptic advances tend to favor technological refinements
over other forms of inquiries. The initial premise is
that designers, with their creativity and user-centered
perspective, can contribute, along roboticists and
engineers, to evolve the next generation of haptic
interfaces. Designers might not have the technical
affinity to develop cutting edge haptic technology, but
they possess skills, processes and expertise that can
definitely bring new perspectives, applications and
considerations for haptics. The main takeaway from
the workshops reveals that quick, creative, explorative
works is not only possible but can be valuable and
rewarding. The observations also stress the necessity of
making, prototyping, materializing ideas and sketching
in hardware when dealing with our sense of touch. By
communicating the structure, activities and outcomes
from the workshops, I hope to inspire educators and
designers to survey and embrace the nascent field of
haptic design, help develop our haptic design toolbox,
and ultimately democratize haptic interfaces.
Despite decades of research and advances in our
understanding of the sense of touch, designers are
generally unfamiliar with haptics. As new technology,
tools and approaches are reshaping our design activities,
I believe that designers and particularly design students
are now in a position to learn and develop a heightened
sensitivity to haptics. Designers, with their creativity and
user-centered perspective, can now join roboticists and
engineers to help evolve new haptic interfaces that are
ultimately more humane, valuable and meaningful to us.
As the haptic, design and Human-Computer Interaction
(HCI) communities are constantly expanding and
reaching out into new territories, new areas naturally
overlap and provide new grounds for collaboration and
fresh inquiries. The amalgam of Haptics and Design is
fairly new and poses interesting challenges for designers,
HCI professionals and researchers.
Traditionally, haptic advances tend to favor and
recognize technological refinements over other forms
of inquiries. As Design and Haptics are coming together,
what are the activities, challenges and friction points
that underpin this new field of inquiry? How can
designers play an active role in this collaboration? And
more fundamentally, why designers and other creative
professionals know so little about haptics, our sense of
touch and how to go about designing for that modality?
The workshop series is my quest to answer some of
these questions.
180
Design and semantics of form and movement
In this paper I present the background motives,
workshop series characteristics, the structure of the
activities, some results and outcomes, and conclude
with observations and insights.
2 Background
The workshops build on previous works by numerous
researchers [1], [2], [3]. MacLean explicitly advocates
approaches and inquiries from Interaction Design for
haptic design [4].
This workshop series is the result of my professional
interest in haptics, both from a PhD student in design
perspective [5] and from an educator point of view.
Workshop B took place at Chalmers University in
Gothenburg, Sweden. Workshop C was hosted at the
University of British Columbia, Vancouver in Canada.
These two workshops welcomed students who were
more acquainted with computer science and HCI, and
less with design. Both workshops took place in classroom-like environments with limited access to tools,
materials and equipment for model making and prototyping. It is worth noting also that workshop C had an
interesting mix of participants, ranging from first year
Masters students to post doctoral researchers already
experts in Haptics.
4 Schedule and Activities
3 Workshop Series Characteristics
Creating a rich, dynamic and creative atmosphere, for
any workshop, is a very challenging task in itself. Naturally,
the art and science of planning an educational workshop
and adjusting its activities with the participants’ progress is
well beyond the scope of this article. The considerations
and details put forward in this section are restricted to
what seems particularly relevant or significant.
The workshops under consideration were realized
between October 2010 and October 2011. They were
part regular educational activities for the university
students, and part a practical form of research and
inquiries for my doctoral studies. The characteristics
of the workshops are presented in table 1.
Workshops
Host program, level
Group size
A
Interaction Design, MA level
9 participants
B
Computer Science, MA level
16 participants
C
Computer Science, MA, PhD
and Post-Doc levels
9 participants
Interaction Design, MA level
11 participants
D
The four workshops had similar programs in their
structure despite varying time constraints. The
schedules ranged from continuous 5-day period full
time to 3-5 days spread over a longer period or a
weekend, and sometimes running in parallel with other
courses. Table 2 presents a schedule used in one of the
workshops.
AM
PM
Day 1
kick-off presentation +
what is haptics + intro to
movement, mechanisms
and actuation
assignment #1
no technology
(cardboard, glue, tape,
rubber band, etc.)
Day 2
review of assignments #1 work on assignment #2
+ design process lecture + recap Arduino
+ presentation of various
actuators + assignment
#2 (3 different scales of
actuation)
Day 3
review of assignments #2 rework assignment #1 or
+ lecture about motors
#2 with Arduino control
and actuators with
Arduino
Day 4
assignment #3 (significant collective literature
challenge) + code/
review/discussion + work
hardware clinics
on assignment #3
Day 5
work on assignment #3
Table 1. Host programs, levels and group sizes of the
workshops.
Workshops A and D were realized at Umeå Institute of
Design in Umeå, Sweden. The participants were design
students generally familiar with the design process and
model making activities, but novices in programming
and electronics. The participants had access to various
workshops and facilities at the design school for building
their realizations.
final presentations, video
documentation and
debrief
Table 2. Schedule of one workshop.
Design and semantics of form and movement
181
The activities have been deliberately developed with
a progression in the use of technology. The first
assignment involves no technology, where actuation is
human powered and controlled. This first step is also
where the use deception, fakery or other Wizard of Oz
inspired techniques is usually advocated. The second
exploration introduces electricity, motors, switches
and other simple building blocks to expand on speed,
responsiveness, range and power. These new additions
are meant to broaden the range of possibilities, but
also directly highlight challenges in controllability and
durability. The third and subsequent assignments bring
in microcontroller, sensors and simple programming
(via the Arduino platform) to explore the link between
sensor and actuator, input and output, and complexity
to a certain extent. At this stage, the participants are
encouraged to build something that can be run with
minimal intervention of the creator(s). The emphasis is
put on felt sensation and variability. Figure 1 shows an
example of a standalone project that directly reacts to
the user.
Fig. 1. Project running in standalone mode, without the intervention of its creator.
The four workshops were composed of lectures,
demonstrations, impromptu and short clinics on specific
topics and team tutoring sessions. Participants were
strongly encouraged to work in teams of two or three,
and change teammates during the week. Various design
constraints and small work briefs were defined, but
participants were invited to interpret the assignment
very openly.
5 Outcomes
This section showcases some of the realizations made
during the workshops (figure 2). The various images
depict also, although very partially, some of the tools
and processes from the sessions. Additional materials,
presentation slides, photos and videos are available
online at http://www.guchmu.com/sketching-hapticsworkshops.
6 Reflections and Insights
The initial goal of conducting Sketching Haptics
workshops was to explore how the fields of Design and
Haptics could come together during a few days, with
a strong focus on prototyping and tangible outcomes.
Although the activities were generally appreciated
from the participants and the hosting educators, many
challenges remain in the pursuit of bridging Haptics and
182
Design and semantics of form and movement
Fig. 2. Various hardware sketches made during the workshops.
Design. My observations and reflections are presented
under the following two themes: Make to Learn and
Learn to Make.
7 Make to Learn
Our sense of touch has various characteristics and
capabilities that we have grown accustomed to. In our
everyday activities, we barely notice how our perceptual
processes work and the different psychophysic mechanisms involved. To tackle and learn more about our
sense of touch, it is often necessary to build some sort
of apparatus that can sense the world and/or produce
stimuli, so we can isolate and recreate a sensation
‘on-demand’. Discussing and drawing haptics isn’t
satisfactory to develop a complete understanding or to
ground design decisions. Ideas, concepts and proposals
have to be felt or experienced in some way.
This reliance on hardware translates into considerable
challenges for building and making things rapidly. The
haptic explorations are as good as your hardware
implementations. Various haptic devices are
commercially available, and could be used to explore
various haptic concepts quickly. Unfortunately,
these products are often expensive and have a very
steep learning curve, putting them out of reach for
most workshop settings. For specific haptic teaching
activities, Shaver and MacLean built a specific actuated
platform to ease the development of haptic design
explorations [6]. Despite a simplified platform and
proper documentation, it requires advanced low-level
programming and a few days of work to get up to speed
with the device, at a cost of a few hundred dollars.
8 Exhilarating Simplicity
The Sketching Haptics workshops favored a rough introduction of haptics, starting with general construction
and craft materials (paper, cardboard, foamcore, hot
glue, rubber bands, etc.) and mechanical kits (LEGO,
Meccano, etc). The priority is to first get acquainted
with our touch sense, and start discussing and
verbalizing haptic sensations. Realizations can be seen
as rather trivial and simple, but it is surprising to what
extent simple contraptions or pieced-together
rigs can trigger and initiate very rich discussions
between the participants.
As motors and other actuators are added to the
mix, a new world of haptic stimuli can be explored
and experienced, i.e. vibration, friction, pull and push
forces, etc. At the same time, evolving haptic sketches
or embryos of haptic interfaces where physical forces
and movement are present tend to result quickly into
miniature wreckages and ruined constructions. Quick,
non-committal and explorative constructions are
generally not compatible with controlled actuation
and repetitive movement.
Fig. 3. Actuation keywords used as a starting point
for design.
One line of explorations that has been helpful as a
point of departure for haptic exploration was to start
creating interface ideas that are inspired by actuation
keywords like shrink, bounce, and slide (figure 3) or
prepositions that involve movement or time (between,
around, with, etc.). These terms can be interpreted very
openly, and often have visual or auditory references to
build upon directly. It is equally inspiring to learn about
the numerous haptic illusions [7] and try to build an
apparatus to experience one of them.
9 Haptic Qualities versus Available
Resources
One main observation from the workshops was that
the selection of available actuators and materials
greatly regulate what is being built and explored by
the students. In the first two workshops (A and B),
I provided solenoids, vibrotactile and servo motors as
the main building blocks. Vibrotactile stimulus was very
common in the projects being developed, most probably
because vibrotactile motors are the easiest items to use
and control. A few participants used servo motors to
build haptic sketches, but those projects had a tendency
to be unreliable (difficulties securing the servos
Design and semantics of form and movement
183
properly) and difficult to calibrate over multiple runs.
In workshop A, the vast majority of hardware sketches
were built out of wood. Students had limited issues with
solidity and stability, in contrast to the workshops B and
C where sketches were ruined or unusable fairly rapidly.
This could be derived by the fact that all students from
workshop A had quick access to a wood workshop
nearby and a plentiful source of free scrap pieces to
work with.
During the last two workshops, I presented other types
of haptic actuators (voice-coil and piezo) and their
adoption rate seemed to match their ease of use. For
some of the more complicated actuators, I prepared
code samples and special wiring connections so students
could get going fairly rapidly. Ultimately, students only
used the actuators I simplified, no one ventured into
uncharted grounds during these intense days. In the end,
participants were enthusiastic about exploring any kind
of actuation and stimuli, as long as they could manage to
build and control it reliably. More complicated forcefeedback mechanisms and kinetic elements seemed out
of scope for these workshops.
Fig. 4. Actuated project without haptic feedback.
There is one difficulty that appeared during each of
the workshops: participants had a tendency to build
actuated projects with no direct haptic qualities. The
apparatus would be moving, spinning or doing something
without any direct action on the human body or skin
(figure 4). After the debriefing sessions, students
mentioned that the challenges of just making something
work was often overwhelming, and that they forgot the
apparatus needed to interface with humans ultimately.
After mentioning the concern, all groups were able to
quickly rectify their activities and focus on haptic output
in the realizations.
184
Design and semantics of form and movement
Despite a large selection of materials, easier toolkits
and lower barrier to entry in actuation technology, it
remains challenging to successfully experience a large
variety of stimulus and sensations over a period of a
few hours or days. One has to not only develop the
haptic concept, but often has to build the platform
to support it. It would be comparable to having to
physically build a screen or monitor every time you
develop a new visual interface idea.
In conclusion, the workshops highlighted that design
explorations in haptics are definitely possible and fruitful
to some extent. Participants engaged quickly with the
topic and found ways to build interesting apparatus and
sketches that embed haptic ideas. They could express
various details and considerations that were grounded
in their felt experiences, and have other relations to
those sensations too. Although very demanding, building
haptics to learn haptics seems like a beneficial way to get
acquainted and develop a sensitivity of the haptic domain.
10 Learn to Make
Prototyping activities, outside the haptic realm, are
challenging in their own way. Going from idea to
materialization can serve many purposes and goals [8],
[9]. It is up to the designer to identify and govern which
filters, shortcuts and perspectives are valuable and
worth pursuing or building [10].
The workshops’ activities were an opportunity to learn
about haptics, but it was equally a lesson in prototyping
and sketching in hardware [11]. As mentioned in the
previous section, the material qualities are known
to influence how things feel, for both motionless
and actuated artifacts. What made the workshops
particularly challenging was the strong time constraint
imposed on the making endeavors. Generally,
participants had more or less one day (4-6 work hours
+ overnight) to fully realize a particular assignment
or hardware sketch. This strong limitation made sure
that participants were focusing on explorative and
non-committal works, and that they could investigate
numerous tracks, iterations or alternatives over the
course of a few days.
11 Prototyping Skills, from Model Making
to Programming
The participants had varying levels of expertise
pertaining to model making, prototyping, mechanical
engineering, electronics, programming and haptics.
Roughly half of the participants were design students,
with considerable exposure and experience in creative
processes and model making skills. The other participants
had a background in computer science, and were
generally at ease with programming and sensing/control
systems. Each group was able to leverage their current
expertise, and work towards the areas unknown to them.
Haptic researchers got to learn about design process,
user-centered methods and creative explorations.
Design students were able to explore actuation control
and psychophysics, for example. All participants were
able to find a relevant and stimulating challenge, and
tap into their teammates’ expertise to advance
their learning.
The workshops emphasized a progressive use of
technology, starting with only analog and non-tech
materials on the first day, using “humanware” or Wizard
of Oz technique with simple motors the following day,
building towards a complete hardware and software
assignment for the last day. This approach doesn’t
aim to imply that more or advanced technology is
better. On the contrary, the activities are purposely
structured to veer away, as much as possible, from
complex and intricate technology often associated
with haptic interfaces. From a pure haptic perspective,
it is impossible to do without actuation and technical
devices, but I believe design inquiries and creative
explorations can help contribute fresh and relevant
perspectives to the haptic field. Designers’ views of
technology and computation are different, not better
or worse than any other engineering professionals.
a post-doc haptic researcher implemented a virtual
ball-in-a-box haptic interface (fig. 5) just within hours,
using a cardboard box, clever embedded electronics and
programming.
Fig. 6. Haptic jewelry responding to proximity of
nearby humans.
On the other side of the spectrum, a design student
was able to produce actuated jewelry pieces made from
actuators, beads and felt, which responded to proximity
with other human beings (figure 6). In most cases, the
integration of simple actuation and responsiveness
framed in relevant context/situation resulted in
ingenious haptic interface ideas.
The balance between quality and quantity of models or
sketches is particularly difficult to manage when time
and tools are limited. Participants tend to make do with
the available materials, access to tools and combined
expertise from the group’s members. During the
workshops, participants had the opportunity to learn
about prototyping, assimilate new making skills and
bring new ideas to life. In today’s knowledge economy,
the ability to make or build stuff has decreasing value.
For exploring, understanding and advancing haptics,
making skills are very valuable if not essential.
12 Conclusion
Fig. 5. Virtual ball-in-a-box haptic interface built in
one day.
Naturally, the different groups were able to realize
different kinds of haptic interfaces based on their skills
and interests. For example, one group comprising
This paper presents results, observations and insights
from four workshops on the design of haptic interfaces.
The workshops “Sketching Haptics” set out to explore
how the fields of Haptics and Design can come
together for educational purposes. The current state
of haptic development tends to promote technological
refinements over other forms of inquiries. The initial
premise behind the workshop series is that designers,
with their creativity and user-centered perspective, can
contribute and drive, along roboticists and engineers,
Design and semantics of form and movement
185
in the development of haptic interfaces. Designers
might not have the technical know-how to develop
new haptic technology per se, but designers have
instrumental skills, processes and expertise in how
people interact with technology. Haptic design activities
present numerous challenges for designers and other
professionals not familiar with haptics. The exercises
and assignments were developed in order to maximize
exposure to felt sensations and circumvent common
technological constraints.
The overall results from the workshops reveal that
quick, creative, explorative works are not only possible
but can be valuable and rewarding. The various
realizations and projects were diverse and inspiring, but
most importantly they act as catalyst and platforms for
tangibly engaging with haptic notions and concepts. That
new medium makes it possible for newcomers to relate,
feel and develop a heightened sensitivity.
In closing, I hope that by relating and exposing those
workshop endeavors and realizations, other design
students, practitioners, educators and haptic experts
will be intrigued and stimulated to initiate similar
activities in their own community. The field of haptic
design is young but developing rapidly. I foresee a
bright future where designers and haptic experts
will collectively expand our haptic design toolbox,
vocabulary and library, and help develop the next
generation of haptic interfaces that are more humane,
valuable and meaningful to us.
Invited tutorial for the IEEE International Conference on
Consumer Electronics 5. Las Vegas: IEEE.
[3] Moussette, C., & Banks, R. (2011). Designing through making:
Exploring the simple haptic design space. In Proceedings of the
Fifth International Conference on Tangible, Embedded, and
Embodied Interaction (TEI ’11) (pp. 279-282). New York: ACM.
[4] MacLean, K., & Hayward, V. (2008). Do it yourself haptics,
Part II: Interaction design. Robotics & Automation Magazine,
IEEE, 15(1), 104-119. IEEE.
[5] Moussette, C., & Fallman, D. (2009). Designing for touch:
Creating and building meaningful haptic interfaces.
In Proceedings of IASDR 2009, International Association
of Societies of Design Research. Seoul, Korea.
[6] Shaver, M. J., MacLean, K. (2003). The twiddler: A haptic
teaching tool: Low-cost communication and mechanical design,
Technical Report TR-2005-09, Vancouver: UBC Dept. of
Computer Science.
[7] Hayward, V. (2008). A brief taxonomy of tactile illusions and
demonstrations that can be done in a hardware store. Brain
Research Bulletin, 75(6), 742-752.
[8] Buchenau, M., & Suri, J.F. (2000). Experience prototyping.
In Proceedings of the Conference on Designing Interactive
Systems Processes, Practices, Methods, and Techniques DIS ’00, 424-433. New York: ACM Press.
[9] Buxton, B. (2007). Sketching user experiences: Getting the
design right and the right design. Interactive Technologies 18,
p. 448. Morgan Kaufmann.
[10] Lim, Y.K., Stolterman, E., & Tenenberg, J. (2008). The
anatomy of prototypes. ACM Transactions on ComputerHuman Interaction, 15(2), 1-27. ACM.
Acknowledgments
[11] Holmquist, L.E. (2006). Sketching in hardware. Interactions,
I would like to thank Umeå Institute of Design (Sweden)
for allowing me to teach two workshop instances with
their Interaction Design Masters students. Thanks to
Johan Sandsjö for inviting me to teach a workshop at
Chalmers University in Gothenburg (Sweden) with his
students. I am also very grateful to Karon MacLean from
UBC in Vancouver (Canada) for supporting my fourth
workshop during the summer of 2011 with her students
and research group. Her own research and continuous
effort to bridge the haptic and design communities has
been immensely inspiring to me.
13(1), 47-60.
References
Camille Moussette
[1] Hayward, V., & Maclean, K.E. (2007). Do it yourself haptics:
Umeå Institute
Part I. IEEE Robotics and Automation Magazine, 14(4), 88-104.
of Design, Umeå
[2] Levesque, V., & MacLean, K. (2011). Do-It-Yourself Haptics:
University, Umeå,
A Practical Introduction to Haptics for Consumer Electronics.
Sweden, 90187
186
Design and semantics of form and movement
Jeroen Peeters, Bram van der Vlist, Gerrit Niezen, Jun Hu, Loe Feijs
[email protected], {b.j.j.v.d.vlist, g.niezen, j.hu, l.m.g.feijs}@tue.nl
A study on a tangible interaction approach
to managing wireless connections in a
smart home environment
Abstract
1 Introduction
Technological advances in computational, networking
and sensing abilities are leading towards a future in
which our daily lives are immersed with interactive
devices that are networked and interoperable. Design
has an important role in facilitating users to make
sense of the many connections between devices in a
networked environment. Two design solutions based
on a tangible interaction approach have been developed,
that allow users to manage wireless connections
between devices in a smart living room context. One
design (Interaction Tiles) is a centralized approach
based on a high level of semantic abstraction. The
second design (Nodes) employs a distributed and
localized approach, building upon laws of grouping
from Gestalt psychology. A user experiment (n=15)
was conducted, comparing both design solutions in
the form of video prototypes, to gain insights into the
mental models users construct when using the methods.
Findings suggest that users’ mental models of the Nodes
design are more accurate representations of the actual
structure of the network and that it allows for the
projection of different mental models. Furthermore,
findings also suggest that this does not necessarily lead
to increased usability or increased perceived value.
Technological advances in computational, networking
and sensing abilities are changing the domain of
interactive product design. Visions of the future, such
as Ambient Intelligence [1], Pervasive Computing [2]
and Ubiquitous Computing [3] predict a future in which
our daily lives are immersed with devices that are networked and interoperable. Other discourses on the
future of technology, such as the “Internet of Things”
[4] and “Shaping Things” [5] predict all devices to be
connected to, or to form a new, Internet of Things.
This allows individual products and their location in
space and time to be identified.
In such worlds, interactive products no longer function,
or are interacted with, in isolation. Rather, they become
part of a larger network of products. This changes
the field of design from a “one person - one product”
paradigm into that of a world in which many products
and systems form complex networks [6].
For these highly interactive and intelligent systems
to have any merit, it is imperative that users are able
to understand and manage their content. Design plays
an important role in allowing users to make sense of
this content – the devices and connections within the
network – and to help bridge the gap between virtual
and physical worlds.
Various approaches have been developed that aim to
bridge this gap. One example is Tangible Interaction [7],
which builds upon perceptual motor-skills by presenting
Keywords
Gestalt, Product Semantics, Interaction Design,
Smart Home
Design and semantics of form and movement
187
users with physical entities that can be manipulated
to interact with virtual data. The European research
project SOFIA targets to “make information in the
physical world available for smart services – connecting
the physical world with the information world” [8].
In the context of this project, we have previously
designed the Interaction Tile [9]. The Interaction Tile is
a design based on tangible interaction that allows users
to explore and manage wireless connections between
devices in a smart living room context. The design
employs a centralized approach and builds on high-level
semantic abstractions.
We created the Nodes design to explore an alternative
design direction in the same setting. The Nodes design
employs a distributed and localized approach and
builds on Gestalt psychology’s laws of perception.
These hypothetical laws dictate expected perception
of visual information in an organized way. In this design
they are employed to visualize the otherwise invisible
wireless network. In order to gain insights into the
use of Gestalt laws to aid in designs that bridge the
virtual and real, a user experiment was conducted.
The two designs were compared in order to answer
the following research question: Is there a difference
in the user constructed mental models between the
Interaction Tile and the Nodes design? And if so,
what is this difference?
It was expected that the Nodes design would provide
users with a mental model that more accurately
resembles the underlying structure of the network,
compared to the Interaction Tile. The Nodes design
places physical objects that suggest the real architecture
of the system directly in the environment. This allows
users to perceive the network, as it exists within the
context, without requiring users to take a large step
in semantic abstraction.
2 Design
Both designs presented in the research are designed
to allow users to explore, make and break wireless
connections between media devices in a smart home
environment.
2.1 Interaction Tile
The Interaction Tile [10] allows users to explore, make
and break connections between devices in the smart
home environment. It revolves around a high level of
semantic abstraction, based on icons that represent
188
Design and semantics of form and movement
the devices in the environment. The design (see Fig.
1) is based around a central, cube-like object – the
Interaction Tile. The Interaction Tile features 4 LED
lights that provide feedback to the user about possible
as well as active connections. Smaller, cube-like objects
each represent a device in the living room. An icon on
top of the small cubes communicates what device
they represent.
Fig. 1. Interaction Tile.
When an object is placed next to the tile, the lights
give immediate feedback when the object is recognized
(Fig. 2c). When multiple objects are placed near the
interaction tile, it immediately shows the connection
possibilities (feed forward) through lighting colour
and dynamics. The lights’ colour coding is simple and
straightforward. Red colour means no connection
and no connection possibility (Fig. 2d); green colour
means there is an existing connection between the
devices present (Fig. 2a/e) and green pulsing means
that a connection is possible (Fig. 2b). To indicate that
the interaction tile did sense the first object a user
places near, it shows a red colour at the side the object
was detected (Fig. 2c). By placing a second, third and
fourth object, the interaction tile shows the lighting
effect corresponding to their connection capabilities.
By simply picking up the tile and shaking it, the user
can make or break the connection between the
devices present at the interaction tile. The result of
this action depends on the connection’s current state,
and the devices present; if the tile shows a connection
possibility, the action will result in a connection event.
The same action performed when the tile shows an
existing connection will break the connection.
We rely on the symbolic meaning of colour – green
colour meaning “proceed” and red meaning the
different (in fact rather complex) relationships at the
same time, and our expectation is that we need the
richness of all these mechanisms to successfully interact
with our complex environments and the envisioned
smart environments of the future.
2.2 Nodes
While the Interaction Tile is a centralized design –
the connections are made by interacting with a central
device, irrespective of the location of the actual devices
being connected – the rationale with the Nodes design
was to explore a different approach to allow users to
understand and manage connections between devices in
the same context. As opposed to a centralized solution
such as the Interaction Tile, which abstracts the network and takes the connections out of their context,
the Nodes design is distributed and localized.
Fig. 2. Meanings of lighting colour and dynamics.
opposite. Using the association of solid colour and
pulsing colour (indicated with solid and dashed lines)
we aim to refer to the “existence” of something and
the “possibility” of something. This something is a
connection, being invisible but with noticeable results
(functional feedback; i.e. the sound of music out of
a loudspeaker that you just connected to your MP3
player). We rely on iconic representation for the cubelike objects representing a stationary non-mobile device,
and on meaning resulting from direct manipulation of
these objects we just described (representing other
objects). People seem to be able to work with all these
The Nodes design revolves around physical objects
that represent nodes within the virtual network.
The physical nodes are small circular platforms that
are distributed in the environment, meaning they are
placed close to or onto the actual devices a user wants
to connect. Placing the nodes near devices does not
yet establish the connections between the devices.
To establish connections, users need to determine
the start and end points of connections between the
nodes. These are determined by placing flat shapes
that resemble an arrow (start point) or negative arrow
(end point) vertically onto the nodes. (Fig. 3) By aiming
a start point on one node directly at the end point of
another node, the connection between two nodes is
visualized and established (Fig. 4).
Fig. 3. Placing a Node on a device (left), placing a network start point on a Node (middle) and placing a network
end point on a Node (right).
Design and semantics of form and movement
189
Fig. 4. The side view shows how to aim the Nodes to
connect device A to device B. The top view shows two
networks: one in which device A is connected to device
B, and another in which A is connected to both devices
B and C.
The Nodes design is based on laws of prägnanz, the
main principle in Gestalt psychology. Gestalt psychology
revolves around the principle that the human mind is
holistic and that it has self-organizing tendencies in
its perception [11]. The laws of prägnanz (Fig. 5) are a
set of hypothetical laws that allow for prediction as to
how visual information is grouped according to certain
characteristics. Specifically, the Nodes design builds
upon the Law of Closure: The mind has a tendency to
complete incomplete forms, effectively seeing something
for which it does not receive stimuli. In this design, this
principle is used to visualize something that is invisible
(the virtual network) through physical objects that
represent parts of it (the nodes and start/end points).
The design also employs other prägnanz laws:
– The Law of Proximity – objects that are close to one
another are perceived to belong together. Used in the
design to communicate a node belonging to a specific
device.
– The Law of Similarity – objects that are similar in form
are perceived to belong together. Used in the design
to communicate the nodes belonging to each other
and form networks.
– Law of Good Continuation – the mind continues
visual patterns. Used in the design to communicate
connections that cross one another.
Fig. 5. Examples of the Gestalt laws of prägnanz.
solutions to create networks of devices in a smart living
room environment. Two pilot tests were conducted to
identify and repair problems concerning the set-up of
the experiment.
3.1 Participants
Fifteen participants in the target demographic of 45+
were recruited. This demographic was used in order
to gain insights into the mental models of users that
are expected to be less familiar with the networking of
interactive products than generations that grew up with
such technologies emerging. In total, eight females and
seven males were recruited. All participants indicated
that they use multiple electronic products with varying
regularity. The educational background for the
participants ranged from low to high.
3 Evaluation
A user experiment was designed to answer the research
question proposed earlier in this article. The experiment
was set up to collect data about differences in participants’
mental models, when presented with two design
190
Design and semantics of form and movement
3.2 Experiment Design
A within-subjects design was used. We employed the
Teach-Back Protocol [12], an established technique
that allows researchers to gain insights into the mental
models constructed by users. Because users’ mental
models consist of both semantic and procedural
knowledge about the system they are interacting with,
teach-back questions can be subdivided into “what is?”
questions focusing on semantic knowledge, and “how
to?” questions focusing on procedural knowledge [12].
Using such questions, adjusted to our specific situation
and research goal, we aimed to extract the semantic
and procedural concepts that are relevant for our users.
Participants were asked to explain to an imaginary peer
what they thought the system was and was for, including
listing all the components and the relationships and
connections between the components they thought
made up the system. By asking the participants to
explain to an imaginary peer how to perform a specific
task with the system, we aimed to get insights into how
well the participants understood the necessary steps
and devices involved to achieve their goal.
To support and communicate their answers to both
types of questions to the researchers and for recording
purposes, participants were asked to make drawings,
schematics or use a textual representation. The
data was collected by examining the drawings and
descriptions made by participants, as well as from
observations and recordings made by the moderator.
In the post-test discussion, participants were asked for
their feedback and preferences for the two designs.
Video prototypes [13] were used to convey the interaction and functionality of both designs to participants,
using the exact same usage scenario. Video prototypes
allow the researcher to have much more control over
the behaviour of the system, minimizing the interference
of prototyping design flaws or technical instability of the
networked devices and networks formed.
The use of video prototypes instead of real prototypes
influences the construction of mental models by the
participants, as humans learn differently when seeing
as opposed to doing. To minimize this difference,
an adaptation to the Teach-Back Protocol was
implemented: Users interacted with cardboard models
of the designs to act out their use of the systems within
context and were asked to vocalize their thoughts
and ideas during this step. This stimulates users to
form their own mental models despite the lack of
functionality in the cardboard prototypes.
3.3 Materials
The following materials were used in the experiment:
- video prototype of the Interaction Tile design
- video prototype of the Nodes design
- laptop computer to present the video prototypes
to participants
- non-functioning model of the Interaction Tile design
- non-functioning model of the Nodes design
- digital camera mounted on a tripod to record the
experiment
- six non-functioning devices that represented the
devices in the scenario (a VCR, a TV, an ambient light
that reacts to sound, a set of speakers, a CD-player
and a small radio)
- a voice recorder to record audio during the experiment.
3.4 Procedure
The experiment was conducted in a controlled
environment. An entertainment room at a residence
was furnished to resemble a living room, the context
in which both designs would be used. Participants were
presented with a video prototype of the design and
asked to complete a number of task scenarios (see next
section) using cardboard models as well as writing and
drawing. To emulate the spatial dimension of the Nodes
design, the six devices used in the task descriptions
were positioned in the environment. The devices
were turned off and to avoid unnecessary confusion
they were clearly marked. The moderator sat next
to the participants while conducting the session. The
moderator welcomed the participants, introduced them
to the experiment, supported the video prototypes with
an explanation and led the participant through the two
test cycles. The moderator took notes on the behaviour
and comments of participants, answered participants’
questions and asked follow-up questions relating to
observations and problems that arose during the test.
Every session was recorded from a wide angle using
a video camera and the audio was recorded using a
separate audio recorder. The moderator led the session
and made notes.
Tasks. Eight different tasks were created for the
experiment:
- Connect the CD player and the speakers:
The music from the CD plays back on the speakers.
- Connect the radio and the speakers:
The music from the radio plays back on the speakers.
- Connect the CD player, the speakers and the ambient
light: The music from the CD-player plays back on the
Design and semantics of form and movement
191
-
-
-
-
-
speakers and the ambient light responds to the music.
Connect the CD player, the speakers, the TV and the
ambient light: The music from the CD player plays
back on the speakers, and the TV screen and the
ambient light respond to the music.
Connect the CD player, the speakers and the TV:
The music from the CD player plays back on the
speakers and the TV screen responds to the music.
Connect the TV and VCR. Also connect the CD player
and the speakers: The VCR plays back on the TV. The
music from the CD player plays back on the speakers.
Connect the TV, the VCR, the speakers and the
ambient light: The VCR plays back on the TV, the
sound from the VCR plays back on the speakers,
and the ambient light responds to the sound.
Connect the radio, the speakers and the ambient light:
The sound from the radio plays back on the speakers,
and the ambient light responds to the sound.
Every participant was asked to perform all of the tasks
in an order that was randomized for each participant.
Each task was presented on a card, allowing users to
review the task if necessary. In addition to a simple
description of the devices to be connected, the card
also communicated the connections in context (i.e. the
music from the CD player plays back on the speakers),
in order to facilitate the participants’ understanding of
the type of connections needed and their purpose.
Participants were divided into two groups. One group
started the test using the Interaction Tile design,
after which they repeated the cycle for the Nodes
design. The other group went through the procedure
in the reversed order. After an introduction to the
experiment, participants were asked to read and sign
an informed consent form, and to fill in a short pre-test
questionnaire. The pre-test questionnaire aimed to gain
general demographic and background data from the
participant, including some general insights into their
use of electronic products.
Participants were first presented with a video prototype
of the design. This video prototype showed a user
making and breaking connections between devices
using the respective designs. In the videos, the designs
appear to be fully functional. Both video prototypes
involved the same person, in the same context and
managing the same connections. The participant was
then asked to use cardboard models of the design to
manage connections for two of the task scenarios. The
192
Design and semantics of form and movement
participant was asked to think out loud and explain what
they were doing and why, including how they expected
the system to respond to their actions.
Then, employing the Teach-Back Protocol, participants
were asked to write down a short general description
of the design they were using, as well as to explain to an
invisible friend how they conceptualized the connections
in two of the tasks, using drawings. This procedure was
repeated for the other design. Finally, the participants
were asked for anything they would like to share about
either of the designs and their preferences, and the
moderator followed up on problems or observations.
This post-test discussion ended with a short debriefing
by the moderator.
4 Results
The collected data was transferred to small cards and
analyzed using the Affinity Diagram method. Cards
were clustered based on their relation to each other,
resulting in three categories of interest, presented
in the Discussion. The results of each technique are
described in the following sections.
4.1 Acting Out
While using the Interaction Tile, three users forgot
to shake the tile, a required action to establish a
connection between the devices which icons have
been aligned with the tile. Instead, they assumed that
simply placing the icons next to the tile would establish
a connection. This had no substantial influence on the
mental models of the participants, as they still perceived
a network to be formed, and were able to explain how
they viewed the network. The meaning of the icons also
confused some participants while they were using the
tile. This did not influence their mental models of the
network, as their perception of the network and the
devices in it remained the same.
Using the Nodes design, five out of the 15 users made
mistakes in their use of sender/receiver combinations
(e.g. making a connection by pointing two senders at
each other, as opposed to a sender and receiver). Ten
participants succeeded in using the right combination
of senders and receivers consistently. Two participants
quickly recovered from this initial mistake. One
participant realized his mistake as he attempted to
describe the system. Two participants did not realize
their mistake of not using the sender/receiver forms.
One of them only used senders to connect devices.
All participants understood the importance of aiming
two senders or receivers at different nodes towards
each other. One participant placed two nodes at the
same device to establish connection to two other
devices, but this did not influence her perception of
the network and connected devices. The 14 other
participants placed the correct number of nodes at the
correct devices (one for each device). About half of the
participants required a few moments to decide what
nodes were required to send and what nodes were
required to receive. The other half was able to decide
instantly.
Two participants were unsure about the placement of
nodes relative to the device, i.e. whether they should be
on top of the device or whether they could also be in
front of the device. Most participants placed the nodes
in front of the device, while some mixed nodes on top
of and in front of the devices. None of the participants
expressed worries about height difference in placement
of the nodes.
Participants were observed to create similar networks
in different ways. For example, when performing a task
scenario that involved connecting an ambient light that
reacted to music, some participants connected the light
to the source of the audio (CD player, radio), but most
connected it to the speakers that made the audio from
the source audible.
4.2 Description
Roughly half of the participants expressed that they
found it difficult to write down a short general
description of the designs. Two participants were only
able to describe one system (the one only Nodes, the
one only Interaction Tile) and two participants were
unable to write a description at all. When participants
were observed to become uncomfortable by their
inability to describe the system, the moderator skipped
this step. For the Interaction Tile, almost all participants
described a central entity that is used to connect
everything, and which automates the establishing of
connections. For the Nodes, almost all participants
referenced the existence of two elements: a sender
and a receiver.
4.3 Teach-back Protocol
All participants were able to use a drawing to explain
how they perceive the connections in a certain scenario
using a particular design.
Concerning the Interaction Tile design, almost all
participants clearly indicated all connections to be
mediated by the central entity. They perceived all
devices as being connected to the central unit, and that
this central unit managed the connections for them.
Two participants thought the central unit managed the
connections through instructing the main device in the
network to form connections to other devices by itself.
One participant described the connections as moving
around the central unit; i.e. every device connecting
directly to another, unmediated by the central unit.
This participant did not realize the underlying necessity
for some kind of connection to exist between the
devices and the Interaction Tile in order for it to be
able to instruct devices to form connections.
Despite some participants making mistakes in their
use of sender/receiver elements in the acting out
tasks, all participants implemented this differentiation
correctly and consistently in their drawn explanations
of connections in the Nodes design.
4.4 Post-test Discussion
During the post-test discussion, some participants
expressed that they wondered what was happening
inside the Interaction Tile. They perceived it as being
automated. One participant explained that he found
it difficult to understand the system because he was
unaware of what happened inside the Interaction Tile.
5 Discussion
The study aimed to determine whether users’ mental
models differ between the Interaction Tile and Nodes
design, and what exactly this difference is. It was
expected that the Nodes design provides users with a
mental model that is more accurate towards the actual
architecture of the system than that of the mental
model created when using the Interaction Tile.
Analyzing the data using Affinity Diagrams, three
categories of results emerged from the data. The three
categories mirrored each other across the two designs,
and were merged to contrast the differences between
the two designs:
5.1 On Mental Models
For the Interaction Tile, almost all written descriptions
of the system by the participants revolved around
a central device that is connected to all devices and
manages the connections automatically. Participants
Design and semantics of form and movement
193
often referred to the Interaction Tile as the “central
unit”, “the interface to all devices” or “a magic box”.
Almost all participants also indicated in their drawings
(see Fig. 6) that they perceived all connections to go
through the Interaction Tile, where the Interaction Tile
“did something” to the signals and created the network.
This leads to the conclusion that the Interaction Tile
system creates a mental model with a centralized
hierarchy; all devices are connected to and controlled
by a central object, the Interaction Tile.
For the Nodes design, almost all participants wrote
about “senders and receivers” to make connections,
and placement of nodes near devices that need to be
connected to determine the content of the network.
In their drawings, all participants created hierarchical
connections between devices, where some devices
send data and others receive it (see Fig. 7). Participants
created different mental models of the same type of
networks, and were able to adapt the use of the system
to fit their mental model without compromising the
functionality of the network. For example, in a network
of three devices, music from a CD player plays on the
speakers and an ambient light responds to the music.
Most participants directed the signal from the CD
player to a receiver on the speakers, and relayed the
signal from the speakers to a receiver on the light. Some
participants sent two signals from the CD player, one
towards the speakers and one towards the light. This
shows a powerful characteristic of the Nodes design:
it supports users in projecting different mental models
on the system.
It can be concluded that the results of our study
support our hypothesis that the Nodes design provides
users with a more accurate mental model towards the
actual configuration of the network in the sense that
devices are directly connected to each other without
the network being mediated by a central unit. However,
as the design allows for different mental models to
be projected onto it, not every mental model of the
Nodes design is exactly the same as the network’s
real architecture.
5.2 On Symbolism and Interaction
For the Interaction Tile, some participants were
confused about the meaning of the graphical icons
on top of the blocks (i.e. which device was being
represented by which icon). Also, participants wondered
whether the location of the icons relative to each other
194
Design and semantics of form and movement
Fig. 6. A typical mental model drawing of the network
when using the Interaction Tile: all connections go
through a central unit.
Fig. 7. A mental model drawing of the perceived
network when using the Nodes design: Data is sent
from the radio towards the speaker-set, which acts
as a relay to an ambient light.
was important, although they assumed it was not.
For the Nodes design, similar problems surfaced. It
was difficult for some participants to immediately apply
the sending/receiving concept in their acting out tasks,
and some did not realize the importance of using the
right arrow-shape to send or receive a signal. In their
drawings, however, all participants used the sending/
receiving principle correctly in explaining connections.
This indicates that the system could benefit from a
better form design to allow differentiation between
the sending and receiving shapes. Furthermore, two
participants wondered about whether the location of
the nodes relative to the device was important, although
they assumed that it only had to be in close proximity.
These issues for both the Interaction Tile and Nodes
designs are similar and occurred (roughly) equally often
and for a minority of the participants. They did not
influence the mental models, as these were observed
from the Teach-Back Protocol to be consistent for all
participants, whether they identified these issues or not.
They do however point out important design issues that
can be limitations to both systems. This suggests that
further research into the semantics of sign and form in
both designs could lead to a better understanding of
the interaction required for the device, as well as
increased usability.
References
[1] Aarts, E., & Marzano, S. (2003). The New Everyday: Views
on Ambient Intelligence. Rotterdam, The Netherlands: 010
Publishers.
[2] Satyanarayanan, M. (2001). Pervasive Computing: Vision
5.3 On Value Judgment
From the observations and post-test discussions,
some data concerning the participant’s preference for
either method surfaced as well. When asked about
their opinions on both designs, almost all participants
indicated they preferred the Interaction Tile method
over the Nodes method.
They described the Interaction Tile method as being
very easy to use, as they only had to add icons of the
devices and did not need to determine what the role
of each device was. The system was perceived as being
automated and therefore experienced as the most userfriendly. Furthermore, they liked the fact that they were
able to manage the connections without getting up and
moving about the space.
Most participants indicated they found the Nodes
design easy to understand, but that it required too many
actions. Furthermore, some participants indicated that
they would not like having to place additional objects
in their living room, for which they did not see specific
merit. This suggests that although the Nodes design
provides a more accurate mental model of the network,
this does not necessarily lead to better usability.
Further research could explore design directions that
merge the merits of the Nodes design (accuracy and
clarity of the mental model, flexibility towards different
mental models) with those of the Interaction Tile design
(ease of use, perceived value).
and Challenges. IEEE Personal Communications, pp. 10-17
[3] Weiser, M. (1991). The Computer for the 21st Century.
Scientific American, Vol. 265, pp. 94-104
[4] Kranenburg, R. (2008). The Internet of Things: A critique
of ambient technology and the all-seeing network of RFID.
Amsterdam: Institute of Network Cultures.
[5] Sterling, B. (2005). Shaping Things. Cambridge, MA: MIT Press
[6] Frens, J.W. & Overbeeke, C.J. (2009). Setting the stage for the
design of highly interactive systems. In Proceedings of International
Association of Societies of Design Research 2009 - IASDR’09
(pp.1-10). Seoul, South Korea: Korean Society of Design Science.
[7] Fitzmaurice, G., Ishii, H., & Buxton, W. (1995). Bricks: Laying
the Foundations for Graspable User Interfaces. Harlow: ACM
Press/Addison-Wesley Publishing Co.
[8] Mission statement on main page of http://www.sofia-project.
eu/, Retrieved June 6, 2011.
[9] Vlist, B.J.J. van der, Niezen, G., Hu, J., & Feijs, L. (2010).
Design semantics of connections in a smart home environment.
In L. L. Chen, T. Djajadiningrat, L. Feijs, S. Kyffin, D. Steffen,
& B. Young (Eds.), Proceedings of Design and Semantics of
Form and Movement (DesForm) 2010. Lucerne, Switzerland:
Koninklijke Philips Electronics N.V.
[10] Vlist, B.J.J. van der, Niezen, G., Hu, J., & Feijs, L.M.G. (2010).
Semantic connections: Exploring and manipulating connections
in smart spaces. In Computers and communications (ISCC),
2010 IEEE symposium on. IEEE.
[11] Rock, I., & Palmer, S. (1990). The Legacy of Gestalt
Psychology. Berkeley: University of California,.
[12] Veer, C. van der, & Carmen Puerta Melguizo, M. del (2003).
Jeroen Peeters,
Bram van der Vlist,
Gerrit Niezen, Jun
Hu and Loe Feijs
Department of
Industrial Design
Eindhoven University
of Technology
Eindhoven,
Netherlands
6 Conclusion
Mental Models. In J.A. Jacko & A. Sears (Eds.), The Human
The study compared the mental models created by
users when using both a centralized and a distributed
approach as a tangible interface for configuring
networks of devices. A clear difference has been
found in the way users perceive the network and
suggests some speculation as to how this could impact
understanding and usability of such networks.
The success of the Nodes design in allowing users to
create and apply mental models to networks suggests
that Gestalt laws of prägnanz can be powerful tools in
the way in which physical artefacts can acquire meaning
in an ecology of objects, and how they can help to
bridge the gap between real and virtual worlds.
Computer Handbook. Fundamentals, Evolving Technologies
and Emerging Applications (pp.52-80). Mahwah New Jersey:
Lawrence Erlbaum Associates.
[13] Beauduin, M., & Mackay, W.E. (2008). Prototyping Tools
& Techniques. In J.A. Jacko & A. Sears (Eds.), The Human
Computer Handbook. Fundamentals, Evolving Technologies and
Emerging Applications (pp.1026-1028). New York, US: Lawrence
Erlbaum Associates.
[14] Kwak, M., Niezen, G., Vlist, B. van der, Hu, J., & Feijs, L.
(2011). Tangible interfaces to digital connections, centralized
versus de- centralized. In Z. Pan, A. Cheok, W. Muller, & X. Yang
(Eds.), Transactions on Edutainment V (Vol. 6530, p. 132-146).
Springer Berlin/Heidelberg.
Design and semantics of form and movement
195
Daniel Davis, Jane Burry, Mark Burry
[email protected], [email protected], [email protected]
Yeti: Designing geometric tools with
interactive programming
Abstract
Designers scripting geometric tools have had two
options: either use an interactive visual script, or forgo
interactivity to use a text-based script. Within this
paper we consider a third option: interactively writing
text-based scripts. Described is an interactive scripting
environment created for this purpose, which manages
geometry with a Directed Acyclic Graph generated
from the text-based relational markup language, YAML.
The environment is compared to the two existing
scripting options by using them to draw three geometric
compositions. We argue it is possible to interactively
script geometric tools, and that interactivity is a vital
component in making scripting intuitive.
Keywords
Interactive programming; End-user programming;
Design computation; Parametric modelling.
1 Introduction
Since Sutherland’s digital Sketchpad, designers have
aspired to make coding more like sketching. Prior
to Sutherland, computer programs were manually
executed in ‘batches’. The designer would compile the
code, define inputs and parameters, run the program
and wait – often a long time – for the result. When
Sutherland developed Sketchpad in 1963 he sought to
overcome the delays in batch processing and allow “man
and a computer to converse rapidly.” [1]. Sketchpad
196
Design and semantics of form and movement
was one of the first interactive computer programs.
It displayed the results of the designer’s actions
immediately, which facilitated feedback and reflection in
a conversation between designer and computer.
Almost 50 years after Sketchpad, interactive graphics
programs have all but replaced the drawing boards
they once imitated. These design programs each offer
a prescribed palette of design tools and often afford
designers the ability to script their own customised
tools. A script defines a list of operations for the
computer to carry out. Yet when designers attempt to
design their own design tools with scripts, they must
once again design using a batch-processed system. This
is because scripting in its current form involves writing
out a procedural script, pressing compile, setting the
inputs for the script, and waiting for the computer to
draw the result – like the programs prior to Sutherland.
Unlike sketching, or even digital drafting, with scripting
there is a pronounced delay between the action of
the user (changing the code) and the reaction of the
system (redrawing the geometry). Such a delay can slow
the pace of iteration in the design process and hinder
feedback reaching the designer in a timely manner.
Recently a number of scripting languages have emerged
for musicians that enable the interactive modification of
text-based scripts. Using these interactive programming
languages the musician can immediately hear how
changes to a script driving a musical performance will
sound. These languages appear to be a viable method
for achieving a similar level of interactivity in geometric
design. However, as will be outlined in this paper, the
emphasis musicians place on tempo and timing makes
their techniques unsuitable for the computationally
intensive task of generating geometry.
With no existing interactive scripting tools for describing
geometry, this research seeks to better understand the
technical and cultural limits of designing geometry with
interactive scripts. This paper begins with an outline
of an interactive scripting technique that overcomes
some of the computational impediments associated
with the interactive scripting of geometric tools. ‘Yeti’
is an interactive scripting environment developed to
utilise this technique, the implementation details of
which are explained in this paper. Using a reflective
practice methodology, Yeti is tested in a pilot study
by applying it to three geometric design problems and
comparing its performance to that of non-interactive
text-based scripts and interactive visual scripts. The
three design problems are taken from an architectural
context, although it is anticipated this research will be
of interest to designers outside the field of architecture,
particularly those describing geometry with their own
scripted design tools or parametric models. The paper
begins by describing some of the existing scripting
environments designers utilise.
2 Existing Design Scripting Methods
A script defines a list of actions for the computer
to carry out. In contemporary usage, scripting is
essentially synonymous with programming. As such
a script can automate tasks that would otherwise be
performed through the user interface and it can also
define entirely new actions. For designers the primary
motivations to script are: productivity (doing tasks that
would take too long otherwise), and control (linking
various actions together to create customised tools)
[2]. Design software packages often encourage scripting
through inbuilt scripting interfaces, and there are also
applications (like Processing) that are created explicitly
as standalone scripting interfaces for designers. With
design increasingly being conducted on computers,
so too scripting has increasingly become a way for
designers to automate and control the design process.
To run a script, the computer generally has to interpret
(or compile) the script into a machine-readable set of
instructions. This is supported in scripting interfaces
through an ‘Edit-Interpret-Run’ loop, whereby the
designer edits the text of the script, presses a button
to activate the script, and waits first as the computer
validates the script, then waits as the computer
interprets the script into a machine-readable set of
instructions and finally waits as the computer runs this
set of instructions. The notable exception is some visual
programming languages, like Max-MSP, which will be
discussed in the subsequent section. As a consequence
of the Edit-Interpret-Run loop, there is a pronounced
delay between the action of the user (editing the
script) and the reaction of the system (redrawing the
geometry). This delay impacts the rate of iteration
since each variation of the script the designer tests goes
through the Edit-Interpret-Run loop, often with the
designer manually deleting the geometry of the previous
loop between iterations.
3 Interactive Scripting
Interactive programming (also known as liveprogramming) is a method for editing and interpreting
scripts while they run. To the end user there appears
to be no Edit-Interpret-Run loop because any edit they
make is automatically incorporated with the already
running instance of the script. Behind the scenes there
is still an Edit-Interpret-Run loop, where the computer
automatically interprets an edit and in real-time invisibly
transitions the running instance of the script to the new
edited version. The net effect is that the end-user can
experience in real-time the consequences of editing a
script – closing the gap between action and reaction.
The crux of creating an interactive programming
environment is smoothly transitioning a running script
between different versions of the script. The most
obvious method is to abandon the currently running
script whenever it is edited, and automatically interpret
and run the updated version of the script. For certain
applications, such as SimpleLiveCoding for Processing
[3], this is effective. However for the computationally
taxing task of drawing geometry this is less desirable
since it involves abandoning all the previous calculations
and recalculating the geometry every-time the script is
modified, even if the modification only changed a small
and discrete part of the geometry. The method typically
employed in interactive debugging is to maintain the
state of the code – through a call stack – allowing the
code to be rewound to the site of the edit [4], however
all subsequent code still needs to be recalculated, even if
it is not affected by the modification.
Design and semantics of form and movement
197
Perhaps one of the most developed systems for
transitioning scripts has been developed by musicians,
for whom interactive programming allows modification
of scripts driving a musical composition while
immediately experiencing the sonic implications.
The first performance with an interactive script was by
Ron Kuivila at STEIM in 1985 [5]. In 2000, Supercollider
led a revival of text-based interactive programming
for musicians, and was followed by a number of similar
languages like ClanK and Impromptu. All of these
languages share in common the need from musicians to
invoke actions relative to an underlying time signature.
Typically this occurs through scheduling a reoccurring
sequence of actions to be performed, and whenever
the script is modified, adding the modified actions to
the queue [6]. These musical environments have been
adapted to generate geometry but the repetitive cycling
of actions makes it unsuitable for generating anything
other than basic geometry [7].
Therefore despite the scattered implementations
of interactive programming, few – if any – are suitable
for the unique challenges designers face when shaping
geometry with scripts. Designers desiring the interactive feedback of sketching while they script currently
have to make do with Edit-Interpret-Run loops. This is
primarily due to the difficulty of editing and updating an
already running script while handling the computational
intensity of geometric calculations.
is defined through visual interfaces in Grasshopper,
Houdini and Generative Components. Text-based
scripts within these environments cannot not be interactively edited and still use the Edit-Interpret-Run loop.
Yeti is a text-based interactive scripting environment
developed for the interactive creation of geometric
tools. Yeti uses a DAG to manage the editing and
calculation of geometry, but the DAG is defined through
a text-based script rather than the visual interfaces used
by Grasshopper, Houdini and Generative Components.
The language of Yeti’s script is based on the relational
mark-up language YAML [9]. The syntax consists of
‘key: value’ pairs, where the key is assigned the value
to the right (the ‘x:’ in Table 1, has a value of −10).
More complex values can be assigned through a list
of ‘key: value’ pairs, separated from the parent key
with indentation (the ‘point:’ in Table 1, has been
assigned ‘key: value’ pairs for x, y & z). Relationships are
defined by naming keys (names start with the ‘&’) and
referencing them as a value (references start with the ‘*’).
Yeti (YAML)
Directed Acyclic Graph
Geometry
point:
››x: −10
››y: 10
››z: 13
4 Introducing Yeti
The problem of editing a script while it runs geometric
calculations has been elegantly overcome by the interactive visual scripting environments GrasshopperTM,
HoudiniTM and Generative ComponentsTM. These
three environments use Directed Acyclic Graphs (DAG)
to represent relationships between geometry [8].
A relationship may be that a line is tangent to a circle
(the circle is a parent of the line) and whenever the
circle is adjusted, the line moves to satisfy the tangential
relationship. From these geometric relationships the
dependencies of the geometry can be extracted. Thus
when part of the DAG is edited, the only recalculation
required is to the geometry dependent (and therefore
affected) by the edited part of the DAG [8]. Since a
node within the DAG is directly associated with the
geometry it creates, the node can manage the creation
and deletion of geometry without the designer needing
to remove old instances of the geometry. The DAG
198
Design and semantics of form and movement
Table 1. A simple Yeti script in YAML (left) and the
corresponding DAG (centre) with the geometry it
produces (right). Note all keys in the Yeti script map
directly to nodes in the DAG.
The advantage of using YAML is that the ‘key: value’
pairs map directly into a Directed Acyclic Graph, where
the key defines a node and the value defines either:
the property of the node, or its relationship to other
nodes (see Table 1). Whenever a script is modified in
Yeti, the underlying DAG is automatically updated in the
following process:
1. The edited script is tokenised into keys and values.
2. For every key, a corresponding node is generated in
the DAG.
3. The node is assigned properties and related to
other nodes based on the values assigned to the
corresponding key.
4. Once the DAG is created, all nodes dependent upon
deleted, added or modified nodes are recalculated,
creating a new instance of the geometry.
In addition to interactive editing of running scripts, the
YAML language and underlying DAG enable a number
of unique features in Yeti:
‡Error handling: The interpretation of code while it is
being written frequently causes errors because the
computer is often unable to resolve the ambiguity of
partly written code. Yeti interprets and run scripts
with errors by ignoring ‘key: value’ pairs with errors
in them. Typically errors cannot be ignored in other
languages because it interrupts the top-to-bottom
progression of logic.
‡Interactive debugging: Clicking a key in the code
highlights the geometry controlled by the key. This
helps clarify the often-enigmatic connection between
code and geometry that characterises scripting. Yeti is
able to do this because keys in the script are directly
associated with parts of the model’s geometry via
nodes in the DAG.
‡Instancing of objects: The YAML language can be
extended to include new keys. The user does this
by creating a prototype object for the key through
a list of ‘key: value’ pairs. When the new key is used
in the script, a new instance of the prototype object
is created and modified for the unique properties
of that object instance. This is a common feature in
text-based scripts but one that visual scripts often
do not support.
Python
Yeti (YAML)
P1 = Rhino.Geometry.Point3d(0,10,13)
P2 = Rhino.Geometry.Point3d(P1.X, P1.Y+20,0)
myLine = Rhino.Geometry.Line(P1,p2)
doc = Rhino.RhinoDoc.ActiveDoc
doc.Objects.AddLine(myLine)
line: &myLine
start: &p1
x: 0
y: 10
z: 13
end: &p2
x: *p1.x
y: (*p1.y + 20)
Table 2. Comparison of scripts to draw the same
constrained line in Rhino Python and Yeti.
YAML also has its drawbacks. The definition of
geometric properties and relationships in YAML is a
significantly different method of scripting compared
to the ordered list of procedural actions familiar to
many scripters (the two paradigms are compared in
Table 2). Similarly the recursion offered in procedural
languages is not yet possible in Yeti due to the difficulty
of representing recursion in a DAG. For this reason Yeti
is not Turing-complete, and therefore unsuitable for
certain operations like L-systems and cellular automata.
Despite these quirks and limitations, YAML and the
underlying DAG is fundamental to empowering the
interactive scripting of geometric calculations, along
with a number of other advantages like interactive
debugging and error handling.
5 Designing Geometry with Interactive
Scripts
5.1 Method
To explore the viability of interactively generating
geometric tools with text-based scripts, we carried
out three design projects with the iterative scripting
environment Yeti (version 0.3). As a benchmark we
repeated the work with two established methods of
scripting: interactive visual scripting in Grasshopper
(version 0.8.0052), and text-based scripting with
Rhino Python (In Rhino5, version 2011-11-08). The
three design problems have an architectural bias but
the focus of the analysis is towards the shaping of
geometry and designers who do so already through
scripting or parametric modelling. Since this is the first
time interactive scripting has applied in this context, the
investigation is a pilot study designed to identify the major
issues with interactive scripting in anticipation of refining
Yeti further. The three projects used in the study are:
Axel Kilian developed a pair of tutorials in 2005 to
teach the then highly experiential visual scripting
software, Generative Components. The tutorials
demonstrate how to developed a customised geometric
tool with scripting and introduce “several key
parametric modelling concepts,” such as: geometric
constraints, data arrays, modularity, and aggregate
difference from topological similarity [10]. These two
roofs form an interesting benchmark, both because they
employ essential scripting techniques, and because they
hold some historic credence with which it is possible to
track the development of parametric modelling.
Design and semantics of form and movement
199
Fig. 2. Project one in Yeti (left)
the more complicated project
two in Yeti (right).
Fig. 3. Full-scale prototype at
Smart Geometry (left) and
associated digital model (right).
As part of the Responsive Acoustic Surfaces workshop
at Smart Geometry 2011, two acoustic walls were
developed to test the sound scattering of various
plaster hyperboloid tile configurations [11]. Originally
the wall was designed with the interactive visual
scripting environment Grasshopper, used alongside
Digital Project and Open Cascade. From the workshop
it is known the project pushes the limits of interactive
design through the computationally expensive
calculation of the hyperboloid intersections, where
very subtle nuances in the planarity of the intersections
determine the project’s viability.
5.2 Differences between scripting environments
The following section broadly describes the main
differences between the three scripting environments,
with a focus on the technical capacity of each
environment.
Geometric output. The geometric library for Yeti is still
being developed but it was capable of creating the
geometry of the Kilian Roofs and creating the geometry
of the hyperboloid wall, as was Grasshopper and
Python. In all the environments the most challenging
geometric task was to encode the reasoning for which
side of the hyperboloid intersection to keep in project
three. The difficulty of expressing this indicates that
200
Design and semantics of form and movement
certain types of architecture are more amenable than
others to the logic of scripting, a logic Yeti follows.
Script length. The number of lines of code in the Yeti
scripts were essentially identical to the Python scripts,
although the lines of the Yeti scripts tended to be
sparser containing just ten characters on average,
whereas the lines in Python contained 25 characters
on average. The Grasshopper schemas are not directly
comparable to text-based scripts, but it should be
noted that the interface for Grasshopper did many
of the tasks that needed to be explicitly defined in
the Python and Yeti scripts, such as making geometry
visible. In the Python scripts, significantly more of the
script was involved with managing arrays of data, but in
Grasshopper and Yeti arrays of data were resolved by
the software rather than the user [12].
Speed of execution. Yeti remained responsive throughout
the two roof projects. On the more complex roof an
update cycle typically took 100ms, which was faster
than one can type. This is comparable to Grasshopper
and faster than the Python script, which took 2 seconds
to execute (Python’s biggest hindrance seems to be
the way it draws geometry). The intersections in the
geometry of the hyperboloid wall were too complex to
calculate in real-time with either Grasshopper or Yeti.
It was only possible to complete the project by disabling
the interactivity and reverting back to the manual Edit-
Interpret-Run loop employed by scripts like Python.
While interactive programming is useful on simple
projects, batch-processing is still a useful paradigm
to grind out computationally expensive geometry
and a useful paradigm for Yeti to fall back on.
5.3 Discussion: Intuition and Interactivity
In creating Sketchpad, Sutherland not only created
the first interactive CAD tool but also one of the
first programs to “eliminate typed statements (except
for legends) in favor of line drawings” [1]. Sutherland
described controlling a computer with text as “writing
letters to rather than conferring with our computers”
[1]. It is an argument about whether designers find
interactive drawing more intuitive than writing code.
In the past 50 years, despite the increasing prevalence
of scripting, overwhelmingly CAD software consists
of interactive visual interfaces activated with mouse
and keyboard shortcuts.
However for certain types of geometry, like the geometry in the three projects above, scripting is the only
method of productively generating and controlling the
geometry. For these projects designers have no option
but to ‘write letters’ to the computer sent via the EditInterpret-Run loop. In writing these letters, some of
the intuitiveness is bound to the language it is written
in. The scripts from Python and Yeti, while of a similar
length, are strikingly different in approach (See Table 2)
the Python scripts methodically working through a list
of actions while the Yeti scripts begin with the outcome
and describe the necessary parameters. For this
reason Yeti may seem unintuitive to designers already
conversant with procedural scripting languages like
Python [12]. Whether users new to scripting experience
this difference in intuition remains to be studied.
Another factor in the intuitiveness of letters written to
the computer is the speed they are returned. In carrying
out the three projects above, it is clear intuition and
interactivity are tightly coupled. Being able to click on
words in the Yeti script and see the geometry they
control highlighted, helps clarify their purpose. Similarly
being able to edit a parameter and instantly see the
geometry respond, makes manipulating the parameters
feel more intuitive.
While writing code often feels like ‘writing letters’, the
three projects above begin to uncover how interactivity
can make scripting a more conversant and therefore
intuitive experience. It remains to be seen if the
advantages interactivity brings are enough to overcome
the hindrances of needing to use a language like YAML.
The cultural implications of such a change could be
profound, particularly if scripting became intuitive enough
to use on projects other than those that can only be
productively generated and controlled with scripts.
6 Conclusion
Sutherland’s digital Sketchpad placed interactivity at the
foundation of digital design. When scripting designers
have had two options: either use an interactive visual
script, or forgo interactivity in favour of writing the
script with text. This paper has articulated a third
option: writing a text-based script in an interactive
programming environment. Significantly this research
has demonstrated it is possible to interactively program
computationally-intensive geometric tools. This can be
achieved by managing the geometry with a Directed
Acyclic Graph, which can be generated from a textbased relational markup language like YAML. The markup language used to attain the performance necessary
for interactive scripting may seem unusual compared
to the conventions of established methods of design
scripting, however there is a real benefit to being able
to instantly see how a change to the script will affect the
model’s geometry. In the future interactive programming may make the act of writing code as responsive
for designers as the act of sketching in a Sketchpad.
Acknowledgments
This research was funded as part of the Australian
Research Council Discovery Grant ‘Challenging the
inflexibility of the flexible digital model’ lead by Mark
Burry, John Frazer and Jane Burry.
References
[1] Sutherland, I. (1963). Sketchpad: A Man-machine Graphical
Communication System. PhD Thesis, Massachusetts Institute
of Technology.
[2] Burry, M. (2011). Scripting Cultures. West Sussex: John
Wiley & Sons Ltd.
[3] Simple Live Coding. Retrieved November 19, 2011
from https://github.com/fjenett/simplelivecoding.
[4] Johansson, O. (2011). Describing Live Programming Using
Program Transformations and a Callstack Explicit Interpreter.
Masters Thesis, Linkoping University.
[5] Roads, C. (1986). The second STEIM symposium on interactive
composition in live electronic music. Computer Music Journal, 10(2),.
Design and semantics of form and movement
201
[6] Wang, G., & Cook, P. (2004). On-the-fly programming:
Using code as an expressive musical instrument. In Proceedings
of the 2004 International Conference on New Interfaces for
Musical Expression (NIME). Hamamatsu: Shizuoka University.
[7] Sorensen, A. (2005). Impromptu : An interactive
programming environment for composition and performance.
In Paper presented to the Australasian Computer Music
Conference 2005. Brisbane: ACMA.
[8] Woodbury, R. (2010). Elements of Parametric Design.
Oxon: Routledge.
[9] Ben-Kiki, O., Evans, C., & Net, D. (2009). YAML Ain’t
Markup Language (YAML™) Version 1.2. 3rd.
http://www.yaml.org/spec/1.2/spec.html
[10] Woodbury, R., Aish, R., & Kilian, A. (2007). Some
patterns for parametric modeling. In Proceedings of the 27th
Annual Conference of the Association for Computer Aided
Design in Architecture. Halifax, Nova Scotia.
[11] Burry, J., Davis, D., Peters, B., Ayres, P., Klein, J., Pena de
Leon, A., et al. (2011). Modeling hyperboloid sound scattering:
The challenge of simulating, fabricating and measuring.
Proceedings of Design Modeling Symposium Berlin. Berlin:
Springer Verlag.
[12] Janssen, P., & Chen, K. (2011). Visual Dataflow Modeling:
A Comparison of Three Systems. Proceedings of CAAD
Futures 2011. Liège: Les Éditions de l’Université de Liège.
Daniel Davis,
Jane Burry,
Mark Burry
Spatial Information
Architecture
Laboratory, RMIT,
Australia
202
Design and semantics of form and movement
Chin Koi Khoo
[email protected]
Towards a responsive architectural
morphing skin
Abstract
The typical application of responsive architecture
is in the facade or skin of a building with mechanical
joints actuating the kinetic transformation. This
paper investigates the unexplored ‘soft’ approach
using lightweight elastic form-changing materials
provides an opportunity for designing responsive
Architectural Morphing Skin (AMS). This idea is inspired
by the current morphing technologies of aerospace
engineering especially in the area of morphing wing
research. The research aims to develop a method
for designing an AMS with a passive and active design
strategy to minimise mechanical operation and reduce
weight. Using a practice-based methodology, an AMS
prototype is generated through a development process
namely soft kinetic system (SKS). AMS serves as a
‘second skin’ brise-soleil in the form of a canopy to
an existing courtyard space. It responds in real-time
to environmental stimuli to address two fundamental
criteria: comfort and visual.
Keywords
Kinetic, responsive, elastic, form-changing materials,
morphing skin.
1 Introduction
Architecture has typically resisted kinetic motions,
however, the technological advancement for
designing dynamic screens and animated surfaces
are new opportunities for designers and architects
[1]. Precedents in these architectural approaches
often involve the design of kinetic skins that are
transformable. These architectural skins also provide
a screen between people and the natural environment,
offer rich possibilities for visual expression and new
architectural vocabulary [2]. This design approach has
been explored since the 1960s, with one of the first
examples being the responsive brise-soleil of LA County
Hall of Records designed by Richard Neutra in 1962 [3].
However, the design of such kinetic skins often includes
complicated, intricate and heavy mechanistic elements
such as joints, actuators and control system for dynamic
responsiveness. The kinetic skin of L’Institut du Monde
Arabe in Paris designed by French architect Jean Nouvel
in 1987 is a salient example of this approach [4]. These
solutions involving conventional mechanical components
like multiple pistons to actuate transformation require
designers to deal with high energy costs and complex
mechanisms. Those piston components were found to
be prone to fatigue-failure, causing gasket leakage from
the piston [5]. This approach often produces brittle
and vulnerable kinetic systems. Thus, the reliability and
longevity of the system is the main hindrance for kinetic
architectural skin to be a mainstream approach in
architectural design. In this paper we investigate how to
design kinetic architectural skins with fewer mechanical
operations and explore the potential for elastic and
form-changing materials to be used as an alternative
Design and semantics of form and movement
203
approach to designing responsive kinetic skins.
This research investigates beyond the conventional
mechanical kinetic skin approach by exploring the ‘soft’,
lightweight, flexible and form-changing materials for
responsive kinetic architectural skins.
Current projects intend to investigate the ‘softness’
of the kinetic architectural skin in various attempts.
The recently completed Media-ICT building designed
by Cloud 9 Architects in Barcelona demonstrated
the energy efficiency and implementation of the ‘soft’
approach to kinetic architectural skin. The complex
façade made of ETFE is responsive to user necessity.
The ETFE skin protects the interior when there is too
much direct sunlight. When light is needed it opens
itself to let the daylight in [6]. This pneumatic kinetic
shading device sets an early inspiration to conduct this
research work; however, further investigation is needed
especially in terms of air ventilation and shadow casting.
Other relevant projects such as ‘Kukkia’ and ‘Vilkas’
designed by Joanna Berzowska and Marcelo Coelho,
investigated the ‘soft’ approach of kinetic electronic
garments that integrate the Nitinol and custom control
electronics to move and change on the body in slow,
organic motion [7]. Coelho further explores this idea in
the project namely ‘Shutter’ as a permeable surface for
environment control and communication [8]. Although
these projects provided an insightful knowledge of
using active form-changing materials to design ‘soft’
kinetic textile skins, further investigation is needed
especially in terms of responsiveness, adaptability and
scalability of these systems. Although the ‘Living Glass’
project by David Benjamin and Soo-in Yang integrated
the Nitinol actuator and sensor to form a responsive
kinetic skin, it focuses on the transformation within the
local ‘openings’ for ventilation purposes [9]. There is an
unexplored area for global topological transformation
on the entire surface of this project.
This research intends to expend the repertoire of
current responsive kinetic skin design by developing an
alternative method to integrate form-changing materials
with computational processes to design Architectural
Morphing Skin (AMS). This project explores the passive
and active design strategies for AMS with minimised
mechanical actuations. In this context, the proposed
concept of Soft Kinetic System (SKS) embodied through
a design experiment, called Blind, serves as a second
204
Design and semantics of form and movement
skin brise-soleil in the form of a canopy to an
existing courtyard space that responds in real-time
environmental conditions to serve two fundamental
purposes: comfort and visual. The main idea behind
deploying SKS as a development process for designing
AMS is the integration of an exoskeleton structure, and
the surface to act as the actual actuator. As an early
hypothesis, the elastic nature of these structures is able
to accommodate responsive mechanisms with passive
elastic memory while minimising the energy and weight
required for actuation. This paper aims to explore
the passive and active design strategies for responsive
AMS through soft kinetic system integrated with
parametric design tools. This study describes a new
repertoire of responsive architectural morphing skin
ideas using accessible ‘soft’ components such as elastic
form-changing materials embedded with contemporary
sensor devices. This approach intended to provide
alternative methods for designers and architects to
design responsive architectural skins.
2 Soft Kinetic System
The concept of soft kinetic system (SKS) is inspired by
the soft mechanical approaches in aerospace engineering
especially morphing wing technology [10]. In the field of
engineering, the word morphing is used when referring
to continuous shape change that is, no discrete parts
are moved relative to each other but one entity deforms
upon actuation [11]. For example, on an aircraft wing
this could mean that a hinged flap would be replaced
by a structure that could transform its surface area and
camber without opening gaps in and between itself
and the main wing [12]. This concept of morphing skin
is an emerging aerospace technology that has influenced
aircraft wing design. It has remains unexplored territory
in terms of designing responsive kinetic architectural
skins.
Soft kinetic system (SKS) does not require mechanical
joints, hinges, or motors. The kinetic actuation also
takes place in the overall system with the use of
form-changing materials and little use of mechanical
components. SKS served as the proposed method of
inquiry for designing Blind. Table 1 shows SKS included
four focus areas, Elasticity, Tensegrity, Form-changing
materials and Adaptability for individual component
implementation and goal:
Focus areas
Components
Goal
Elasticity
Architectural skin
Flexibility
Tensegrity
Skeleton
Transformation
Form-changing materials
Actuator
Actuation
Adaptability
Sensor
Sensing
Table 1. Focus areas, components and goal of SKS.
Fig. 1. Two different types of triangular skin modules
embedded with tensegrity tetrahedral components.
The first area used elasticity as a design strategy
to explore the new possibilities of the flexibility of
architectural skin. The most common elastic materials
deform when force is applied to them and that
deformation is reversed to return to the same state
once that force is removed. There is potential energy
stored within the material itself that can be harnessed
to activate the acceleration back to its original
state. This offers potential new forms of flexibility,
adaptability and deformation using the memory effect
in architectural skins. The second area focused on the
tensegrity structural approach that reduces the friction
between mechanical joints and achieves a lightweight
structure. Due to the interdependent nature of all the
elements, a slight change in any of their parameters
can result in a significant form transformation [13].
For these reasons the tensegrity skeleton was chosen
as part of the SKS and for its flexibility and lightweight
components. The third area discussed form-changing
materials that by nature dynamic and deformation
occur under electrical stimuli to produce expansion
and contraction which can harnesses for actuation of
the SKS. We proposed shape memory alloys (SMAs)
as the active form-changing materials to investigate
the alternative of actuation. The last area explores
the adaptability of SKS in order to achieve morphing
skins that display elastic properties, able to respond to
digital and physical stimuli. This idea is developed using
parametric design tools discussed in subsection 3.4.
3 Designing a Prototype for Architectural
Morphing Skin
In order to implement the soft kinetic system (SKS),
this section discusses the design experiment, called
Blind, to investigate AMS to exhibit comfort and
visual applications. Blind is the multilayer AMS that
included four components: Skin, Skeleton, Actuator
and Sensor formed by two basic types of triangular
modules (Fig.1). Each of the components works to test
the implementation of Blind as a responsive AMS. The
four-pronged developmental process of SKS: Elasticity,
Tensegrity, Form-changing materials and Adaptability
as the methods of inquiry for designing Blind, will be
discussed in subsections 3.1-3.4.
3.1 Elasticity
We explored the flexible performance to the skin
of Blind using lightweight elastic material. The elastic
material used for this experiment was silicone rubber
and it forms the basic non-load bearing membrane
surface for the architectural envelope. The elastic
nature of this skin is able to accommodate responsive
mechanism with passive elastic memory while
minimising the energy and weight for actuation [14].
The skin surface of Blind is fabricated using silicone
rubber because of its heat resistance and elastic
capacity (Fig. 2). In general, silicone rubber offers good
resistance to extreme temperatures from -55 to 300
degrees Celsius. Under these extreme temperatures,
the properties in terms of the elongation, compression,
tear and tensile strength are far superior to conventional
soft and elastic materials. Conventional organic rubber
has a carbon to carbon backbone which can make it
susceptible to UV, heat, ozone and other ageing factors
that silicone rubber can withstand [15]. Thus, this heat
tolerant material property makes silicone rubber a
suitable material to integrate with active form-changing
materials (SMAs) to form morphing skin, addressing
elasticity and actuation respectively. In addition, the skin
itself serves as part of the actuation as well as structural
component of the overall modular tensegrity system of
Blind (Fig. 3).
Design and semantics of form and movement
205
Fig. 2. Silicone rubber in the mould (left). Laser cutting the openings of ‘eye’ apertures with embedded SMA wires (middle).
RModular silicone skin (right).
Fig. 3. Initial experimentation for elastic silicone rubber skins set as the structural components as well as actuators.
Fig. 4. ‘Reduced’ version of tetrahedral module to minimise weight (left). Tetrahedral modules formed the
elastic tensegrity skeleton of Blind (right).
Fig. 5. Physical fabrication of Type 1 and 2 tensegrity skin modules.
206
Design and semantics of form and movement
3.2 Tensegrity
The term tensegrity coined by Buckminster Fuller
by combining the words tensional and integrity is
a structural principle based on the use of isolated
components in compression inside a net of continuous
tension, in such a way that the compressed members
do not touch each other and the pre-stressed tensioned
members delineate the system spatiality [16]. This
approach provides the structural flexibility that is
suitable to implement for the skeleton of Blind. The
materials used for assembling this skeleton included
using accessible ABS (Acrylonitrile Butadiene Styrene)
as a primary lightweight and strong explicit material for
a ‘reduced’ version of tetrahedron. It integrated with
stainless steel wire as tension component to fabricate
the tensegrity tetrahedral module with reduced
components used as exoskeleton structure (Fig. 4).
The elastic silicone rubber embedded with tensegrity
tetrahedral modules formed two different types of
larger scale modules (Fig. 5). They served as the basic
components to fabricate Blind for structural integrity
as well as actuation purposes. The integration of the
modular tetrahedral skeleton and elastic silicone skin
that fabricate the overall continuous surface of Blind
in order for transformation to take place.
3.3 Form-changing Materials
As a result of comparing multiple form-changing
materials, we found that shape memory alloy (SMA)
is the most suitable material for designing soft
kinetic system (SKS). Although EAPs (Electro Active
Polymers) have been used widely in robotic research,
EAPs-based actuators are still exhibiting low force
below their efficiency limits, are not robust, and are
not available as commercial materials for practical
application considerations [17]. Furthermore, they
require high activation field (>150V/um) close to the
breakdown level. Since the 1960s, SMAs have been
the most accessible form-changing materials in the
present market, and there are many applications in
the aerospace and automobile sector [10]. They are
commonly used in a wire or spring form that contracts
in length when heat is applied. The heating can be
done directly via electricity to give electrical actuation.
SMAs expand by as much 8% when heated and cooled.
When the SMA is below the ‘transform’ temperature
(60 degrees) the material takes on an ‘elongated’ and
neutral form, but if heated it contracts and returns to
the ‘memorised’ form. This process creates a dynamic
range in the way that the SMA wire expands and
contracts for various state changes (Fig. 6).
Fig. 6. Deformation of SMA wire occurred when
heated by electric stimuli.
Ordinary metal alloys have an internal structure that
is not altered by small temperature or electric current
changes. Electrical stimuli create heat causing the atoms
of the metal to vibrate faster and this makes it easier to
bend when an external force is applied. The molecular
form of the metal is not normally altered by heating.
However, form-changing materials such as SMAs are,
by nature, dynamic and deformation occurs under
electrical stimuli in this experiment, using: 5V for a 3
amp current. There are two stable crystalline states in
their structures that are triggered when a temperature
change occurs. Thus, SMAs are selected to implement
in this research and develop further because of their
accessibility, reliability and low electric current usage.
This form-changing process produces expansion and
contraction that can be harnessed for actuation of
the whole kinetic system.
Figure 7 shows four potential profiles for ‘Soft’
actuation based on the process of expansion and
contraction in specific parts of the SMA wire (Fig. 7).
While profile one and two show the potential for
the pull and push actuation, profile three and four
functioned as the spring system that can actuate
greater distance and force. They demonstrate that an
alternative actuation system can be embedded in the
overall tensegrity structure discussed in subsection 3.2.
for various transformation purposes. Profile one and
four are selected to use the form of actuation to Blind
in terms of transformation for their robustness and
stronger pulling force (Fig. 8).
Design and semantics of form and movement
207
screen provide a morphing patterned atmosphere that
suggests a continued relationship between exterior and
interior by shadow casting and modulation of the direct
sunlight (Fig. 11). Blind is an alternative approach to
the conventional inert brise-soleil, the design of which
generally lacks the consideration of its effect on the
interior condition especially in different timezones and
seasons.
Fig. 7. Four potential profiles for ‘soft’ actuations
of SMA wire.
Fig. 9. Blind in the form of a semi-ellipsoid canopy
performed morphological transformation for optimal
sunlight manipulation.
Fig. 10. Grasshopper and FireFly schema (left).
Responsive process of Arduino microcontroller for
Fig. 8. SMA springs embedded with tetrahedral
digital simulation (right).
skeleton and silicone rubbers for actuation through
electrical stimuli.
3.4 Adaptability
The application of the adaptability of Blind is as a
prototypical active shading device that to regulate
shading and shadow control to improve the comfort
level of existing spatial conditions. Blind can also be
responsive to ambient conditions or live data streaming
that addresses two applications termed comfort and
visual. The comfort application of Blind in the form of
a semi-ellipsoid canopy creates a ‘transition’ space inbetween Blind and the existing courtyard space
(Fig. 9). This application is embodied in two experimental processes. The first process used design tools
such as Grasshopper and Firefly parametric software
together with Arduino microcontroller and photo
resistor for initial digital simulation (Fig. 10). Full-scale
digital simulations demonstrated the shadows cast
into the existing courtyard space under the Blind
208
Design and semantics of form and movement
Fig. 11. Shadow casting of morphological
transformation (top). Morphing shadow patterns
of porosity transformation (bottom).
The second process is a physical experiment involved
through the use of SMAs spring that serve as the
actuators to reduce mechanistic components to test
the initial physical transformations of Blind (Fig. 12).
This experiment also using a photo resistor as light
sensor and the torchlight mimicking the path of sunlight
to embody the initial essence of comfort: a digital and
physical responsive skin model in various morphological
states for optimal performance as a sunlight modulator
(Fig. 13).
responsive undulating silicone rubber surfaces and
actuated by SMA wires and springs. This analogy of
an ‘eye-like’ permeable aperture functioned as a skin
muscle mechanism in the eye which allows various
changing porous patterns in binary form on the skin
(Fig. 14). This perforation process created a potential
application for the surface of blind as the ‘analogue
media screen’, to display binary images or even motion
graphics. It is also adding a new layer of aesthetic for
visual communication between existing courtyard space
and external surrounding environment.
Fig. 12. The expansion and contraction of the SMA
spring responding to direct light through a photo
resistor.
Blind allows transformation to take place in the
exoskeleton structure and skin. It develops the
skin surface transformation in two different types:
morphological and porosity transformation.
Morphological transformation explores the global
surface curvature of Blind to be modifiable. It allows
contraction and expansion while maintaining the
continuous topology of any undulating or flat surface.
It can respond to various functional drivers.
Fig. 13. Morphological transformation of the triangular
modules of Blind responding to direct light for optimal
performance.
Porosity transformation, on the other hand, focuses
on the visual application through permeability of Blind
that is generated by the individual ‘soft’ aperture to
cast shadows in response to sunlight penetration. This
transformation manipulates the spatial conditions of
the interior and exterior spaces through the dynamic
patterns of the skin surface. The initial geometry of
the membrane aperture is inspired by the performance
of the eye. The ‘eye-like’ apertures in the geometry
are determined by their relative curvature on the
Fig. 14. ‘Eye-like’ apertures open and close actuated
by SMA wires to perform porosity transformation.
4 Conclusion and Future Work
Architectural skins are part of the crucial components
for architecture in terms of climatic control and visual
appearance. In general, the materials used to construct
these components (steel and glass) have not changed
for a century. Their ‘hardness’ is literally true from
the standpoint of both system and materials. Current
advancement of material technology provides the
alternatives especially the ‘soft’ textile and form-changing
materials such as Aramid and electro active polymer.
They became increasingly relevant to architecture as well
as responsive architectural skins design for lightweight
and flexibility purposes. The study provided in this
paper initiated the investigation towards this direction
using scaled physical prototypes, digital simulation and
physical computing set as the methods of inquiry.
The design experiments explored in this paper
through the passive and active form-changing
materials embedded in the elastic tensegrity structural
components. It is an alternative method for less energy
and simpler actuation to control, and regulate the
behaviour of responsive morphing skins in terms of
the comfort and visual. The form-changing material
that operates inside the tensegrity system becomes
the new kind of ‘structures’. It can actuate the elastic
Design and semantics of form and movement
209
component exposed to the ambient environment to
be functionally adaptable. This alternative approach
for actuation can create multiple states of stability
in terms of transformation for architectural skins. It
has potential, and is more economical and silent than
conventional mechanical approaches. The research
in this paper demonstrated the new possibilities of
implementation for responsive architectural morphing
skins evaluated through the small-scale design
experiments. The design method developed through
Blind provides designers with a mix of passive and active
design strategies for responsive architectural morphing
skins. This experiment while in an early stage, evaluated
through the criteria of comfort and visual, point to the
potentials, and identified the challenges, for full-scale
architectural applications,
AMS revealed the new territory of responsiveness from
mechanical components operation towards a ‘solidstate’ kinetic matter for actuation and transformation.
Future work will focus on this ‘solid-state’ approach
to further investigate the potential of the harness,
elasticity, thermal, magnetic and sensing properties of
the form-changing and kinetic matter embedded with
physical computation for AMS application. The fullscale physical experimentations also included explore
the possibilities of AMS for feasible implementation in
architecture.
[6] Ruiz-Geli, E. (2011). Media-ICT. Barcelona: EI Consorci de
la Zona Franca de Barcelona.
[7] Berzowska, J., & Coelho, M. (2005). Kukkia and Vilkas: Kinetic
electronic garments. Proceedings of the 9th IEEE International
Symposium on Wearable Computers (ISWC’05). Osaka: IEEE
Computer Society.
[8] Coelho, M., & Maes, P. (2009). Shutters: A permeable surface
for environment control and communication. In N. Villar, S. Izadi,
M. Fraser, & S. Benford (Eds.), Proceeding of the 3rd Tangible
and Embedded Interaction Conference (TEI’09). Cambridge:
ACM Press.
[9] Benjamin, D., & Yang, S.I. (2006). Life Size: Volume 2.
New York: Graduate School of Architecture, Planning and
Preservation of Columbia University.
[10] Harti, D., & Lagoudas, D. (2007). Aerospace applications
of shape memory alloys. Proceedings of the Institution of
Mechanical Engineers, Part G: Journal of Aerospace Engineering,
221(4), 535-552.
[11] Till, C., Etches, J., Bond, I., Potter, K., & Weaver, P. (2008)
Morphing skins. The Aeronautical Journal.
[12] Wiggins, L.D., Stubbs, M.D., Johnston, C.O., Robertshaw,
H.H., Reinholtz, C.F., & Inman D.J. (2004). A design and analysis
of a morphing hyper-elliptic cambered span (HECS) wing.
American Institute of Aeronautics and Astronautics.
[13] Frumar, J., & Zhou, Y.Y. (2009). Kinetic tensegrity grids with
3D compressed components. Proceedings of the 29th Annual
Conference of the Association for Computer Aided Design in
Architecture (ACADIA 09). Chicago, 255-258.
Acknowledgments
[14] Khoo, C.K., & Salim, F.D. (2011). Designing elastic
The author would like to thank the contributors to
this paper: Professor Mark Burry, Associate Professor
Jane Burry and Dr. Flora Salim. This research is funded
by Australia Postgraduate Award (APA) and supported
from Spatial Information Architecture laboratory
(SIAL), Melbourne, Australia.
transformable structures: Towards soft responsive architecture.
Proceedings of the 16th International Conference on Computer
Aided Architectural Design Research in Asia. Newcastle, 145.
[15] “Silicone rubber”. Wikipedia, The Free Encyclopedia.
Retrieved August 30, 2011, from http://en.wikipedia.org/wiki/
Silicone_rubber
[16] Gomez-Jauregui, V. (2010). Tensegrity Structures and Their
References
Application to Architecture. Santander: Sevicio de Publicaciones
[1] Moloney, J. (2011). Designing Kinetics for Architectural
de la Universidad de Cantabria, 296.
Facades: State change. Oxon: Routledge.
[17] Bar-Cohen, Y. (Ed.), (2006). Biomimetic: Biologically Inspired
[2] Olgyay, A., & Olgyay, V. (1957). Solar Control and Shading
Technologies. Boca Raton: CRC Press, 31.
Devices. New Jersey: Princeton University Press, 15.
[3] Sample, H. (2012). A Brise-Soleil without a building. In
G.P. Borden, & M. Meredith (Eds.), Matter: Material Processes
in Architectural Production (p.332). Oxon: Routledge.
Chin Koi Khoo
[4] Ritter, A. (2007). Smart Materials in Architecture,
Spatial Information
Interior Architecture and Design. Basel: Birkhauser, 6-7.
Architecture
[5] Chun, M.K. (2007). Investigation into the cause of
Laboratory (SIAL),
pneumatic actuator failure on the HypoSurface. Unpublished.
RMIT
210
Design and semantics of form and movement
Mizuki Sakamoto, Tatsuo Nakajima, Todorka Alexandrova
{mizuki, tatsuo, toty}@dcl.info.waseda.ac.jp
Digital-physical hybrid design:
Harmonizing the real world
and the virtual world
Abstract
Keywords
Embodied interaction technologies make it possible
to enhance our real artifacts. Various displays and
projectors are already embedded into the artifacts,
which makes it possible to create virtual forms into
them. The virtual forms present dynamically generated
visual expressions containing information that ascribes
some additional values to the artifacts, and enables
users to consider them as being more attractive.
Using virtual forms is a very promising way to enhance
artifacts surrounding us, and to make our daily life
and business richer and more enjoyable. Recently,
many people are anxious about the future that is why
increasing daily pleasure is one of the most important
social issues to be considered. In this paper, after presenting an overview of three case studies that enhance
traditional artifacts with virtual forms, we extract five
values that define a frame how a user feels about them.
We describe the values and explain how they are used
in the case studies. Then, we propose a framework to
design digitally enhanced artifacts with these five values.
Moreover, we show that the values are of significant
importance in order to make a user’s activities with
them richer and more enjoyable by adding virtual forms
or changing values dynamically with virtual forms.
Embodied interaction, semiotics, empathetic value,
persuasive value, informative value, economical value,
ideological value, gamification, digital-physical hybrid
design.
1 Introduction
Embodied interaction technologies make it possible to
enhance our real artifacts and the various displays and
projectors that are already embedded into the artifacts,
allow us to attach virtual forms to them [1]. The virtual
forms show dynamically generated visual expressions
containing information that provokes a user to feel some
additional values of the artifacts, and enables him/ her to
consider the artifacts as more attractive. Technologies
have become mature enough to realize the virtual
forms. Sensors retrieve various information about the
real world, and the virtual forms reflect what happens
in the real world and offer more value to a user. Using
virtual forms is a very promising way to enhance the
artifacts that surround us, and make our daily life and
business richer and more enjoyable, since increasing daily
pleasure is one of the most important social issues to be
considered. Traditional artifacts enhanced with virtual
forms bring an augmented reality features to them and
from now on in this paper we call them “AR artifacts”,
where “AR” stands for Augmented Reality.
Computation becomes a primary method to realize
AR artifacts. The virtualization of the real world
Design and semantics of form and movement
211
through computational materials makes the distance
between artifacts disappear, and the communication
within a community more asynchronous. In addition,
information reflecting the current situation is embodied
in AR artifacts. There are many advantages in using
virtual forms to design AR artifacts. The virtualization
makes our real world more flexible, and there is a
possibility to incorporate more trading and gamification
aspects, which will encourage a user’s motivation in his/
her daily life.
However, how to “successfully” and “harmoniously”
design the virtualized real world with AR artifacts
remains an important and challenging issue. In order
to suggest some clues for solving this problem, in this
paper, we describe three case studies and based on
the experiences with them we are able to extract five
values, which allow us to discuss and consider some
good design implications for the design of AR artifacts.
The proposed five values are significant in order not
to lose the reality in traditional activities by adding
virtual forms.
A frame in social theory consists of a schema of
interpretation, which is a collection of anecdotes and
stereotypes. People construct a set of mental filters
through biological, emotional, economic and cultural
influences. The choices they make are influenced
by their creation of a frame. Framing can affect the
outcome of a choice problem. The framing effect, one
of the cognitive biases, describes that presenting the
same option in different formats can alter people’s
decisions [2]. The values discussed in this paper are
very useful for defining a frame to characterize activities
with AR artifacts. We can recognize how each of the
proposed values is defined depending on the current
frame associated with the current AR artifact. The
values could be changed by incrementally adding virtual
forms to some AR artifacts. Thus, this approach offers
the possibility to apply and design gamification elements
and mechanisms for the current activities with AR
artifacts in a more systematic way without modifying
the infrastructure heavily.
The main contribution of this paper is to propose five
values that can be used to change the frames in which
people use artifacts explicitly. The values are useful as
a design tool to make the artifact more attractive.
The paper is organized as follows. Three case studies
that use virtual forms to enhance our real world are
presented in Section 2. In Section 3, five values are
212
Design and semantics of form and movement
extracted from our experiences with the case studies
and we show how these values are used in the case
studies. Section 4 shows how to consider maintaining
the reality even when virtuality is introduced in the real
world. Finally, we conclude the paper in Section 5.
2 Three Case Studies
2.1 Persuasive Ambient Mirror
Virtual Aquarium is an AR artifact called a persuasive
ambient mirror [3] that has been developed and has
the objective of improving users’ dental hygiene by
promoting correct toothbrushing habits. It is set up in
the lavatory where it turns a mirror into a simulated
aquarium as shown in Figure 1. A virtual form in Virtual
Aquarium represents an aquarium located in the
lavatory, and the form reflects a user’s toothbrushing
behavior. Fish living in the aquarium are affected by the
users’ toothbrushing activity. If users brush their teeth
properly, the fish prosper and procreate. If not, the fish
become unhealthy and may even perish.
In this AR artifact, we use a 3-axis accelerometer
sensor that is attached to each toothbrush in a
household and a user brushes his teeth in front of
Virtual Aquarium using such a brush. Since toothbrushes
are usually not shared and each sensor has a unique
identification number, we are able to infer which user
is using the artifact at a given time. Toothbrushing
patterns are recognized by analyzing the acceleration
data. The toothbrush is able “to observe” passively
how the user brushes his/her teeth and this is the only
interaction needed to use this AR artifact.
In this AR artifact’s case, the ideal user’s behavior model
is defined as follows: 1) a user brushes his/her teeth at
least twice a day; 2) one session involves at least three
minutes of brushing; and 3) brushing involves patterns
that ensure the teeth are properly cleaned. Each user’s
behavior is compared to the ideal one and translated
into a feedback as described below.
As shown in Figure 2, when a user begins to brush his/
her teeth, a scrub inside the aquarium starts cleaning
the algae off the aquarium’s wall. At the same time,
a set of fish associated with the user starts moving in
the aquarium in a playful manner. When the user has
brushed his/her teeth for a sufficient period of time,
the scrub finishes cleaning and the fish dance becomes
even more elegant. When the user finishes brushing,
Fig. 1. Virtual Aquarium.
Fig. 2. Proactive Information Feedback in Virtual Aquarium.
the fish end their dance and resume their normal
activities. Both the activities of the fish and the
movement of the scrub are designed in such a way as
to give the user hints regarding the correct method of
toothbrushing. However, if a user does not brush his/
her teeth sufficiently, the aquarium becomes dirty, and
the fish in the aquarium become sick. The feedback
information is returned immediately according to
the movement of the user’s toothbrush. We call this
feedback immediate feedback.
The health of the fish is visibly affected by how clean
the aquarium is. If the user neglects to brush his/
her teeth properly, fish health worsens. In contrast,
faithful brushing may result in the fish laying eggs as
shown in the right pictures of Figure 2. At first, the
eggs are not very likely to hatch. If the user continues
to brush consistently for a number of days in a row,
the incubation ratio increases. This way, the longterm feedback gives clues to the correct behavior and
attempts to maintain motivation over a period of time.
The long-term feedback is called accumulated feedback.
While designing this AR artifact, we consider the
association between a user’s healthy lifestyle and the
cleanliness of the aquarium. Our design takes into
account the fact that the user feels empathy for the
virtual fish.
In our daily life, a mirror reflects our appearance
and allows us to know whether we look well or not,
whether our makeup and clothes fit or not and so on,
and has the power to make what is invisible from us
visible. We believe that mirrors are adequate devices
to reflect our current behavior that return immediate
feedback on the current situation. Virtual Aquarium is
a new type of mirror that reflects a user’s current state,
encourages him/her to change his/her behavior and
motivates desirable lifestyle.
2.2 Augmented Board Gaming
Go is a traditional board game for two players, where
the goal is to occupy a larger portion on the board
than the other player. Black and white stones are used
to control the territory and a board with a grid of
19x19 lines is used as the game field. The rules of Go
are relatively simple, but the underlying strategies are
extremely complex and rich. As in chess and reversi,
a numerous set of strategies have been invented to
reduce the complexity, but studying them requires the
player to actually understand the strategic concepts.
Thus, it takes a long time for a beginner to play well
with an experienced player and to feel pleasure during
the play. Augmented Go [4] supports several gaming
modes to play a game. The basic idea is to offer useful
information to beginners without extra interactions
and intrusive devices as shown in Figure 3. A virtual
form in Augmented Go is superimposed onto the real
Go board. Proactive feedback information is offered
visually by superimposing guidance information onto
the Go board by a projector. A web camera connected
to a personal computer is used to detect the position
of each Go stone. The OpenCV library is used for
Design and semantics of form and movement
213
Fig. 3. Augmented Go System.
Fig. 4. Proactive Information
Feedback in the Augmented
Go System.
visual analysis and the core logic of the enhanced
artifact generates information presented to the players
according to the current game situation.
The AR artifact supports several gaming modes. As
shown in Figure 4(a), players can interact with the AR
artifact by placing Go stones on a menu that is projected
onto a board. Here, we explain briefly some of the modes
and how players interact with Augmented Go.
Normal Play mode. The normal play mode is the basic
form of the Go augmentation. In this mode, two players
play Go as usual, but useful information is projected on
the board to help beginners recognize the situation and
make better decisions. The rules of Go are simple, but
the vast number of possible moves in each turn makes
it hard for beginners to make decisions. Moreover, on
the large 19x19 board, beginners tend to concentrate
on localized fighting in a narrow region and lose the
big picture. It is difficult to recognize invaded areas,
since an invasion process gradually progresses as new
214
Design and semantics of form and movement
stones are put on the board. For choosing good offense
and defense strategies, recognizing the links between
the Go stones is important, but it requires some skill.
Moreover, the normal play mode visualizes the strength
of links between the Go stones. As shown in Figure
4(b), same-colored stones are connected with lines.
If a dangerous situation occurs somewhere on the
board, a warning message appears for the players to
avoid losing the area. The sequence of stone moves
is also recorded into the database, which facilitates
replaying the game for self-training. Replaying allows
us to review and analyze the play by projecting the
stones on the board later.
Tsumego mode. Tsumego is a type of exercise where
the player is given a game board situation. The aim is
to find the best sequence of stones’ placement in a
given board situation. In this mode, the positions of
the stones are visualized on the board. Players can try
different moves by placing stones on the board, with the
results and comments explaining key important points
Fig. 5. Augmented Trading Card Game.
displayed as visual feedback (Figure 4(c)). The Tsumego
mode prepares questions for a player with different skill
levels, and the level of difficulty can be selected in the
menu (Figure 4(d)).
The advantage of our approach is to allow players to
receive information through the normal interaction
with the Go board and the stones. By superimposing
information onto the board, players can concentrate
on the normal play or self-training without fragmenting
their attention by taking an instructional book and so on
into their hands. This is important to make it possible
for the players to allocate enough cognitive resources
for understanding the current situations in the game.
2.3 Augmented Trading Card Game
A trading card game is also commonly referred to
as a collectible card game, a customizable card game,
or CCG. For our purposes here, we will use trading
card game (TCG) to refer to all the three varieties of
games. In a nutshell, a TCG combines the collectability
of trading cards with a strategic game play. Typically
a player purchases a starter set, containing a playable
deck of cards and a manual that includes an explanation
of the rules and the mechanics of the game in an introductory fashion. One of the biggest problems faced by
any new TCG player is the need for an opponent to
truly engage in the game play, as it is extremely unusual
for any TCG to support a solitaire mode. Players usually
begin playing with a friend, at a particular location such
as a hobby game store that offers organized gaming
opportunities and includes a tutorial component, or
via an online portal.
Computer-based TCG is also becoming popular, and in
our project we make a comparison between the real
TCG and its virtual one running on Nintendo DS [5]. An
important conclusion resulting from that comparison
is that the computer-based TCG loses a lot of realities
offered by the real TCG. For example, the sense of real
cards is essential for many TCG players since making
and completing collections of cards is a significant
source of pleasure for them. Also, the computer-based
TCG implies some communication limitations, as it
allows a player neither to have an eye-to-eye contact,
nor to look at or chat with the opponent player.
As described above, although most of the current
computer-based TCGs lose the realities of the real
TCG, we claim that ubiquitous computing technologies
may help to recover these lost realities and encourage
and attract players to enjoy the computer-based TCG
in a very similar way to the real TCG. Moreover, adding
special effects and virtual forms to the computer-based
game might even increase the excitement and the
enjoyment of the game more than the real one.
Figure 5 shows Augmented Trading Card Game
(Augmented TCG) that we are currently developing.
This system extends the trading card game running
on Nintendo DS, where two players are usually located
in different places while playing the game.
In Augmented TCG, the opponent player is represented
as a virtual character. The movement of the character is
synchronized with the movement of the real opponent
player by using MS Kinect, and the behavior of the
character is determined by the information retrieved
from a biosensor attached to the opponent player, i.e.
the virtual character’s behavior reflects the real player’s
behavior and emotions to some extent. In Augmented
TCG, two virtual forms are used. The first form is
superimposed onto the playing table to show the virtual
trading cards and some special effects during the play.
The second virtual form is installed on the wall and
represents the virtual character of the opponent player.
The trading card itself is also enhanced in the AR
artifact. Cards presented on the display of the Nintendo
DS are retrieved by Web cameras and projected on a
real table. Then, these cards can be enhanced by adding
effects reflecting the player’s emotions, enjoyable battle
effects or empathetic effects to the characters shown
on the cards.
Design and semantics of form and movement
215
In the original computer-based game, a player usually
cannot see the opponent player. However, the proposed
Augmented TCG enables us to recover this lost reality
by adding a virtual character whose movement and
behavior are synchronized with the movement and
behavior of the real opponent. In addition, the virtual
trading cards carry some special effects that increase
the sense and the excitement of the game. Similarly,
if the character drawn on a trading card shows some
empathic expressions, a player feels more empathy for
the character on the card, and feels more enthusiastic
and committed to the game. We believe that such
special effects would help to compensate the lost
realities of the real trading cards. Moreover, virtually
attached rarity to the virtual trading cards would also
bring a feeling of reality and would encourage a will
to collect virtual trading cards.
3 Design Implications
In this section, we extract five values based on our
experiences with the design of the case studies
described in the previous section. We mainly focus
on the experiences that provide us some clues as to
how to integrate virtual forms into the real artifacts
by increasing their values without losing the reality for
a user. The virtual forms offer the possibility to enhance
a user’s experience beyond the original artifacts. The
values presented in this section are a useful tool to
analyze the current artifacts and to develop AR artifacts
with virtual forms. As shown in the next section,
designing AR artifacts incorporating the five values
described here enables us to consider which frame
is chosen that corresponds to the current AR artifact.
The approach provides the possibility to change the
current frame explicitly in order to gamify the usage
of the AR artifacts according to a user’s personality
by changing or adding the five values.
3.1 Semiotics of Virtual Forms
The meaning of the visual forms should be easily
understood by the user [6]. When instructions on the
use of the virtual forms are needed, they should be
simple so that the user can understand them completely
with ease. It is not a good idea to assume that the user
will read a manual. One of the solutions to this problem
is to use metaphors. Understanding a metaphor relies
on the user’s prior knowledge. If the user has been
216
Design and semantics of form and movement
acquainted with similar information in the past, the user
can learn the meaning of new virtual forms through
the use of an appropriate metaphor. A metaphor does
not require too much information for making a better
decision. For example, as described before, in Virtual
Aquarium, the cleanliness of the aquarium is a metaphor
of the cleanliness of the user’s teeth. Also, Augmented
TCG’s special effects surrounding the battlefield
become metaphors to show the will and strength of the
virtual characters drawn on the trading cards. Another
solution is to use affordance. In Augmented Go, a player
chooses a command to the system by putting a Go
stone on a circle projected on the Go board. In this
case, the accuracy to identify commands is important
because a user cannot distinguish the misunderstanding
of the offered affordance and the inaccuracy to
recognize commands.
However, users sometimes misunderstand the meaning
of the visual forms, and this is one of the dangers of
relying on metaphors and affordance. Users may find
unintended meanings in a visual form. For example, if a
supposedly unattractive picture is used to discourage
undesirable behavior, that picture may actually have
the opposite effect on an avant-garde or ironic art
consumer. Moreover, the way a picture is understood
by a user strongly depends on the cultures and
personalities of the users. It is not easy for a designer
to attach a single meaning to a specific expression for
all people. The interpretation of the expression could
be left to the user. This open interpretation [7] allows
the user to feel pleasure or good surprise on one hand,
but on the other hand it is not easy to predict the effect
of the interpretation by the user in a controllable way.
The presentation of the information according to a user’s
current attitude is a key issue for representing the
values described in this section. The information may
appear in an ambient way. For example, as shown in this
section, a metaphor is a useful tool for that purpose.
However, visual information representing a metaphor
should be tangibly manipulated to present more detailed
information [8]. One of the solutions is to offer a visual
expression that offers an affordance presenting more
detailed information by opening the visual expression.
After a user can choose one of the visual expressions,
detailed and concrete information appears on a visual
form for the user to make a better decision. This style
of information design is effective
to show enough information when designing AR artifacts. A skillful user chooses several necessary abstract
pieces of information and opens the information to
show more detailed information. Choosing how much
information is necessary depends on the user’s current
attitude. Virtual forms attaching a value require us to
choose an appropriate feedback strategy according to
a user’s current attitude, but it is not easy to detect
the user’s current attitude with the current sensing
technologies. The interaction design enables the users
to explicitly retrieve necessary information according
to their current stage because a more skillful user has
a strong incentive to know more detailed information
for raising up his/her current level to the next level.
3.2 Five Values Extracted From Our Case Studies
Empathic Value and Virtual Characters. Using an
empathetic form is an effective way to evoke the user’s
emotions. Empathy is a strong social incentive to feel
values on a virtual form. Empathy engages the user to
feel close to the empathetic form. Virtual pets are a
typical example of an empathic form and they are very
popular in many online services. Social robot pets also
make our daily life happier. The pets evoke the user’s
empathetic emotion and encourage him/her to change
undesirable behavior as a consequence of negative
emotions. The emotional impact is very effective in
making the user keep desirable habits. One interesting
theoretical result is the media equation [9]. A user
feels empathy for even non-living things like a personal
computer. The result indicates that there is a possibility
to use various expressions or products that do not
represent living or animated characters. On the other
hand, if the form showing a pet is too realistic, a user
may feel uncomfortable although he/she feels empathy
for unrealistic characters. The phenomenon is called
the uncanny valley [10]. When the user considers that
something expressed by information services has a
personality, the user feels empathy. If the personality fits
the user’s personality or he/she feels altruism towards
the personality, he/she feels a close relationship.
Therefore, designing a good and appropriate personality
is an important topic when designing AR artifacts. In
Virtual Aquarium, fish in the virtual aquarium evoke
emotion to the user. In this case, one of the important
design issues is to synchronize the current situation of
the virtual fish with the user’s toothbrushing practice.
In Augmented TCG the virtual character representing
the opponent player has a strong impact on the player’s
feelings during the game play. If a player has a good
feeling about the virtual character, the player tends to
satisfy the game play sincerely even if he/she does not
know the real opponent player well.
Media tend to be used by a user for a longer time
if the user feels empathy for it. However, it is not
easy to offer empathetic experiences to users. One
promising way to solve the problem is to make
it possible for a user to customize his/her own
experiences. For example, decorating a mobile phone
is very typical for Japanese people, which makes it
unique and differentiates it from others. It is also very
typical for people to customize their avatars in online
games by changing their avatars’ accessories, clothes,
shoes, and hairstyles even if doing so involves paying
extra money. The product attachment theory [11]
explains why people like customized things more than
uncustomized things. According to that theory, people
prefer a product whose personality matches their own
personality. The customization is a process to make the
products more preferable to a user.
Persuasive Value and Feedback Information. An effective
and easy way to change a user’s behavior is to offer
feedback information to the user [12]. When a user
behavior is desirable, a positive expression is returned
as a feedback, while if he/she behaves in a bad manner,
a negative expression is returned as a feedback.
Designing emotional incentives is one of the important
aspects to offer the persuasive value on the virtual
form. However, the approach requires that the stimulus
continues forever in order to maintain better habits.
A user may get bored after the same stimulus is
received over a period of time and the stimulus itself
can not be strengthened infinitely. Also, the feedback
information should not reflect the real effect if the
effect does not appear soon. If users do not receive the
appropriate feedback information according to their
efforts, they will lose interest and motivation. Moreover,
the feedback information needs to be synchronized with
the real situation of the users to make them feel the
sense of the reality.
The transtheoretical model defines a five-stage process
involving the progress to change the user’s undesirable
behavior [13]. The returned information needs to be
changed according to the current stage of the user’s
Design and semantics of form and movement
217
attitude, skills and knowledge. In earlier stages, the
user prefers emotional reinforcement not to give up
his/her current efforts. On the other hand, for the
user who is in a nearly final stage, enough information
for making a better decision through rational thinking
is more suitable. Virtual Aquarium offers emotional
stimulus to a user to continue to wash his/her teeth.
Also, Augmented TCG offers special effects on the
trading cards to encourage more plays. The design of
the case studies is suitable for encouraging users to play
the corresponding game continuously. On the other
hand, in Augmented Go, the projected information on
the real Go board is useful to make a better decision,
but the AR artifact does not offer emotional stimulus to
encourage a beginner to play the game.
A beginner may not have an interest to continue using
the applications described in the case studies over
a period of time. Especially, if the activity to use the
AR artifact requires some additional efforts, it is not
easy to continue the activity since the curiosity does
not motivate more than the exertion of extra effort
demotivates. This is one of the reasons why most
people give up many interesting activities quickly. It is
important to offer extrinsic motivation like joy, comfort
or reward in the early stage.
In the next stage, self-efficacy is the key to raise
the stage. A user feels self-efficacy when he/she has
a confidence to continue the target activity. The
confidence comes from the evidence that he/she has
the ability to do the activity well and the user needs to
be offered enough information showing such evidence.
In order to move to the later stage, the user needs
to perceive self-efficacy. Since a positive attitude is
necessary to grow the user’s self-efficacy, it is essential
to use positive feedback, and the user should be aware
of his ability to change his/her current lifestyle.
The feeling as information theory [14] is useful to
consider how AR artifacts with virtual forms evoke
the user’s emotions. The theory indicates that it is
difficult to think rationally while in a positive mood and
users tend to think more rationally when they are in a
negative mood. The results indicate that positive stimuli
are effective in early stages, but in later stages, negative
stimuli are desirable under the transtheoretical model.
Virtual Aquarium provides positive stimulus when
the user’s current behavior is desirable, but negative
stimulus is returned when he/she behaves undesirably.
One of the important findings is that negative stimulus
218
Design and semantics of form and movement
alone is not effective, because the user becomes
rational, and he/she considers the effectiveness of his/
her behavior. He/she needs enough information to
think about the importance of the activity in a rational
way. When rational decision-making is important, it is
desirable not to evoke a user’s positive emotion too
much. It may lead heuristic thinking to make a wrong
decision. The fact is important when designing an AR
artifact for a game. A game usually evokes a user’s deep
emotion, but it may not be better to win the game
without rational thinking.
Informative Value and Decision Making. The informative
value is useful to offer information to a user in order
for him/her to make a better decision. When using the
persuasive value, a user usually makes his/her choice
unconsciously, but the informative value enables a user
to feel his/her decision explicitly. If users believe that
they make a decision by themselves, they usually do
not try to change their decisions, and the effect is
useful to increase intrinsic motivation.
We believe that proper supports of decision-making
are very important and the informative value should
be incorporated in various future AR artifacts using
virtual forms [15]. We hope the experiences described
in this paper are useful to design future AR artifacts.
In Augmented Go, extra information is projected on
the physical GO board. The user can still use normal
stones and a board without attaching any artificial
objects like visual tags. Also, the user does not need to
be equipped with special devices like a head mounted
display. Projecting information directly on the GO board
is useful not to fragment a user’s attention. While doing
Tsumego, a user needs to look at a book. This forces a
user to look at the GO board and a book alternately,
and this prevents them from concentrating on learning
the strategies of a GO play.
For designing informative value, it is important to
consider how much information is necessary to make
better decisions [16]. If hidden information becomes
explicitly visible for a user, his/her decision-making will
become more rational, counteracting the biases that may
creep into their or his/her decision-making processes.
For attaching informative value, how to offer the nudge
proposed by Richard Thaler to make rational decisions
is an important design decision [17]. However, if proper
information is not given, a user may get more confused,
and in such case it is hard to make a rational decision.
For example, if there are too many choices, a user tends
to not choose at all [18]. A large amount of information
also requires heavy cognitive efforts, so it is important
not to give too much information when asking for a
decision. Also, too much information is neither effective
nor helpful for helping a user think rationally. In some
cases, heuristics are dangerous and lead to mistakes in
decision-making [2], and the bias in heuristic thinking
may cause the user to make a wrong decision. However,
heuristics are necessary to make better decisions from
many choices within a reasonable time. The amount
of information should be carefully designed for better
decision-making.
Economic Value and Virtual Items. Not surprisingly,
we also found that economic values are a powerful
technique to motivate people to change their behavior.
An economic value is a tangible reward that the users
consider valuable, yet not necessarily actual money or
goods. In online games, millions of players work hard to
obtain rare and valuable virtual goods, and even trade
these goods for real money at a rate of three billion
dollars per year [19].
In [20], Lendonvirta proposed three attributes that
make virtual items valuable in the game. The first
attribute is the functional attribute consisting of
two categories: performance and functionalities.
The performance is the skill to play a game well and
the functionality of the equipment increases the
possibility of winning the game. The second attribute
is the hedonic attribute. The attribute consists of six
categories: visual appearance and sounds, background
fiction, provenance, customizability, cultural references
and branding. The hedonic attribute offers the value
to satisfy a user’s emotional desire. The third attribute
is the social attribute. This attribute consists of one
category: rarity. The value is strongly associated with
the ability to distinguish a group of owners from nonowners. The above attributes are effective to provide
economic value to virtual forms by making the items
with the attributes shown in the forms exchangeable
with other people.
Adding economic values in our case studies is a very
important issue. In Virtual Aquarium, we can buy fish
and plants for the aquarium. If a user becomes sick,
he/she may not wash his/her teeth properly and it
may make fish ill. The user may feel helplessness and
hopelessness about using Virtual Aquarium if there is
no way to solve such problems. If a user can use virtual
currency and buy medicines, it motivates him/her to
continue using Virtual Aquarium. In Augmented Go, it is
useful for a user to buy more Tsumego patterns in order
to improve his/her skills or to buy new software to
analyze his/her current play in details and advise better
ways to play in the future Go games. In Augmented
TCG, a player can buy a new trading card and
strengthen his/her current card deck. It is also possible
to buy a new virtual character and a new pattern to add
special effects on the virtual trading cards.
There are different virtual money systems in many
online services. In each service, a different virtual
currency is defined. One of the most interesting issues
is the exchange of virtual and real currencies. The rate
to exchange the virtual currency and the real currency
may be changed according to the value of the virtual
currency. This means that the value of the virtual
currency is decreased if the trust of the virtual currency
is decreased. We believe that the economic value will
change the understanding and attitude to virtual and real
money and will be useful for the design of a money system
that uses the virtual currency in a more effective way.
Although the economic value is a powerful tool to
motivate behavior, it may lead to unpredictable results
if not used carefully [15]. One additional effect that
could be utilized is reciprocity, or the desire to
reciprocate gifts and favors received from others.
Virtual gifts are frequently exchanged in online environments, strengthening the relationships between users.
Another example of economic incentives in persuasive
technology is the activity-based billing system [22],
which uses automatic micropayments and microrewards
to coax users towards desired behaviors. The approach
utilizes the heuristics to make a decision to change a
user’s behavior and to make the economic incentive
more effective.
Ideological Value and Aesthetics. What is here referred
to as an ideological value is the notion of influencing
users’ behaviors through influencing their attitudes
and values; in other words, educating the users on a
deeper level. Attitudes and values influence the users’
behavior in the long term. The ideological value makes
it possible to motivate the user by himself/herself. The
user raising the ideological value has a belief called
self-efficacy that makes him/her believe he is able to
achieve his/her goal. In our current case studies, we
Design and semantics of form and movement
219
choose simple metaphors that could be understood by
the user easily, but the metaphors have shallow impact
on the understanding of the importance of maintaining a
desirable lifestyle.
The virtual form that has the ideological value may also
include a user’s dreams or expectations for the future.
Especially, an art form is a useful style to express future
human’s dream or expectation in an ambient way.
A virtual character may speak some words reminding
a player’s future dream or expectation. This may be
useful to help a player mature through the game play.
Also, virtual forms can be drawn surrounding virtual
trading cards or a virtual character and they may include
the importance of recent serious social problems such
as sustainability and human wellbeing. This makes it
possible to learn these important issues during game
play by incorporating these social problems in the game
design. Also, a special effect in Augmented TCG can
represent the wish of a character in a trading card,
and the effect can heavily influence a player’s current
motivation. Using an art form is a useful tool to give
meaning to virtual forms. We believe that artistic ways
of thinking will help for digital-physical hybrid design
that offers more stimulative experiences to consider
the importance of a desirable future. For example,
contemporary conceptual art uses complex metaphors
to provoke deep reflections on issues like sustainability
and peace in the world.
The ideological value is brought by intellectual
stimulation. For maintaining desirable behavior, it
is important that the user is aware of the importance
of the desirable behavior. The association between the
effect of desirable behavior and the virtual form offered
to the user as feedback is an effective intellectual
stimulation. Other values described in the previous
subsections cause an extrinsic motivation that makes it
difficult to maintain the user’s desirable behavior for a
long time. On the other hand, the ideological incentive
may cause an intrinsic motivation, and thus the changed
behavior to be maintained for a long time [23].
Aesthetics is an important concept to design the
ideological value. Especially, in Japan, incorporating
the ideological value in Japanese digital products is a
promising way to sustain the Japanese economy [24].
Of course, Japanese traditional folk craft represents
the aesthetic value, and we need to investigate how
to incorporate the esthetic value into AR artifacts
as a next step.
220
Design and semantics of form and movement
4 Achieving the Reality in the Virtualized
World
In this section, we show a brief and ongoing framework
to design a new AR artifact with the proposed five
values. It is important to note again that the values
are very closely related to each other, which can be
seen from the descriptions in the previous sections.
For example, the rarity in the economic value strongly
depends on the feeling of how important it is to a
user. Therefore, how to orchestrate the extracted
values defines different frames for the different users.
Although it is not easy to change the value attached to
the real form, the virtual form allows the value to be
changed by replacing the values attached to the virtual
form. This means that the currently attached values
can be changed according to the situation, which on
the other hand offers the possibility to gamify the
AR artifact successfully.
The framework consists of the following two steps.
The first step is to consider which values are used
to design a target AR artifact. The second step is how
to orchestrate the five values and design an AR artifact
to change the current frame by changing the association
of the values and the virtual forms without losing the
reality of the artifact. In the remainder of the section,
we describe the two steps briefly.
As a first step, we consider how to use virtual forms
to offer additional values to a user. In our framework,
there are five core values: empathic value, persuasive
value, informative value, economic value, and ideological
value. For using the values, we first choose a virtual
form that can offer the target value. The value should
offer a feeling of materials to the virtual form as in the
real artifact because the newly attached values may
change the meanings of the original artifact.
As a second step, we need to consider how to obtain
and keep the reality in the AR artifacts. The virtual
forms should convey some realities to users. For
example, in Augmented TCG, if there is no real feeling
for the opponent player, then the player is more likely
to cheat. Also, it is hard to feel empathy for the virtual
trading card. Rarity is an important economic value
to make a user feel empathy for virtual items [20].
The value makes it meaningful to collect trading cards
because the owner of the rare card has a feeling that
the card is unique. Recently, many Japanese animations
include scenes that can be manipulated from real photo
scenes. Realistic scenes are important for making a
viewer feel the reality in fictional stories. Also, in a
game, when the season in the game is synchronized with
the season in the real world, a user feels more reality.
If a virtual character in a game uses humorous words
or jokes, the reality of the character is significantly
increased. Recently, Apple iPhone 4s offer an assistant
application called Siri. The application offers a new
way to control a user’s phone with the user’s voice.
Since Siri sometimes answers very humorous words,
it increases the reality of the answers. Also, if a user
can customize his/her virtual room with furniture that
represents some realistic items, a user feels the reality
better. One of the most important design issues is that
the virtual form should be natural like traditional real
materials. However, the virtual form may exaggerate
the effect in the real world, which might make the AR
artifact with the virtual form even more exciting and
attractive to the user than the original one. A feeling
of the reality is important so that a user continues to
use an AR artifact for a long time. However, it is hard
to say and define, when and how a user feels the reality
on an expression presented in a virtual form. It is not
just enough to make the virtual form as close as possible
to the realistic expression because a user may not feel
the reality and may not be satisfied or attracted by
extremely realistic expression. A user sometimes feels
the reality to unrealistic exaggerated expressions or
to expressions that include reality elements partially.
It is an important issue to investigate the question of how
a user feels the reality in a next step of our research.
This means that real products are becoming more
virtual, and virtual products are becoming more real.
Maintaining the reality in our real world is extremely
important to make our daily life meaningful, and keeps
us interested in the real world without escaping in the
virtual world [26].
Digital-physical hybrid design causes real forms and
virtual forms to be seamlessly integrated. In the next
step, we are interested in using our concept to gamify
our daily and business environments. Gamification
recently became a popular concept for making daily
and business activities more enjoyable [27]. The basic
idea is to use a game mechanism to control activities in
daily life and business. Our framework can be used to
successfully design AR artifacts and is also very useful
for realizing the gamification of daily and business
activities. We believe that Digital-physical hybrid design
offers additional values to an AR artifact, and these
values are used to define a new frame to make our daily
and business activities more exciting and enjoyable.
In this paper, we assume that traditional displays and
projectors are used to realize virtual forms, and we do
not consider the tangible aspects of the virtual forms.
We found that the sense of physical touch is important
for a TCG player [5], and we will consider the issue in
the near future. Future tangible technologies will help
to discuss the possible solutions [28] to this issue.
References
[1] Marzano, S., & Aarts, E. (2003). The New Everyday View
on Ambient Intelligence. Rotterdam: 010 Publisher.
5 Conclusion
[2] Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgement
Based on the experiences with the three case studies,
we draw some important conclusions for the design of
AR artifacts. The most important issue is how to define
the meaning of the virtual forms. The forms need to
offer some values to users and to include empathetic,
persuasive, economic, informative and ideological values
that wield influence over users. Attaching the meanings
to forms of products is necessary to design the user’s
experience. Baudrillard proposed that the consumption
becomes more symbolized and additional values become
more important than the products as materials [25].
For example, a brand offers significant additional values
to fashion items, and a consumer feels the value on their
virtual properties. On the other hand, adding values to
virtual items makes a user feel the realities into them [20].
Under Uncertainty: Heuristics and Biases. New York: Cambridge
University Press.
[3] Nakajima, T., & Lehdonvirta, V. (2012). Designing Motivation
in Persuasive Ambient Mirrors. Springer Personal and Ubiquitous
Computing.
[4] Iwata, T., Yamabe, T., & Nakajima, T. (2011). Augmented reality go:
Extending Traditional game play with interactive self-learning support.
In Proceedings of the 17th IEEE Conference on Embedded and RealTime Computing Systems and Applications. Chicago, Illinois.
[5] Sakamoto, M., Alexandrova, T., & Nakajima, T. (2011).
A study on the effects of a computer-based trading card game.
Waseda University Ambient Intelligence Research Technical
Report 2011-1, (In Japanese).
[6] Krippendorff, K. (2005). The Semantic Turn: A New
Foundation for Design. Boca Raton, Florida: CRC Press.
Design and semantics of form and movement
221
[7] Sengers, P., & Gaver, B. (2006). Staying open to
[23] Reeve, J. (2005). Understanding Motivation and Emotion.
interpretation: Engaging multiple meanings in design and
(4th ed.). Wiley. New York.
evaluation. In Proceedings of 6th International Conference
[24] Hara, K. (2011). Japanese Design – Aesthetics makes the
on Designing Interactive Systems (DIS 2006).
Future. Iwanami Publisher.Tokyo.
[8] Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless
[25] Baudrillard, J. (1994). Simulacra and Simulation. University
interfaces between people, bits, and atoms. In Proceedings
of Michigan Press.Michigan.
of the CHI 97.
[26] McGonigal, J. (2011). Reality is Broken: Why Games Make
[9] Reeves, B., & Nass, C. (1998). The Media Equation: How
us Better and How they Can Change the World. Penguin Press.
People Treat Computers, Television, and New Media like Real
New York.
People and Places. New York: Cambridge University Press.
[27] Zicbermann, G., & Cunningham, C. (2011). Gamification
[10] Mori, M. (2005). On the Uncanny Valley. In Proceedings
by Design. O’Reilly.
of the Humanoids-2005 Workshop.
[28] Dunne, L. (2010). Smart Clothing in Practice: Key Design
[11] Ruth, M., Schoormans, J.P.L., & Schifferstein, H.N.J.
Barriers to Commercialization. Fashion Practice, 2(1). Berg
(2007). Product attachment: Design strategies to stimulate the
Publishers. London.
emotional bonding with products. In H.N.J. Schifferstein,
& P. Hekkert (Eds.), Product Experience, Amsterdam: Elsevier.
[12] Fogg, B.J. (2002). Persuasive Technology: Using Computers
to Change What We Think and Do. Waltham, Massachusetts:
Morgan Kaufmann.
[13] Velicer, W.F., Prochaska, J.O., Fava L., Norman, G.L.,
& Redding, C.A. (1998). Smoking cessation and stress
management: Applications of the transtheoretical model
of behavior change. Homeostasis, 38.
[14] Schwarz, N., & Clore G.L. (2007). Feelings and phenomenal
experiences. In E.T. Higgins & A.W. Kruglanski (Eds.), Social
Psychology: Handbook of Basic Principles (2nd ed.). New York:
Guilford Press.
[15] Ariely, D. (2008). Predictably Irrational: The Hidden Forces
That Shape Our Decisions. New York: HarperCollins.
[16] Todd, P.M. (2007). How much information do we need?
European Journal of Operational Research, 177, 1417-1332.
[17] Thaler, R.H., & Sunstein, C.R. (2008). Nudge: Improving
Decisions About Health, Wealth, and Happiness. New Haven
& London: Yale University Press.
[18] Schwartz, B. (2005). The Paradox of Choice: Why More
Is Less. New York: Harper Perennial.
[19] Lehdonvirta, V., & Ernkvist, M. (2011). Knowledge Map
of the Virtual Economy. Washington DC: World Bank,
http://www.infodev.org/en/Document.1056.pdf.
[20] Lehdonvirta, V. (2009). Virtual item sales as a revenue
model: Identifying attributes that drive purchase decisions.
Electronic Commerce Research, 9(1).
Mizuki Sakamoto,
[21] Lehdonvirta, V. (2009). Virtual Consumption. Publications
Tatsuo Nakajima,
of the Turku School of Economics, A-11:2009.
Todorka Alexandrova
[22] Yamabe, T., Lehdonvirta, V., Ito, H., Soma, H., Kimura, H.,
Department of
& Nakajima, T. (2009). Applying pervasive technologies
Computer Science
to create economic incentives that alter consumer behavior.
and Engineering,
In Proceedings of ACM Ubicomp 2009. ACM.
Waseda University
222
Design and semantics of form and movement
Maxe Fisher, Simon Fraser, Tim Miller, Ross Stevens, Jerad Tinnin, Annelies Zwaan
[email protected], [email protected], [email protected], [email protected],
[email protected], [email protected]
Digital craft in digital space:
A paradigm shift in the making
Abstract
1 Introduction
With every technological innovation in Industrial Design
there is an accompanying conceptual change about how
materials are used, how forms are put together, what
they symbolize and how their existence changes the
way we reflect upon the design and the design process.
We are currently in the midst of such a change. Digital
technologies offer designers a new context to challenge
the methods of mass production with inventive and very
different means of making, forms of manufacture, and
avenues of distribution within the broader context of
virtual systems, services and networks.
Over recent decades we have seen vast changes
occurring in many areas of our existence. With the
birth of computing came numerous new opportunities
in the development of technologies, completely
changing the way we function in everyday life and
bringing a resounding transformation to the domain of
Industrial Design. Here we believe lies an exceptional
chance for design to help shape the future of our world.
As creators of many of the objects and systems that
proliferate in our lives, design is in a unique position
to explore the possibilities of these new technologies.
This paper documents an ongoing sequence of projects
by both design academics and students from Victoria
University of Wellington seeking to explore these
issues. Through the process of practice-based research
we demonstrate that digital technologies are capable
of both expressing and expanding traditional notions of
workmanship and craftsmanship – to revive and embody
its emotive and meaningful qualities within these new
emerging social contexts and systems of making.
What is important, however, is that we should not overlook the role of design in facilitating the absorption of
the technological within human consciousness. For it is
precisely design that facilitates the connectivity that lies
at the heart of machinic processes, and lubricates the
processes themselves. And it is design that fosters the
‘sensuous correspondence’ with the world, that flares
up at that vital moment of assimilation afforded through
visual expression [1, p. 12].
In starting to facilitate this absorption of technology,
not only do we have the chance for more agile, flexible
and sustainable forms of production, we also are at a
point to investigate the changing role of the designer
as facilitator rather than design dictator [2], redefining
the machine to consider the social machine as a conduit of
desire [8], and digital communities such as virtual guilds [3].
Keywords
Digital craft, industrial design, digital guilds, co-design,
digital manufacture.
Design and semantics of form and movement
223
Fig. 1. Beyond the Smooth group (2002),
Fig. 2. Dan Emery (2005), Craft
Fig. 3. Ross Stevens (2006), Worn
Beyond the Smooth: CNC Machining with
without Hands © Dan Emery.
Out or Worn In © Ross Stevens.
Meaning © Maxe Fisher.
A starting point for such investigations is the notion
of digital craft and the shift from mass-production to
more customised, individualised methods of designing
and making. An initial theoretical framework for such
investigations is provided by David Pye’s eloquent
critique of mass-production in the age of the machine.
In The Nature and Art of Workmanship he argues that
the failure of mass-production is not that it is incapable
of producing quality products, but that it has created a
system of undifferentiated, uniform and characterless
products; a material culture which gives little value to
workmanship and craft, and the potential of both. Pye
noted that since mass-production is a designed system
of our own making, and, to an extent, that it is no
longer producing something we desire, then we should
construct another [4].
The workmanship of the motorcar is something to
marvel at, but a street full of parked cars is jejeune
and depressing; as if the same short tune of clear
un-modulated notes were being endlessly repeated.
A harbour full of fishing boats is another matter.
Why do we accept this as inevitable? We made it so
and we can unmake it. Unless workmanship comes to
be understood and appreciated as the art it is, our
environment will lose much of the quality it still
retains [4, pp. 2-3].
In his lament, Pye was attempting to raise the
consciousness of not only the designer and/or the
maker, but also that of the intended market, to the
world that we have produced through mass-production.
Unfortunately, Pye died in 1993 and was unable to
see the unfolding of a new revolution in design and
making, and the potential for a new material culture;
a revolution grounded in the burgeoning landscape
of new tools of digital design and manufacturing.
To understand the potential of our new material culture
of digital design and fabrication, we must understand
the fundamental background of the technology and
its inherent potentials. The field of digital fabrication
224
Design and semantics of form and movement
is composed of two general types of fabrication
technologies; subtractive and additive. Subtractive
technologies fall in the line of CNC routers and
associated fabricators, which work by eliminating
material from a larger form, until it matches a computer
model. One such example would be taking a large
plaster block and using a CNC router to remove bits
of plaster at increasing depth levels until the desired
pre-determined shape is achieved. The second group
of technologies are those that fall into the additive
camp. This includes the machines often referred to as
3D printers; machines where materials are added until
a final desired form is obtained. This can be achieved
through the hardening of material through a liquid bath,
or through the additive layering of a material.
The significant impact of both technologies is the
relief for the designer or the maker of some of the
restrictions of previous manufacturing technologies.
In the mass-production model, economy and viability
were only achieved through repeatability and restriction
of variation. The goal of which was quality control of
similar objects, through which economy of scale was
achieved. The new digital fabrication technologies
require no such repetition. They are as capable of
producing two completely different products in a
row as they are two completely identical products in
a row. In both cases there is no need for retooling, no
expensive set up; it is similar to printing two separate
documents from your computer. It is easy to see how
such flexible technology can provide a transformative
opportunity, however, when we consider the rapid
innovations and reductions in price which are making
these machines available to the general public, the
potential is all that much greater.
In 2000, the Industrial Design programme at Victoria
University of Wellington began extensive practice-based
research within this emerging field of new technologies.
Our interest in this subject was twofold. Firstly out of
a sense of necessity; the decline of mass-production
was already more than evident in our industrial neighbourhoods with the migration of manufacturing to
Fig. 4. Beyond the Smooth group (2002), Beyond the Smooth: CNC Machining with Meaning © Paul Hillier.
Fig. 5. Beyond the Smooth group (2002), Beyond the Smooth: CNC Machining with Meaning © Paul Hillier.
Fig. 6. Beyond the Smooth group (2002), Beyond the Smooth: CNC Machining with Meaning © Paul Hillier.
China. Seeking alternatives, we changed focus to the
potential of digital manufacturing technologies as a more
agile, flexible and sustainable form of manufacturing,
more suited to our geographical isolation. Secondly; it
presented an opportunity to discover, test and create
innovative new ways in which the digital and physical
worlds may begin to merge and how this may effect
on-going changes in the discipline of design. By using
the digital to push beyond our known ideas of craft as
a degree of workmanship and as a social practice, it
raises the question, is digital craft an opportunity to
explore new forms of production and social practice
while recapturing all the richness and diversity we
value in traditional forms of craft?
2 Projects
This paper documents a sequence of projects undertaken at the school, by both academics and students,
seeking answers to this question. Our first explorations
into digital craft started with a project called ‘Beyond
the smooth; CNC machining with meaning’. The project
was a collaborative study undertaken by academic Tim
Miller with undergraduate students. Their task was to
explore issues of surface, texture, form and meaning
through the direct manipulation of digital milling data.
Digital milling generally leaves the surface with a series
of fine concave grooves or tracks. The technician or
engineer will probably regard such a surface as unfinished
and will do everything possible to eliminate these traces
of the making process to achieve a perfectly smooth
surface. However Miller identified a design opportunity
in these irregularities and encouraged the students to
take control of the soft ware and to prescribe every
cut taken by the tool.
Each student was asked to investigate an architectural
project that has added to the ideas and theories of
surface and structure, and to use it as a starting point to
engage the computer as a generative tool to produce a
series of models, transforming cast plaster blanks with
increasing complexity from 3D surface through to 3D
form and space. The project engages a form of digital
technology which Pye was certainly aware of, but it
had not yet revealed its true potential at the time of
writing The Nature and Art of Craftsmanship, when he
noted that diversity in shapes and surfaces could also,
no doubt, be achieved fairly crudely by numerically
controlled machine tools, and perhaps something more
can be hoped for there in the course of time [4, p. 73].
Close on the heels of this project the School acquired
its first laser cutter. This initiated a journey of discovery
for one of our students, Dan Emery who threw himself into an intensive sequence of experiments, to
explore and exhaust the limits of this new device. His
thesis ‘Craft without Hands’ challenged precisely that
monotony and sameness achieved as a by-product of
mass-production while seeking to engage digital craft.
In a self-imposed challenge he restricted his experiments to materials like MDF and extruded plastic
sheet; industrial materials which have been processed
to the point of lifelessness in the name of technical
consistency and uniformity. In a process reminiscent
of the apprentice gaining an intimate understanding
of his tools, Emery explored not only the effects of
the speed, intensity and pathway of the laser on the
surface and structure of his samples, he also took the
investigation a step further by exploring the effects of
subverting the settings, or corrupting the digital process
with deliberate hand interventions such as mounting
the work askew. His prolific output of samples revealed
remarkable visual and tactile richness in what are
otherwise dull and banal materials.
To further emphasize his point Emery identified the table
as an everyday object with a long tradition of being both
hand crafted and mass produced – as a kind of datum, a
suitable reference point against which he could further
test his thesis. This resulted in an exploration of new
structures, new forms of construction as well as the visual
expression of the digital processes involved. As if to
acknowledge the expressive success of his approach it
Design and semantics of form and movement
225
Fig. 7. Dan Emery (2005),
Fig. 8. Dan Emery (2005),
Craft without Hands © Dan Emery.
Craft without Hands © Dan Emery.
very quickly became known in the faculty as ‘pixel’ table.
Similar opportunities exist with 3D printing or additive
manufacturing technologies as demonstrated by a series
of experiments from academic Ross Stevens’ thesis
‘Worn out or Worn In’ exploring variations in build
topography from a fused deposition (FDM) printer [5].
Fig. 9. Ross Stevens (2006), 0.5° build angle.
Worn Out or Worn In © Ross Stevens.
Fig. 10. Ross Stevens (2006), 1.0° build angle.
printing, is an opportunity for industrial design to
draw on the traditions of craft and recapture all the
depth, subtlety, overtones, variegation and diversity
which Pye celebrates as being fundamental to design.
Diversity on the small scale is particularly delightful
in regulated workmanship because there it maintains
a kind of pleasantly disrespectful opposition to the
regulation and precision of the piece seen in the large:
as when, for instance, the wild figure of the wood sets
off the precision of the cabinet-work. Diversity imports
into our man-made environment something which is
akin to the natural environment we have abandoned;
and something which begins to tell, moreover, at those
short distances at which we most often see the things
we use [4, p. 72].
Worn Out or Worn In © Ross Stevens.
Fig. 11. Ross Stevens (2006), 10° build angle.
Worn Out or Worn In © Ross Stevens.
The variations in surface textures are derived from the
printing process which builds 3D forms from very fine
(0.3mm) layers deposited by the printer. The digital
file used to create these samples was identical in each
experiment. However, intervening with the digital
process and building the samples on minutely different
angles causes the layers to create different surface
textures as an individualized and seemingly decorative
pattern over the surface of the product. The pattern
is inherent in the process and not a superficially
applied decoration.
Such interventions introduce an element of unpredictability or serendipity to the digital process and we see
a new understanding of a machine aesthetic emerging
that is not unlike the irregular structure and growth
patterns or natural diversity and variegation we find
in organic objects such as plants and shells. To what
extent the algorithm will be able to mimic these manual
interventions remains to be seen. In any event, these
projects demonstrate that engaging the digital in this
way, whether CNC machining, laser cutting or 3D
226
Design and semantics of form and movement
In terms of design education, the principles behind
digital craft – the flexibility of digital control combined
with the freedom from commitment to tooling – also
make it fertile ground for students to experiment
in ways which were never previously possible with
traditional processes of mass-production, such as
injection moulding or metal pressing. For that reason
we regard it as critical to offer these digital technologies
in-house where we can encourage students to do what
the instruction book, or external service providers,
might say cannot be done, and let them test the technologies to their limits. By customising their tools,
the designer harks back to the evolutionary role as
homo-faber or tool maker using a medium to affect
the medium, comes to know how the tool works to
open up new possibilities in the virtual stage of digital
design [where] a new level of control and exploration is
exposed [6, p. 1].
Having expanded the accepted boundaries of our
proprietary equipment as far as possible, Ross Stevens
then challenged our undergraduate students with the
impossible; to design and build a fully functioning 3D
printer in four and a half weeks.
MIG-me is an investigation of 3D welding as an additive
Fig. 12, 13, 14. MIG-me group (2007), MIG-me © MIG-me group.
process, or building thin walled vessels on a digitally
controlled rotating bed with a continuous bead of
stainless steel MIG welding wire feeding through an
electrode gun mounted on two axes. Curiously, the
build process lies in principle somewhere between
hand-coiled and wheel thrown ceramic vessels,
capturing the advantages of both; asymmetry and
vertical fluidity.
Fig. 15, 16, 17. MIG-me group (2007), MIG-me
© MIG-me group.
While the form is digitally controlled the physical
process of making, or the ‘tectonic’ forces of welding
and the fluid quality of the molten steel, are very evident
in the build quality and surface structure of the vessel.
It is a form of mass-production, but one where no two
vessels are the same – in either form or surface quality.
The point of this exercise is not so much the design
of the machine, it is what the machine produces that
is of interest to us, in both cases new and unexpected
material qualities, original and infinitely variable surfaces,
forms and structures that, with some engineering
assistance, may ultimately lead to new forms of
production. But therein lies the design challenge;
to make an aesthetic judgement in refining these
processes without losing the expressive quality of
making. Or in Pye’s terms, maintaining ‘the contrast
and tension between regulation and freedom, uniformity and diversity’ [4] in the pursuit of richer
and more poetic manufacturing processes.
This project questions our established uses of digital
tools and starts to look at how the digital can create
things that were imperfect yet perhaps more creative
than with the existing systems. It seems that it was
more interesting to look at what didn’t follow the
exacting standards of the digital and how the digital
and physical responded to and translated information.
Or as Masterton puts it in Deconstructing the Digital:
“Perhaps it will be more appropriate to learn how
to abuse, rather than use, these systems in order
to develop a creative niche” [7, p. 24].
However we are only just beginning to scratch the
surface of digital craft and the next group of students
suggest that there is very definitely more to come.
Calling themselves NEXUS, their project was inspired
by forms and structures found in nature such as slugs’
eggs and frog spawn. They identified the bubble as a
unit capable of generating complex 3D forms and
additive structures and explored possibilities with
an experimental approach best described as research
by ‘making’.
They started with the obvious; materials such as hot
glue or wax, followed by plaster and resin. There were
many others, including snap frozen water droplets, corn
starch and blown glass, as well as the ultimate material
of choice – gelatine – a water soluble, colorless, nearly
tasteless solid substance derived from collagen in
animals’ skin and bones. Being translucent it allowed
them to explore transparency as a quality of bubbles
they wanted to celebrate and thereby supported the
aesthetic intent underlying all of their explorations.
A further advantage is that gelatine sets almost instantly
in contact with air. It also proved to be versatile in terms
of scale, accommodating a wide range of bubble sizes,
thereby suggesting it may serve as a model for a more
natural process of making, not unlike cell biology with
applications ranging in scale from the micro to macro.
Design and semantics of form and movement
227
Fig. 18, 19, 20. Nexus group (2008), Nexus © Nexus group.
Fig. 21. Nexus group (2008), Nexus © Nexus group.
Experimentation with additives – initially with natural
food colouring to achieve different colours suggested
that the gelatine could also be used as a carrier for
other substances. This later proved to be an
important insight.
Material experiments slowly gave way to experimenting
with the machine itself. Bubble size was controlled
with a miniature peristaltic pump while a three axis
build platform completed the exercise and allowed
the students to generate complex 3D forms.
Fig. 25. Nexus group (2008), Nexus © Nexus group.
Fig. 26. Nexus group (2008), Nexus © Nexus group.
Fig. 22, 23, 24. Nexus group (2008), Nexus
© Nexus group.
228
Design and semantics of form and movement
The resulting structural forms combined with their
unusual expressive and material qualities invite and
inspire new and very different interpretations of what
the ultimate physical product might be. The opportunity
to individualize the form along with ability to suspend
nutrients or medication in the gelatine offers potential
in unexpected areas.
The NEXUS group suggested applications as diverse
as architecture, horticulture or health and well-being.
It is shown here as a medicinal mask or skin wrap which
can be customized to individual scanned anatomies and
infused with patient specific medication or regeneration
agents that leach out as the gelatin softens in contact
with body moisture. The customization takes place
on two levels, the physical form of the mask and the
chemical configuration of the medication. So this open
ended form of exploration without preconceived ideas
as to what the ultimate product might be, not only
suggests new and very different types of products, it
also takes the notion of digital craft beyond the crafting
of objects to include the crafting of new forms of
product services and systems.
As a form of exploration it also engages Pye’s workmanship of risk by engaging unconventional materials for
unforeseen purposes. Or as Patrick H. Harrop reflects
upon Pye’s proposition in his paper ‘Agents of Risk’ by
writing,
communities – or guilds – is characterized by the
master-apprentice model, where practitioners devote
significant time passing on their skills to the next
generation. The open source software movement
embodies the communal and highly skilled practices
of craft guilds, but without the traditional economic
function of those organizations. [3, p. 180] Participants
engage in a type of ‘technological craftsmanship’ where
the quality of the work can be its own reward [3, p. 180].
When we make, instead of predetermining action, we
discover a map of engagement. We play by challenging
and resisting material. It in turn reveals an intentional
resistance that provokes another challenge, and on and
on. In fact, craft excels in the less-than-ideal situations.
When challenged by aberrant materials, geometry and
craft are forced onto innovative discovery [8, p. 1].
As 3D printing technology continues to become
cheaper and thus more accessible to the general public
the opportunity for products to be [email protected]
emerges. To maximize the ecological and economic
potential of this it needs to be complimented with
a de-making process allowing [email protected] [9].
However, as time and methods of making progressed,
a new challenge for students arose – the challenge to
put these new forms of making into the broader context
of virtual systems, services and networks.
Or, to make full use of these new digital technologies,
not just developing new ways of making but to take it
a step further, to put these technologies into context
and develop new social and cultural scenarios of making,
in the so called ‘digital space’.
This introduces the idea of virtual guilds and the
potential for a massive shift in the way products are
designed, manufactured and distributed. With the
opportunity for learning and communicating across
time and space via the internet, it is quickly becoming
apparent that the role of the designer is changing.
Already we see online communities working together
to improve, reiterate, redesign or simply alter existing
resources by trading ideas and skills with the joint aim
of gaining knowledge, much like the craft guilds of old.
Craft is a social activity, shaped by communal resources
and motivations. The collective approach of craft
Not only does this talk of new methods of learning
but also a change in attitude towards digital making,
not simply using the technology as a tool to simplify
or speed up an existing process but a change in the
reasoning behind digital creation. To encourage this
type of thinking in yet another project, Ross Stevens
put the following proposition to the students:
The vehicle for this investigation was a $750 kitset
printer from Makerbot which the students were
expected to assemble and if necessary, to modify
it themselves. The Glomus group is a good example
of the type of response.
They envisioned the Makerbot as a creator of
prostheses for broken objects in a scenario where
broken, abandoned, unused or unloved artefacts
would not only be repaired but also enhanced with
a printed prosthesis. Their manifesto is compelling –
their proposition is that the act of making increases
the intimacy and sense of ownership between the user
and the product and thereby enhances its perceived
value and extends its lifespan. In other words you
have invested in the product, and you have invested
something more valuable than money – you have
invested time, thought and care.
They capture this philosophy in a lifestyle scenario
where, for instance; cutlery bought second-hand
is personalized and becomes your own with newly
enhanced handles, sunglasses found discarded on the
street are taken home and given a new lease on life
Design and semantics of form and movement
229
Fig. 27, 28, 29. Glomus Group (2010), Glomus, © Lulin Ding.
and the eternal problem of the wineglass with a broken
stem is solved by revitalising it with a replacement.
Interestingly, this project moved beyond the brief of
recycling products and created a process of re-making
which allowed objects to be [email protected]
It is a scenario that uses new technologies to reassert
emotional bonds between people and products.
Ironically, it is the new technology that connects us
to what we used to do – repair, rejuvenate, reuse and
recycle – and the group has achieved it with a visual
integrity and empathy for the obviously low resolution
capabilities of the Makerbot.
They foresee a future where domestic 3D printers
would be widely used and available in every household.
An accumulative online database of default digitally
modelled prosthetics could be produced for people to
access in order to mend their broken items; similar to
modern online databases like the Wikipedia community,
where all parties will benefit through information sharing [10].
Essentially, this would break down existing manufacturing
processes and mean that production takes on some of
those guild-like qualities and becomes more akin to the
creating and fixing practices of a bygone era such as
darning socks and mending furniture, but though digital
means. It also highlights the translation between both
physical and digital as a means to inform and deepen
our understanding of design.
These same issues are discussed by Richard Nelipovich,
whose thesis looked at the use of digital manufacture in
the creation of silverware.
Certainly there is value to pure formal and visual
exploration in the virtual environment, but that exploration
can only be taken to a level of synaesthetic haptic interaction with a physical material object. There is a wealth
of experience in the moment a spoonful of honey touches
the tongue that just cannot be conveyed on a screen [6].
Another project that evolved from the brief was known
as the Recyclebot. While the Makerbot 3D printer has
created the potential for digital craft to be [email protected]
230
Design and semantics of form and movement
and the Glomus project has extended this to include
found objects that are [email protected], the Recyclebot
project goes one step further, offering [email protected]
In this scenario domestic plastic (bottles, toys) are
harvested and transformed from specific and redundant
objects into generic plastic filament that can feed through
the Makerbot to produced completely new objects.
This offers the potential for a closed material loop
within the home where plastic is constantly transformed
depending on needs, desires and whims of a new
generation of digitally empowered craftspeople. They
are no longer at the mercy of a far off designer’s taste
but are now part of the design process, becoming
creators in their own right.
3 Digital Space and Collaborative Creation
Klaus Krippendorff, Professor for Cybernetics,
Language and Culture at Pennsylvania University, who
like David Pye before him, also questions the social cost
of mass production; but in this case the disenfranchising
or disconnecting people from designing and making as
a fundamental human activity.
Before the industrial era, there were millions of craftsmen,
artists, poets, and thinkers who invented new technologies, created new visions, and experimented with new
practices of living. The industrial era eradicated most
of this creative activity by enforcing the distinction
between creative designers and uncreative consumers
who had to be told how to live and what to do in the
service of mass production.
The idea of THE user is a fiction conveniently
maintained by designers who believe in their superiority
over those for whom their design is intended. These
terrible conceptions have lost their force. Increasingly,
ordinary people demand making their own choices and
designing their own environment with what they find.
I am suggesting that designing is fundamental to being
human and contemporary society increasingly realizes
the fact that making things is fun and the opportunity
Fig. 30. Recyclebot group (2010), Recyclebot, © Paul Hillier. Fig. 31, 32. Recyclebot group (2010), Recyclebot, © Jason Mackie.
to play with possibilities, and to invent rules rather than
follow those imposed by others, enables people
to realize themselves [2, pp. 14-15].
In his paper Krippendorff suggests a number of solutions
to this observation – one being the concept of
“Cooperative design” as a new paradigm or approach
to design where, as he puts it, stakeholders are involved
in the process of design. This is based on the idea of codesign and indeed, the Glomus proposition of making as
a form of social networking where default designs would
be made available through a Wikipedia type online
community starts to address at least some of the issues
Krippendorff is referring to.
We can begin to see here a potential for the digital and
the physical to mutate and become embedded in our
lives through such machines as the 3D prototyper.
practices of sociable creation – transforming production
from the anonymous activity it has become, to a more
personal and communal activity through digital guilds.
It also raises questions about the future role of
designers in society and the Glomus and Recyclebot
groups are not the only ones asking these questions.
With the expanding presence of online communities,
the role of the designer is becoming an increasingly
prevalent question. As at-home production technology
becomes more readily available, what then constitutes
a designer, maker or user and where are the lines to
be drawn, if at all?
Even if they are mechanical machines, their purpose
is to connect. They form a rhizome with the world –
a symbiosis, a symphony. Moreover, the nature of this
connectivity is dynamic. It is based on free flows and
nomadic intensities [1, p. 10].
the most significant boundary not only being crossed
but being dismantled is the boundary between
professional and amateur, or more pertinently, between
‘designer’ and ‘user’. Recent design methodology has
stressed the importance of taking a user-centered
approach, but has not envisioned a position where
designer and user are essentially one and the same.
This change in perspective has the potential to transform design education, design practice and the
consumption of design [12, p. 137].
This brings a new dimension, not only to the social
context of making, but also to the process of
manufacture and distribution. Along with the ability
to create from home, we are seeing the beginnings of
localised manufacture, the process by which products
are dreamt up in any corner of the globe then, via the
internet, sent to the digital manufacturer closest to
the products’ final destination, and turned into real
material. With these changes in making and producing
through the use of digital technologies, the use of
materials, production times and even the simple lack
of tooling cost become hugely significant to the field
of industrial design. “Freedom is fostered when the
means of communication technologies are dispersed,
decentralized, and easily available” [11, p. 5].
It is an interesting proposition – making as a form of
social practice, recapturing in virtual space our older
Across the globe, there is a bourgeoning guild of online
digital fabricators and makers. Here in Wellington we
have one of the first companies to tap into the digital
making revolution: Ponoko. Started in 2007 and now
with over 5,500 users, they are an online platform
from which anyone around the world can upload,
buy, sell or create products.
This is somewhat akin to a specialised commercial
Fablab, or Fabrication Laboratory. An idea created
at MIT and now with labs from Kenya to Afghanistan,
Iceland to India and across much of the states and
Europe, Fablabs are small-scale workshops that use
digital technologies to create personalised fabrication.
What is clear in both these examples is that the user
and ‘designer’ are both involved in the creation process,
at times even as the same person, a potential answer
to Krippendorff’s idea of ‘Cooperative Design’.
Design and semantics of form and movement
231
However, if you talk to any of the pioneers of online
design and making like Ponoko, Shapeways in Holland,
Materialise in Belgium or Fluid Form in Austria (which,
curiously, was co-founded by a New Zealander) they
will all tell you that the real challenge is to actually
facilitate the creative act of designing. What is second
nature to us as designers is in reality a complex process
and constitutes a serious inhibiting factor for anyone,
particularly the uninitiated, wanting to engage with
online design and making at a satisfying
and sophisticated level.
This leads us back to the concern of digital craft. We
are now only at the start of the digital era in making
and there is much still to be learnt, understood and
imagined. This is part of an ongoing dialogue but by
challenging students to push the boundaries of our
existing understanding of the digital, we are already
seeing the ability to create crafted products through
the dialogue of digital to physical and back again.
4 Conclusion
The interplay between the value of digital craft and
the potentials of digital space is currently a focus
of much discussion. The processes, materiality
and outcomes of our comprehensive practice-led
experimentation reveal and vividly illustrate the vast,
nearly infinite, possibilities within the digital and stand
to highlight the current shifting paradigms in the design
world. As the digital proliferates and our physical world
reels with the impact, how do we, as designers and
design educators take the seeds of these changes and
grow them into a new world of design, where digital
and physical are the mutual founders in our new lands
of creation?
Where the sea meets the land, life has blossomed
into a myriad of unique forms in the turbulence of
water, sand, and wind. At another seashore between
the land of atoms and the sea of bits, we are now
facing the challenge of reconciling our dual citizenships
in the physical and digital worlds. Our visual and
auditory sense organs are steeped in the sea of digital
information, but our bodies remain imprisoned in the
physical world [13, p. 1].
digital input to physical outcome. There is a need to
view these processes instead as a constant feedback
loop to allow for richer, more contextualised and more
intriguing design. To move beyond the old models of
‘file to factory’ [14] and begin to use the potential of
evolving design through both the digital and the physical
so that each is the better for it.
‘File to Factory’ protocols have indeed pushed ahead
our vision as designers with regards to efficient CAD/
CAM/CAE processes and yet the other way around,
‘factory to file’, has never been considered. In other
words, machine execution should not merely be
regarded simply as a service tool for materializing design
but rather an opportunity to inform the design process
as one which integrates machine-logic across all scales
of production [14, p. 2].
As digital technologies continue to proliferate, it is easy
to be swept away by their potential, and it is, indeed,
incredibly important to embrace and examine the world
of the digital. But, by moving fluidly between the digital
and the physical realms, we have the opportunity to
gain powerful insights into the possibilities of both.
The challenge now will be to not only understand
the landscape ‘growing’ from new expressions of the
material qualities we value within the traditions of
craft, but to also understand that the design innovation
potential of digital fabrication technologies is but one
small part of a larger network of design thinking, which
is ultimately changing notions of what design is and what
designs will mean in our imminent future.
It is certainly a design challenge and a paradigm shift
that we are already witnessing in industrial design,
manufacturing and mass production and it is yet to be
seen how the notion of [email protected] will challenge the
power of made in China mass manufacture. It may well
not be a question of one or the other, simply a matter
of how they co-exist. But as, Jamer Hunt hypothesised
in his Manifesto for a Postindustrial Design, the seeds
of change “are implanting themselves in the cracks of
the industrial foundation. And with that, new species
of products will soon emerge.” [15, p. 122].
Acknowledgments
A part of what can be learnt from the projects
mentioned is that a possible way forward is to cease
to view the digital world as a linear progression from
232
Design and semantics of form and movement
Beyond the Smooth Group
Course Coordinator: Senior Lecturer Tim Miller.
Students: Marc Addis, Kelly Baker, Stu Barr, Christopher
Bisman, David Chambers, Craig Chesterman, Matthew
Clark, Ashton Cogdale, Nimesh Dahya, Shelly Farrell,
Luke Feast, Liely Faulkner, Nicholas Hardman, Peta
Hird, Vera Lasut, Kenneth Mackenzie, Matthew
Moriarty, Josh O'Neill, Callum Ross, Diccon Round,
Brooke Teague, Trudy Ward, Marcus Warren, Ho Yu.
[4] Pye, D. (1968). The Nature and Art of Workmanship.
Cambridge: Cambridge University Press.
[5] Stevens, R. (2008). Worn Out or Worn In? (Master’s thesis,
Victoria University of Wellington, Wellington, New Zealand).
[6] Nelipovich, R. (2005). Adapting Craft, Adopting Technology,
Expressing Nature. (Master’s thesis, Cranbrook Academy of Art,
Michigan, United States). Retrieved July 13, 2011, from
Craft without Hands
BDes Honours Thesis. Supervisor: Professor Simon
Fraser. Student: Daniel Emery.
http://www.designercraftsman.com/thesis_draft.htm
[7] Masterton, D.H., (2007, July). Deconstructing the Digital.
Paper presented at New Craft- Future Voices, Dundee
University of Scotland. Retrieved November 7, 2011, from http://
Nexus Group
Course Coordinator: Senior Lecturer Ross Stevens.
Students: Gina Rawlings (Group Leader), Sean Antrobus,
Lisa Howell, James Mayne, Ulrich Meyer, Mark
Owbridge, Andrew Porter, Mauricio Vera, Jasun Woods.
falmouth.academia.edu/DrummondMasterton/Papers/76396/
Deconstructing_the_Digital
[8] Glomus. (2010). Manifesto. Retrieved, November 5, 2011,
from http://glomusproducts-manifesto.blogspot.com/
[9] Stevens, R. (2010). Course Outline, IDDN 341: Production
Materials and Processs [Course Outline]. Wellington, New
Mig-me Group
Course Coordinator: Senior Lecturer Ross Stevens.
Alex Keegan (Group Leader), Jonas Claudy, Solomon
Fry, Fraser Hadley, Kane Lochhead, Henry Reed,
Palarat Ronokait, Karna Sigurðardóttir, Jay Wang.
Zealand: Victoria University of Wellington.
[10] Hunt, J. (2005). A manifesto for postindrustrial design.
ID, 52(8), 120-122.
[11] Pool, I. (1983). Technologies of Freedom: On Free Speech
in an Electronic Age. Cambridge: Harvard University Press.
[12] Atkinson, P. (2010). Boundaries? What Boundaries?
Glomus Group
Course Coordinator: Senior Lecturer Ross Stevens.
Gina van Berlo, Christoph Brautzsch, Anindita
Candrika, Justin Davies, Lulin Ding, Luisa Fonseca,
Tenzin Heatherbell, Cody Law, Stewart Mcgregor,
Daniel Starkey.
The Crisis of Design in a Post-Professional Era. The Design
Recyclebot Group
Course Coordinator: Senior Lecturer Ross Stevens.
Daniel Collinson, Amelia Diggle, Jenny Drinkard, Avid
Kadam, Ben Kitchen, Jason Mackie, Lou MutsaersHoyte, Jono Watkins, Luke De Villiers.
the Age of Digital Production. (PhD thesis, MIT, Massachusetts,
Journal, 13(2), 137-155.
[13] Ishii, H. (2008). Tangible Bits: Beyond Pixels. (Master’s
thesis, MIT Media Laboratory, Massachusetts, United States).
Retrieved November 5, 2011, from http://fluid.media.mit.edu/
courses/2008-MAS672/files/TBits_TEI08_FINAL.pdf.
[14] Oxman, N. (2007). Digital Craft: Fabrication-based Design in
United States). Retrieved November 6, 2011, from http://
ambient.media.mit.edu/transitive/ubicomp07papers/oxman.pdf
[15] Harrop, P.H. (2004, November). Agents of Risk: Embedding
resistance in architectural production. Paper presented at AIA/
ACADIA Fabrication conference. Retrieved July 12, 2011, from
References
http://www.ocular-witness.com/wp-content/uploads/2011/03/
[1] Leach, N., & Weiguo, X. [Eds.] (2010, October). Machinic
agents_of_risk.pdf
Maxe Fisher,
Processes: Architecture Biennial Beijng. China Architecture
Simon Fraser,
& Building Press.
Tim Miller,
[2] Krippendorff, K. (2008, November). The diversity of
Ross Stevens,
meanings of everyday artifacts and human-centered design.
Jerad Tinnin,
Paper presented at DeSForM: Meaning.Matter.Making.
Annelies Zwaan
Retrieved August 15, 2011, from http://www.northumbria.ac.uk/
School of Design,
static/5007/despdf/designres/2008proceedings.pdf
Victoria University
[3] Bonanni, L. & Parkes, A. (2010). Virtual Guilds: Collective
of Wellington,
Intelligence and the Future of Craft. The Journal of Modern
New Zealand
Craft. 3(2), 179-190.
Design and semantics of form and movement
233
Acknowledgement
Acknowledgement
We would We
like to
would
thank
likethe
tofollowing
thank thereviewers
following for
reviewers
their valuable
for their
contribution
valuable contribution
to
to
the DeSForM
the 2012
DeSForM
conference:
2012 conference:
Georg Christof
GeorgBertsch,
ChristofAngela
Bertsch,
Blachnitzky,
Angela Blachnitzky,
Tom Djajadiningrat,
Tom Djajadiningrat,
Jeng-NengJeng-Neng
Fan,
Fan,
Maxe Fisher,
Maxe
Simon
Fisher,
Fraser,
Simon
Anne
Fraser,
Galloway,
Anne Galloway,
Paula Gardner,
Paula Morten
Gardner,Gjerde,
MortenLeon
Gjerde, Leon
Gurevitch,Gurevitch,
Bart Hengeveld,
Bart Hengeveld,
Michael Hohl,
Michael
Jun Hohl,
Hu, A.Jun
Baki
Hu,Kocaballi,
A. Baki Kocaballi,
Sloan Kulper,
Sloan Kulper,
Chih-I Lai,Chih-I
Patrick
Lai,
Janssen,
PatrickBenny
Janssen,
Leong,
Benny
John
Leong,
Lewis,
John
Ming-Huang
Lewis, Ming-Huang
Lin, CarolLin, Carol
MacGillivray,
MacGillivray,
Margaret Maile
Margaret
Petty,
Maile
Thomas
Petty,Markussen,
Thomas Markussen,
Jules Moloney,
Jules Bettina
Moloney,
Neu,
Bettina Neu,
Anne Niemetz,
Anne Niemetz,
Elif OzcanElif
Viera,
Ozcan
Edgar
Viera,
Rodríguez
Edgar Rodríguez
Ramírez, Stanislav
Ramírez, Roudavski,
Stanislav Roudavski,
Benjamin Benjamin
Salem, Marc
Salem,
Aurel
Marc
Schnabel,
Aurel Schnabel,
Kin Wai Michael
Kin WaiSiu,
Michael
Choi Siu,
Yin Soh,
ChoiHector
Yin Soh, Hector
Solis, Dagmar
Solis,Steffen,
DagmarHsien-Hui
Steffen, Hsien-Hui
Tang, BigeTang,
Tuncer,
BigeBram
Tuncer,
van Bram
der Vlist,
van der
Charles
Vlist, Charles
Walker, Anders
Walker,Warell,
AndersRegina
Warell,
WRegina
Y Yang,WAlan
Y Yang,
Young,
Alan
Salvatore
Young, Salvatore
Zingale. Zingale.
Faculty of Architecture
Faculty of Architecture
and Design, and Design,
Victoria University
Victoria
of University
Wellington,
of Wellington,
New ZealandNew Zealand
Image: Paul Hillier
Image: Paul Hillier
ISBN 978-0-475-12389-3
All rights reserved.