metacube-2014

MetaCube Paper

MetaCube: Using Tangible Interactions to Shift Between Divergent & Convergent Thinking

A research Paper submission By Haig Armen, 2015

ABSTRACT
For many decades, we have observed and studied how people create, what the characteristics of creative people are and what the process of creativity is. Many of these studies have focused on the cognitive abilities of individuals – what happens in our minds when we are creative? This paper describes a research tool for building a better understanding about how creative teams move between divergent, exploratory and convergent ways of thinking. With the proliferation of embedded technologies, there are emerging opportunities for employing tangible or embodied interaction within the creative process. In this paper, we make the case that the creative process can be augmented, observed and supported by metaphorical interactions via a hand-held tangible computing device.

Author Keywords

Interaction design; Tangible user interfaces; Embodied interaction; Design research; People-centered approach; Metaphor; Creative Process; Divergent-Convergent Thinking

ACM Classification Keywords

Human-centered computing, Interaction design theory, concepts and paradigms, Human-centered computing, Collaborative and social computing devices

General Terms
Design; Human Factors; Theory

INTRODUCTION
Today’s contemporary design teams have a wide array of tools to aid in the design process and even the most digital savvy teams still use tangible tools like whiteboards to help in the brainstorming sessions. There has been a great deal of studies in the area of creative process in the context of design and

brainstorming, predominantly about the varying exercises in divergent (generative), exploratory (connecting & combining ideas) and convergent (analytical) cognitive modes. Yet how teams or individuals transition between these modes of thinking relatively unexplored. This project explores how a tangible object might emphasize meaningful gestural interactions not as a departure from, but rather as an integrated part of the creative process. We propose that a tangible user interface will help in the creative process by shedding light on the transitions between modes of thinking. Tangible analogical interactions can be a powerful way to support modes of cognitive activity and ultimately provide a better understanding of when different strategies may be most effective. In this paper, we call to question the connection between tangible gestural interactions as analogical mappings to abstract modes of cognition by way of a conceptual prototype called the MetaCube.

To best understand how a tool could improve the creative process we first observe that creative teams are most productive when shifting between divergent and convergent modes of thinking. The ability to efficiently shift between modes may be an important feature underlying the capacity to be creative [12], and possibly, of particular importance in professions such as design [8]. There are a wide variety of creative activities, exercises and games that have been categorized into divergent and convergent categories [7] that act as useful frameworks for creative thinking and conceptual development. Physically interacting with an analogical concept makes the abstract become more concrete.
Building from the theory of embodied interaction we propose a tangible computing device that helps to bring a clearer collective understanding of how we shift cognitive modes using tangible interaction. Beyond embodied interaction, this case additionally considers the importance of flow within creative sessions as well as their collaborative nature. We hypothesize that by building a better understanding of how, when and why we shift our cognitive modes in creative sessions we can begin to create frameworks of knowledge around the collective creative process. The MetaCube project revolves around the following research question: Does rotating a tangible computing cube help creative teams better observe and gain insight into shifting between divergent, explorative to convergent cognitive modes based on specific time intervals?

Case studies of this type are important at this juncture in the area of tangible computing; as designers strive to understand what the most natural gestural affordances are for tangible user interfaces (TUI). Discovering ways of encouraging people to interact using analogy are crucial for the Interaction Design field to create a vernacular around these gestural interactions. Does turning an object towards you imply ‘inward-looking’ convergent, logical and critical thought? Does rotating an object to the right signify thinking into the future or conversely the act of rotating an object to the left representing thinking about the past or precedence of a problem space?
THEORETICAL BACKGROUND
Although cognitive modes in the creative process have been well documented, it is unclear that there are best practices in the frequency and periods in which to transition from one mode to another. Furthermore, though there are many generative and analytical activities, little has been discovered about whether certain combinations of activities are better or worse than others, or whether randomization of activities fosters effective creative thinking. Furthermore, flexible thinking involves the ability to shift cognitive functioning from common applications to the uncommon; namely, breaking through cognitive blocks and restructuring thinking so that a problem is analyzed from multiple perspectives.[12] Yet “Most do not easily switch divergent and convergent thought, but they need to do so because continued learning that blocks ideation is not helpful to the overall effort, and neither is continued ideation that blocks solution choice [2,9].
By decoding the transitions in cognitive mode we can begin to understand where we have trouble shifting and can address and improve our abilities to move easily between cognitive modes. The MetaCube aims to demystifying these mode transitions by employing theories in embodied interaction. Using tangible tools to help in brainstorming can prove to be extremely effective. As Lakoff and Johnson [10] point out, metaphor and analogy are more than mere language and literary devices, but rather conceptual in nature and represented physically in the brain. As a result, such metaphorical brain circuitry can affect behavior profoundly. For example: you may recognize that Shakespearean tragedies have a similar structure: a phase of increasing conflict between opposed sides or characters, a major confrontation between the opposed characters, and a phase in which the opposition is worked out and resolved in one character’s victory and the other’s defeat. It may then occur to you that this structure is very like the shape of a pyramid isosceles triangle, which rises from a baseline to a central point and then falls back to its baseline. You have then perceived an analogy between a temporal phenomenon and a spatial one. In the case of the MetaCube, the device represents a noun, in the context of a brainstorming session this may be the problem at hand and the act of rotating the cube is analogous to seeing the problem from another perspective. In another case study Antle [1] elaborates: Gestures may lighten the cognitive load because they are a motor act; because they help people link words to the world (e.g. deictic gestures); or because they help a person organize spatial information into speech (e.g. iconic or metaphoric gestures).
Along with modulations in cognitive modes, flow is a crucial aspect of the creative process, specifically in brainstorming sessions. In Csikszentmihalyi’s seminal book, Flow [4] is described as a state of concentration or complete absorption with the activity at hand and the situation. When exploring the requirements of our MetaCube, we must consider the flow of the individuals in the creative team. The momentum and immersion can only be achieved with the absence of interruptions from the creative team. Achieving momentum in a creative brainstorming session requires time management. Commonly time is blocked out and a facilitator is tasked with being timekeeper. A number of questions arise; what should the time period between cognitive modes be? Should each mode take the same amount of time? One of the widely adopted time blocking methods for focused periods of concentration is the Pomodoro Technique [3], which suggest 25-minute increments of activity followed by 5-minute breaks. The research on collaborative creativity is extensive and widely varying based on the type of creativity and field. The most relevant conclusions that can be draw are that shared engagement fluctuates with changes in activities within creative teams. This finding suggests that careful consideration must be taken in designing a device that will keep people’s attention on brainstorming and topics of discussion rather than on the tools being used. It is clear that a device for collaborative creativity will require the affordances of many to interact with it, not just an experience for an individual. The device will require the capability of providing a feedback mechanism that will communicate to a number of people within the context of a room and not necessarily one person like most computing devices.

Related Work
Although there are no examples of work directly related to this area on inquiry, there are a few good examples of conceptual design projects that are at all related that we may draw possible considerations from. The research project, “A cube to Learn” by Terrenghi, Kranz, Holleis and Schmidt [10] describes a Learning Cube as a novel tangible learning appliance used as general learning platform for teaching vocabulary and 3D views to children through gestures and test-based quizzes. In 2001, Terry [11] outlines a project called Task Blocks that employs blocks as the tangible interface representing computational functions for creative exploration within the programming context. The design of the system encourages hands-on, active experimentation by allowing users to directly insert, delete, or modify any function in the computational “pipeline”.

DESIGN CONSIDERATIONS

The goal of the device is to aid creative teams to collectively shift modes of thinking without losing their momentum as well as regulating the frequency of the mode transitions. The MetaCube has the potential to become a powerful tool for facilitating creative sessions by providing users with gestural affordances that create analogies while modulating through various creative thought modes. The design of the prototype must reflect the collaborative nature of creative problem-solving teams. When providing feedback to the user/team it is important that the device is able to communicate to more than one person. If color is the main mechanism to communicate the cognitive mode, it is imperative that the color be visible from all viewing angles if the team is sitting around the cube. Although seemingly unimportant, the shape of the cube is instrumental in implying specific gestural affordances. Unlike a sphere a cube’s physicality suggests rotational gestures on the X and Z-axis. Additionally, the device could possibly communicate the changing of cognitive modes using sound or wirelessly transmitting information but these options were shelved to concentrate on the core of the study, opting for a subtle non-digital form of user feedback.

DESIGN SOLUTION & RATIONALE

By creating MetaCube – a small hand-held tangible prototype capable of measuring its own rotation, we are able to address our research question. Participants use the MetaCube by rotating its orientation to mark the transition from one way of thinking to another. Imagine the scenario where a member of a creative team in a brainstorming session is prompted to pick up and rotate the MetaCube tool. The MetaCube’s orientation triggers a new glowing color that marks the transition between one way of thinking to another. The team has been told in advance the following light mappings:

1. Blue glow indicates divergent (generative) thought mode
2. Green glow represents exploratory mode
3. Red glow signifies convergent (analytical) thought mode
4. Flashing light of any color prompts rotating the cube
For example rotating the cube in one orbit would yield divergent thought mode and rotating the cube in another orbit indicates that participants proceed with convergent activities. The working prototype will be able to detect rotation and its own orientation. Once rotated on its X-axis or Z-axis the object is triggered and communicates its new cognitive mode to the team. The cube will utilize an Inertia Measurement Unit (IMU) – 5 Degrees of Freedom IDG500/ADXL335, which is essentially a combination integrated circuit board with both accelerometer and gyroscopic sensors to sense orientation and rotation. An important key feature of the MetaCube is the specific time intervals that prompt the members of the creative team to interact and change cognitive modes.

In the initial stage of exploration the modes will be communicated by use of contrasting colors and later iterations by include broadcasting activities via web applications served by the cube to surrounding computers. With a built-in web server the MetaCube could dynamically creates activity cards that are served to the client-side browsers of the team connected to the cube via a wifi network. These last features were not included in the original prototype as it was beyond the core research question.

INFORMAL OBSERVATIONS

To begin to validate the hypothesis of this study the MetaCube prototype acts as a proof of concept. The basic prototype was assembled and programmed to test amongst participants in a number of informal settings. The purpose of the cube is first explained to participants prior to their brainstorming activities. A simple creative process will be facilitated and the use of the cube will be observed and captured to later reflect upon. During the session participants’ reactions were observed, anything they said and their facial expressions, we tried to capture. Participants were then asked the following types of questions: Did the cube help or distract the team’s creative flow? Did rotating the cube strengthen the idea of shifting modes of thinking? Did the colored light help users understand the shift in modes? Could this method of observing shifting cognitive modes be useful for creative teams? We were able to informally test our assumptions by putting the prototype into a brainstorming session and explaining how the team could use it to help them shift between divergent and convergent creative activities. Our observations were generally positive but further formal studies would be necessary to draw definite conclusions.

The people within the observation session welcomed the idea and felt it was an intriguing idea in the context of creative problem solving. The interaction paradigm was easily understood and the team was able to integrate the MetaCube into their flow. The following findings were discovered from our informal study: 1. The cube helped the team creatively once the members of the team all understood its purpose. 2. Rotating the MetaCube did indeed strengthen the idea of shifting modes of thinking both for individuals and for the team as a collective. 3. The colored light did help understand the mode changes but a legend mapping the colors to mode was frequently glanced at. 4. There was a great deal of agreement that by observing shifting cognitive modes both teams and people would become more effective during creative problem-solving sessions. Additionally, we observed that although the cube was able to indicate the change in cognitive mode, the team still broke their flow by having to discuss which creative activity they would proceed with. This suggests that there is the opportunity for the device to communicate an activity.

DISCUSSION

After creating MetaCube and later presenting and explaining its purpose to various designers and writers the response was generally of interest and many began to think of other analogies to apply to the rotational interaction. Ideas were generated about ‘hinging’ from one way of thinking to another and using the metaphorical expression of “taking a 180 degree turn” to represent a pivot in direction. There was a slight cognitive disconnection between the six sides of a cube and the three cognitive modes. This added an element of unpredictability to using the MetaCube, which not all participants understood. Although this research tool was created primarily to experiment with ideas for the creative process, the prototype and its reception act as an informal validation of a possible product. In the initial conceptualization of the MetaCube it was decided that not having the cube display any digital information to minimize the perception of a computing device was in retrospection, a good decision and any further exploration of this idea will be to continue following this same line of reasoning.

8. CONCLUSION

In this paper we present a short study that investigates the benefits of a tangible computing device that enables hands-on interaction to help creative teams while brainstorming. Our contributions include a concept-driven design project and prototype. We concluded that the MetaCube shows promise as a unique tangible non-disrupting way of conducting collaborative creative brainstorming sessions. The physical interactions gave the creative teams a concrete way of thinking about when and how to transition from one way of thinking creatively to another. We concluded that effective Tangible User Interfaces (TUI) design can result in epistemic, exploratory, collaborative and cognitive benefits within the context of collaborative creative contexts.

REFERENCES

1. Antle, Alissa N. Exploring how children use their hands to think: an embodied interactional analysis. Behaviour and Information Technology (2011)

2. Brophy, D.R. Comparing the Attributes, Activities, and Performance of Divergent, Convergent & Combination Thinkers. Creativity Research Journal. 2001

3. Cirillo, Francesco. The Pomodoro Technique FC Garage GmbH 2013

4. Csikszentmihalyi, Mihaly. Flow: The Psychology of Optimal Experience. Harper Perennial (2007)

5. Gray, Dave, Brown, Sunni and Macanufo, James. Gamestorming. O’Reilly. 2010

6. Hatchuel, Armand. Le masson, Pascal. and Weil, Benoit Teaching innovative design reasoning: How concept knowledge theory can help overcome fixation effects. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, Cambridge 2011

7. Lakoff, George and Johnson, Mark. Metaphors We Live By. The university of Chicago press. 1980

8. Pringle, Andrew J. Shifting between modes of thought: a mechanism underlying creative performance. 8th ACM conference on Creativity and Cognition, 2011

9. Sak, Ugur and Maker, C. June. Divergence and convergence of mental forces of children in open and closed mathematical problems. International Education Journal, 2005

10. Terrenghi, Lucia, Kranz, Matthias, Holleis, Paul and Schmidt, Albrecht. A cube to learn: a tangible user interface for the design of a learning appliance. Personal and Ubiquitous Computing April 2006

11. Terry, Michael. Task Blocks: Tangible Interfaces for Creative Exploration. CHI ’01 Extended Abstracts on Human Factors in Computing Systems (2001)

12. von Oech, R. (1992). Creativity Whack Pack, Stamford, CT: U.S. Games Systems, Inc

last-cartographer

Where the Wild Things Are: Research Paper

Seeking Improvisation on the Open Web Platform

Haig Armen – Emily Carr University of Art & Design
John Maxwell – Simon Fraser University
Kate Pullinger – Bath Spa University

Abstract
Teaching creativity in digital media comes with its challenges. Obvious questions arise: what is the most effective creative process? what tools should be used? what is the right format? As Maslow’s Hammer reminds us, if you only have a hammer, everything looks like a nail; perhaps the challenge is that of perception. In this article we reflect on an experimental digital fiction workshop that took place in the summer of 2014. The workshop aimed to explore the boundaries of digital fiction using open web technology. The article discusses how process and tools forge the pathways and ultimately shape our outcomes. As we work to establish fully digital workflows and widely-adopted tools for creating digital narratives, we inadvertently limit our possibilities and fall into producing the same formats and modes of interaction over and over. In this nascent form, digital fiction requires efforts aimed for the fringes, to explore the boundaries of medium and genre. Today’s software authoring tools are designed for carefully composing and organizing content, not collaboratively improvising to create new forms of content, navigation and experience. Without a better understanding of the materiality of digital media gained through code proficiency and improvising on the form, we are limiting our ability to achieve more sophisticated forms of expression.

The universe of improvisation is constantly being created; or rather, in each moment a new universe is created… At any moment, an event may occur for no reason at all, with no relation at all to the preceding event… In this universe each moment is an entelechy, with both its cause and its end contained in itself.
Introduction
In June, John Maxwell, Kate Pullinger and I ran a week-long digital fiction workshop hosted by Simon Fraser University in Vancouver, BC. The experimental workshop was billed as a way to explore collaborative writing and production of digital fiction, inspired and directed by Kate Pullinger, a pioneer of the form.

Inspired by a conversation we had at Books in Browsers 2013 and a mutual admiration of Adam Hyde’s BookSprints , the idea of collaborating on a digital book workshop was born. Apart from gathering people together for an intensive limited timeframe to produce an outcome, this workshop had little resemblance to an actual BookSprint. We liked the idea of creating a framework – planning the activities and the technologies we’d use around creating a piece of digital fiction in a tight, high-intensity collaborative environment.

Setting the Stage
As the Digital Fiction workshop became a reality in the spring of 2014, we prepared by establishing the structure, content and technology for the event. Months before the scheduled workshop John and Haig discussed establishing a structure for the week of the gathering. Yet not knowing exactly what type of people the workshop would attract made it difficult to forge a strict structure. We wanted to bring together a group of interested authors and creators to participate in a real-time collaborative effort, writing and designing a multi-modal work over a short period of time. So we needed to carefully scaffold things to allow our workshop participants to move forward quickly.

John and I saw eye to eye about the kind of tools to provide to our workshop participants. For perhaps the first time in the history of publishing there is a common platform and software ecosystem that is free and open to all. The wildly popular technologies underpinning the Open Web Platform act as an alternative to the specialized, expensive toolsets that have dominated publishing (Linotype, InDesign, even Flash), the open web lets the finely tuned skills and sensibilities of authors, designers, and producers be decoupled from the exclusivity of proprietary tools. We both see Open Web technology as the only probable and preferable way forward.

In the weeks leading up to the workshop, we created a workflow that began with a wiki as a central collaborative writing hub where participants of the workshop were able to write in plain text, markdown, and HTML markup: markdown for formatting text and HTML to specify code for differentiating data types and insert media elements.

The next step was to find a way to generate HTML from the wiki. After a number of attempts John settled on using Pandoc, John MacFarlane’s free command-line tool for converting just about any flavour of markup and markdown. The final piece of our workflow was our target production framework, Caleb Troughton’s excellent Deck.js, a jQuery library designed for slide presentations that proved to be both flexible and extensible. The Deck.js library gave us the ability to create visual transitions from slide to slide, trigger html elements to animate across the screen and control audio/video—all of this, in an elegant, full-screen presentation that was compatible across multiple browsers and platforms.

Our Plan
Our intended workflow was to have our workshop participants begin sketching and storyboarding on sticky notes, index cards and whiteboards, then move onto laptops, from writing in markdown on our wiki to HTML5 via Pandoc. In the wiki, we could write a “script” for each story segment, including not only the text, but the images and audio as well. The wiki allowed everyone in the group to edit, make quick changes, create new segments, and quickly see and click through their results in a browser window.

John and I worked out extensions to Deck.js to facilitate audio and video triggers at different places, a global navigation system, and some custom effects for builds. The idea was to keep the gory details of the HTML5 production contained in an iterative production pipeline, in order to o facilitate a group of writers working out the story and how to tell it, letting them work through it in an agile, iterative way.
Observations
We gathered 12 people in June 2014 for the workshop: Kyle Carpenter, Alexandra Caufin, Jodie Childers, Jennifer Delner, Bob Fletcher, Rochelle Gold, Nicola Harwood, Inba Kehoe, Shazia Ramji, Kaitlyn Till, Jessica Tremblay. They were mostly graduate students and academics, but also writers, editors and photographers.

As Adam Hyde has pointed out, to run a successful Book Sprint you need: good people, a good venue, good food and an experienced facilitator. We had just two of these: a good venue and good people. We didn’t have the budget for catering and we were certainly not experienced at this sort of engagement!. What we did have was 15 hungry people in a room with lots of computers and technology on tap, and paper of all sizes.

Over the space of a week we came together on a storyline, a script, imagery, and ideas for animation and audio. Our workshop goal was to create, experiment and discover new tools, to explore new methods, to collaborate ‘digitally’, to forge new territory. Collectively we called what we created The Last Cartographer, a ‘neo-diluvian’ tale of love and loss, set in near-future Vancouver. By the end of the week-long session, we had a five-part, not-quite-linear narrative, mostly created using text and images, but with audio, video, and animation elements as well. (fig-0-LastCartographer.png – “The Last Cartographer’s opening screen”)

As a separate effort, The Last Cartographer was later further developed by my 3rd. Year design students and I at Emily Carr University of Art & Design in September. There are now 5 versions of the Cartographer story, based on the same story, but differing in visual and interactive treatments.

Working on paper
We liked to say that The Last Cartographer was produced entirely using free open web technologies. That was our intention but the reality is that this piece was produced mostly on paper. (fig-1-Collaborating using paper)

Our participants largely worked on paper, whether on stickies, flipcharts or index cards and whiteboards. They drew and talked, wrote and talked. In fact, probably the bulk of the week was spent sketching on paper. Why? Because paper affords extremely quick, easy, social development of ideas.
(fig-2-cardsorting.jpg – “Moving story fragments around on paper”)

It’s also worth mentioning that the workshop attracted people who identified as writers, not designers, nor coders. But despite that, they mostly produced sketches and storyboards, rather than any amount of text. They thought primarily in terms of a small bit of text in the foreground and a background image, much like a picture book. Where did this model come from? Was it from Kate’s influence via Flight Paths? Or perhaps presenting my work on CBC Radio 3 magazine during the week? Or something else?
(fig-4-sequencing.jpg “Storyboarding, sequencing and visualizing”)

We’ve all read our fair share of picture books. Could this pattern be so ingrained that it seeps into our thoughts of onscreen fiction inconspicuously enabling us to fall into the patterns – the modes of storytelling – that are most comfortable and familiar to us? John and I provided a straightforward production workflow but it wasn’t fluid enough to really allow our writers to work creatively within it. The ideas were being generated outside of the tools.

Jamming it out: The Wolf
In the weeks before the workshop, while John and I were working out how we would scaffold things together, and while we were evaluating frameworks, we threw together a little story about a lonely wolf, using Deck.js. The idea was mostly to test out what we were doing: image, text, placement, backgrounds, masking. John created a few screens, I created a few, John added more, etc. We bounced a story back and forth between us. It was a jam session, akin to two musicians tuning up our instruments and warming up. Between the two of us we have something approaching 40 years of web development experience; we improvised our Wolf story, in markdown, raw HTML, CSS, jQuery, while ‘tuning up’. In contrast, in our June workshop, we improvised using paper: on stickies and flipcharts, in words, in sketches and in arrangements.

Reflections
With hindsight we are now able to look critically on our Digital Fiction Workshop experience. We see three main points of reflection. First, the role of proficiency in creating digital fiction. Second, how profoundly the tools shape our outcomes. And third, the important distinction between composing and improvising in the context of creating digital media.

The Role of Proficiency
Our workshop participants mostly defaulted to creating on paper and pencil. Although they were all proficient with digital tools, when it came to creating collaboratively, paper simply didn’t get in the way of their ideas and creative process. Paper does a surprisingly good job of enabling fast, iterative idea development—something not lost on designers, nor software developers.

We also put software in front of our writers: easy to use tools like wikis. Yet software only gets out of the way of one’s creative inspiration when you know it intimately. This is something that John and I were able to do, jamming through our improvised Wolf story, because of many years of coding, designing, and web development. Chimero puts it well, “Let’s talk about making tools. The things we make should either reduce pain, increase pleasure, or do some mix of the two.” Yet this was not an experience open to our workshop participants. Fortunately, they found ways around it, photographing their storyboards and sequencing their narratives in a variety of crafty ways that often approached but perhaps never quite reached final resolution.

In our talk at Books in Browsers 2013, we spoke about the craft of publishing and the importance of understanding the grain or materiality of digital media. With a better understanding of materiality we are able to craft with nuance in mind. Taking advantage of the idiosyncrasies of the medium, to create more sophisticated forms of expression, forms that are appropriate for and unique to digital narratives.

In the workshop, there was a comfort with exploring and experimenting with characters, setting, themes and plot. This is where most of the improvisation occurred. Our group was less able to experiment with the presentation level with software. We believe that with a stronger understanding of the possibilities that come with digital media and some solid coding skills, our group would have been able to jam out some amazing concepts directly in the digital realm.
Tools & Frameworks
In our workshop we gave our participants tools that we thought would be open and malleable in shaping a digital piece of work. Yet even these loosely connected open web tools implied a very specific way to present content. To move beyond the prescribed way that, for instance, Deck.js wants to be used requires a level of comfort and proficiency with software of that kind. It appears that the tools we chose clearly dictated the type of narrative that would be produced. Many of the commercially available software tools for creating digital books are just as limiting.

My colleague Celeste Martin at Emily Carr University of Art + Design has catalogued the various interaction patterns that a majority of electronic books follow . Each pattern clearly details a style of navigation & content delivery – essential learning for designers with digital books. Yet these rudimentary patterns are directly linked to specific software platforms. Each platform is optimized for a specific interaction pattern and we are not easily able to push beyond the prescribed navigation and content presentation paradigms.
(fig-6-ebook-patterns.png – Celeste Martin’s Book Patterns)

Software can transcend mere utility when it allows its user to simultaneously think expansively while providing abstract constraints that help organize and build mental models. Consider how Jazz musicians use the harmonic structures of standard jazz repertoire as a shared foundation to improvise upon:

Many of the most popular jazz compositions — the standards — are repeatedly transcribed and compiled into Real Books and often used as learning tools. Real Books, as well as their many variations (Fake, Latin Jazz, Jazz Rock and, latterly, iReal Books), provide conventional harmonic sequences and phrase components that are acquired and employed as parts of each new musician’s improvisational complex vocabulary.

Just as Haftor Medbøe explains the importance of creating frameworks for Jazz musicians to build shared understanding and foundational structure for improvisational collaboration. Liz Danzico also described how designers of software could approach improvisation as a goal:

“Just as Miles Davis created a new form of jazz that allowed a new generation of musicians to play beyond themselves, so do we have the opportunity to create frameworks for audiences to create in realtime.”

Apart from iBooks Author, all the other platforms require a dual mode of creation: compose then preview, compose then preview, and repeat. This switching of modes is slow and tedious and makes the tools too opaque. Where as the visual (direct manipulation) tools are the ones that seem to melt away, becoming transparent and allowing for improvised moments. How can we seriously approach creating digital fiction without having (a) a mature, visual, Direct Manipulation toolkit, or (b) ten years or more experience with the code so that you can visualize what your code will produce before you see it rendered? The latter seems to be what experienced web developers do, thinking in code but imagining what appears in the browser. It’s a mode that clearly works, but it seems to limit creative engagement.

Composing & Improvising
Composing, in music, requires that same kind of abstract sense, where you can imagine the orchestra playing the notes you write. In the same sense, coding with web technology requires you to imagine how the browser will render your code. Most of the software we use on our personal computers today are designed for composition not improvisation. Yet improvisation is critical if digital fiction is ever going to be art.

As Alan Kay puts it,

There is the desire of a consumer society to have no learning curves. This tends to result in very dumbed-down products that are easy to get started on, but are generally worthless and/or debilitating. We can contrast this with technologies that do have learning curves, but pay off well and allow users to become experts (for example, musical instruments, writing, bicycles…)

We seem to be always striving to improvise with consumer level software but unable to reach interesting results because those tools support general patterns and conventions rather than the deep engagement that facilitates improvisation. Alan Kay urged us to improvise on the early generations of the personal computer. He wanted us to build our own tools that would allow us to put things together dynamically.

Designer Bruce Mau also suggested we make your own tools in his Incomplete Manifesto for Growth: “Hybridizing your tools in order to build unique things. Even simple tools that are your own can yield entirely new avenues of exploration.”

Creating small open tools for publishers to build their own unique platforms will amplify our capacities, allow the improvisation and fluid ideation that is necessary in creating new works of digital fiction. Even though software companies urge us toward fully-digital workflows, we are reminded that creative people collaborate well using plain paper. We aspire to carry the spirit of improvisation and brainstorming through the entire creative process and not compartmentalize it within the early pre-digital phase of a project. Improvising throughout a project has the potential of introducing media experimentation along with conceptual ideation.
Media researcher Paul Nemirovsky argues, “in order to facilitate this kind of exploration, (1) computational tools must actively participate in the creative process and (2) the interaction framework must allow structural exploration of media. This leads to our main claim: improvisation should be considered a valid and appropriate paradigm for media interaction.”

We might ask ourselves how might Stein, Woolf, Vonnegut, Burgess, Burroughs or Dahl have approached digital fiction narratives? How would they have found ways to experiment with media and create experiential narrative structures? It is not until we forge our own tools that allow us to engage deeply and fluidly with media, and meaningfully explore creativity and new genres.