Communication Design Innovation Software & Hardware Technology Usability

Interactive technology: sight, sound and touch

What we see, touch and hear pretty much defines our interpretation of the world. Stimulating these senses often has its own rewards, just talk to Apple about their iPhone. So how else might technology stimulate our 3 senses — of touch, hearing and of sight?

What we see, touch and hear pretty much defines our interpretation of the world. Stimulating these senses often has its own rewards, just talk to Apple about their iPhone. So how else might technology stimulate our 3 senses — of touch, hearing and of sight?

Stimulating the 3 senses — interactive technologies

Ever since the introduction of the “Desktop” metaphor in computing, first pioneered by Xerox PARC — then later commercialized by Apple — a familiar metaphor was merged with the new to help bridge the gap between the known office and the unknown of the new desktop computer.

Since those times, all kinds of interfaces have emerged, some of which are departure from the Desktop UI most of us are familiar with.

Many of these different user interfaces rely on the operator learning something entirely new, owing little or nothing to the things we’re used to, which most people would balk at.

But that doesn’t stop those with the ideas from thinking differently.

Moving music

As a means of input, speech is great for dictation, but it’s pretty much rubbish for anything else. Here, touch is the undisputed Iotola of input.

Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music.”

I’m reminded of Musicovery, the interactive internet radio station, which is a surprisingly simple and enjoyable way of finding new music.

You’d think that the Audiopad system would be ideal for a touch screen, certainly something larger than the iPod Touch. But that question’s already been asked:

“We believe that one of the most important aspects of the design of Audiopad is the use of multiple physical objects that move, rather than a single large touch screen with interactive buttons, sliders, etc. The main difference is that it is much easier to interact with real physical objects because your hands get passive haptic feedback from the objects which helps you move them without requiring a lot of visual attention.”

Which certainly doesn’t preclude Microsoft’s Surface technology, assuming Audiopad is Windows-based.

Audiopad is a wonderment of music production. Making music becomes one of visual sensations as much as it is about a knowledge of the sounds themselves.

As the videos illustrate, both deft and daft strokes of the hands are accommodated for equally.

But the key differentiator here has to be the shear pleasure of just fooling around and discovering new sounds just by chance.

For the more accomplished musician, those detailed and nuanced wafts of the hand must surely be a pleasure all their own.

What emerges is ultimately all about the skill of the hands and the timing.

A time to touch

“The goal of the Khronos Projector is to go beyond these forms of exclusive temporal control, by giving the user an entirely new dimension to play with: by touching the projection screen, the user is able to send parts of the image forward or backwards in time.”

As a concept, Khronos Projector certainly makes you think about space & time in a more interchangeable way. Which — if you’ll indulge the wayward amateur theoretical physicist in me — forces me consider those obscure areas of physics I’ve read about, where the 4 dimensions we live within are joined by another 7, possibly where time and space coalesce into something that can be deformed by physical means.


Dipping back into more prosaic and arguably dismissive language, the Khronos Projector can be thought of as a multi-touch video scrubbing dial.

Khronos Projector is currently a highly-sophisticated art project. My own feelings are that the art is probably less important than the idea, or even the technology hidden behind the art.

I have to wonder what kind of device might emerge should Audiopad and Khronos Projector be combined in some fashion.

What about being able to pull the surface towards you to create higher pitched tones, then pressing deeply to draw out lower, more resonant sounds.

Interactive anywhere

Imagine being able to turn any surface into one that’s interactive. That’s certainly what Xiang Cao and Ravin Balakrishnan have been working on with their “handheld projector and a pen” concept.

The thing to remember here is that the technology is very much conceptual, or it certainly was circa 2006.

In one hand is held the projector, while the other hand is free to use the pen. However, the projector itself is a pen of sorts. So it’s not like one hand is totally redundant.

As the video illustrates, it’s not like the projector moves the interactive area around the room. The entire surface of the room is itself a canvas. The projector simply limits what you can see of that canvas at any one time.

But what if you wanted something that offered you all of the benefits of collaboration, but without the limited canvas area, or the darkened room?

Jeff Han has the touch

I first highlighted Jeff Han and is revolutionary multi-touch systems back in May 2007 when he previewed them at the TED Conference of that year.

So much was made of this new take on multi-touch technology that Apple’s iPhone was met with rapture as opposed to repulsion. As I’m sure Steven P. Jobs will tell you, timing is everything.

Jeff legitimized complex interactive displays, which I understand he’s managed to sell into the military and education among others, whereas Apple commercialized and made mainstream such technology.

Taking the Apple iPhone and iPod Touch as examples of commercially-engineered touch-screen technologies, what’s the logical progression — what’s the next class of device we’re likely to see following these technology trends?

A vision of the future

Sony’s Flexible, Full-Color OLED Display certainly fits the bill.

What will “dead tree” media fast becoming an anathema in its own lifetime, the tireless pace of technology seeks out a successor to the humble newspaper.

The thing is, the newspaper is just about the most widely used interactive medium there is, albeit of a different era.

That simplicity and flexibility are still desirable qualities not to be sacrificed on a whim. So it’s understandable why these technologies have taken so long, relying as they do on advanced materials.

Way back in February 2007, a company called Readius debuted their electronic paper PDA. But much more has happened since then, clearly.

Indeed, the environmental pressures, the requirement to recycle and municipal wireless technologies, as well as micro-payment e-commerce will only hasten the arrival of electronic newspapers, like those seen in scenes from the film, Minority Report, which also includes future technologies similar to those of Jeff Han today.

We’re reaching a point in time when the unimaginable is imagined and made real. Where the unreal is commercialized and turned into a product our parents might even use.

So long as we look to the obvious are dare to imagine the unattainable, the future is visibly within reach…

Recommended reading

By Wayne Smallman

Wayne is the man behind the Blah, Blah! Technology website, and the creator of the Under Cloud, a digital research assistant for journalists and academics.