Documentation - Untitled 8 - Team Super Awesome

Untitled 8 is an ambient gesture based installation. Responsive to who ever walks in its path, it lies in wait for a curious user to interact. Using an independently run animation of a small group of sprite images, at first the installation appears to be almost a simple animated display. However when the user enters within the threshold ranges of the tracking camera, the sprites gravity to the person and follow them. And with bit of audio interaction thrown in, if a loud noise is heard, a bright flash of colour will appear and the sprite image will change.

The piece has changed slightly from when we first started designing. Originally meant to be an interactive canvas, where the users movements left a ‘drawing trail’. With brush size based off speed and brush colour off various colours in the environment. Audio was also incorporated through the use of animated effects at key noise levels. Our motivations for this were to have a playfully interactive installation that satisfied a users creative curiosity and was only limited by their own expectations. With simple controls and an inconspicuous set-up, it was not our intention to create something attention pulling.

We have done our best to achieve this even with complications. The biggest challenge was becoming familiar with the software MaxMSP and its limitations for what we need to achieve, but technical issues will be covered further on. Our installation is fully self-dependent and from that inconspicuous. User movement is still the main form of interaction and audio events trigger colour and shape changes. The only large difference is the lack of the drawing tool.

As we are still learning about interactive installations our work has a few key influences. Our orginal idea was based of the concept of a ‘digital graffit’ canvas and a version of this is Graffiti Research Lab’s L.A.S.E.R Tag. By using a camera to track a laser pointers trail on a large surface area, a project then shoots back spray can effect. Used against the sides of buildings it appears as though vandals have defeaced it. But as soon as the projector turns off the ‘paint’ disappears. See: http://graffitiresearchlab.com/?page_id=76. Camille Utterback also explores the concept of intereactive drawing installations with multplie works. Untitled 6, a coincidental name as I only found it after we named ours, creates abstract paintings based on user movement within a defined space. The paintings are also highly animated and evolutionary in their design creating a variety of shapes. Her ideals of human movement being used to express creativity in a visual way reflects our original ideals.


The technical set-up of ‘Untitled 8’ took some development. We intended to keep it all within Max but our quest for a suitable drawing tool was in vain. We did find a drawing object (jit.lcd), but found this jaggity and to basic for our needs. We also had a lot of issues with tracking. Originally we used “jit.features” to find ‘interesting’ points to draw on. But no matter what we did we would always end up with large quantities (10+) and they generally where just bright stationary points like lights. Max does not seem to classify frustrated humans waving their arms ‘interesting’.

With little luck here we moved to using “jit.blobs” to select a large area based on threshold levels in a black and white image. We limited it to only track 1 large object, instead of multiple smaller ones. In this way we can successful track a person as a single object. To display our movement, we found the drawing idea a bit daunting for our abilities and shifted it to a sprite based trail. The benefit of this is that when no ‘blob’ is found, the sketch generates its own values to continue animating. The entire visual display component is handled by Processing. MaxMSP has now been limited to being a back end of tracking both movement and audio. They are linked together by a programming Nathaniel showed me called MaxLink. This allows MaxMSP to output data to Processing easily and simply. Our technical needs are therefore limited to a camera for tracking, microphone for audio events, a computer to run the software and a wall to display on.



The output from Max is quite simple. With us selecting only the 1, largest blob. We get 3 numbers. And X value, a Y value and a radius. At the moment we are only using the X and Y. By matching the dimensions of the tracking camera to the Processing output we get a pixel perfect match of the blob location. This X and Y value are sent to Processing which then in turn creates an affectionately named “Bob” on that location. By increasing the number of ‘Bob’s’, more sprites appear. They are all in slight different locations thanks to a process of taking the x,y value and selecting a number between a small range and adding/subtracting this to the value. The other data sent by Max is the audio event trigger. To simplify things we used a simple ‘bang’ object that sends a “1” or “0” to Processing. When a “1” is received, Processing creates a large colourful burst on the background, as well as switching the ‘bob’s’ to use a different sprite image.

While I cannot comment on user experience for a wide audience, as I haven’t had the change, my own user experience was fairly satisfying. BY limiting the threshold to only pick up hair when sitting in front of my laptop, I could move my head and let the ‘Bob’s’ dance after me. By clapping, yelling or mumbling silly noises, my screen would light up with a bright colour and the ‘Bob’s would change their display style. If I moved out of the viewing range the ‘Bob’s’ would float around on their own whim, but as soon as I stuck my head back in they would stop and switch to tracking me. While not ground breaking the experience is fun and playful. It’s not ‘In-your-face’ and lets you be the boss.

Im happy with our result, even if it is not entirely what we set out to achieve. The basic principles we strived for are still existent in our installation. It has been a good learning experience and has taught be a lot about the necessary aspects and sensitivities of human interaction.

Attraction - Final Report

The installation that my group has created this semester is titled Attraction; the installation demonstrates gestural interaction; it is responsive to human movement providing both visual and audio feedback.

Our key influential work was TMEMA Messa Bounce; we all found this work to be very entertaining and were engaged by the main concepts it presented. However we wanted to make the installation original and have the ability for outside users to be able to ‘play’ and interact with. This was different from TMEMA Messa Bounce which was a demonstrative or performative work.

As a group we developed a number of ideas that were responsive to human movement however our lack of knowledge of Max/MSP and ability to effectively implement the project resulted in disregarding a number of options. Our final solution was to create an installation where movement was captured by a video camera. A moving point/s would be selected and objects would be attracted to that location.

Through the development process we struck a number of problems that arose because of our inability to successfully connect a number of objects in Max/MSP. One instance of this is that we had wanted to track movement of one or more objects simultaneously should the occasion occur that several people would interact with the installation concurrently. While it would be possible to develop this component we were unable to do this, so the tracking system that we ended up using picks multiple spots of movement and the objects flock conventionally to the largest blob/centroid.Another problem that was experienced during the development process was that after a lot of deliberation as to what form each of the boids would take on we decided upon a bee. This proved to be a bad choice as we were unable to obtain a mtl (material library file) to go with the only obj (object file) that we found of a bee. We were unable to successfully modify any other mtl file so this resulted in us using a dragonfly instead for which we had both files.

The basic flow of the Attraction installation was to obtain video data, track movement and use this to send objects to the spot of movement and change the frequency of a “buzzing’ sound. First of all we used cv.qt.grab to obtain the video data from the attached webcam. We then used cv.jit.blobs.centroids to track the movement. To be able to successfully use cv.jit.blobs.centroids we first had to convert the video to grayscale and modify the threshold so that movement was could be clearly tracked. The value from centroids was then sent to the boidroids algorithm which is a flocking system. Each boidroid took on the shape of a dragonfly and together they had a set of common characteristics which modified the outcome of the flock. We found here the most useful variable to be separation, which is quite self-explanatory but basically changes the space between each of the boids/dragonflies. Sound was added by using the x and y values from the largest centroid to modify the frequency of a sound created by line~ and modified with cycle~. We used a dac~ object for the sounds output.

When developing our initial ideas for the project and also through the process of refining our ideas, we constantly considered the user’s experience and how it would affect them. Our final work provides immediate response to a users’ movement thus a participant is able to immediately see their effect on the system and so being encouraged to further interact and play with the system. It also allows a number of people to interact with the installation simultaneously which also encourages more people to interact with it. In the installations current state, participants are required to ‘fight’ for the attraction; that is because only movement of one point is tracked so users need to become the point being tracked to see a response from their own individual movement. A key aspect of the installation and the user’s experience is that is easy to use; no knowledge is required of the technical aspect and users are immediately able to interact with it without requiring any special gear such as a particular coloured clothing. Attraction also is able to be used by a wide variety of people first because it is intuitive and easy to use and secondly because it tracks varying levels of movement so moving one arm only a small amount will see a response just as jumping up and down and running from side-to-side would elicit a response.

When observing participants interacting with the installation we found that everyone seemed to find it very entertaining and recognised the relationship between their own movement and the dragonflies. However, we also found that some people found the audio feedback to be annoying because of the ‘buzzing’ and the high pitch that was used. The image to the left is a picture from the installation, if you look closely you are able to see some of the dragonflies which move in response to a participants movement.

To further improve the project had we had more expertise in developing such works, I would like to improve the tracking system so that the flocking was more obvious thus being able to engage the user more effectively because of the increased level of responsiveness. I think it would also be beneficial to be able to include multiple tracking points that are responded to by the dragonflies, so if two people moved into the correct area then more dragonflies would appear and would flock in two separate groups for each of the moving people. I think the realism of the dragonflies being attracted to a person would be increased if each of them appeared slightly differently, such as if the rotation of each dragonfly was different to the next, whereas currently they all point in a vertical direction; also if some were different sizes it could be more intriguing.

References:
http://www.tmema.org/messa/photos.html

Traffic Nodes

This is a data visualization program that will provide a graphical representation of web server data. The primary data input is in the form of two arrays; one describing the websites themselves, and a second describing the number of visitors to each website.

The program then uses this data to generate nodes that correspond to each entry, and provide output in the form of a visual representation of visitor data, and allow the user to interact with the program to inquire as to which website each node belongs to.

The program is capable of generating the nodes and connections independently, without the need for external data.

The very first prototype was an interactive, animated version of a demonstration model produced by Drew. It was originally intended to be a ‘data cloud’ of sorts with various nodes that would allow access into a deeper level of data abstraction, with streaming data in the background.

However, while this was easy to accomplish on the top most level, establishing any amount of atomicity with recursive structures, as well as the exhaustive quantities of data required for a multi-tiered visualization, the exact scale and nature of this endeavor soon became obvious. It would not be possible to create such a complex program, given the limited time frame, limited programming experience and the already complex nature of the web server data.

An attempt was made to scale down the nature of the project, truncating the level and quantity of data, as well as simplifying the visualization model considerably.

The subsequent model was initially inspired by an example from the Traer Physics processing website, which used recursive functions to generate a tree-like pattern. [1]

Using this method of structuring levels of data abstraction was also inspired by a processing sketch found by Drew. I’ll shove this in some time once I have the link. Essentially, this example created a similar recursive structure, using individual web pages as nodes and represented connections between websites using the branches.

Using examples from the Processing website, a new design, featuring branches and recursion was developed. [2] This was a modified tree structure that grew branches at random rotations, rather than in a single, uniform direction. It also included a colour function that would change the colour slightly with each branching of the program.

This utilized simplistic array structure to control the external data; namely, a single number to determine the number of levels, and a single number to determine the number of branches per level.

While this worked initially, the project progressed beyond the data input capabilities of the program. Eventually, it was recognized that to implement interaction, each node would have to be defined as an individual object. Furthermore, a new requirement was the need for each node to support an independent number of nodes.

While this was feasible on a smaller scale, it became unwieldy and overly complex when the program was required to generate a larger number of nodes.

Another considerable problem was the nature at which the nodes were generated. By rotating each frame of reference to draw a node, this leant no history to the program; movements could not be tracked back, and it was very difficult to return to a previous frame once the frame was rotated. This made the placement of objects very difficult to implement. As such, an alternative solution was sought.

Another approach to the project was developed based on the CHIPlace navigation system, PeopleBrowser. [3]This system utilized collections of data arranged into modular circles, each representing an individual entity, grouped around a larger category. This inspired the idea that data could remain represented by nodes, all tied to a central category.

This new program reused code from the previous program. Modified functions that previously controlled the rotation of branches were throttled to rotate at a set increment, thus creating circles that surrounded a node. If a data quantity exceeded the number of circles, the program would simply generate another ring of concentric circles around the previous ring until the number of entities matched the data quantity.

These data entities would be tied to a central entity, namely a node representing a particular webpage. Thus was presented the webpage itself, albeit in a simplistic form, and a direct visualization of visitor information for each respective webpage. These webpages were then linked to a central category; in this case, Australia. As such, this program would represent the number of visitors on webpages in Australia.

This system also did not rely on recursion, and was therefore easier to organize. The system relied on a series of variable arrays to store both local and external data. This allowed the program to generate the patterns without the need for explicit external data. Additional interaction was added, allowing a user to select a particular node and receive information from the external data concerning that node. The input system was designed to be flexible, accepting any input device that produces an X and Y value.

Default dummy data was inserted into the program as a proof of concept and demonstration while awaiting the properly parsed data from Drew’s program.

This program would therefore grant users a visualization of the most important data present; the who, what and where. Which websites are concerned, the visitor data describing the website, and where this data is relevant. It also condenses the data down into its most salient and important points, thus removing the danger of information overload.

Furthermore, it was important to recognize that this was merely a demonstration principle. As such, the program was deliberately structured to be expandable and adaptive to the amount of data it received. User input was also designed to be flexible, required only an X and Y coordinate to map to the screen. This left the program open to adaption in a wide range of interaction technologies. The technologies considered were initially web-cam tracking/touch, markers and finally a graphics tablet.

Users to whom this was demonstrated (none of whom have a web-administration background) were able to understand, through brief interaction, that the data presented concerned webpages and Australia. However, many did not immediately understand the relevance of the circles, and instead assumed that it was a visualization of the hyperlinking structure of a central webpage. After some brief explanation however (literally one or two sentences), users were able to identify with the intention and data representation presented by the program. A few users also cited overlapping issues which made it more difficult to distinguish the data. This was addressed by adding a new function that would check the cursor location to node location, and allow users to drag nodes around.

---

[1] http://www.cs.princeton.edu/~traer/randomarboretum/

[2] http://processing.org/learning/examples/tree.html

[3] http://chiplace.fxpal.com/people/browser.jsp

Game of sound

___________________________________intended interface design
The intended design was to design a USB game controller/joystick as a gestural interface, so in essence a kind of digital instrument. The basic idea we began with was to use a game controller with analogue sticks, where a user would use a controller to manipulate either a pre-existing audio stream, such as changing its tempo, or to assign individual sounds to each button so a user becomes the composer. We were going to extend our idea to include the manipulation of basic 3D visuals, but decided against it as it could become too ‘visually’ dominant. Our final idea is using Logitech game controller with analogue sticks as a kind of Digital Instrument. Four tones are assigned to the buttons 1, 2, 3 and 4, and four bass tones are assigned to the bumper buttons located at the top of the controller. The left analogue stick is used to control harmonicity and modulation, while the right analogue stick is used to change volume. The d-pad buttons have no function, as there would be a function overload of the controller.


__________________________________related and influential works
The main influence for this idea was my knowledge in music. I have been playing the piano since the age of seven and hence I have a strong background in music. We wanted to create something which we had familiarity with since it would give us a confidence that it would work.
Our first idea was to design a digital instrument which would draw the sound of music. We wanted to visually represent the term “music was beautiful”. Our idea was to have a piano play and then every key would draw different lines and eventually when you play a piece, it would draw a picture. However, this idea became too difficult to implement. However, we still wanted to design something to do with music. The next major influence was the MAX/MSP website [1]. It had a tutorial about the producing sound with a game controller.


____________________________________________design motivation
A lot of my motivation came from the playground experimenta exhibition. Since I have never had much interest in interactive works and had little knowledge on interactive artworks I was not so confident in the assignment. I found it difficult to imagine this digital instrument working since I had never had much contact with interactive work. After going to the exhibition, my view changed. I was able to see the different things you are able to do with interactive art. After seeing different types of interactive works I was able to motivate myself in thinking that what we want to do is possible.
Chloe and I both have a musical background and we both enjoy video games. Video games have become increasingly popular in the recent years. Different types of video game controller and have been introduced. With this musical background we had, we were able to understand what we could do with music. Other influences include a Myspace video[2] Chloe and I found on the internet. This person took the tutorial/patch that we found on the max/msp website [1] and edited it. He came up with something very similar to what we were aiming to do. When were looking at this video. We had possibly little idea of how to change patches. We had difficulty understanding the tutorials we found online. Many of them were very complicated and beyond our abilities. However, when we found the Myspace video [2] of this person performing his masterpiece, we had hope because our idea was possible to implement. Along with this motivation, there is another motivation which influenced me more than Chloe. There is a movie called “Raise your Voice” This movie is a music movie about a girl who gets into this school but has trouble adapting to the lifestyle because of her brother dying. At the end of the movie, there is a violinist who performs and her performance turns into an electronic violin performance which makes extraordinary sounds. [3]



____________________________________________technical method
The first thing we needed to do was to buy a game controller. We did not know if we wanted to use the old game controller which was used in most video games or if we wanted to use the newer developed ones such as the Wii and Xbox 360 controller. We decided to stick with the old controller as this would be simple and because the patch used the old controller it would be easier to follow.
Before we started any implementing, we first worked out what each button would do.





We started off by using the tutorial patch we found on the max/msp website. We plugged in the game controller we bought and started playing with it. It was making horrible sounds. They were deep and had a low note. It sounded very depressing.


This is the final patch:




_____________________user experience / usability considerations

We wanted to base the user experience on two types of people; people who play video games and people who do not. We wanted to know if the people who play video games often would pick up what to do in a faster time frame than people who did not play video games or vice versa.
The interactive work would be set up like a video game to attract video game users. There would not be any instructions on how to use controller. Basically, the user has to play around with it until they make some sounds.
The game controller device was given to two people, one who was a heavy video game user; he played them almost every night for at least two to three hours.
The other who had some video game experience, but the last time she played was maybe ten years ago.


Video game users:
He started off my pressing the d-pad buttons. However when he noticed that none of the d-pad buttons didn’t work, he moved onto the analogue sticks, which did not work either. So his last choice was to move to the four buttons. When he realised he was making sound from these buttons, he tried to manipulate them by pressing the d-pad buttons. It didn’t take him too long realised that the d-pad buttons actually had no function so he moved onto the analogue sticks. After playing around with the joysticks he eventually made the conclusion that the right stick would manipulate volume and the other pitch.


Non video game users:
She started off by playing with the analogue sticks. When nothing happened, she started playing around with the four buttons and moving the analogue stick. She was starting to manipulate noise at a sooner time frame than the videogame player.










I was not surprised with the results I was able to gain because I do not think working out how to use the game controller had anything to do with how much video games the user would play. It depends on how the user would see the game controller and what they want to touch first, the analogue stick or the d-pad buttons.

_________________________________________resources/references
[1] MAX/MSP joystick patch
http://www.cycling74.com/story/2007/3/12/113645/135

[2] Myspace video
Title: dGpad v1.0
Author: The Digital Garden
http://vids.myspace.com/index.cfm?fuseaction=vids.individual&videoid=7172676

[3] Raise your voice
http://www.youtube.com/watch?v=4GECmH__yZA

Documentation of Attraction

Brief overview

Initial design

Our initial design proposal was to have entertainment in a gestural controlled interface. The body movements direct flying magnets which then stick or ‘attract’ to you. The physics came from the technical department known as electromagnetism. This occurs when an electric field changes, become a magnetic field. It happens when there is an electric current – a movement in the flow of electrons or ions. What we want is to make interface seem like it is attracted to the center of the motion which is the person. Thus he or she will fell like as if the ball is attracted to them. For example, it may start off with five balls, almost invisible as they lie on the bottom of the display. When a person moves into the area, those five balls become semi-transparent. The more movement that is perceived by the software, the more of these magnets fly to the body and the more solid in form they become. We also need to track the person so we had thought to use cv.jit.track.

Final design

Our final project actually became something similar. But instead of just balls, we used the boidroid patch and changed them into dragonflies. We originally wanted bees, unfortunately, we only had an object file and we were missing a material library file which contained the colour, texture and reflection map syntax to map the texture onto the 3d model. Nevertheless, we had dragonflies that chase after the biggest mass on the screen, produced by finding a contrast on the screen. Because we used a flocking algorithm, the dragonflies are still moving like as a real dragonfly would in its natural state. The movement also affected the sound; it included cycle~ and if there is movement, the frequency will be altered. We mapped one sound range (scaling between a set value) to the x axis, and one to the y axis. Thus, there are two sounds which was the audio output. The visual display was by using the projector, and outputting it onto a white screen. The person then stands in front of the camera

Related and influential works

Our main inspiration was the TMEMA Messa Bounce as it made a deep impression in the group. We wanted to just use body movements to impact the whole installation, which included the visuals and audio. Kirsty said not to have smile faces because it was neurotic and not have balls because it seemed too much like the Messa Bounce, so we changed to something less abstract and it was to have a swarm of insects chasing you. Unlike Messa Bounce, the dragonfly was attracted to you, and did not bounce off, but hover around you.

Design motivation

We wanted to create an interaction that was fun, but also allow a user to not need any directions to do anything. That is, to have an intuitive design that could be used by everyone. It is not so hard to figure out that there is a swarm of dragonflies following you. If we had more time and a bigger budget to work on this, we could have made the interaction at a bigger space, such that the user has ability to efficiently run away from them. Also, when there is nobody there, the dragonflies could land on a plant to rest. And when the user touches a dragonfly, it would fly away, and come back again.

User experience and usability considerations

We wanted to have the user experience a fun way to interact with insects, without having to be outside. It is easy to use the piece, except the user might end up having limited interaction as they might get tired after a while, or it might just be the same. We added audio to make it so they can create noises when they move, and be as entertaining as jumping around trying to collect all the dragonflies to you. Another way we could have approached this was to have each individual dragonfly have its own characteristics. Some may take more movement to fly to the person, and other dragonflies ‘curiosity’ might be little, so would fly away after a while; and the main challenge was to gather as much as possible. This would be better in having groups of people, trying to attract the attention of the dragonflies.

Technical method

Our project is all based in Max/MSP + Jitter. We began by grabbing a visual input from a webcam with the jit.qt.grab. We then used cv.jit.blobs.centroid to track the largest mass. If it was on a black or distracting background, we would change the threshold and make it so that it tracks white. We grabbed the x and y coordinates and scaled it to affect the position of the dragonflies. The dragonflies are controlled by a boidsroid patch which is a flocking algorithm that includes many physical aspects such as separation, cohesion and gravitational pull. Each position is varied, and the coordinates are transferred to a model patch. The patch contains the code to create a model and rendered with the textures. We also have code to clear the whole screen, so there would leave no trail. Adding a background, then put that with the jit.gl.render and open a jit.window in fullscreen – there we have our project.

Evaluation

The most difficult task in this was the implementation; Sam helped us enormously, and without the knowledge of each function in Max/Msp + Jitter, it was difficult to try and put everything together. Nevertheless, we (Chelsea, Christine and I) managed to come up with an installation that is fun and enjoyable to play with. Overall, though we didn’t manage to do as much as we wanted, we are satisfied with the end result, and hope that others will find it just as appealing.

Here are some pictures: (originally 5Mb each.. took a while to upload..)

'Mayer Hands'

General Description:
Mayer Hands can be described as an interactive sonification and visual installation. It can be categorized into the ‘gestural interactive installation’ as it uses computer vision (camera tracking) to produce an audio-visual experience.

The name ‘Mayer Hands’ comes from the use of Lindenmayer system that are implemented in the installation. The audio the participant hears is in fact a L-System. Most people are used to seeing L-Systems represented graphically whereas this installation produces increasing strings of midi notes upon hearing sound produced by the participant.

How it Works:
The systems works by tracking the colour black from a webcam mounted on the computer giving a field of view about half that of our eyes. A Max MSP patch tracks the X and Y coordinates of this colour in the frame and does two things with the results.

1. The X co-ordinate is used to select a L-System axiom. In ‘Mayer Hands’ we have implemented two systems that are distinctly different – one being a higher tone and the other a lower. As the participant is tracked they are either in one of two planes. Each plane has its associated L-System that is used.


2. The X & Y co-ords are also past onto the visual element which was produced in a Processing application. The visual treatment is a burst of particles that rain down when the user makes a sound (usually a clap). The starting position of this burst depends on the X&Y values fed into it from Max.

When a microphone picks up a noise over a specified threshold, the patch sends a few bangs into a L-System. This also triggers an encapsulated patch to traverse the L-System and output the result as midi tones. Once the L-System has been traversed the sound stops. If a user makes a sound whilst the midi downs are playing the tune will stop.

What we want the user to try is stay within one plane and produce a sound and listen to the result. Then if they clap again they will add more productions to the L-System increasing its complexity, which increases the diversity and length of the midi output. Then when they switch to the other plane and repeat the experiment they will here a different output because of the different L-System they would have triggered.

Design Motivation:
‘Mayer hands’ was the result of trying to create a playful yet intriguing piece of interactive installation. We wanted the users to have fun playing with the system and for the system to operate without guidance. Users own investigation with the piece and natural intuition would be enough for them to see the installation working. Although they might not know the complexity of the sounds they are hearing they would know that moving produced different sounds and affected the visuals they received.

As a team, Nathaniel and I were interested in using camera technology and the Midi capabilities in Max MSP and that was what we used as a starting point for our ideas. We had several ideas before settling on ‘Mayer Hands’ some were scrapped due to our technical limitations and others because we felt the idea wasn’t good enough. But the need for midi and tracking stayed with us throughout our brainstorming sessions.

As I mentioned earlier about L-Systems being represented graphically more often then with sound I was intrigued by the unique sounds they can produce. I just wish my knowledge of L-Systems were more concise as I would have liked to produce my own L-Systems to give us more control over the sound output. But the fact that a very simple axiom combined with simple ‘rules’ can create such complex and beautiful sounds was a major point in us developing this particular installation.

Related and Influential Works:
A lot of help and inspiration came from the Max MSP help patch for jit.linden and examples for how lindenmayer systems had been used to produce sound output. These were the starting points for the project and we owe a lot to the usefulness of these patches.

In terms of other works I was originally inspired by John Conway’s Game of Life as that has had plenty of re-incarnations in a variety of other artists projects- some also utilising sound. They are basically the same idea as ours – have an algorithm and then output a sound based on the output you receive.

I have had a little experience with fractal programs in the past and was intrigued into how a simple repeating rule could produce very impressive and varied results.

Technical Method:
The System’s inputs come from two sources:

1. The users position in space which is tracked by a camera looking for the colour black.


2. Sound input from a microphone that picks up sounds levels and depending on the amplitute sends the message on to another part of the patch.

The system acknowledges two levels of sound input – either quite/normal or loud. With a low amplitute the visual element will display as normal but will not be accompanied with the sound treatment. Only when the input sound is above a certain threshold (currently at the sound of a normal clap) will the L-System part of the patch kick in and produce notes. This was partly implemented to stop the system over loading with the slightest ambient noise. But we also wanted the installation to react even though no body was specifically interacting with it. So for example a conversation nearby could trigger the visual response and draw users into interacting with the installation.

We found a usual freeware program called MaxLink that enabled us to easily transfer values from max into the processing program responsible for the visuals. Processing was used as we felt we could produce a better visual treatment within this program rather then using Max MSP. The two work seamlessly together with no lag at all.
Our final patch can be divided into the following segments each having a specific role:

• Camera tracking colour black – giving X&Y co-ords.


• Scaling and inverting the X&Y values for use with Processing


• Using the X value to select a specific Lindenmayer system to use and sending that system remotely to the Lindenmayer patcher.


• Gathering information from microphone and deciding if it is above the threshold


• Passing on a bang to Processing to let it know a sound has been heard.


• The Lindenmayer patcher dedicated to traversing the axiom and outputting midi notes.

Usability & Bugs
The installation does not work flawlessly or as we work ideally like. There are still several bugs that we would like to address such as:

1. If a user moves from one plane to another whilst sound is being played (so whilst the L-System is being traversed) a new L-System gets and abruptly stops the music currently playing. Ideally we would like the current music to continue if a user has switched sides and only change/stop if a user claps or makes a sound.


2. The colour tracking is the most accurate thing, sometimes producing unexpected results which then effects the placement of the visual element.

Usability wise I think it is a very simple installation to understand and use. It requires no skill and or previous experience. If clapping was the only form to get the sound to be outputted then we could have some issues with usability for physically challenged people but as long as you are able to make some sort of sound (clicking, whistling, shouting) then the system is usable.

Overall Impressions:
Overall the experience in building this installation has been extremely enjoyable and it’s been interesting seeing the project develop from the initial brainstorming stage till its final state as it is today. I have been introduced to new software, which I am looking forward to using again in the future.

Image: Visual Element to 'Mayer Hands'

Report - Traffic Circles

Traffic Circles is a web-server traffic visualization work. By processing three-months of raw server log data, it generates a series of clustered objects representing visitor counts.



The core inspiration for this project is a common problem in large business - the creation of useful reporting data. Often, data from multiple sources must be compiled into an Excel spreadsheet, involving a high amount of manual manipulation of data – this wastes a significant amount of an employees’ time and introduces a rapidly-increasing potential for error as the complexity of a report increases.

As part of my role with my now former employer, I was responsible for the creation of reports regarding the company website and online marketing campaigns. These reports were created by combining data from three sources:

• Web-server log data – Provides aggregate information regarding the number of visitors, number of visits and page views.
  
• Online dashboard – an in-house attempt at presenting data in an easier-to-use form. While it made pretty charts, it usually posed more questions than it answered.
  
• Search Engine Marketing tool – provides data regarding the effectiveness of paid online-advertising campaigns
  

While the server log-data provided the most accurate and in-depth information, it could take between half-an-hour and four-hours to return results – depending on the time-span being reported on, and the load placed on the service by other employees. Below is a sample of the data after Sam's processing advice is provided below:

visitors,visits,pageviews,Levels,Level1,Level2,Level3,Level4,Level5,Level6,Level7,Level8,Level9,
89105,105970,138908,1,australia,,,,,,,,,
1,1,1,2,australia,contactsap?baseconfig=A1SRequestxxx,,,,,,,,
1,1,1,2,australia,contactsap?baseconfig=A1SRequestxxx',,,,,,,,
1,1,1,2,australia,contactsap?baseconfig=A1SRequestxxx&response=yyy,,,,,,,,
1,1,1,2,australia,index_events.epx,,,,,,,,
1,1,1,2,australia,platform?pageview=print,,,,,,,,
1,1,1,2,australia,sitemap?pageview=normal,,,,,,,,
1,1,1,2,australia,sitemap?pageview=print,,,,,,,,
1,1,2,2,australia,contactsap?baseconfig=A1SRequestxxx');,,,,,,,,
1,1,2,2,australia,index_configurator.epx,,,,,,,,
2,2,2,2,australia,partners?response=shared,,,,,,,,
2,2,2,2,australia,platform?pageview=normal,,,,,,,,

This project was intended to streamline the process of obtaining simple statistical information regarding the web-site in a near-real-time way. By taking advantage of the reporting tools ability to schedule complex reports, a report could be automatically generated overnight and saved to a CSV file, ready to be processed by a custom application.

While web-server log-data is a common business process, most reporting tools provide either simple statistics – with the most complex visualization available usually being a line or pie chart.

Traffic Circles aims to improve this experience by providing a simple, but rich data visualisation, using an unconventional (by business standards) interface.

Some inspirations for this project, came from the works of others. HTML Graph at mkv25.net, is a Processing based applet which draws hierarchical structures of individual web pages. The visual representation was, in my opinon, to be close to the ideal structure for this project, showing the relationship of pages to their parents.

Another work is Stack, a visualisation of the number of votes for the most popular pages on Digg.com. It simply represents each page as a bar chart, increasing the number of votes in real-time by through drips falling from the top of the screen. The traffic flow between sections was intended to be represented in a similar manner.

As another part of my role attempted to focus on the user experience and usability of the website, I took interest in addressing the problems of the original reporting solution by ensuring the projects ability to respond in real-time to user interaction. This could only be achieved by pre-processing data as the application loads, storing a large amount of simple data in memory to be visualized. Computational power is still an important consideration as the computer-vision system will compete with the visualisation for resources. Multi-core processors massively improve performance of the application.



Another user experience concern revolved around the ability to quickly manipulate the results. The fiducial marker based vision system appealed to me, as unique physical objects could be assigned significance to the application. And the relatively fixed structure of the website could allow an object to be assigned to each top-level folder. By adding or removing markers, areas of focus could be defined. The spatial position of the markers could explore relationships between site areas, and rotation of markers change the resolution. An additional marker - or set of markers – could be used to represent and manipulate time-scales.

For a more complex version, I envisage the use of fiducially-marked objects representing major sections. Rotation of these objects changes the resolution of data – showing more or fewer pages; with on-screen proximity allowing for exploration of site visitors clicking from one section to another. Given the time and resources, this would ideally be using a bottom-projected system where the markers and graphics can co-exist.

The prototype design used clusters of circles held together by an attractive force to represent each level of the website, smaller circles would orbit within this cloud representing individual pages and other files (for example: PDF or podcasts).

As the project development continued, the complexity of the initial idea became readily apparent. The first hurdle to be overcome was that of data-mining. As the project relied heavily on the processing of data into the appropriate sections, a technique for splitting the data by URL needed to be developed. At this point, a decision was made to shift development from Max/MSP/Jitter to the Java-based Processing.org to enable the large amount of raw-data (~7000 records) to be captured, processed and outputted in a useful form.

Unfortunately, the programming ability of the author didn’t allow for the creation of a solution capable of the complex data-mining required. A simplified visualization structure was developed for the prototype, instead splitting the website by level. The programming required for this proved to be much simpler and provides the core of the project as it appears today. For example, the URLs /australia/solutions/sme/product1/ and /australia/services/education/courses/ are both in level 4.

A sample structure of the company website is shown below.


The second complexity encountered was in the visualization, the initial idea of transparent, dynamically animated objects required the use of OpenGL techniques beyond the scope of this project. Given the purely aesthetic nature of this feature, it was simplified to a simple 2D representation of circles, clustering together based on the website hierarchy. This concept was undermined (excuse the pun) by the difficulties experienced in developing the data-mining, and consequently had to be rethought to represent the simplified structure provided by the processed data.

This too was overcome, with the final design being inspired by an example shown in the Traer Physics (website) package for Processing. In this schema, layers are represented as circles, with a collection of smaller circles around it to represent visitors’ counts. These are of course scaled down – as having 450,000 circles on screen at once would be both visually and computationally overwhelming.

Attraction - Design Documentation

Attraction can be seen as alluring or enticing. For our major project presentation we wanted to create a gestural interactive data driven work. That is, a visual representation of a live data input, used to create a gestural performance. The proposal in this project is for objects to hover and fly across a screen and follow a human and its movement within an environment. The idea was initially to have inanimate objects flock onto and a person's silhouette, in a magnetic form.

Inspiration was taken from the interactive installation artwork TMEMA Mesa Bounce. [i] In this video the movement and placement of the user is tracked and object on the screen react to the positioning of the person. The objective of this artwork is for balls to move towards the user, yet repel from them once they touch. This is in contrast to our artwork, where objects consistently flock towards the user as they are the centre of motion. We were intrigued by the fact that the image of the user was also part of the artwork and aimed to re-create this in our display. However due to changes in our plan, this idea was no longer feasible.

Our initial idea was to have inanimate objects stick to the user’s silhouette in a magnetic form. We then realised that this could become difficult as different objects would have to be mapped separately. This would not only take alot of time, but strenuous on the computer’s processing ability. We also realised that the objects wouldn’t necessarily tessellate, meaning that it didn’t look realistic either.

The final theme for this project was for dragonflies to hover in a familiar environment. As users moved past the sensory tracking area, the dragonflies would follow the path that the user takes, in a flocking motion where a whole swarm moves in the same direction. Movement also triggers sound, where frequencies are mapped to the positioning of the user and the amount of movement that they make.

The motivation for this piece is to create an installation that is both intuitive and fun. We also wanted it to be used by all people, even those with disabilities or mobility issues as they are still able to move and see the relationship between their movement with the audio and visual display. We were very adamant about providing both visual and auditory feedback as it was a multi-faceted display. That made sure that the user is stimulated and therefore entertained in numerous ways.

When creating this installation, we had to think about many aspects about the user’s experience. We made sure that the user will be able to interact instantaneously with the displayed objects. We also made consideration that more than one user could be detected at any given time, so we made multiple points trackable. By doing this, the user not only interacts with the project, but also with other users also being engaged by the artwork at the same time. Levels of movement were made fairy sensitive so that small movements resulted in small changes to the display, as opposed to lager movements, which display larger more erratic changes to the installation. Our plan was to make this project as easy to use as possible, so that people with no technical knowledge can still interact with the artwork, and not need to have any context or wear and sensory devices.

We were also required to evaluate the efficacy from the user’s perspective. We had to find a way to entertain the user, so that they are stimulated by the displays provided and therefore feel inclined to spend more time ‘playing’ with the artwork. We also wanted it to be as logical as possible so that all users were able to interpret and understand what they had to do to create a visual response in the artwork. We also tried to make it easy for users to understand not only visual but auditory feedback. We tried to use the co-ordinates provided in the tracking to provide x and y co-ordinates which would change frequencies as movement was detected. This is similar to visual feedback where the x and y co-ordinates provided a point for a boid, which would in turn be the centre of a flock of objects.

Looking at the project from a technical point of view, the entire project was created in Max/MSP and Jitter. We used numerous patches that we altered to suit our needs. ‘cv.jit.blobs.centroids.draw’ was used to detect objects as well as their movements and positioning. This was done by a threshold filter, and areas with difference in colours were tracked. The sensitivity and colour being tracked can be changed to suit various environments. From this numerous spots were generated, and the largest blob was used to provide input into the projection and sound display.

http://www.christinelam.com/uni/deco3005/mainpatch.gif

The x and y co-ordinates of the largest points were scaled and then used as input for the ‘boidroids’ patch as well as the frequencies of the sound. The image of rounded circle as a boid was replaced by an .obj file which is a 3d image. The use of the more realistic medium such as dragonflies meant that the user understood that they were supposed to fly across the screen. A background image was also included for this reason, and this was done so with the patch ‘jit.gl.videoplane’ where the image is also scaled and zoomed into at the desired points. The sound display is a direct link from the co-ordinates of the largest point tracked in the ‘cv.jit.blobs.centroids.draw’ patch. The sound was created by inputting the co-ordinate values into ‘cycle~’. To create more than one dragonfly hovering on the screen, the boids patch was unpacked 8 times. Each package was assigned different x and y values according to the coherence and friction assigned in the ‘boidroids’ patch.

Upon reflection, we were happy that although we had to change our interactive piece on numerous occasions, we were able to maintain the themes of video tracking and the flocking of objects to the positioning of the user. Due to technical difficulties, we were not able to explore all of our ideas and get maximum potential, but we managed to create an artwork that was able to stimulate the user on more than one level. We were happy with the setting we created, although the sound display was not as interactive or responsive as we had intended.

---

[i] http://www.tmema.org/messa/messa.html

Documentation

We set out to design a gestural interaction installation which picks up the movements of passers by and “draws” according to their movements. Different movements and sounds were to cause different brushstrokes, colours and reactions on the “canvas” which was a blank, white wall in a public area (we planned for it to be on the back wall in the hearth). The installation was to be inconspicuous and catch a member of the public’s reaction when they unsuspectingly walk by and something is created on the wall just because they did so. It was intended to satisfy the creative curiosity of the public and just be playful and fun.
As our idea developed we found that it was similar to the “Laser Tag” concept created by GRL [1]. Our idea was to “grafitti” a blank wall based on movements, whereas GRL intended for the graffiti to be created from the movement of a laser. Apart from that, our main difference was that the user was to be unsuspecting of the installation. Where the Laser Tag required users to know to pick up the laser and draw with it, our installation was to catch users off guard and then spark an interest once they realise they were able to affect the wall just by walking passed.

When creating the application, we found that it was best to use Max for our input. We needed tracking data and audio input. However, when it came to displaying the information, Processing produced more aesthetic results. Using Maxlink, the two programs were connected and the information was able to be read and displayed.
Our biggest obstacle was our tracking system. We found it difficult to track movement without the need for someone to be at the computer and selecting a point to follow. To overcome this, we used a tracking system which picked up on blobs of colour and followed them. However, this then lead to problems with too many blobs being tracked and wrong lighting causing no tracking to occur at all. With Sam’s legendary help, we found a patch which was able to track only one blob. It did this by labelling all the blobs according to size (biggest to smallest), and then taking the first blob and tracking it, by unpacking its x and y coordinates. We were able to keep to our original concept of an “inconspicuous placement” by changing the threshold and having it based on the closeness of the user from the camera. However, lighting was the main issue, as bad lighting would cause for no tracking to occur. The enivronment needed to contain a good contrast in colour so that users can be differentiated regardless of their surroundings.
Initially, we connected our x,y coordinates into “jit.lcd” so that it “drew”, however, results weren’t very visually appealing. To overcome this, we used Processing. Rather than having the user draw, our original concept was adapted so that the user’s movements manipulated the installation image. Nathan’s ambient sketch, which was pre-made with Processing, was used as the main focus of the installation. The x,y coordinates were sent into processing so that the sketch would follow the movements of the user (based on the coordinates). This concept also contrasts with Laser Tag, as Laser Tag needed user interaction in order to “exist”. Our installation doesn’t necessarily need human interaction to “work”. It is able to “exist” on its own due to the ambient nature of the sketch. Nevertheless, the sketch can be “influenced” by users who choose to interact with it.
To add another sense of interactivity to the installation, we wanted to use audio input from the audience to change parts of the installation. Initially, sounds were to cause splatters and sudden colour theme changes. However, as our concepts were changed to overcome obstacles, so did our intentions for the audio inputs. Max was used to calculate the amplitude of the sounds which occurred, sounds being claps or loud noises by the user. A threshold, which is adjustable (can pick up low sounds such as chatter, to louder sounds such as claps), was made so that a sound which was higher than a certain number would cause a bang, which in turn triggered a quick background colour change or a shape change in the Processing sketch.

Starting out, we wanted our installation to draw in users and allow them to express themselves through our interactive piece by creating new “paintings”. Our final project, is unable to produce “paintings” which display user creativity, however, it is able to give users a means of finding a connection to the installation, because they are in a sense, able to control part of it.
Whilst overcoming obstacles, we needed to take into account whether or not the changes would still be able to allow the installation to keep the attention of users. Using Processing for the visual component of the installation was important as it was much more capable of producing more aesthetic results, which would in turn, keep users interested. Allowing users to then manipulate this visual display in a manner which was straightforward was also important, because if it was too convoluted and users were not able to pick up on what they needed to do, they would have lost interest. Finally, even though users may not immediately know there was an audio component to the installation, the fact that any loud noise would set off a reaction in the piece, would allow for this realisation to occur. Hearing a loud noise and seeing and immediate reaction, is a clear indication that there is a link between the two. As it is so clear, users would be able to pick up on this and so would be able to interact with the installation in a new way.

Overall, we were able to maintain our basic concept of tracking a user and having a resultant change. Though we weren’t able to achieve everything we had set out to do, our interactive component was able to attain the interest of users to an extent as its ambient nature was just as appealing as its interactivity.



[1] Graffitti Research Lab Website: http://graffitiresearchlab.com/?page_id=76

Game of Sound: Design Document

Brief overview description

The final assignment is to create an interactive artwork of one of the following a digital instrument, an interactive data driven work, or a gestural interaction installation. Coming from a musical background, and having previously researched digital instruments, I became interested in creating one.
The final design is using a Logitech USB game controller as a kind of digital instrument, where four tones are assigned to the buttons 1, 2, 3, and 4, a further four bass tones are assigned to the bumper buttons 5, 6, 7 and 8. The left analog stick is used to control the harmonicity and modulation and the right analog stick is used to control volume. The d-pad has no function, to prevent function overload of the controller.Related and influential works

A major influence for the final project design was web examples of how to use game controllers as a sound creator and manipulator. But here are two websites that have been most helpful:
1. The Create Digital Music was the website that first triggered my interest in using a joystick/game controller as a gestural interface. The topic ‘Making Game Joysticks Work with MIDI’ provides a simple explanation of how game controllers can be used to operate music and/or live visual setups.
2. The Cycling ’74 Max/MSP website provided the tutorial joystick patch, providing a step-by-step tutorial on how to get HID data (Human Interface Design gadgets – something that provides some sort of interaction between humans and computers e.g. keyboards, the mouse, and including game controllers), into Max. This patch was used to execute my final design project.

Design motivation

A big motivation for this project was video games, in particular the interaction between a user and the game controller when they play a video game. Video games are becoming increasingly popular in this technological savvy world. An example is the November 2005 Nielsen Active Gamer Study surveyed 2000 regular gamers, found that the U.S. games market is diversifying, with male players expanding significantly in the 25-34 age group, and what use to be recognized as ‘traditionally male’ games are now considered cross-gender with a rise in female gamers (GameDAILY, 2005).
However, the biggest motivator is a myspace video, which was outstanding. In the video the creator (under the name ‘the Digital Garden’) had implemented the Max/MSP joystick patch to create the dGpad v1.0, something that resembles a DJ, ‘beat box’. By watching the video I was able to catch a glimpse on how he created such a rhythmically centred audio stream. Four buttons were assigned four separate tones; the two left bumper buttons acted as bass beats, whilst the right bumper buttons acted as a snare drum beat. To add more sound manipulations, the left analog stick controlled harmonicity, while the right analog directed modulation. This simple setup had a very effective result.

Technical method

Even with such a simple design idea I still found it difficult to technically implement it. Using Cycling ’74 tutorial as a reference guide, I began playing around the patch, trying to make sense of it. The first thing is to get the controller’s HID data into Max, which could be done using the patch below: By clicking the menu, the umenu fills up with a list of available HID devices attached to your computer, where you would select Logitech Dual Action to tell hi to focus on it. By pressing poll 10 it will start polling the device for incoming data, pressing poll 0 will stop polling. When the controller’s buttons are pressed a stream of numbers will appear in the two number boxes, where the first number is the index number, the second is the value of the control. The route object used after hi, allows the controller index to be sent to a different outlet, where route takes the first index of the list, so each output will be the value of the associated control.

At first when I tried implementing this, I had all the index numbers of each button assigned to one route. So, when the DSP was turned on and each button was pressed, it all produced the same sound. I then realized that in order to make the controls have their own unique sound, I had to create a route object for each button.

A closer look at a section of the Max/MSP joystick patch and its implementation

Once this was established, and with the help of Sam, I was able to assign different tones to each button. By connecting number boxes through the input channels of p scaling (allow smooth audio transitions), I was able to add harmonicity and modulation control with the left analog stick. Then connecting the right analog stick into the input channel of the volume number box, allowed volume to be controlled. Another way of varying the sound is changing the $ sign argument, which alters the sharpness of the tone.

Usability considerations/user experience

The popularity of video games means that a majority of people will already have some sort of knowledge of how a video game controller works. Generally people associate game controllers as a manipulation of virtual objects in a virtual graphical environment, what my design will hopefully do is challenge people’s expectations of using a game controller. A user is put into a position where instead of controlling a virtual object, they will be using it to control sound. There are no set rules on how they interact with the controller; it will purely be based on their instincts.

I wanted to see if there was a difference in interaction between a person who is a ‘familiar gamer’ and a ‘non gamer’. Although the test was not entirely accurate, as I only had two test participants, it was still interesting to see their method of interaction. The ‘familiar gamer’ being an x-box gamer, used the control in a similar manner. They were also quick to use the analog stick, and therefore found that the sound could be manipulated more if the analog stick was swivelled in time with pressing the buttons. The ‘unfamiliar gamer’ was more cautious, pressing the buttons one at a time, and not using the analog stick much, but eventually through careful exploration, they managed to manipulate the sounds through the analog stick. So, overall all, I think I have fulfilled what I had set out to create - a sound producing controller acting as a Digital Instrument, its simplicity means that the design is open for further exploration, in particular its sound generation.

Reference

GameDAILY, Neilsen Studies The ‘Active Gamer’, 2005 retrieved 4 June 2008 from
http://www.businessweek.com/innovate/content/nov2005/id20051122_489515.htm

Final Report: Design Interactive Project

Our gestural interactive installation explores the areas of sound sonification and camera tracking aided with a visual representation of the user’s motion and actions.  The installation allows the person or persons to interact inside a specified area.  The person can also clap and walk around this area.  When the person claps, it will trigger 2 things in the installation:

1.     Depending on the position of the person in the area, 1 of 2 lindenmayer sound systems is chosen and the amplitude of the clap determines which tune to output.

2.     Depending on the position of the person in the area when the sound system is triggered a burst of objects appears on a visual screen according to the position of the person, falling from the position it started to the bottom of the screen.

One of my influential public art works was one of the interactive installations that were exhibited at the Carriage Works exhibition space: Shadow Monsters.  Shadow Monsters uses shadows to create visual monsters that use the shapes of shadows to create teeth, ears and mouth.  After seeing this installation it gave me an idea on how we could track our own person…colour black.  With the precise background of a white wall, anything that is not white will be a shade of black and will be tracked using Jitter.

When Adam and I brainstormed ideas, we both brought different things that we wanted to explore, Adam really liked the use of colour tracking and in particular using coloured gloves, on the other hand I wanted to explore the realms of camera tracking and using that in a visual space.  We were both highly motivated to join our two exciting areas, as this assignment was very “free” in that there were many possibilities that could be created.  The journey from our initial idea to our final idea was very complicated with many changes, though one of the things that didn’t change was our choice of themes: sound manipulation, enjoyment, and sense of fun, playfulness.  We had these four themes and each of our ideas encompassed all of these themes, though when we technically thought about the idea, most of the time proved too complex for the expectations of this project.  We endeavoured to include both of our design motivations and areas of interest in each idea, though sometimes the ideas became unbalanced, nevertheless we both feel our final design encompasses the areas we were interested in and to a certain extent areas we had never thought of.

The technical method is outlined below in a step-by-step order:

1.     Camera tracking:

a.   Using jit.qt.grab to receive real time video, the video then passed through jit.rgb2luma to change the video’s colour matrix from a 4-plane (alpha, red, green, blue) to a 1-plane monochrome (black and white).  We used jit.rgb2luma so that we could use jit.findbounds to find any shade of black compared to white.

b.  We inserted a jit.op which converts the video so that depending on the shade of black or white the color is changed so that it is either black or white, this way the camera can track just one black color.

c.     Once Jitter had found a black patch (using the jit.findbounds) it used a series of expressions to calculate the centre of the tracked colour, with an output of a single x and y coordinate.

2.     Lindenmayer Sound system:

a.     After acquiring a single x and y coordinate we experimented with two MIDI keyboards, though it was not very complicated and we thought we could push our knowledge, so we researched and experimented with the Lindenmayer Sound System.

b.     We found we could use the Lindermayer Sound System in a way that we could use its different tunes dependent on the x and y coordinates outputted.

c.     The x coordinate is primarily used, the coordinate is fed through firstly a scale object to scale the x coordinate into 2 sections, (or a left and right section) once the x coordinate is outputted as either 0 or 1, this chooses which set of sound tunes to use.  At the moment we were able to get 2 sets of sound tunes to work efficiently.  For example, if the person was standing on the left of the screen a low set of sound tunes would be chosen.

3.     Hand Clap sound input:

a.     Our next step was to introduce a new concept that we had not previously discussed, at first we were just going to use the x coordinate to play a tune, but to make it more interesting we used a peakamp object to determine the amplitude of a single hand clap.

b.     2 levels of amplitude (soft and normal) are used:

Soft: affects the visual output

Normal: affects the audio and visual output. When a person claps normally the Lindenmayer system produces a tune, if they clap whilst the tune is playing the system will stop.

4.     MaxMSP/Jitter integration with Processing:

a.     The visual display utilizes the x and y coordinates as discussed in 1.b above.  We used a freeware program called MaxLink to link a relationship between MaxMSP and Processing.

b.     Using mxj jk.link we could input the x and y coordinates to Processing.

c.     The x and y coordinates don’t do much in Processing without the hand clap sound input.  I used a multiple particle system sketch created by Daniel Shiffman to create a visual explosion.

d.     The x and y coordinates are used to determine the position of the particle system on the visual display however it is not until the hand clap sound input is triggered that the particle system is displayed.

e.     Dependent on the x and y coordinates the particle system displays when a bang message is received and then translated as a mousePressed Processing function.

I asked a few of my friends doing Design Computing to play with our design.  One person commented, “that’s so cool” whilst another asked why every time she clapped, the particle system always displayed in the same spot, after a few goes she swapped with another person that was in close proximity to the edge of the camera’s vision, they worked out that because the person sitting on the edge of the camera was wearing a very black shirt, the tracking system tracked that colour over the grey shirt that the person (who was initially interacting with the design) was wearing.  When the person wearing the black shirt moved into the design area, the particle changed spots more regularly and they both commented that it “logically made sense”.  However I found that with a group of people in the same area the tracking system found it difficult to track all of the shades of black.

One positive feedback I received was that once they had found that just by simply moving around the space did nothing to the visual display they resorted to other forms of interaction (sound input), which was when the hand clap sound input was triggered and they enjoyed discovering what happened if they clapped quietly or moved to a different spot but reached across the space to clap, and where would the particle system display.

Overall I enjoyed this project as it allowed us to be creative and to think outside of what we already knew and to explore the areas of human computer interactivity.  We also used a blog to brainstorm and communicate ideas as well as document the final design with audio recordings, a video of myself interacting with the design and a few pictures of the visual display, which can be found here: http://web.mac.com/vlmresearch/deco3005/Blog/Blog.html. 

Advance interactive multimedia design--Interactive exhibition

Experimeta playground

Sopa (2006) Irene Iborra and Jossie Malis

This was a video produced by Irene Iborra and Jossie Malis who are based in Spain. The video starts off with a person who is hungry and looks in his kitchen cupboards to find that they are all empty. He instead uses unexpected ingredients such as sliced corks, cut up doll bits, shaving bristles and a scrubbing brush. He cooks all these ingredients in a saucepan and adds a camera (which he cracks) like and egg into the mixture. The animation used is stop animation; the sound effects that are used make the video very realistic. [1]

Once he has almost finished eating the “food” there is a bit of sauce left over in his bowl, instead of using the spoon, he uses a sponge to soak up the sauce and he eats the sponge. Now, although I thought that was very creative, I also thought it was very disgusting. I never liked the idea of sucking the juice out of a sponge, let alone eating the sponge.

I really like the idea of cracking the camera like an egg and then the computer parts come out like egg yolk comes out of an egg.

The main motivation of this art work was mainly to show the irony between culture consumption. This art piece was reversing the notion of materials transforming humans.



The manual input station (2004-2006) Tmema (Golan Levin & Zachary Lieberman)

This interactive artwork consisted of an overhead projector and a digital video projector. These two are aligned to that they overlap each other. A user places their hands over the projector and the hands are analysed by the computer vision system, the shadow of the users hands can be seen on the projected walls. In response to these postures and gestures produced by the user’s hands, it generates synthetic graphics and sounds.[1]

A better explanation may be the white shapes are produced in the gaps between the fingers, when you move your fingers, the white shapes drop and when they drop they produce a sound.

I worked out that if you place a number, it will produce a sound such as a piano note, but if you place a shape such as a love heart, and try to move your hands and fingers, no notes will play but the sound of wind will be produced instead.

If you move your hands like crazy, the notes will play intensely, but after a while, the wind noise will take over and it will not stop until everything in under control.

Coming from a musical background, this artwork really appealed to me. The aesthetics and the sounds produced were quite “pretty”. What I really enjoyed about this artwork was that when I first started playing around with this, I did not know how it was producing sound. I knew it was, but it took me a while to realise that the shapes will form when I just keep my hands still for a few seconds. There was a lot of trial and error until I realised how it worked out. This is good because it keeps the audience engaged for a longer time.




[1] Experimenta playground
http://www.experimenta.org/playground-touring/ date accessed 20/05/2008

Review event interactive experience – Help Your Self – Narinda Reeders

DECO3005 Advanced Interactive Multimedia Design-
Review event interactive experience



With the modern communication standards of the world, where information can be accessed on a whim on nearly any topic, the phrase ‘Help your self’ has never been more relevant. It is so easy to jump on the web and research into the particular topic you need advice on, but even though this might appear useful, the lack of human contact might be disconcerting. The use of call hotlines still exists, in fields such as anti-smoking and mental illness these hotlines provide us with the sense of a more personalized advice, rather than a generic mass-market answer.

“Help Your Self” by Narinda Reeders is a reflection of this simple access of self-help, but with a human based voice/visual twist. Presented in a device similar to an ATM, there is a small screen with a microphone pointing towards you at roughly head height. Your guide and advisor is Lydia, a well-dressed woman with an air of knowledge and power. If one was to stereotype you might assume she was a high-level corporate office manager or similar. Communication is based on the users voice. By speaking into the microphone, Lydia listens and replies, advising you on the answers you seek, or so you assume.

The first hurdle is her understanding you. As its voice driven, often Lydia has difficulty understanding exact words. I found this slightly encouraging and amusing more than an annoyance. But I approached this work from an enjoinment point of view, not as a serious form of advice. As I proceed through various questions I realized the piece was also humor based, and not designed to be serious.

My name was mike, I was 80 (I did say 86, but what’s a few years difference), and I had relationship trouble. I was feeling very guilty. Apparently Lydia advised that I wasn’t confronting enough in my relationship, and I need to take more power and express my unhappiness and guilt. Lydia’s responses with thoroughly entertaining, one of my favorites was a loud “No!” when I said I rarely express my emotions, it got a turn or two from some bystanders.

The technical properties of the device are kept hidden from the user. The only input being the microphone and feedback is generated on the screen and through the speakers that Lydia speaks through. Lydia is presented as still images, with a small selection of poses that match her tone and response. With the process hidden from the user, we don’t really think about what is occurring or how Lydia computes her answers. Often if she ignores responses, I said “banana” at one stage, but she didn’t pick up on it. This shows there is a database of user responses she responds to, namely “ rarely, sometimes, all the time, yes, no, back, start again etc” and a flow chart style hierarchy of her response after.

From the piece I realized how abstract you could turn a simple idea of helping someone into a harsh, unloving piece of technology. With no grasp on the events, only the principles of what’s occurring, generic answers for complex situations result in only a humors response because of how unconnected to the situation they feel. Only a fool would take the advice seriously. But maybe that’s the point, should we search around for advice on our problems, finding answers that make sense on the surface, but really have no merit. Or stick to our own guns and solve them ourselves?

Images: Flickr.

event interactive experience review: beta-space +-now

As part of the collaboration between the UTS Creativity and Cognition Studios and The Powerhouse Museum, there is a permanent exhibition in the Powerhouse Museum called beta-space. The current display in this exhibition is called +-now by Jen Seevinck.

+-now is an interactive art system using a tangible interface and a simulation agent being developed through iterative methods.[1]

The piece consists of a wooden box with perspex on the top, and sand placed on the perspex. There is a video camera inside the box facing the perspex which would track where the sand is and isn't. This camera is to sense where there is an isn't sand, presumably to provide the system with an on/off input which controls what is being displayed. There are also 2 projectors which display imagery according to the positioning of the sand. One projection is on the wall in front of the box, and another is onto the sand itself.

In terms of usability, this installation is extremely easy to use, although not necessarily easy to understand. Anybody with the ability to move sand can interact with this tool, although they may find it difficult to understand. The theory is simple enough - move sand around on a piece of perspex and where there is sand, the screen shows a different shade to where there isn't sand.

However, due to the sketchy 'çrawling' nature of the exhibition, the extreme sensitivity as well as the extremely long response times, I saw that many people found it difficult to understand how the installation worked, especially young children who were expecting more of an instantaneous display.

Jen Seevinck is an electronic artist and researcher with a particular interest in interactive art, emergence, perception of the Gestait, tangible computing, virtual reality and medical simulation. Her works are largely art-practice based. +-now is an extremely fast based environment with a theme of getting lost within the activity. The idea was for users' gestures to be tracked and monitored, and displaying it back to the user in an echoed liquid form. Because of her aesthetic approach to her artworks, Seevinck has included two projections in this piece. This is presumably to engage the participant in more than one level.

I don't consider there to be too much consideration towards context and culture in this piece as it is not site specific and there is no targeted audience. This way the artwork can be installed in any enclosed area facing against a blank wall. Because there is no sound in this installation, it can be installed in busy areas although it is much more effective when it is in a secluded room as it is in this case. That way the user can then immerse themselves in the visual aspect of the installation.

I found that this installation wouldn't have been too difficult to create, as my boyfriend and I were able to create a very similar device with a webcam, cardboard box and a clipboard. Even though we only made one display, this proves that the concept itself is extremely basic, yet still has the potential to engage the user through its different projections and the use of sand as an input mechanism. I found that the display was too sensitive and it was difficult to use as there was clearly less sand on there than when this piece was first installed. The echoing nature of the images and the long response times made it difficult to create the image I wanted on the screen, although I was still engaged by the piece. I was also particularly interested in this piece as it has many similar characteristics to the project that I intend on making for my final project and therefore I am interested in discovering the way that it works.










[1] http://www-staff.it.uts.edu.au/%7Ejseevinc/

All images and videos by Christine Lam and Mark Priest

Assignment – Review Event Interactive Experience

Recently, I visited the Performance Space at CarriageWorks which showed the Experimenta Playground, the works proving “that play is a means through which we are able to make sense of the world”.[i] The most insightful piece of installation was Immersion (2007) by Angela Barnett, Andrew Buchanan, Darren Ballingall, Christian Rubino, and Chris Mackellar. [ii] This group of five, involving 3D animators, game developers, film makers and audio engineer, has extensive background in interactive multimedia which ultimately produced this installation.[iii]

Immersion is a simple screen-based display involving a light shining down to the mat which projected various seascapes and sea creatures which the audience to interact with. It is a 3D real time display as you use your shadows to interact with the animals which swim by. The director, Angela Barnett, was inspired by the rock pools in Newcastle. The motivation was to have a “total body engagement” and "the reaction of the user, and how the user's going to interact with it”.[iv] It made me feel like as if I was standing on a glass bottom boat, except what was more exciting is that the underwater creatures reacted.

I liked this project because it represented the fish like our major project using boids – a simulation of flocking behaviour. Our shadows make the fish scatter and they may also change their structure and behavioural aspects. The most interactive animal I played with was the puffer fish, bloating on contact. You can also place your hand over starfish, and they would change colour, however trying to twirl the limbs require more movement, and there is a slight lag in their responses. Unfortunately, due to the 3D rendering in real time, much desired features (and speed) had to be forfeited.



It took a short while to realize that standing on the screen did not impact the organisms more than standing on the outside and extending arm to wave around. Each animal had its own response to the interactions. One, shown in the video, acted as if you were tickling it then swims away. Another is repelled and the other seems to follow the direction of the shadow. If there were more creatures, the installation would have been too confusing to deal with. Much of the surrounding installations influenced the sound, or lack of, which was heard in the exhibition. It would have been more mesmerising if the project was placed in a studio room with no disturbances.

Immersion is a creative work which uses technology (but hides it from audiences) where only by interacting, a story unfolds. The innovative method of using shadows as the controller draws the attention away from ‘mainstream’ art as intuition guides us to play with the underwater environment. This artwork transforms the notion of going to quiet galleries, look and not touch, but to play, interact, even walk on the project. It has become personal.


[i] “Experimenta Playground: International Biennial of Media Arts” (Accessed via IE, 22nd May 2008) http://www.unisa.edu.au/samstagmuseum/exhibitions/upcoming.asp
[ii] “Exerimenta Playground” (Accessed via IE, 22nd May 2008) http://www.experimenta.org/playground-touring/
[iii] “Experimenta Play ++” http://www.experimenta.org/top_experimentaPlay.htm
[iv] Maunder, P. “All grown up” (Accessed via IE, 22nd May 2008) http://www.smh.com.au/news/technology/computer-animation-finally-comes-of-age/2007/09/19/1189881523650.html?page=3

Performance Space @ Carriage Works

(Yes, everyone knows about Carriage Works Drew and Eric, I'm glad I went! Here are some sample images and footage)

Images

Performance Space @ Carriage Works

Experimenta Playground

Shadow Monsters

What's Yours Is Mine

Game Over Project: Pong, Space Invadars, and Pole Position

The Manual Input Station

Charmed

Guide to Recent Architecture: Fountains

Movies

Drew and Eric in Shadow Monsters

Florence and Kristina in Shadow Monsters



Drew and I in Shadow Monsters



The Manual Input Station

For more information visit http://www.experimenta.org/playground-touring/

For more pictures visit http://www.facebook.com/album.php?page=1&aid=21157&id=500257437

Review Event Interactive Experience

Experimenta Playground

Experimenta Playground is great, highly interactive exhibition exhibiting featuring media work from artists around the world, including interactive artworks, video installations and extreme sport on screen.

I have chosen three interactive works, which I absolutely enjoyed playing with:

The Manual Input Station, Golan Lexin and Zachary Lieberman (2004-2006), USA

This interactive artwork physically consists of an overhead projector, a digital video projector and custom interactive software. The overhead and digital projectors are aligned so that their projections overlap, creating a striking, hybridized pink light. As the user, I interacted with the system using my hands and shapes provided, by placing them on the overhead, where they were analyzed by a computer vision system. As a response to my movements, the software generated artificial graphics of simple shapes and colour, and accompanying sounds, depending on the forms and movements of my actions. What makes this work so attractive is the artificial responses are projected in line with the shadows generated by users, resulting in a magical, augmented reality shadow play.

I think the artists have created this work as a way of exploring new modes of reactive expressions, in particular to explore the nature of communication and the hazy boundary between what is visible and the invisible.

Immersion, Angela Barnett, Andrew Buchanan, Darren Ballingall, Chris Mackellar and Christian Rubino (2007), Australia

Immersion- a still image of a person interacting using their hands and feet to generate shadows that influence the deep sea environment [1].

Enter into a under water scene, where you can literally ‘swim’ and play with exotic sea creatures. Using Immersion this group of artists have created a virtual seascape- a shallow rock pool, an underwater cave and the dark depths of the ocean, allowing for a fantastic interactive 3D real time experience, where the audience can connect with animated creatures of the sea. The movement of the shadows generated by me (the user) can change the movement, shape and colour of the sea creatures. Similar to real life, schools of fish scatter, and puffer fish swell, as well as more imaginative responses to shadows where electric eels generate lightening bolts, while jellyfish glow.

Along with the visual display, the virtual contact also produces an underwater symphony, where sea creatures produce their own unique sound. I discovered that as you dwell deeper into the ocean the sea creatures become increasingly lively, exaggerated and unreal, pushing the imagination to find new ways of interacting with the sea life.

Shadow Monsters, Philip Worthington (2005), UK

This interactive work is something you would envision in a nightmare. The audience is invited to stand in front of a light box, that immediately captures our bodies, gestures and clothing. These shadow then metamorphose into monster-like creatures, with the addition of horns, hair, beaks, eyes, teeth, tongues, claws and triangular scales. To manipulate these ‘monsters’ the audience is encouraged to move- I found that the more dynamic your movement results in continual transforming of the monsters, some even recognizable such as crocodiles with razor teeth. Shadow Monsters uses vision recognition technology, heightened by a soundtrack layered with animal noises, presenting an outstanding audio-visual representation of human shadows as monsters. The audience is challenged to involve themselves in spontaneous play, competitive posturing and narrative interaction, making for a fun and frightening experience.

Reference:

[1] Immersion [Image], retrieved 20 May 2008 from http://www.unisa.edu.au/samstagmuseum/exhibitions/upcoming.asp

Event Review: Shadow Monsters

Yes, another review of Shadow Monsters. As Eric said, this is partially Rob's fault for mentioning it, partially out of geographic convenience for those of us who don't live close to the City but mostly because Shadow Monsters is just super-cool! But without further ado:

Shadow Monsters is a computer-vision based interactive artwork by Philip Worthington, currently shown at the Experimentia Playground exhibition at Carriageworks, Redfern.

A wonderfully high-quality cell-photo sneaked in despite the watchful eye of the attendant...

Shadow Monsters plays on the concept that monsters lurk in the shadows – beyond what people can see. It creates digital manifestations of monsters in the shapes and outlines of participants’ shadows. Based on my observation of several exhibition attendees, the silhouettes of sharks or dinosaurs in the persons own form is initially disconcerting. However, the responsiveness of the work quickly allows the visitor to realise that the monsters react to their movement, and can be controlled to some extent by specific movements – participants were generally amused by the unexpected behaviour of their shadows.

The metaphor of Shadow Monsters also allows people to behave as if they were casting shadows on a wall using a torch. The hand gestures used in this activity were recognized by the computer software, creating complex monsters complete with cartoon teeth and burping effects.

Of the interactive art exhibitions currently shown in Sydney, I felt this was the most relevant to my project due to its transparent use of computer-vision based gestural interfaces. By removing the computer from view, it enables participants to interact with the system without the usual preconceptions of computer interfaces. While observing other exhibition visitors using the system, it was apparent that these participants quickly discovered how to make the shadows react and were generally amused by the unexpected behaviour of their shadows.

The usability of this work was inherently high as an interaction could be formed simply by being within the field of view of the camera, while motion altered the behaviour of the on-screen monsters it was not crucial to the monsters creation. For example a mobility-impaired participant would still be able to create monsters – it would also be conceivable that unexpected reactions occur to the silhouette produced by a user using a wheelchair. The work also produced a aural component to the users presence and motion allowing visually-impaired users an opportunity to interact with the system, and again unexpected results (from the designers perspective) may be created.

From a technical perspective, Shadow Monsters is an installation artwork, requiring the manufacture and installation of specific hardware for its exhibition. Based on the exposed components of the work, it required a surface created from frosted Perspex, backlit with fluorescent tubes as a back, this served to provide a high-contrast silhouette of users for the camera to track. The digital video camera is an external Apple iSight mounted below the projection screen. A projected located behind the installation rear-projects the monsters onto the screen – allowing users to step into the space without blocking the projection. A computer system running the custom software is hidden somewhere inside this enclosed space.

Crucial to the success of Shadow Monsters is the responsiveness of the system. With interactive systems, the latency (time between a user-action and the generated response) can greatly impact the participants perception of their interaction – monsters with poor reaction time aren’t going to be very scary. Most modern computer systems are powerful enough to process video in real-time, but during software development, care must be taken to optimize program code to remove – where possible – sources of unwanted latency.

Finally, Shadow Monsters allows for people to interact with others within the system, by attacking each others shadows on-screen and by standing at differing distances from the camera, utilising the scale and perspective effects to tower over another users shadows (or conversely be smaller than the other user).

Interactive Exhibition: The Wedge

The Wedge
Sydney Powehouse Museum

About the Wedge
A recent visit to the Powerhouse Museum in Sydney had me interacting with an installation piece called ‘The Wedge’. Situated in the permanent exhibition ‘Cyberworlds’; The Wedge is a stereoscopic projection theatre – that enables users to immerse themselves in a 3d virtual space and interact with full 3d versions of scientific visualizations. A project started by the guys from the Australian National University- notably Drs Rod Boswell and Henry Gardner.

The version of The Wedge at the Powerhouse museum is a specific version designed to be suitable for the museum space, to withstand high numbers of untrained participant’s, and provide a short demo showcase of a limited sample of objects to encourage visitors to circulate

Interacting with the Wedge
Approaching the Wedge I was met with two large rear projected screens and a approximately 5 special headsets places along a rail for users to place on their heads. When looking at the screen without the headset I saw a flat, not particularly clear or high resolution, image of some scientific model/molecule. Although I had initially missed the instruction/introduction screen which informs the user to mount the headset it wasn’t difficult to understand the user requirements.
Placing the headset on changes your whole perception of space. Before I was staring at a 2d version of a ‘Bucky Ball’ split down the middle between the two screens (which join at a right angle), now I’m staring at a full 3d version of the same object which is floating mid air less then a meter away from me – so close I can almost touch it.

Placed in front of me is a joystick which enabled me to interact with the object. I was able to move, scale and mould the object by shifting this joystick on 2-axis and pressing the two buttons available to me. This was a very enjoyable part for me as I was able to move this 3d object so close to me that I was practically inside the object – a very strange but enjoyable experience.

Roughly every 2 minutes the object would change to the next object (see here for entire images. What I noticed was that some of the models worked better then the others. The ‘Gyroid’ worked best in my opinion for creating that illusion of 3d but I’m not sure why. I also noticed that the 3d effect was reduced the more you moved the object to the far corners of the screens – on some modules completely losing the 3d effect.

In terms of responsiveness the installation worked very well. There wasn’t any delay in the system responding to my input from the joystick. The only delay I experienced was actually waiting for my go as it was apparent that this is a very popular exhibit at the museum, and rightly so.
.
The headsets I thought were rather on the large side but apparently this is down to the fact that they’re a little more hi-tech then your average 3d glasses. It was slightly clumsy adjusting them to your head size - something I noticed the smaller children having particular difficulties with. You wanted them to fit properly so they wouldn’t slip off - leaving one hand free for the joystick and the other hand free to reach out in an effort to try and ‘touch’ this object.

One thing that I wish was different was the ability to show something other then scientific models/structures. Although this is what the Wedge is designed for I somehow wished for the possibility for something more personal to me be shown in 3D – perhaps a human body or motor vehicle etc.

Overall Impressions
A highly enjoyable and interactive installation that transforms you into a new 3d virtual space and gives you a sense of seeing things in an entirely different perspective. A great asset to the Cyberworlds exhibition at Sydney’s Powerhouse Museum.

Links / Technical References
Powerhouse Cyberworlds Exhibition
ANU Wedge
Vizlab Virtual Enviroments
The Wedge Virtual Reality Theatre

Interactive Experience Review

I visited Experimenta Playground at Carriageworks for my review.
One of the more entertaining installations, I found, of the playground was “Help Us Help You Help Centre” by Narinda Reeders. The piece was basically a voice activated installation which interprets what’s said by the user in an attempt to “help” them with troubles in “relationships, emotional health, psychological health and career”. Questions, which were quite personal, were blatantly asked and even though answers were sometimes misinterpreted, they just made the installation even more amusing, even if her reply reprimanded you in public. Whilst answering the questions, quite dishonestly (just to make it fun, and something I assumed a lot of users would’ve done), I was thinking how lame it was and if anyone actually found help from this. And also, if this was to help people, did they really think that placing it out in public where such personal questions (of which were so cliché) and bland responses were given quite loudly, would actually get users to respond honestly and get the help they were after?
THEN, I read the statement and realised the piece aimed to “parody the potential for self help gurus to prey on our vulnerabilities and play with the increasingly blurred line between public and private in our society” [1]. Then I thought, well played. The artist was able to make me think about everything she had aimed to do in the few short minutes I was answering the questions.
The interface of the installation allowed for users to know exactly what to do when they came across it (even though I wasn’t aware that it was part of the playground until it was pointed out to me). The screen had clear instructions on it and the microphone was in clear view. Even though she thought my name was Thea, I found the piece entertaining and interesting because it was a different sort of interaction I expected from the playground.
However, the most memorable installation for me was a stop motion animation piece “Sopa” (Soup) by Irene Iborra & Jossie Malis. Though it wasn’t interactive, the short film drew me in and kept me watching even though I got queasier (keeping in mind I don’t have a strong stomach) whilst watching it. The film was a parody of cooking shows, where the protagonist, finding nothing to eat in his kitchen, improvises for food and ends up throwing together a range of “different” ingredients into his “soup”. Ingredients consisted of corks, screws, keys, dolls and mobile phones, all of which was prepared in way as if they were proper ingredients. They were sliced, chopped and even sautéed, and all thrown into a pot to be cooked. The stop motion accentuated the uneasiness of the film for me, as the quick and jerky movements made it even more uncomfortable to watch as he shoved the food into his mouth and chewed it, as if it were normal. The film brought me to a new level of perturbed, but I enjoyed it.

[1] Experimenta Playground National Tour Site: http://www.experimenta.org/playground-touring/

Images also from: http://www.experimenta.org/playground-touring/

Interactivity Review

(Yes, there are (and will be) a lot of reviews covering Experimenta; blame Rob, for not only informing us in class, but emailing all of us and reminding us to go…)

The Experimenta Playground, currently on exhibition at CarriageWorks, is an art exhibition featuring a multitude of video, installation and interactive artworks. Proudly proclaiming itself as the “leading contemporary arts organization [1]” for Australian video and interactive art, Experimenta boasts an array of innovative and engaging means of interaction with artwork, as well as ingenious non-interactive video works [2], the most notable among them being Shadow Monsters, one of the largest installations at the exhibition. Shadow Monsters is an interactive artwork by Philip Worthington, unveiled in 2005 [3]. In it, users are able to interact with both the work and one another using computer-generated shadow puppets, recalling back past memories of childhood shadow fights, bringing its audience back to a more youthful age, to have fun without a goal, and highlighting the notion that artwork can be a fun and enjoyable experience. [4]

“Magical Monsters appear from the shadows cast by the hands of participants” [5] describes the creator Phil Worthington. From the traditional media of shadow puppets, Shadow Monsters grows various fantasy-type features, including horns, teeth, eyes, barbels and tongues. The constantly changing nature of the shadow puppets, as well as the ability for multiple users to participate, is a fundamental component to the success of the engaging method of interactivity utilized. Gesture recognition adds features such as sound and animations, features normally devoid in traditional shadow puppets.

Shadow Monsters builds upon the traditional limitations of shadow puppets. Whereas traditional shadow puppets are constrained to the imagination of the participants, Shadow Monsters builds upon this imagination, adding new qualities to the shadow puppets and prompting users to further explore the possibilities presented by the new medium.

This was most exemplified by actual usage, and observation of the installation in use. While demonstration videos, promotional images, and indeed the words of the artist himself, [6] only presented the notion of hand puppets being converted into monsters by the artwork, users experimented with the system using

their entire bodies, including their arms, heads, legs and even running in front of the screen, interacting with the artwork through the ‘language of play’ [7]

The novel means of interaction, as well as the appealing and innovative concept, made this installation by far one of the most enjoyable artworks presented by the exhibition. Users were noted to spend considerably longer sessions experimenting with this artwork than any other installation, and would frequently return after exploring other installation works. Shadow puppets admirably accomplishes its goal of creating “an intuitive and magical experience… to play with body posture for creating crazy narratives” [8]

Shadow Monsters has a different mechanism from a similar shadow-based installation, The Manual Input Station. Unlike the Input Station, the Shadow Monsters input Unit, named the “Leave No Trace Unit” [9], does not use the direct shadows of its users and overlay a projection over them. Instead, Shadow Monsters appears to detect shadows using the light screen and appends the afore-mentioned monster features to the shadows using a computer before projecting the images onto a screen. [10] While this creates smoother blending and interaction between the user generated shadows and the computer generated images, it also increase latency considerably, and caused a small amount of confusion, as the active area of the light screen was smaller than the actual area of the light screen. It appears to be written in a combination of Processing [11] and Sonia, although the details of its operation have not been released.

Shadow Monsters is an intuitive and ingenious installation. It combines an innovative means of interaction, with a childhood pastime to promote imagination and interactivity between both audience members and the computer, reviving the concept that art should be fun, as well as an expression or statement. Shadow Monsters is an invaluable contribution to the Experimenta exhibition.

Owing to the fact that Experimenta is free, and Shadow Monsters is possibly the most engaging and promising installation there, the Experimenta Playground is an invaluable and worthwhile exhibition in it’s contributions to interactive artworks, and well worth the visit, even if some of the other artworks smell funny…

References:

[1]
http://www.experimenta.org/about.htm

[2]
http://www.carriageworks.com.au/whats_on.php?event=experimenta

[3]
http://www.experimenta.org/playground-touring/

[4] [7]
http://www.performancespace.com.au/program_details.php?programid=194

[5] [6] [8] [9]
http://www.worthersoriginal.com/viki/

[10]
Unlike certain students of the Faculty, Worthington has avoided the potential fire hazards, and placed ventilation holes in the Unit. These also allow a brief glimpse of the mechanism inside.

[11]
http://www.processingblogs.org/2006/06/28/shadow-monsters-new-software-from-philip-worthington-added-to-the-exhibition/
[12] [13] [14] [15]
http://www.worthersoriginal.com/viki/
The exhibition did not allow photo taking. Hence, all photos are obtained from external sources.

Interactive Experience Review: + - now

After visiting the Powerhouse Museum and interacting with Jen Seevinck’s installation titled ‘+-now’ I now have a deeper understanding of interactive installations and how they need to suit their target audience.

The installation was located in a darkened area of a partitioned room with no distractions other than the audio from installations in the near vicinity. This lack of disruption in the environment proved to be a success with people directly attracted to the stand in the middle of the room containing the ‘box’ of stand with which to interact.

As noted on part of the installation’s blurb, it has two aspects which contribute to the playfulness and appealing nature. The first is that as user’s play with and move the sand around the area revealing the glass below and thus producing patterns and images, multi-coloured lights are projected onto the sand creating a direct sense of play. The second aspect is that the patterns produced in the sand are also projected onto a wall with “varying shades of opacity”.

While first interacting with the installation by myself I was rather impressed with its intuitiveness and the level of feedback provided as I moved the sand creating a number of patterns. However, I also noticed the time delay that was clearly evident with both the light projected onto the sand and the patterns being projected onto the wall. While a small delay allows the participant time to think and reflect on the project, it proved to be frustrating as the delay was not consistent resulting in the participant waiting up to 20 seconds at some stages to see any response. Nevertheless, I was still quite pleased with the overall effect and outcome of the installation. However, my view started to change as I saw a group of children interacting with the sand. I found their reaction to be justified and something that would be expected of children with them immediately thinking that they had broken the installation on two occasions firstly when it failed to produce any response after a few seconds and secondly when they moved a large portion of the sand to one side which also failed to produce any result at all on either of the projections. This reaction forced me to think further about the target audience who I believed to be almost anyone – a very large target audience; for those who like to play – the young and the young-at-heart. I expect that this was the target audience of the electronic artist Seevinck because of its playfulness.

Taking the intended audience and my observations into consideration I would have to say that the installation failed to fulfil one of its expectations, which I believe would be providing a quick response to the participant’s action thus to further engage and attract them to its playfulness. However, in saying this I also do believe that the installation was fun and intuitive with all participants that I observed immediately understanding how to interact with the installation and being amused by the projections that it produced.

As a side note I must mention that I was quite interested to see that below the glass the camera filming the sand’s movement was just a handy-cam which many people to do would own. The only other piece of equipment that could be seen was the projector used to produce the colours and the wall projection. I expect that this installation was made with Max MSP as I am starting to understand how all the components would link together to create such a project.

Review of an interactive installation: The manual input station

The interactive event I visited was called Experimenta Playground at Carriageworks near Redfern Station.  I interacted with most of the interactive installations, some I really enjoyed and some I asked myself…what is the point of this installation.  Below is my user experience of the exhibit installation that I liked the most.

The installation I really enjoyed was called The manual input station by Golan Levin and Zachary Lieberman.  Unfortunately this installation was directly opposite one of the exhibition curators so I was unable to take photos illustrating myself interacting in this installation (though a youtube video documents a persons’ interaction), however I did manage to take some other photos that I have inserted at the end of this article.  The way this installation works is by using shadows cast by objects or human body parts using an overhead projector a special coloured purple light is used to determine when the shadow cast creates a closed prism, where the purple light is converted into a digital coloured physics based object.  The installation was both visual and audio, each general type of prism has an assigned sound, so that when different prisms were formed a musical symphony was created.  My first experience with interacting with this exhibit was one of fascination and exploration in how the prisms were created.

The installation primarily used a combination of MaxMSP/Jitter for the shadow casting analysis and audio output and processing for the prism parameters.  The installation was relatively easy to learn, as most things related to an overhead projector, it’s a natural action to place your hands on top of the glass to discover what exaggerated shape you can form.  The installation made me want to interact and explore different ways in which I could create and manipulate the digital objects, for example, once the digital shape is created you can take your cast shadow out of the light and the object takes on several physical properties such that I bounces and spins when it hits the bottom of the screen and then disappears after a few seconds.  I could also create an object and then cut or reshape the object by placing more cast shadows in the middle of the object.

I really liked this installation because it used some of the elements Adam and myself want to use in our own project design idea, using some sort of camera tracking to produce sound, except this installation introduced an extra visual experience (through the digital objects) that made the overall installation look aesthetically appealing.

Some of the other interactive installations that I was able to sneak a photo shot included What yours is mine (explores the vulnerability of the human mind as it yells out degrading comments at the participants) and Immersion (an interactive floor installation where schools of fish and sea creatures interact with the shadows that are created from a ceiling spot light, wherever a shadow exists; fish will start to congregate).

Interactivity - Zachary Liberman

Interactivity. A simple concept of involving, responding, engaging and taking part of something. Often a 2 directional process with no set limit on the amount of times it exchanges, something is presented, you interact, it reacts/responds/changes, which then follows with your own response and so it continues. Interactive art highlights and involves the audience and environment around it, bring them closer and deeper into the work. Sculptures, websites, installations, computers and some performances can all breach over into this field. Rather than be static and inanimate, interactive art presents a new way to be involved with a creative outlet of another.

Whilst there are many various artists in this field, their goals are individual and personal. Some may use interactive to present information, such as “The Secret Lives of Numbers” [1]. This project shows the popularity of a very large array of numbers based of user interaction. Others like “Skipping Digital” [2] are more of a fun based, ‘play’ piece that each person participates in the manner they choose, whether for entertainment, understanding or just for a giggle. One of these more ‘play’ based artists is Zachary Liberman.

Zachary Liberman describes himself as a “researcher as I feel that my artistic practice is a kind of research” [3] A researcher that’s work tries to develop and express various forms of what it is to be human. As he himself says, his “work takes a playful look at these essential parts of human existence and creates playful, organic myths that aim to shed some light on their nature and origins.” [4] Having displayed his work all over the world, in both interactive installations and performances, Zachary Liberman’s bring a joyfulness to some of the more ‘taken-for-granted’ ideals and interactions in everyday life.

To me, his installation/performance “Drawn” highlights this outlook clearly. “Drawn” is a work that takes physically drawn images and brings them to life, so to speak. Originally a performance piece exhibiting “painted forms appear to come to life, rising themselves off the page and interacting with the outside world.” [5] The performance was also supported by Pardon Kimura to add an audio level to the piece. Kimura is a sound artist with various albums and musical credits to his name and he brings his experience from various fields into this piece. The performance was toured around Europe and Asia during 2005.

In 2006, “Drawn's” success and audience interest prompted an installation version was developed. “After every performance, crowds would gather to peer at the table, and often ask to try the software themselves… their delight was immediately evident. [6] Very similar to the performance version, anybody could come up, draw and then experiment. The drawings were collected and displayed across the walls surrounding the installation, “creating an increasingly dense tapestry of ink forms.” [5] Like the performance version it was developed from, Drawn (installation) was displayed in a variety of museums are Europe.

But how does Drawn actually work? As the artist draws onto a sheet of paper, a camera mounted above captures the drawn image and sends the data to a computer. This computer stores and transforms the image in its memory and turns the drawings into vector shapes. The artist then manipulates the shapes by various hand motions around them. Pushing, pulling and twisting are all possible. The shapes themselves become almost organic as they bounce and flex at the users ‘touch’. Pardon Kimura’s part comes in to play when an ink stroke is touched. Various sound effects are created in response, and reflect the movement and action of the ink stroke. For example, if a stroke is shaken side to side, the audio might be a swishing sound, tracking the movement of the ink. This also occurs in the installation version where movement from the ink sparks an audio response. This to me adds another dimension and mentally engaging level to the piece as it gives the lifeless ink strokes more life.

I find that Drawn is an excellent example of Zachary Liberman’s goal to create playful yet exploration artworks. (See [3]) By combing something primordial like drawing with modern day computer graphics and blending them together, Liberman succeeds in creating a successful interactive and engaging piece. I find the childish nature of the piece a better way to draw the audience’s focus, as we are all children at heart. From examples I have witness, the more simple and basic the drawings, often the more enjoyable and endless possible manipulations are possible. I see this piece as more ‘fun’ than mental stimulating, but at the same time, often this is the best way relate to a wider range of people. As Liberman said in response to the use of Drawn used in a workshop for kids, "There was a lot of laughter. "[6]

Another piece of Liberman that future develops on the idea of play to stimulate the mind is “Motionscapes”. Designed specifically for children with learning difficulties and special needs “an intuitive and fluid means of self-discovery and expression.” [7] This ideal of letting these children develop their sense of the world and surroundings in a playful manner future reflects Liberman’s goal to highlight human existence in various forms.

“Motionscapes” use a fairly basic hardware set-up in its approach. A simple camera is placed in front the user, and using a projector for image display. As images are projected onto the screen, the users move in front of the camera. The camera takes these motions, computes them and then causes the projected image to respond to the motion. This allows children just simple movements to thoroughly engage with the projection. One thing Liberman has done by focusing on motion rather than the physical being, if the child has a carer, they can sit there being still and let the child engage on their own. This helps individualise the Childs exploration of the piece.

The piece has various ‘modes’ to create very different types of interactive and display. These modes are twirl, fluid, loops, bounce and stripes. Each mode presents a new way for the child to get involved, with their movements creating a variety of effects on the screen. Twirl involves a ball rotating around a centre point. The more the child moves, the bigger the ball gets. It also has an audio file where as the size increases, so does the pitch. Fluid is plasma like image of abstract colours and fluid forms. Movement creates new particles in the image. Movement also determines the direction of the fluid, with larger movements creating more of a ‘push’. Once again, audio is accompanied based on particle position and movement. Loops uses multiples of a single image. Movement causes the loops to start in a flowing order from top right to bottom left. Sound is again based of motion. Stripes uses movement to create coloured bands. Different ways of moving affect the colour of the bands. Sound is generated from the colour and position of the bands. Bounce is a collection of balls on the screen. Movement triggers the balls to move and react to the Childs gestures. They collide with the screen walls and each other. Sound is based of position and speed, including collisions.

When you compare Drawn and Motionscapes, you can clearly see similarities and what must ultimately be Zachary Liberman’s ‘style’. With both pieces there is a child like essence and playfulness. Simple movements create an action on the screen that a user can react and cause to react. There is no political stance or message, no attempt to force a user to think a certain way. Its an opportunity for a users self fulfilment, what they see and do is a reflection unto themselves. Liberman gives us a chance to be ourselves and change and manipulate objects and worlds we normally do not have access to.

The comparisons between informative interactive art and interactivity for the sake of the users enjoyment is a blurred line. One may argue that information presented in an interactive format is not ‘art’, just a prettier way to do it. At the same time, is an interactive art piece designed for fun and enjoyment anything more than a glorified technological toy? Each individual has their own stance, mine leans more to the enjoyment of toys, such as Zachary Liberman’s work. They do however present a user the ability to create their own information from the work, information only bound by their creativity. You could even say this information, in total could provide more information than a piece specifically designed to give you a set piece of data. Informative interactivity has its place however. It provides you with a thoroughly more engaging way of learning and exploring a particular field of data. It can also preset opportunities to look at the data and sort through it in a way that you previously had not thought of.

In conclusion there is a fine line between the set goals of informing with interactivity and providing a medium and creative outlet for a person via an interactive artwork, something I don’t feel justified to complete decide on. I do however feel that Zachary Liberman's work is an excellent example of a way an artist can present a variety of people a simple way to interact with totally individual and varying results and response, all in a fun and play focused way.


References:

[1] (2002: Golan Levin, Jonathan Feinberg, Shelly Wynecoop and Martin Wattenberg – The Secret Lives of Numbers - URL)

[2] (2007,ENESS – Skipping Digital – Melbournes QV Square - URL)

[3] (July 26, 2006: We Make Money Not Art - Zach Lieberman's talk at Futuresonic By Regine - URL)

[4] (June, 2006: Herraiz Soto & Co - Zachary Lieberman: pintar la voz – Interview by San Cristóbal - URL)

[5] (Zachary Libermans Website – Drawn - URL)

[6] (Zachary Libermans Website – Drawn Installation - URL)

[7] (Zachary Libermans Website – Motionscapes - URL)

Task 3 – Short Research Report: Camille Utterback

Programmer and artist, Camille Utterback has developed a number of interactive installations that have been exhibited in museums across the globe and also in private spaces. Becoming a fan of computers and technology at an early age, Utterback taught herself to program in BASIC and even made programs to complete her maths homework at age ten. After losing interest in computers during school Utterback focused her attention on art and eventually “rediscovered how cool computers were” by accident [1]. Having already completed a BA in Art from Williams College, Massachusetts, Utterback’s rekindled interest in computers led her to complete a Masters degree from The Interactive Telecommunications Program at NYU, where she now also lectures as a professor. Utterback also works as an occasional professor at the Masters of Fine Arts in Design and Technology Program at Parsons School of Design as well as running her own business, Creative Nerve, which she founded as a means for developing future commercial installations [2].

One of Utterback’s oldest and most playful installations is Text Rain developed in partnership with Romy Achituv in 1999. Text Rain is an interactive installation that projects a mirrored video of participants on to a screen in black and white. Coloured letters, which together form a poem, fall from the top of the screen and can ‘land’ on participants just like real rain [3]. The participants movement is tracked by a video and enables them to catch, lift and play with each of the letters individually or as a cluster. The intelligence behind this effective installation is based on a simple tracking device – “falling text will ‘land’ on anything darker than a certain threshold, and ‘fall’ whenever that obstacle is removed” [4]. If participants are able to catch enough letters, words and phrases became apparent and participants and observers are able to see the relation and meaning of the poem. New phrases of the poem appear as the falling letters slowly fade enabling participants to delve further into the poem. “Reading the phrases in the Text Rain installation becomes a physical as well as cerebral endeavor” [4].

Text Rain has three permanent public installations (in Germany and USA) and has also been exhibited in Spain, Brazil, Taiwan, Poland and Japan to name a few [4]. Text Rain’s popularity and attractiveness is attributed to its accessibility by the everyday folk; its simplicity and the fact that the participant is not required to wear any expensive equipment encourages people of all ages and education levels to play with the installation. Several people are able to interact with Text Rain at any time, some of which may not even be aware of their interaction, as it is merely the person’s presence that affects the falling letters. This installation breaks down the barrier between real and virtual space because users are able to “simultaneously shift between the two, or to feel present in both simultaneously” [3].

Utterback’s Text Rain is one of her most well-received installations because of its playfulness. Participants are also able to directly see the effect of their involvement in the installation and do not have to think about any abstract themes that influence the installation. I believe it is also an inspirational piece of work as its simplicity enables novice artists to envisage other installations and thus encourages them to create installations of a high calibre while understanding that complexity does not necessarily add effectiveness.

“I am passionate about creating experiences that show people that their interactions with computers do not have to be frustrating, deadening, and potentially debilitating. Instead, we can imagine and create a world where this interaction is seamless, intuitive, playful and inspiring. By using video cameras to create physical-digital systems that engage people's bodies instead of just their fingers and eyes, I hope to refocus attention on the embodied self in an increasingly mediated culture.” - Utterback [3]

See/Saw developed by Utterback and her business associate Adam Chapman is an interactive installation that uses the movement of an actual see-saw to tell a story. As participants see-saw a word is projected onto a screen behind both participants; with each movement up and down the words on both screens change simultaneously. The words that appear on each screen are responsive to the angle of the see saw in that the person who is close to the ground will see a word associated with negativity while the person elevated will see a word associated with power. The installation also takes note of when the see-saw motion stops and a short phase is played to each of the participants that incorporates the word that is shown to that user. Each participant only hears one phrase when they come to a stop as the speakers are embedded within each end of the see-saw and thus the other participant does not hear the opposing phrase to their “point of view”. The words shown to each user do not follow a linear pattern but are completely jumbled to make no sense. “Its structure as well as its meaning derives from the temporal and spatial interactions with the see saw” [5].

The See/Saw installation is an effective piece of art due to its originality; few exhibits in any place over the world allow for such ‘hands-on’ interaction. The relationship between the angle of the see-saw and the words displayed to each of the participants conveys depth and meaning to a participant but also enables them to experience the installation. The sound that is played to the participant as they slow down the see-saw motion enables them to discover even further the level of interaction and it adds a drop of playfulness.

Utterback’s works thoroughly convey her interest in art while also displaying her technical capabilities. They encourage people to take a broader look at the world and consider the relationship between the virtual and physical worlds. I believe Utterback’s works are an inspiration to young aspiring designers and artists pushing them to develop further installations that are both appealing to the eye and show some level of abstract thinking, thus creating a number of completely revolutionary works.

“By refiguring the possibilities for interaction with digital media, I question and explore the space between the symbolic and the corporeal; between the virtual and the real. By creating poetic relationships between these spaces I hope to engage people both emotionally and viscerally.” - Utterback [4]

[1] Utterback, C., 1999, GirlGeek of the Week, (Accessed via Safari, 5th April 2008). (Online). http://www.girlgeeks.org/innergeek/gkwk/gkwk_utterback.shtml

[2] Creative Nerve, Inc. (Accessed via Safari, 4th April 2008). (Online). http://www.creativenerve.com/

[3] Utterback, C. Design Systems for Human Interaction Not Human-Computer Interaction, (Accessed via Safari, 4th April 2008). (Online). http://www.core77.com/reactor/utterback.html

[4] Utterback, C., 2008, (Accessed via Safari, 4th April 2008). (Online). http://www.camilleutterback.com/

[5] Utterback, C., 2004, Electronic Book Review, A Riposte to: Camille Utterback, (Accessed via Safari, 6th April 2008). (Online). http://www.electronicbookreview.com/thread/firstperson/tabular

Short Research Report - "Seen" by David Rokeby

Interactive art is artwork that not only engages the user, but the user is also an integral part of the artwork. The user can input data through many mediums including video, colour sensors, motion, heat and other input devices. Generative art, electronic art, andimmersive art all integrate the user as a vital towards the final outcome of the piece. Although some interactive art pieces are designed site specifically and therefore created to adapt to the users and environment that it is housed in, many installation pieces are created so that they can be replicated in many different gallery spaces and still achieve similar outcomes.

David Rokeby

[i]

David Rokeby is a sound and video installation artist born in Ontario, Canada. Since 198, he has been based in Toronto, Canada. He has designed many interactive installation pieces over the years which predominantly engage the human body and its interaction with an artificial system.

“These exhibitions serve as a public research laboratory where my ideas about interaction and experience are tested, affirmed, or shot down. This is a condensation of the results of my free-form research.” - Rokeby[2]

His exhibitions have been displayed all over the world, from The USA, Canada, Asia and Europe. Such venues include:

- The Venice Biennale in 1986

- Ars Electronica (Linz Austria) in 1991

- The Mediale (Hamburg Germany) in 1993

- The Kwangju Biennale (Korea) in 1995

- The Biennale di Firenze (Florence, Italy) in 1996

- Alien Intelligence (Kiasma, Helsinki) in 2000[3]

Some of his recent exhibitions and installations by David Rokeby include:

- Machine For Taking Time (Boul. Saint-Laurent) (2007)

- Cloud (2007)

- San Marco Flow (2005)

- Gathering (2004)

- Sorting Daemon (2003)

- Cheap Imitation (2002)

- Taken (2002)

- Seen (2002)

- inter / face (2002)

- Machine for Taking Time (2001)

- n-Cha(n)t (2001)

- Guardian Angel (2001)

- Shock Absorber (2001)

- Watched and Measured (2000)

- Universal Translator (1999)

- The Giver of Names (1991-)

- 60 (1995)

- Watch (1995)

- Silicon Remembers Carbon (1993-2000)

- Petite Terre (with Erik Samakh) (1992)

- Measure (1992-94)

- (Perception is) The Master of Space (1990)

- Liquid Language (1989)

- Echoing Narcissus (1987)

- Very Nervous System (1986-90)

- Body Language (1984-86)

- Reflexions (1983)[4]

Seen (2002)

[5]
One of David Rokeby’s most well known pieces is the video installation piece “Seen“ from 2002. The installation was commissioned for "Next Memory City", the Canadian Pavilion of the Venice Biennale of Architecture 2002. It is set in “Piazza San Marco“ in Venice, one of the most prominent tourist attractions in the world. The area is used as a source material for the actual installation. The source material consists of 4 video projections over the square, all from the same camera mounted in the same position. These projections are 30 minutes each and record the architecture visible from that point, as well as the people and pigeons who walk through the square. The movements of the people and pigeons are also recorded, and turn out to be an integral part of this installation piece.

[6]

The first and fourth projections show the moving objects (the people and pigeons) and seperate them from the still objects (architecture and souvenir kiosks). The second and third projections then show different perspectives on the traffic flow patterns through the plaza. The second projection is a variation of the movement image on the first projection. It takes the first image and feeds it back on itself on a half second delay, turning all the moving subjects into a procession of themselves. This creates an interesting visual effect as areas with more human traffic flow would appear to be much more densely packed than areas which had less traffic flow. That showed a great overview of the areas within the plaza where more people would travel through, and areas which had a high probability of people passing through. It would also leave an interesting pattern where the infamous pigeons would move. The third projection then traces the trajectory of the occupants in “Piazza San Marco“ and creates a colour gradient relating to the direction that they are moving in.

As all the video projections were processed in the video’s full resolution, every person and pigeon left a mark. As pigeons flew across the plaza, the arcs on their flight left a trail; as did running padestrians leaving trails as they dodged and turned through the crowd. Interesting patterns were therefore created, especially amongst the flocks of pigeons and their footprints.

This installation has the capacity to amaze audience as it transforms regular movements by pedestrians and pigeons into an explosive mix of colour and abstraction over numerous screens within the same room. Within this, the audience can see different views of pedestrian movement in the same area silultaneously, making the movements of the people in the installation almost timeless and almost ghostly. The thought of social spaces is easily ignited amongst the viewers.

“This explains our attraction to optical illusions and mind-altering experiences (chemically-induced or not). Those moments of confusion, where identification and resolution aren't immediate, give us a flash of the raw experience of being. These moments of confusion are also the fulcra of paradigm shifts. It's only when our conventional way of dealing with things breaks down that we can adopt another model, another way of imagining and experiencing a scenario.” - Rokeby[7]

The installation “Seen” sparks many thoughts into the minds of the viewers, by far exceeding the great aesthetic awe. Viewers are set to think about aspects such as social presence, the tension between the architecture and the people who use it, and the use of space in general. It visualises life itself in a way that many people can imagine in their minds, but cannot express in any way to other people. It may be seen as an appreciation of a gathering of people in a well congested area, or it can be seen as a critique on the use or even miss-use of a public area.

The video footage in “Seen” is not live data, but rather a pre-recorded batch of video which has been processed by Rokeby. This would suggest that he is more concerned about the visual aesthetics and informational themes of the piece than the accuracy of the data. This has been how much new media has worked in the past decade, as artworks are beginning to have more of a public and social statement. This follows the mindset of social theorist and English professor Mark Hansen.

“the humanities must embrace technology and that humanists must enter full-scale into the informatics revolution by, for example, contesting the meaning and value of “information” and rethinking what it means to be human in a “realtime,” digitally-networked, global world in which we often cognize in concert with intelligent machines.” - Hansen[8]

The installation “Seen” is created in a style very typical of David Rokeby, as images of users are collected and therefore used as part of the artwork. A very similar installation to “Seen” is “Taken” which was created almost immediately after “Seen” and was displayed in The Art Gallery of Hamilton, Hamilton, Canada. Similarities include:

- Collecting images of the audience and relaying them back onto large screens

- Using multiple viewpoints of the users to collect different projections

- Morphing the different projections to create a sense of fusion in time

- “Freezing time” as a way of analysing social density in a public space

- Collecting and re-presenting interactive experiences of users within a set environment

[9]

[10]

“Seen” can be seen and interpreted in many ways. It can be interpreted as a piece of entertainment or a form of social commentary. These interpretations create a sense of the user’s experience, which is intended to be magical as it is not an experience that the user endures on a day to day basis.

“...as an artist, it’s my traditional right to use every trick in the book to create a magical experience” - Rokeby[11]

Fantasy and illusion can be created in the piece by real life images are presented to an audience, yet it is altered to create a sense of surrealism and enchantment. Rokeby achieves this sense of enchantment by freezing time and creating lapses of movement. Not only is this an intriguing piece to look at, but it simulates a response from the user in the form of a social realisation. A viewer in such an installation or artwork is seen as an integral element of the artwork, and they can single handedly control the outcome of how the visual piece looks. A person can also understand their role in the set environment, as they can see how the environment around them interacts. They are therefore a part of the artwork and as a result of this they are more engaged in the artwork.

“Because the computer removes you from your body, the body should be strongly engaged”-Rokeby[12]

The way that Rokeby has expressed the content is fairly subtle, there is no writing involved where the viewer is bombarded with and kind of propaganda or other written forms which can influence a user’s mind set. It is simply observing an area and the people who interact within it, and then presenting it back to the user in a way that they may or may not have previously realised or anticipated. Because there is no real meaning behind such abstract artworks, there are different interpretations of the same artwork, drawing many different responses from the audience.

David Rokeby has exceeded in interactive art installations as he has found a way to subtly engage his audience and make them feel like they are part of the artwork without making them feel like they are overly exposed to the public. As many of his pieces involve collection of images of large amounts of people, the users don’t feel like they are being targeted and they all serve an equal amount into the final product that the installation crates. Although Rokeby’s pieces aren’t directly live inputs like other artists have created, he leaves a lasting impression as he usually tries to create a social commentary for the user to think about – both regarding themselves and the environment that they interact in.

---

[i] http://www.canadacouncil.ca/canadacouncil/archives/prizes/ggvma/2002/images/hires/dr_portrait.jpg

[2] Dodsworth, C (1998) Digital Illusion: Entertaining the Future with High Technology

[3] http://www3.sympatico.ca/drokeby/home.html

[4] http://homepage.mac.com/davidrokeby/installations.html

[5] http://www.youtube.com/watch?v=0Ai49w7QPPY

[6] http://www.rokebyshow.org.uk/images/seen.jpg

[7] Rokeby, D (2002) The Construction of Experience

[8] Hansen, M Department of English, The University of Chicago

http://english.uchicago.edu/graduate/amer/m-hansen.html

[9] http://youtube.com/watch?v=ipsz4ALgUi0

[10] http://homepage.mac.com/davidrokeby/taken.jpg

[11] Dodsworth, C (1998) Digital Illusion: Entertaining the Future with High Technology

[12] http://www.medienkunstnetz.de/works/very-nervous-system/

Assignment 2

KEN RINALDO-Standby Deliver

Interactive art is an art which involves the spectator in some way. Sculptures achieve this by allowing the observer to walk in, on and around the piece. Other works such as computers and sensors allow response from motion, heat and other types of input. There can be many types of interactive art, some being contextual theatre, installation art, computer-generated art, electronic art, interactive film, internet art, kinetic sculpture, new media art, performance art and video game art. Ken Rinaldo is an interactive artist like many other who makes their artwork have a message. This message is usually to emphasize the original problem in a more practical way than written information. The following information will be present in the following, an introduction to Ken Rinaldo, the problem which he wishes to address to people who view his artworks and information on his artwork “Standby Deliver”


Ken Rinaldo is an installation artist who focuses on the intersection between natural and technological systems. The combination of organic and electro-mechanical elements give a connection and a co-operate evolution between living and evolving technological materials. His artworks are mainly influenced by theories on living system, artificial life, the communication and interaction between species between and the beauty and fundamentals in the nature and organisation of energy, matter, data and information. His works emphasis on the concern for ecological issues, which are often not highly regarded or taken in account for within the domination of technological and cultural progress. Ken’s motivation in becoming an installation artist comes from the encouragement and self determination with his interest in relationship between humans, machines, nature and culture. [1] He believes that the relationship with a work of art can demonstrate and become both real living system as well as a metaphor for a co-evolved and symbiotic coupling that can exist between nature and culture. Ken Rinaldo work mainly focuses on interactive multimedia installations that blur the boundaries between the organic and inorganic. He has been working with the connection between art and biology for over two decades and his works include interactive robotics, biological art, artificial life, interspecies communication, rapid prototyping and digital imaging [1]


One of Ken Rinaldo works is “Stand By Deliver”. It is based on sugar tweaked cognition. Sugar tweaked cognition is the research of sugar consumption by humans. The increase of sugar intake in our daily lives has increased dramatically in the past decade. Even sugar consumption in the third world countries has increased dramatically. Sugar consumption has been rising at over 1.7 percent a year for the last decade, higher than the U.S. population growth rate of about 0.8 percent. [2] An average American would have the daily consumption of thirty teaspoons of sugar. This does not include natural sugar; it comes from cakes, lollies, chocolate and soft drinks. The increase of sugar consumption has increased due to the amount of added sugar products which have increased in our daily consumables. Examples of this is the spring water we buy can now be flavoured in different fruit flavours, soft drink are also adding extra flavour such as vanilla, lemon and lime and a lot of our cereals have increased in sugar intake. Our gum which was originally peppermint or spearmint has now increased to multi fruit flavours. Another reason for the increase of sugar consumption is that a lot of food is said to be “fat-free”. However foods which are fat free tend be high in sugar. Sugar over the past ten to twenty years has managed to gain a positive image. With the high demand for a healthy diet, many people are aiming towards the fat free products and hence thinking they are not consuming the kilojoules but in fact they are increasing their intake of sugar.
Sugar consumption has raised many issues. The two most common issues that are raised are health issues. The increase of heart disease and diabetes and increased dramatically due to the high in take of sugar. Another issue is has risen are environmental issues. Most of the sugar is grown in third world countries; sugar and salt cropping are the main source of income for people living in third world counties. However the price that the Pilipino farmers have to sell their sugar has lowered dramatically over the past thirty to forty years due to the ineffective world pricing controlling mechanism of the International Sugar Agreements (ISA). This has forced the farmers to leave their crop land and move to other places to which have cheaper rent or a smaller land to manage. This has caused the increase of deforestation and soil erosion. Furthermore, the expansions of human settlement, forests have been cleared to grow crops, wetlands have been drained, and grasslands have been irrigated. [3] The sugar product which has caused a great environmental issue is chewing gum. The littering of gum has caused great problems. Gum is very difficult to remove, the deposal of gum that we chew usually end up on our pavements, hair and the bottom of our shoes. Councils receive countless complaints about the amount of gum in the environment. Gum as well as being difficult to remove is also very expensive to remove. One piece of gum will cost around ten pounds to remove. [4] .

When gum ends up in our greenery, animals eat the gum, thinking they are food and they eventually choke on it as it does not digest. Although Ken’s “Stand by Deliver” is trying to send the message across about sugar consumption, it focuses highly on the damage chewing gum can do to our environment.


Stand by Deliver is a very interesting piece of interactive art. In comparison with his other art works which mainly focused on robotic interaction and artificial life. This took on a different approach.

For images click here

His interactive art piece which consists of steel plates facing each other and moving back and forth attached to activating motors. Under the steel plates is a lit glass sugar molecule. Spectators have the chance to chew gum, which then they can stick to the plates. The steel plates come together and come apart. The gum will stretch which will create long colourful stings of the sticky gum. After many cycles of the plates moving back and fourth, the glass sugar molecule is coated with colourful goo. Although the idea of placing chewed gum on a piece of steel may be quite intimidating, but the reason behind it is to let participants know what the consequences of dumping gum on a pavement can lead to. Ken got the inspiration to design this installation from his great love for sweets, especially chewing gum. He would go through four packs of gum within fifty minutes. He would start of with spearmint, peppermint, fruit and finish off with cinnamon. By the time he got to the cinnamon flavoured gum, he would only be able to taste the cinnamon flavoured the most. After chewing all this gum, he would place the gum on a piece of paper and he thought that that the colours and arrangement were really “pretty” hence he developed this machine which would create a picture from chewed up gum. He always thought that during the process of developing this collage or picture with the chewed up gum that maybe the sugar in the gum had got to him in his body and brain. He thought that maybe the mouth candy that was in his mouth had changed to eye candy hence that is why he found the collages so interesting and beautiful. His message that he wants to send to his participants is that the damage gum can do to our environment. [5]


This piece of installation art is very different to his other works. His other works focuses highly on robotic-human interaction. His method of sending his message through to his participants is quite subtle. You would not automatically think that it is to tell us about the damage of chewing gum. The message is not only towards chewing gum problems but all sugar related products. The packaging of lollies, chocolate and soft drinks have led to a huge problem cleaning our beaches, neighbourhood and coastline. The amount of littering in these areas is damaging our environment and killing our animals. However many people do not realise how damaging it can be because they can not see it before them. Some people have this theory in their head that if they cannot see it they will not believe it. His idea of creating these machines which creates pretty gooey pictures with chewed up gum is also trying to send through the message that although this picture is pretty, there is also a lot of damage this pretty picture can do to us. It is almost contradicting himself.
His artwork is very creative; it has a way of attracting spectators because of the uniqueness in this art piece. The idea of being able to stick your chewed up gum to some people may seem disgusting and unhygienic, but it is because of how odd this process is that it becomes interesting and spectators want to know more about it.

References
[1] Rinaldo K 2004, Emergent Systems: Artist Statement, viewed 3 April 2008 http://accad.osu.edu/~rinaldo/

[2] Lord R 1997, ‘US Sugar Consumption Continues to Grow’ in Agricultural Outlook March 1997 [pdf], viewed 9 April 2008 http://www.ers.usda.gov/publications/agoutlook/mar1997/ao238g.pdf

[3] TED Case Studies 1997, ‘Phillipine Sugar and Environment’, viewed 5 April 2008 http://www.american.edu/TED/philsug.htm

[4] Adams T 2005, ‘This is a stick up’ in The Observer, viewed 5 April 2008
http://www.guardian.co.uk/politics/2005/jul/03/localgovernment.uk

[5] Rinaldo K 2004, Emergent Systems: Stand By Deliver, viewed 3 April 2008 http://accad.osu.edu/~rinaldo/

Asg 3: Hyperinstrument Design

Tod Machover who was attempting to invent new technology for music launched Hyperinstruments in 1986. The concept he worked with was to augment musical instruments with the aid of digital technology to give the performers extra ability for expression and creativity. It wasn’t until after 1992 that this concept was expanded to build stylish interactive musical instruments, satisfying both entertainment and human performance.

What is a hyperinstrument?

Most classic hyperinstruments are built from traditional musical instruments. Having been a cellist and a piano player I know that sound is generated using structures called vibrators. To make a certain note, a vibrator must produce a stable pitch of constant frequency. The physical structures of vibrators are how they generate sound, and is also how the musical instruments are classified e.g. wind or strings. To go into more detail a string instrument produces sounds through the vibration of strings. Frequency and tone quality of the sound depends on varying factors such as strings’ length, thickness, linear density and tension.

Hyperinstruments, on the other hand, are divided into two types. There are those that imitate traditional musical instruments, usually combined with a computer to analyse collected data. These are referred to as hyperinstruments e.g. hypercello, hyperviolin. Others are built from scratch looking unique in their varying structures. These are also referred to as hyperinstruments, but do not have ‘Hyper’ as a prefix e.g. Musical Jacket, Beat Bugs [1].

Typical characteristics: hyperviolin

The main goal in designing hyperinstruments is to expand the power of traditional instruments, providing musicians with more ways to interpret and express music [2].
Looking at most Hyperinstruments their structure are alike traditional musical instruments, even though they are electric. This is because they are designed in mind to suit musicians, so that they do not have to learn new techniques, as original look and feel will certainly improve playability. However there are still variations between each generation.

Here is an example:

Figure 1: Old version of a hyperviolin, which is more or less similar to a traditional violin [3].

Figure 2: New version of the hyperviolin, with a more streamlined style it is certainly a break away from the traditional violin [4].

By expanding these instruments it also means putting in new functionalities, such as the addition of sensors and transmitters. One particular example of this is the hyperbow. The aim of its creation was to capture the most intricate aspects of violin bowing techniques. It has been generally agreed that string instruments are one of the most difficult instruments to play, and it is through experience in which I have to agree. What makes it so difficult is the need to master the coordination between the left hand, which controls the finger, and the right hand, controlling the movement of the bow. The bow is essential to the instrument. Like a singers voice, the bow is used to produce sound, in particularly variations of loudness, the duration of notes, and to distinguish different musical expressions.

What appears to be a simple task is actually quite complex. To create a ‘good tone’ a person must coordinate between the speed in which the bow is drawn, the force applied to the string and the distance between the bow and the bridge of the instrument. It is these interactions in which researchers are keen to investigate. To enable this study of violin playing techniques is a measurement system that can be used to measure parameters in real time, maintaining portability and playability. This system consists of a carbon fibre violin bow and electric violin, in which custom electronics were installed. The bow holds a light and compact electronics board marked by a set of gesture sensors, a small battery and a transmission device [5]. Combined with the audio data provided by the electric violin, the gesture data is collected from this system and recorded by a laptop computer.

Data from these studies can be analysed, and can be used to help understand distinctions between novices and experts as well as the differing bowing techniques between experts.

Example: Tod Machover's design of the hypercello

In 1986, Tod Machover and Joe Chung began the development of hyperinstruments at the M.I.T Media lab. The purpose was to enhance and expand the performance intelligence through technology. The aim was to develop techniques that would allow the performers normal playing technique and interpretive skills to control computer extensions to the instrument, combining the ‘personality’ of human performance with the accuracy and clarity of digital technology. The driving inspiration behind hyperinstruments was in fact Machover’s general musical philosophy to “convey complex experience in a simple and direct way” [6].

The hypercello was designed for the famous cellist Yo Yo Ma in 1990. Along with its development Machover also composed a 25-minute long solo for the hypercello- ‘Begin Again Again’, where the performer controlled a variety of sound producing and transforming devices, allowing of elaboration in new and unexpected ways. Thomas Levenson [7] provides a good description of the hypercello, where it serves as an input device to a stack of machines that process the data created by a performer and generates a variety of different types of responses.

As the cellist plays the cello, sensors pick up the angle of the cellists bow hand wrist forms, where the bow is on the cello’s strings, how much pressure the cellist is placing on the bow and where the cellist is fingering the strings on the finger board. A special circulatory hooked up to the cellist sensors then translate that information into digital information, transmitting the results to the systems main computer. This is synchronised with the electronic signal of the tune actually played, which is then fed directly to the synthesizer, which can either manipulate the data or pass it through as a reproduction of a conventional cello sound.

To accompany this is a software tool, also developed by Machover and Chung to control the flow of information from a performer to his machines. The bow setting vibrations on the string also acts as a conductor’s baton. Machover’s interest in timbre in particular, further led him to develop a technique, which authorizes the performer to add the pure cello sound from the strings to a sampled, or synthesized timbres stored as MIDI data’s in the computer. This allows the performer to decide precisely how much of their playing they want to hear. The computer then generates an accompaniment to the cellist’s solo, in which the performer can control pace and rhythm. As it carries out each function the main computer processes the data it receives, modifying or even creating new sounds, depending on the instructions given within the software.

Yo Yo Ma’s performance of ‘Begin Again Again’ illustrates the vastness of Machover’s hypercello and its software. The technique of Yo Yo Ma enables the computer to measure, evaluate and then respond to many aspects of the performance. These responses could be used in different ways and at different moments of the piece. While at times the cellist playing sounds electronic other times the interweaving of the sounds are indirect and mysterious, where “the entire sound world is conceived as an extension of the soloist…but as a new kind of instrument” [8].

Other interactive instruments

Discussed above was the first type of hyperinstruments mentioned in the introduction. These were the first generation of hyperinstruments, designed in mind for skilled, professional musicians, such as Yo Yo Ma. These hyperinstruments measured many shades of performance expression, and using this to enhance the instruments capabilities.

The Brain Opera: What is it about?

Starting in 1991, Tod Machover continued inventing hyperinstruments, but the purpose this time was directed for non-professional music lovers. Anyone who wanted to express ideas, experiences and feelings in music and sounds was able to create the Brain Opera, and participate in live performances. Each performance of the Brain Opera consisted of two parts. First was an introductory period where the audience could explore, experiment and play with a variety of Machover’s instruments, following was a 45-minute musical event orchestrated by three conductors. The music incorporated recordings made by the incoming audience, along with material and musical contributions from participants from the World Wide Web.

Breaking down the Brain Opera

The Brain opera is divided into three movements, similar to many musical compositions. These final performances are what make the Brain Opera unique. There have been many experiments in the past that have incorporated audience response or contribution to musical environments, but non-so far have tried to recreate it as a real ‘piece’. Machover has attempted this by creating a structure that lets people play with the individual elements of the music, get to know them, add to them, and then see the fragments fit together like a giant puzzle [9].

Although it is difficult to pick out, there is a lot of pre-composed music in the Brain Opera. The music for Melody Easel and Harmonic Driving (in the introductory period) is all composed, but modifiable by the audience. The Gesture Wall, Rhythm Tree and Singing Trees are more dependent on improvisation, where sounds and limits are set and the design and playability of the system, but do not determine the actual music that can be played. An example is the Speaking Trees, where the system sounds out pre-composed stimuli, response is recorded is completely up to the audience member.

In the performance, Movement 1 is improvised, where there is some music but plenty of room for new sounds, thoughts and texts. Movement 2 is completely composed, with room for interpretation. Movement 3 is a mix, Machover has composed basic elements, but has left plenty for the imagination, allowing for addition, modification, mixing and producing a very different version of the piece. Movement 3 also acts as a solo for participants from the World Wide Web. Using ‘The Palette’ the ‘musical instrument for the web’ [10], which is an advanced, multi-user Java applet, active only during Brain Opera performances. Participants are also urged to submit audio self-portraits and images to the Brain Opera, which will then incorporate this to each live performance. The more material the Brain Opera can work with the more exciting the piece will look and sound.

What does the Brain Opera achieve?

With a name such as the ‘Brain Opera’, it is interesting to try to understand what Tod Machover was trying to achieve. Machover designed Brain Opera as an experience to stimulate audiences to reflect on questions revealed by Marvin Minsky, a great inspirer of the idea, whom questioned in a philosophical like manner- why do we like music? Why does music make us feel? And think? Brain Opera also serves to see how independent fragments and layers of music come together to form a complex, yet integrated sonic images. The deepest hope for the Brain Opera is to encourage people to excite their own minds, and the desire to “look inside and hear what is going on” [11].

The title ‘Brain Opera’ is provocative. It is the audiences’ involvement, not the manipulation of the hyperinstruments, that makes the Brain Opera an ‘opera’. Unlike traditional operas with a set narrative, Brain Opera steers away from any narrative, however it does have lots of ‘voices’, both professional and amateur, singing, speaking, individual or communal, where the whole generated texture is very vocal. What is more significant is the “psychological journey with voices” [12]. Participants embark on their own voyage, navigating through the process of understanding the situation of each instrument, and seeing how these can evolve into full musical structures in the performance, creating a rich experience.

Machover combined music and interactive electronic technology to stimulate. The goal of the audience, whether young or old, is to get involved with the performance and the whole experience, not just to pay attention to the technology. Those with an open mind would have a more fulfilling experience. What Machover is doing is more than just piecing notes together to make music; instead he is trying to “touch people’s lives” [13]. What the Brain Opera is trying to achieve is to take the seriousness of how people perceive music and instead making them listen and enjoy the performance.

Reference:
[1], [2], [3], [4] Nguyen, T 2006, ‘Hyperinstruments’, retrieved 5 April 2008 from http://www.tml.tkk.fi/Opinnot/T-111.5080/2006/FPaperit/hyperinstruments_final.pdf
[5] Young, D 2006, ‘Studying Violin Bowing’, retrieved 5 April 2008 from http://www.acoustics.org/press/151st/Young.html
[6], [7] Machover, T 2000, ‘The Hyperstring Trilogy’, retrieved 6 April 2008 from http://brainop.media.mit.edu/hyperstring.html
[8] ‘Hypercello/ Yo Yo Ma’, 2000, retrieved 7 April 2008 from http://brainop.media.mit.edu/Archive/Hyperinstruments/hypercello.html
[9] ‘Welcome to the Brain Opera’, 2000, retrieved 8 April 2008 from http://brainop.media.mit.edu/indexold.html
[10] ‘Interact with the Brain Opera’, 2000, retrieved 9 April 2008 from http://brainop.media.mit.edu/online/net-music/main.html
[11] Machober, T 2000, ‘The Brain Opera and Active Music’, retrieved 8 April 2008 from http://brainop.media.mit.edu/Archive/ars-Electronica.html
[12], [13] Scientific America, 1996, Interview with Tod Machover, retrieved 9 April 2008 from http://www.sciam.com/article.cfm?id=0004C81C-0C01-1C75-9B81809EC588EF21

Art drives a car. Programming provides the fuel to get me there.

-David Rokeby [1]

A hundred rectangular plates, suspended in mid air, ripple about as if brushed by pulses of air from the Ontario Science Centre’s air conditioning. The vast network of floating plates rotates slowly, sending large pulses through the interlocking mass. This is not however, the world’s largest baby mobile, nor a chandelier on steroids. This is ‘Cloud’; the ambitious brainchild of Canadian installation artist David Rokeby. [2]

Rokeby describes ‘Cloud’ as a combination of three disparate states of matter; solid, liquid, gas, as well as the interrelated energy and space. Rokeby’s ‘Cloud’ throws into contrast the large and imposing space of the Ontario Science Centre’s Great Hall, with a work that he describes as ‘largely transport or diffuse’. As such, his work occupies a large volume of the room, but animates and interacts with the room, rather than obscuring and hindering it. [3]

The numerous rows of blue plates ostensibly represent the atoms within matter. Computer-controlled motors rotate the rows at different speeds, sending fluctuating waves throughout the work. These movements symbolize the energy and state transitions of matter, representing the different states of matter in an easy to digest manner. By allowing the user to look directly up into the heart of the ‘matter block’, Rokeby achieves his goal of placing the viewer inside the space of a wave or matter state. [4]

Rokeby, who has been producing similar scaled installation works since 1982, is a decorated installation artist, whose works have received international recognition. He has hosted exhibitions in numerous locations throughout Europe, Asia and North America. He has received considerable acclaim and recognition, receiving the BAFTA award, Governor General's award in Visual and Media Arts and the Prix Ars Electronica Golden Nica Award. [5]

Many of his works feature a prominent theme of the relation between human and computer-based intelligence, and how they interact with the physical environment. [6] They are interactive installations that directly engage the users in conjunction with computer-based perception systems, although unlike other installation artists, Rokeby provides no direct interface for the user to touch or manipulate. As he has put it, his intention was to help viewers learn about themselves, by contrasting people to the areas where computers fail imitate human behaviour. [7]

One of Rokeby’s most recognized and acclaimed work that exemplifies this concept of human/computer intelligence is his ‘n-Cha(n)t’ sonification work. For this work, Rokeby was awarded the 2002 ARS Electronica award. [8] An evolution upon a previous Rokeby exhibition, the ‘Giver of Names’, ‘n-Cha(n)t’ builds upon his previous sonification technologies, creating a network of computers that talk amongst one another in an attempt to reach a consensus concerning the naming of a given object. [9]

Left alone, the computers communicate amongst themselves in their own trademark, idiosyncratic way. However, users are also able to interact with the computers, ‘interrupting’ them in a way, by speaking into a microphone. This form of input allows the computer to add more information to the ongoing ‘discussion’ between the computers, as the agents attempt to make sense of and decipher the words provided to them. [10] Visually, the computers communicate their current level of receptiveness using images of an ear, representing various states such as listening, thinking and ignoring. [11]

While the work throws into light the contrast between human and computer based communication, these interlinked agents are not truly intelligent. As articulated by Rokeby himself, his intention was merely to create a group of intelligent agents, capable of the most rudimentary form of verbal communication, “granted a fraction of some freedom they are utterly incapable of desiring.” [11]

Over 20 years ago, Rokeby produced his first large scale installation. Over 20 years, Rokeby still endeavors to impart his message of human intelligence vs computer intelligence. Over 20 years, Rokeby has refined and received critical acclaim for his first and possibly most influential installation: ‘Very Nervous System.’ [12]

Very Nervous System is a large scale installation that utilizes cameras, image processors, synthesizers and computer processing to create a ‘space’ that captures the movements of the viewer’s body to create coherent music. [13] Motion capture and video software records not only the movements of the user, but the location and movements of various body parts, allowing to computer to interpret and translate the user’s interaction into a truly unique and intuitive interaction. [14] Utilizing top of the range motion tracking and video processing capabilities, VNS is an incredibly flexible system, allowing Rokeby to capitalize on the commercial potential presented by the installation. [15]

Rokeby understood that as the computer was a logical device, the use of a computer to interpret human based activity was biased. Rokeby understood that interaction should be intuitive and natural and strove to impart this upon his computer-controlled installation. [16] To achieve this end, Rokeby replaced the random number generation of computers with the ‘complexity of sentient human response’ to create his art. The result was an interactive sound based environment that also prevented the user from controlling the installation or producing predictable results. [17]

The interaction between Very Nervous System transcended the traditional human-computer interaction of other contemporary systems at the time of its conception. In essence, the environment is being adjusted to the user, rather than allowing digital information and technology to transcend the capabilities of the human body. [18] As Rokeby has described the relationship, it is an open ended feedback loop, where the computer and person exchange and respond the information the other presents; the viewer dances and moves while the computer interprets the movements and generates respective music for the user to dance to. [19] This form of mutual feedback between the computer and viewer was the foundation of Rokeby’s journey into the mechanisms of computer-based intelligence and it’s relation with human intelligence.

David Rokeby has spent over 20 years of his life working to define computer intelligence, and applying his knowledge to contrast human intelligence in an effort to make people more aware of themselves. He is both an inventor and an artist, and his works exemplify this combination of design and practicality to achieve an ends. [20] After 20 years, Rokeby perseveres, exemplifying the boundaries between human and digital interaction, and endeavors to teach something new to both computers and his audiences.


References:

[2]

http://www.youtube.com/watch?v=r1kYDwyu8pU

[3], [4]
http://homepage.mac.com/davidrokeby/cloud.html

[5]
http://homepage.mac.com/davidrokeby/cv.html
[7]
http://www.horizonzero.ca/textsite/invent.php?tlang=0&is=3&file=17

[1], [6], [20]
http://www.horizonzero.ca/textsite/invent.php?tlang=0&is=3&file=3

[8], [10], [11]
http://homepage.mac.com/davidrokeby/nchant.html
[9]
http://www.swr.de/swr2/audiohyperspace/engl_version/interview/rokeby.html
[11]
http://www.youtube.com/watch?v=FMIjxnNllMA

[17], [19]
http://www.medienkunstnetz.de/works/very-nervous-system/
[15], [18]
http://www.horizonzero.ca/textsite/invent.php?tlang=0&is=3&file=11
[16]
http://www.fondation-langlois.org/html/e/page.php?NumPage=80
[12], [14]
http://homepage.mac.com/davidrokeby/vns.html
[13]
http://www.wired.com/wired/archive/3.03/rokeby.html

All images obtained from:
http://homepage.mac.com/davidrokeby/installations.html

Assignment 2: Hyperinstruments Report

Introduction

The hyperinstruments project was started by the composer Tom Machover at the Massachusetts Institute of Technology Media Lab in 1987, continuing from work started in 1978 at IRCAM in Paris. Hyperinstruments aim to capture data regarding the performance of music, process the data using computer software and produce output in real-time (Machover 1992).

Early hyperinstruments were designed for use by expert musicians and the data captured focused on nuances in performance to alter the sonification. Later versions of the technology would extend the scope of the focus to help teach performance skills to novice musicians by stimulating imagination and providing precise feedback (Machover 2004; Grindlay 2007).

Today, aspects of hyperinstrument design have entered mainstream culture as interactive music artwork and games. Additionally, research into hyperinstruments takes place at a wide range of commercial and academic institutions (Blaine 2007). This report however will focus on the works of the MIT Hyperinstruments Group.

Early Hyperinstruments

The early work done by the Hyperinstruments Group at the MIT Media Lab, focussed on the creation of software capable of capturing the output parameters of an electronic instrument, processing this data in some way and producing an output in real-time. As their work became more progressed, the group started making traditional instruments augmented with sensors to measure subtle parameters of the musician’s performance. The following section details the work from some of the projects undertaken by the MIT Media Lab group.

Keyboard Based Hyperinstruments

For the project Fusion Figace, which premiered in 1982, the early hyperinstruments developed by Machover used electronic keyboards to report extremely detailed data to a computer on 32 parameters – information such as pitch, amplitude and phase. This data was interpreted by the computer to create an electronic orchestra influenced by the musician’s performance. The software for this system was written in FORTRAN - In Hyperinstruments: A Progress Report, Machover comments that this system gave much more information than later MIDI based systems, although the process of manipulating the data was far more complex – as unique code had to be written for each manipulation (Machover 1992).

MIDI Based hyperinstruments

In 1987 a new system based on the Musical Instrument Digital Interface (MIDI) standard was developed. In this system, data was captured directly from the MIDI data-stream giving simpler data which was far easier to process. The musical-input device remained an electronic keyboard, but percussion devices were added creating a richer audio-environment.

The captured data was then processed according to one of four rule-sets, and was finally outputted to a digital synthesiser for output. These systems were entirely electronic – from data capture to output. These techniques developed for Machover’s opera VALLIS illustrated major design issues for future hyperinstruments – primarily that the absolute control of the instrument be with the musician in an easily accessible way (Machover 1992).

“Too often in computer/musical systems, control over the computer is inadequate, forcing the human to severely compromise musicality for the lack of a responsive partner. In the worst case, a computer can be worse than a tape recording; while it may be equally deaf and uncontrollable, it may also be playing the wrong thing” (Machover 1992).

By 1990, the BUG-MUDRA project combined input from multiple instruments – MIDI electric and acoustic guitars, and for the first time gestural data from the conductor. This enabled the creation of more controlled and complex manipulations of real-time audio. The continued development further outlined design issues and started a shift away from MIDI based devices to interpreting data directly from acoustic instruments (Machover 1992).

Hypercello

The ongoing research lead to the development of the Hypercello in 1991 and the project Begin Again Again... For this project a cello was designed with augmentation of sensors to measure a suite of parameters such as finger position and pressure, bow position, wrist angle and the pitch and amplitude of each string; a computer would capture this data via the MIDI protocol and pass it into a specially designed software environment for processing; finally, the output would be passed to hardware synthesisers and signal-processing equipment for playback. The result of this processing created the expected sounds of a cello, accompanied by a complex, changing electronic sequence of notes.

“It starts out with the performer exerting careful control over the electronic extensions, each bow change, each accent chosen to elicit a specific response. Gradually, however, the computer part starts to develop on its own - as if a Pandora's Box had been opened - becoming denser and more complex than a single human could control.” (Machover 2004)

With the collaboration of cellist Yo-Yo Ma, Begin Again Again... was performed live on numerous occasions. The score written by Machover utilised the capabilities of the hypercello with some movements being largely unaltered cello sounds, and others with the computer generated sound leading the performance. This was achieved by the computer following a pre-programmed set of rules which change throughout the performance (Machover 1992).

More Recent Hyperinstruments

Following the success of the early projects, the hyperinstrument group expanded its focus to creating other forms of computer-mediated musical devices. These projects had a greater emphasis on interactivity for amateur musicians, expanding the use of hyperinstruments as learning tools. The hyperinstrument group achieved these goals by embedding digital sensors into objects, creating instruments with integrated sensors as learning tools, visualisation of music.

One project undertaken by the group was Brain Opera which ran from 1996 until 2000, project aims included the creation of inexpensive, fun, easy-to-learn hyperinstruments designed for children (Machover 2004). These instruments combine elements of tangible computing – by changing the control-interface to a tangible object, used by touching, squeezing and shaking.

Mainstream Hyperinstrument Works

The development of powerful, low-cost game consoles has allowed the development of commercially released music games which utilise concepts from the hyperinstruments project. The game Guitar Hero for Playstation 2 by Harmonix Music Systems (comprised of former students of Machover) allows players to play complex guitar parts through a simplified interface – the software automatically handles pitch, while the player focuses on rhythm and playing the correct sequence of notes. This system simplifies the creation of music by being limited to pre-composed elements, but still gives players the ability to personalise their performance (Blaine 2006).

Guitar Hero is designed purely for entertainment, but many of its elements could be used in a learning environment, the scoring system keeps track of timing accuracy, pitch accuracy and other parameters. These could be used to provide detailed input on where a musician needs to practice their skills and provide suggestions to do so (Blaine 2006).

Conclusion

Hyperinstruments in the hands of professional musicians are able to provide uniquely detailed insights into a performance through the tracking of additional parameters only subtly revealed in normal performance. By allowing these nuances in a musician’s performance to be captured and emphasised, the musician can create far more complex expressions of a musical work.

Conversely, in the hands of an amateur musician, hyperinstruments can provide additional feedback not present in traditional instruments while providing a fun, easy-to-play interactive experience. This opens new possibilities for the process of teaching music, and for allowing people to interact with music, rather than simply passively listening.

References

Blaine, Tina. “New Music for the Masses.” Adobe Corporation, 2006 Retrieved from: http://www.adobe.com/designcenter/thinktank/ttap_music/
Grindlay, Graham C. “The Impact of Haptic Guidance on Musical Motor Learning.” MIT, 2007.
Machover, Tod. “Hyperinstruments: A Progress Report.” Cambridge, MA, MIT, 1992.
Machover, Tod. “Shaping Minds Musically.” BT Technology Journal, London, Vol. 22, No. 4 2004
Rossing, Thomas D. “Flutes to hyperinstruments.” Nature, 18 August 1994: Pages 509-510.

Short Research Topic – Nigel Johnson

Nigel Johnson is a British artist focused within the domain of interactive media and the development and production of computer-controlled, two and three dimensional, "real-time" interactive installations and digital artworks [1]. Johnson is currently a research professor at the School of Media Arts and Imaging at the University of Dundee.

This report will focus on a few examples of Johnson interactive Art installations including his internationally recognised ‘Fire-Fly’, and ‘A-Life’ - his large-scale adaptation of John Conway’s "Game of Life", incorporating elements of artificial intelligence, cellular automata, artificial life and gaming.

Fire-Fly:

“The inspiration for Fire-Fly came about from witnessing for the first time one early summer’s evening in New Jersey, USA, large points of luminous, glowing, green light, ‘dancing’ in the night air”[2]

Fire-Fly is a piece created by Johnson in 2002 and is comprised of a structural grid above the participants heads, LED’s, ambient sound, ultra sonic motion detectors and user movement to mimic the gleams and glows of a firefly. Obviously for any installation in which light is a key part the ambient lighting should be subdued in order to maximise the impact from the installation. Hence why Fire-Fly is displayed in almost total darkness to intensify the green light emitting from the LED’s. The artist explains that the ‘lightweight, thin, phosphor-luminescent discs [that hang from above] mimic the chemical [of a fire-fly] and produce a diffuse but bright green glow’ [3].

Interestingly Johnson’s experience with fire-flies are in contrast to my own experiences with fire-flies - in my experience the fireflies emit more of a sharp distinct piercing light rather then a ‘diffused glow’ that Johnson describes, so I’m not too sure why the disc are used if he’s trying to imitate the fire-fly’s chemical reaction. However there are supposedly over 2000 species of fire-flies and the ones I’ve had experiences with are from Queensland, Australia which most likely are a different species from the ones Johnson saw in the USA. [4]

In my opinion a better effect could be created if Johnson had made the installation look more natural; perhaps by entwining bush leaves through out the grid in which the led lights would reside. As participants look up at the light they would see flashes of light coming from inside a bush rather then a bare grid structure. This could easily be implemented and the only consideration would be to make sure the sight of the motion detectors was not obscured.


The ambient sounds of beetle wings flapping can be heard as participant navigate the installation. The sound is distributed around the space via a switching matrix connected to amplifiers and four loudspeakers positioned at the outer edges of the installation space and approximately 2 metres above floor level. [5]

The sound is really important in this installation. It would really help give meaning to the participant as without it a variety of interpretations could be derived from blinking green lights – Morse code signals, green eyes blinking etc. It’s a shame Johnson hasn’t manage to implement his idea of mounting smaller speakers spread out amongst the lights to enable a more dynamic movement of sound as participants move from one side to the other. The earlier idea of using branches and leaves to make a more natural look would be beneficial for this idea as smaller speakers could be mounted onto the grid structure itself and then purposely hid by the foliage.

Despite my critisim of the piece Fire-Fly has gained significance and international recognition through their inclusion at peer reviewed, festivals such as European Media Art Festival and the World Wide Video Festival.

Digital Gateway:

Digital Gateway was created in 2004 for the entrance to a new gallery called ‘Wired Worlds’ at The national Museum of Photography Film and Television in Bradford, UK. Johnson’s ‘Digital Gateway’ utilised multi-processing and l.e.d. technology, dealing with the transition from one state to another, from the physical analogue world into the digital domain. [6]


Its structure consisted of two opposing clear glass vertical walls embedded with 128 autonomous modular circuit-boards, consisting of 6,912 red light-emitting diodes, infra-red sensors, 128 micro-computers.[7] The idea would be for participants who were entering the exhibition to see themselves represented as blocks of light at the beginning of the gateway and as they progressed through these blocks would become less coarse; eventually being represented as binary i.e. 1’s and 0’s. Johnson decribes it as “[taking] the human form as the vehicle for its central theme and imparts the idea that this representation can be broken down into fundamental building blocks.”[8]

In terms of showing analogue to digital or real world to the digital world this concept is an extremely clever idea. After all, digitization consists of binary – 1’s and 0’s, which in computers constitutes On and Off, so it makes sense to use LED’s which also only have two states – they can either be on or off.

As it was an entrance piece it had to deal with fluctuations in traffic. In the building process Johnson identified several criteria that the installation had to meet – one of them being that visitor flow can vary from zero to highly active [9]. The installation also had to be ‘real-time’. If there had been a delay in displaying the visualization it wouldn’t work so well and there wouldn’t be that ‘seamless’ transition of which he speak about.

In its period of inactivity Johnson had programmed the installation to kind of experiment with itself by giving it the opportunity to ‘play’ with individual imagery.

To give a true sense of immersive-ness perhaps the use of 4 walls arranged into an enclosed corridor through which the participant walks through would give a greater effect. Being enclosed by thousands of LEDs all changing depending on your movement would give a heightened sense of what the Digital Gateway is trying to convey. But as I mentioned earlier about the criteria he identified before building the installation – one of them was budgetary constraints. If you had an extra two walls of LEDs the costs go up significantly; not just because of the extra walls but the extra processors needed to control them, and the use of strengthened glass so participants could walk along the bottom wall.

A-Life
“At one level, this installation is a simple metaphor encompassing the complexities of life, distilled down into the basic components of life, birth and death governed by the rules that control these events. At another level it is full of the subtleties and complexities of life itself.” [10]

A-Life is Johnson’s re-creation of the earlier work of John Conway’s “Game of Life” [http://www.tech.org/~stuart/life/rules.html] which is a simple yet fascinating algorithm with has fascinated numerous people of various studies for over 35 years.

Using equipment similar to the Digital gateway Johnson constructed a wall 3.4m wide x 2.9m high x 1.9m deep and a remote interface unit.

This is a large scale, real-time, interactive computer installation which Johnson describes as “incorporating elements of artificial intelligence, cellular automa, artificial life and gaming.” [11]

This work follows on from some of Johnson’s earlier research into artificial life and eco systems from the late 70’s which the artist says was partially inspired by Conways ‘Game of Life’. Johnson’s Walk II (1979) and Eco.II (1980-81) were his initial attempts at creating artificial life forms in computer software.

A-Life allows the participant via the remote interface unit to play the ‘Game of Life’. Consisting of three buttons and a joy stick allowing movement in 4 directions – up, down, left & right. The first button allow the participant to initially select a starting pattern of cells to be ‘live’ and the subsequent buttons allow the participant to pause and clear the current game.

Although many people have re-created the ‘Game of Life’ most of them have been on the small scale so its pleasing to see somebody bring this intriguing idea of bringing “simple rule-sets that could lead to the development of complex systems” [12] to a larger scale and implementing a wireless interaction between the participant and the installation. But in terms of originality this piece is lacking as the credit for the algorithm and initial implementation goes to Conway.

Nigel Johnson has been in the game of interactive installations for a long period of time and if you visit his site you can see a varied amount of interactive work.

References:

[1] Nigel Johnson: Research Summary
http://imaging.dundee.ac.uk/research/profiles/research-profiles/view/nigel-johnson/research-summary/; last accessed 10/04/2008
[2] Fire-Fly: Proposal
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/firefly/firefly.html; last accessed 10/04/2008
[3] Fire-fly: General Description
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/firefly/ff_gen.html; last accessed 10/04/2008
[4] Wikipedia: Fire-Fly
http://en.wikipedia.org/wiki/Firefly; last accessed 10/04/2008
[5] Fire-Fly: Installation
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/firefly/ff_inst.html; last accessed 10/04/2008
[6] [7] Research History : Digital Gateway (2004);
http://imaging.dundee.ac.uk/research/profiles/research-profiles/view/nigel-johnson/research-history/digital-gateway/; last accessed 10/04/2008
[8] [9]Digital Gateway: General Description;
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/nmpft/nmpft_gen.html; last accessed 10/04/2008
[10] Project – “A-Life”
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/alife/alife.html;
; last accessed 09/04/2008
[11] Prof. Nigel Johnson: Rearch Summary
http://imaging.dundee.ac.uk/research/profiles/research-profiles/view/nigel-johnson/research-summary/; last accessed 09/04/08
[12] A-Life – Context
http://imaging.dundee.ac.uk/people/njohnson/content/proposals/alife/al_context.html; last accessed 09/04/08

Assignment Task 2 - Short Research Report - Design Work by Karlheinz Essl

Note: I am not at all familiar with sound so I researched on someone who did know about it. Then I realized how little I know about sound and how very difficult it was to write about it. Forgive me if I missed out on things. (Edit: referencing didn't look like it worked with the blog).

Design Work by Karlheinz Essl

[i]

Karlheinz Essl is an Austrian top performer in Electronic music. He also does the impossible: composing unending electronic music pieces, writing music samples, creates software for Max/MSP, develops generative sound and video environments and has to power to improvise music over and over again. All this is featured in his music. Essl believes that ecstasy and trance’ always meant something for him. He tries to feel and share it through his music. To him, “composing is a pain”[ii]; he likes to improvise, for “things can happen that the composer never planned on having.” [iii]
The music is influenced from rock to jazz, and even music from the Renaissance period through to the sounds of computer –generated music. With all this impact, he breaks the normal barriers of a composer. He contradicts the meaning of a music play, what it is, how does it begin, how does it end, and how it must be structured. And amazingly, he does this through the use of MAX/MSP software.
The compositions from Essl are widespread and range from instrumental classical music, to mixed forms of live-electronic, to real-time composition and improvisation concepts to sound installations and internet projects. Essl uses m@ze°2 (Modular Algorithmic Zound Environment)[iv] - an electronic keyboard instrument he created that allows users to compose and improvise music. It is connected to his RTC-lib (Real-time Composition Library), a Max/MSP software library for his algorithmic compositions. It contains random, harmony, rhythm and structural generators. [v]
Most of times, Essl is a composer and performer in public. He believes that in today’s society that musicians are very limited to the composers of the music and the performer of the music. What makes the composer unique is that he should not only be able to compose for himself but also compose music in all categories and for all occasions. Essl has been studying closely to the principles of organization and drama through his music. Some of his works have been commented of being “constructive” from listeners, Essl says “[construction] only helps me get into areas where I would normally not be able to reach”. [vi]

Lexikon-Sonate

[vii]
Lexikon-Sonate is a program that composes an endless piano playing piece. Basically, there is no score for the computer to read from, and there is no pianist playing a piano, or a composer to direct it. The algorithm is comprised of "which key should be pressed how quickly and held down for how long"[viii] which then directs the piano into playing the note.
The goal was to create a program which can compose in real life – synchronous. The user constructs the sound from their own character personalities (personal taste), and henceforth becomes the composer him/herself.
He used music gestures from “Bach, Beethoven, Schönberg, Webern, Boulez, Stockhausen and Taylor” [ix] and placed them into ‘modules’. These modules are then mixed and a part will be extracted from the piece from different constraints placed on it. The technology he wrote this in was Max/MSP as seen in the image above. There are the structure generators seen on middle left hand side, that controls the melody, chord, texture, repetition and pauses.[x]
Lexikon Sonate is surprising it can create sound from nice soft tones to fast high pitches, varying in volume, pace and style. It is truly amazing that something can be created from a piece of software and sound system that would never repeat itself. It is very humane, yet contains none of the classical composition, for it can be relaxing at one point, but so intense the next. It would definitely have been more interesting if he not only used the piano, but other instruments. Yet even though there is still a limit in the sounds of a keyboard, the variables Essl’s program Lexikon Sonate can process, is unimaginable.
Sequitur
This new work comprises of performances using solo instruments including the flute, clarinet, trumpet, violin, cello, toy piano, e-guitar and voice mixed in with live electronics. The Sequitur program receives the input from the instruments, and differentiates it into eight parts. The random generator plays a different amount of parts each time, at various intervals all random. What is interesting about this is that there are inputs of music, but what comes out as the result is totally different to what was played. The processing is done in real time, with sound transformers altering the sonic shape. [xi]
The song Sequitur III for violin and live-electronics begins high pitched the setting for a film - a torturing scene. Then I feel the lost romance in the next tones. At the middle, it sounds like Chinese music for Peking Opera which elevates the mood. It becomes sad, and then the high keys of the piano makes it light again. That was the only instrument I can fully recognized, the process by which the violins and other music go through in Sequitur III it becomes intertwined and mysterious. What is different in their location to our context is that it all seems classical, yet modern without any harsh beats of a drum. Some parts make you tap your feet; other melodies make you concentrate more to the composition of the song.

Sonnez la cloche!

In 2003, he used church bells to create a live performance. This piece was performed in Tyrol, Schwaz and he firstly took a sample of the church bells, which then turned it into a never-ending piece. Meanwhile, he varied the sound using his Max/MSP program and waited till the real church bells rang and synchronized his music along with it. An audience member commented that the piece “changed the very Alpine air we breathe”. [xii]
The sound of Sonezz la cloche is so complex to figure out, and what Essl has created in his programs are more farfetched than you can imagine. Watch this video to see.



Other Works

Through the Curly Rain Taps is a 2005 generative visual sound installation composed by Karlheinz Essl in collaboration with Ramón Gonzalez-Arroyo. It contains the “collaboration, comination of image and sound, infinte flow, relationship to the work of the two different and similar painters”.[xiii] Essl provided the real time endless piece of music which synthesised beautifully with the images.

Panta Rhei is a 2006 generative sound environment for an art installation by Jürgen Messensee. The sounds depict water in its natural course – from destructive waves to the peace and soliloquy of the ocean. This piece is randomly generated with “the rising and falling of the sounds, the glissando movements and the spectral sound shifts are controlled by a time-variant wave model, the parameters of which are constantly changed by random operations.”[xiv] In Panta Rhei, everything is meant to be fluid.

From all his works, I realized how nothing is absolutely the same, how electronic sound can be generated, processed and portrayed as an art form.

[i] Lackinger,H., (2004) “Karlneinz Essl”, Image: http://www.sammlung-essl.at/shared/photoshop/KHE/lackinger/KHE-strudel.jpg (Accessed via Mozilla Firefox, 10th April 2008) (Online).
[ii] Transcript by KHE, “Karlheinz Essl interviewed by Bruce Duffie“ (1997) WNIB Broadcast, Chicago (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/bibliogr/essl-duffie.html (Online)
[iii] “Karlheinz Essl”, (2008), Myspace network (Accessed via Mozilla Firefox, 10th April 2008) http://www.myspace.com/karlheinzessl (and http://www.freetranslation.com/ for German translations) (Online)
[iv] Essl, K., (2007), “m@ze°2” (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/works/maze.html (Online)
[v] Essl, K., (2007), “Real Time Composition Library” (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/works/rtc.html (Online)
[vi] “Karlheinz Essl”, (2008), Myspace network (Accessed via Mozilla Firefox, 10th April 2008) http://www.myspace.com/karlheinzessl (and http://www.freetranslation.com/ for German translations) (Online)
[vii] “Lexiton-Sonate 3.2 User Interface” (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/works/lexson-modules/dep.html (Online)
[viii]
[ix] Essl, K., (2007), “Lexikon-Sonate” (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/works/Lexikon-Sonate.html Image: http://www.essl.at/pic/lexson/lexson32.png (Online)
[x] Essl, K., (2007), “Lexikon-Sonate: An Interactive Realtime Composition for Computer-Controlled Piano” http://www.essl.at/bibliogr/lexson-sbcm.html (Online)
[xi] Essl, K., (2008), “Lexikon-Sonate: An Interactive Realtime Composition for Computer-Controlled Piano” http://www.essl.at/works/sequitur.html (Online)
[xii] Ferguson, S., (2003) “I hear the Bells” (Accessed via Mozilla Firefox, 10th April 2008) http://www.essl.at/works/sonnez-la-cloche.html (Online)
[xiii] Essl, S., (2005) “Through the Curly Rain-Taps” (Accessed via Mozilla Firefox, 10th April 2008) http://sammlung-essl.at/deutsch/musik/archiv/klangwelten/rain-taps.html (Online)
[xiv] [xiv] Essl, K., (2008), “Panta Rhei” (Accessed via Mozilla Firefox, 10th April 2008). http://www.essl.at/works/panta-rhei.html (Online).

Short Research Report - Interactive Installation Art

Interactive installation art uses various materials and other media to change and/or enhance the way a particular space can be experienced by users. It is not necessarily limited to gallery spaces and can be installed anywhere in everyday public or private areas.
Almost any media can be used to create and interactive art experience in a particular environment. New media such as video, sound, performance, computers and the internet are often used, along with the conventional materials, ranging from natural to man-made. Though some installations are site-specific, in that they are designed to only exist in the space for which they were created, others are able to be adapted to other areas, thus creating a slightly unique and different experience each time.

Rafael Lozano-Hemmer is a Mexican-Canadian artist. Lozano-Hemmer is best known for his large-scale theatrical interactive installations displayed in spaces across Europe, Asia and America. Using his custom made interfaces as well as new technologies, such as: real-time computer graphics, film projections, positional sound, video and ultrasonic sensors, LED screens and other devices, his installations aim to provide "temporary antimonuments for alien agency" [1]. “I always use the word alien instead of the word new. The precedents for new are so large, but if you say alien, it is simply something that does not belong there” [2]. His works seek to break up the constant and ever increasingly uniform urban state that society is becoming by providing critical platforms for involvement and participation.

UNDER SCAN (above) is a large-scale public art project commissioned by the East Midlands Development Agency in England. Thousands of "video-portraits" taken in Derby, Leicester, Lincoln, Northampton and Nottingham will be projected onto the ground of the central squares and pedestrian streets of these cities [3].

The work uses very sophisticated, integrated technologies. The world's brightest projector (110,000 lumens of intensity) is installed to inundate the particular area with bright light, which would allow for the creation of the shadows of people, which is a vital part of the work, walking through it. A computerised surveillance camera will sense people as they walk though the vicinity, it predicts where the individual would go and sends the information to the 14 video servers and 14 robotic video projectors which can project anywhere within the site. Thousands of video portraits, which would have been previously recorded, of people from the local area, are selected at random by the servers. These will then be displayed in the viewers’ paths by the projectors and will come to life through interaction with pedestrians' shadows.
Though this is interactive art, the participants, the everyday members of society who view the work, do not actually directly manipulate the work themselves. They cannot actually have their own input and influence how the work is made. However, their physical presence is essential so that Lozano-Hemmer’s work is able to “come alive”. If nobody takes part in the installation, the work will not exist.

“In interactive art the viewer is an integral part of the artwork, so many artists attempt to foster intimacy by personalizing experiences, by establishing close relationships … I am interested in more collective and connective experiences that several people partake in.” [4]. As Under Scan has moved from country to country, the recording of the various locals of the area in Lozano-Hemmer’s work has already formed an “intimacy” with the participants of the installation, as they are all from the same place. They will feel some sort of familiar presence, as they are all locals and have the same sense of community. Furthermore, by allowing those that are filmed to express themselves as they like, Lozano-Hemmer has enabled a “relationship” to form between the participants and the video portraits. Due to the fact that those who are recorded are free to act and move as they like, the projected images will have their own individual traits and characteristics. As the video portraits would have their own personalities, viewers of the work would be intrigued by the individual reactions of the portraits and as a consequence, treat it as though they are meeting another person, thus forming the “relationship”. They would be able to interrelate with them, in that they are able to wave back if they were waved to, imitate the portraits’ actions or even just smile and watch what the portraits are doing.

Under Scan did not use sound projections at all. The portraits were mute. However, each of the portrait subjects was trying to do something to stimulate the public (talking, dancing, rowing etc.). In response, people were trying to communicate with the subjects of the video portraits. The non-verbal gesturing carried out by the portraits engaged the viewers even more as it caused them to focus even more on the message the portrait was trying to get across, even if the portraits did not want to say anything, the fact that there was no sound coming from the portraits, allowed the viewers to think about what was happening and interpret it in their own way.
More intimacy and connection is created as the images are projected from the participants own shadows. It is as though the viewer was able to bring forth the portrait thus causing more intrigue and curiosity. Those being recorded were asked to look at the camera at some point so that eye contact will be made, allowing the viewer to feel more involved and connected and not just a spectator, it would again; cause a bond between the viewer and projection.

“Call my work relationship-specific, not site-specific” [5]. Lozano-Hemmer, through Under Scan, and many of his other works, such as “Subtitled Public” and “Body Movies”, creates not only a relationship between the viewers and his artworks, but also between the viewers themselves. As a result of the fact that the computer system programmed by Lozano-Hemmer would point 14 robotically controlled projectors to interrupt the path of pedestrians, 14 individual portraits were simultaneously projected over the site; meaning 14 individuals would be able to interact at the same time. However, most of the time two or three people would gather around one portrait and start to understand what they were trying to tell them, thus creating more “relationships” between the viewers as they interacted with each other.

His other work, "Body Movies" (seen right), was also able to create various relationships between his viewers. This project, following a somewhat similar concept to “Under Scan”, transformed a public space with 400 to 1,800 square metres of interactive projections. Thousands of photo portraits, which were taken on the streets of the cities where the project is exhibited, are shown using robotically controlled projectors. However, the portraits only appear inside the projected shadows of local passers-by, whose silhouettes measure between 2 to 25 metres high, depending on how far people were from the powerful light sources placed on the floor of the square” [6]. Depending on how big images were, and the ability of the participants to “bring forth” the images using their shadows, it created a great contrast in size.
This created various reactions and relationships between the participants. “Every time we show this piece, the behaviours are totally different, ranging from playful parading to erotic performances to aggressive stances … In Rotterdam, after the work had been showing for a few days, participants started using props. Breakdancers appeared. People brought their pets. A man in a wheelchair projected his shadow 22 metres high and he seemed to derive a lot of pleasure from crushing everybody around him. It is a learning experience to see how people self-represent” [7]. Thus, all his works not only create a relationship between the user and the installation, it also creates for a means of communication between those involved in the work.

So that the "Under Scan" system was able to redistribute the portraits, the computerised severs needed to be stopped. During this time, the lights as well as the projectors were turned off. This period was called the Interlude. This occurred every 7.5 minutes, and in this time, the tracking mechanism was revealed to the public. It was displayed as a number of grids, as seen in the picture to the left. Though this was never meant to be an important piece of the installation, it was found that the Interlude provided a breather or “a moment when participants could suspend their disbelief or their act of faith” [8] and turned out to be quite popular and enjoyable. Audiences found it intriguing to be able to go behind the scenes of the installation to see the basis of how it worked [9]. This again allowed for more interactions between the viewers themselves, as they discussed what they were seeing, as well as interaction between the viewer and the installation, as they comprehend how they are being tracked within the installation site.

“I am interested in how local people use their public spaces and how they relate to each other in them. Under Scan will create an interactive shadow play where members of the public become actors” [10]. Lozano-Hemmer’s works not only requires the audience to interact with the installation, but also allows for interaction between each other, which ultimately makes the work more engaging. “Under Scan” was able to bring a different sense of interactivity, as it caused the participants in move around differently, as the way they moved in the space of the installation was different from the normal movement in public space. “This is perhaps due to the fact that the original concept behind the work was centered on the idea of the shadow, which is directly linked to body movement” [11].

The general nature of the installation called for interactivity as it brought out the curiosity within. Older people took a contemplative (observer) role, while younger adults and children were more playful and engaged with, and curious about, the piece. The Interlude, on the other hand, offered another type of experience. The participants’ behavior switched from a calm/contemplative attitude to a more active and intuitive way of behaving as they reacted to the realisation that they were actually being tracked [12]. Overall, Under Scan was able to allow Lozano-Hemmer to see how people do use their public space and how they do relate to each other as the nature of his work/s instigates and encourages interactivity from all those involved.

REFERENCES

[1] Rafael Lozano-Hemmer Website: http://www.lozano-hemmer.com/einfo.html
[2] [4] [5] [7] Glandman, R. (2006) “Rafael Lozanon-Hemmer at ARS Electronica”, Akrylic. http://www.akrylic.com/articles/34/1/Rafael-Lozanon-Hemmer-at-ARS-Electronica/Canadian-Art-Vol-19-No-4-Winter-2002.html
[3] [6]Rafael Lozano-Hemmer Website: http://www.lozano-hemmer.com/eproyecto.html
[8] [11] [12] Walker, S. et.al. (2007) “Ethnographic Interventions: A Strategy and Experiments in Mapping Sociospatial Practices” http://www.humantechnology.jyu.fi/articles/volume3/2007/mounajjed-peng-walker.pdf
[9] Audience interview: http://www.youtube.com/watch?v=Bfn14sLJmyU (7:10)
[10] “News from arts council England, east midlands” Issue 16. http://www.artscouncil.org.uk/documents/newsletters/phpuJBaWK.pdf

IMAGES

Under Scan Images:
Left:http://www.bbc.co.uk/northamptonshire/content/images/2006/01/30/under_scan_body_250x180_180x250.jpg
Right: http://www.threecitiescreate.org.uk/_images/1underscan074.jpg

Body Movies Image:
http://www.akrylic.com/articles/34/1/Rafael-Lozanon-Hemmer-at-ARS-Electronica/Canadian-Art-Vol-19-No-4-Winter-2002.html

Interlude Image:
Screenshot from YouTube video: http://www.youtube.com/watch?v=Bfn14sLJmyU

Short Research Report - Alex Davies

This short report reviews and discusses the interactive installations created by Australian artist Alex Davies, in particular four of his installations: Dislocation, Pugilist449, Drift and Reflux. 

Alex Davies studied a Bachelor of Fine Arts with Honours between 1995 and 2001, in 2006 he continued his research at The University of New South Wales.  “His art practice involves sound and time-based image production, spanning a diverse range of media including film, network, real-time audio-visual manipulations and responsive installations.”[1]

Dislocation

What is Dislocation?

This installation combines a mixture of audio, visual deception and audience participation. The installation combines the virtual and physical realities together by ‘dislocating’ the participant’s location in the installation space and placing the participant in a series of pre-recorded video clips. 

How does the installation work?

The interactive installation provides the participant with 4 small holes that are cut into a wall.  Each hole displays a monitor that displays the room the participant is standing in from the rear end.  The participant can see the other three participants peering into their own small hole, at this point an illusion of mixed reality is synchronized behind the real time visual images displayed on the participant’s monitor.  Such illusions could include a person entering the room with a dog; initially the participant’s first reaction is to look behind them with an expectation to see a person and a dog. 

What sorts of computing technologies are used in the installation?

Alex Davies primarily used Max/MSP and a green screen environment.  The artist used Max/MSP to program the sequencing of virtual and physical realities, using CCTV cameras during the pre-recorded scenarios and during the installation.  Max/MSP was programmed so that it would choose different scenarios based on the audience’s location within the installation space.  For example, if three people were peering through three different holes in the wall, the program would track and determine that the fourth hole is not being used and so executes a scenario where a person enters the same installation space and peers into the fourth hole giving an illusion that another person is standing next to the participants.

Did the installation achieve its primary goals?

According to the artist, the interactive installation was a success because it allowed the participants to experience a virtual world where they became the subject.  The installation goals at first seemed easy to execute and perform, but the artist commented that trying to make the pre-recorded scenarios as ‘real’ looking as possible was quite difficult because many of the scenarios relied heavily on the participant’s location within the environment.  If a participant stood close to the rear of the room, they would collide with the virtual ‘phantoms’ (that is, the pre-recorded scenario).

I think the installation proved to be a success because participants enjoyed the experience and were compelled to introduce their friends and family to the installation, which signified that the integration of the virtual reality with the physical reality was very much ‘real’ enough that the participant was compelled to show others of their new found discoveries and experiences.

Pugilist449

What is Pugilist449?

Pugilist449 is a video production installation where the audience is taken into a boxer’s viewpoint during a one-on-one boxing fight.

How does the installation work?

The installation is screened on a white screen, with full surround sound speakers.  The audience sits in the middle of the installation similar to a movie cinema.  The artist wears a custom made boxing helmet with 3 CCTV cameras attached and various microphones placed in and around the helmet.  All the sound and visual information is captured onto high definition digital cameras and edited with time-distortion effects such as slow motion.

What area of interactive art is the artist exploring?

The installation explores the area of space within an environment; the artist is exploring the fusion that is created between the audience and the boxing opponent.  The artists’ aim is to immerse each member in the audience with a distinct feeling of space invasion from the boxing opponent.  For example: every human being has an imaginary space ‘bubble’, each person’s ‘bubble’ varies in size depending on their personalities, whether quiet and shy or outgoing and communicative.  If a person’s personality is quiet and shy their space ‘bubble’ will be very large as they avoid contact with other people’s ‘bubbles’, however a person that is very outgoing and communicative will alternatively have a smaller space ‘bubble’ as they can handle with close proximities with other people.  In the same way with Pugilist449, the artist creates a pre-determined space ‘bubble’ between the boxing opponent and the victim (audience’s point of view).  With constant physical and metaphorical punches towards the victim, the artist attempts to explore the way certain people react to this space ‘bubble’ intrusion through time edited audio and visual aids.

Drift

What is Drift?

The Drift interactive installation is an outdoors site-specific installation, the installation was exhibited during a cruise voyage, and the interactive object was disguised as a normal sightseeing telescope.  The enhanced interactive object allowed the viewer to view surrounding areas through a manipulation of time and area.

How does Drift work?

Alex Davies worked closely with another interactive installation artist Daniel Heckenberg.  The installation works by the viewer panning the telescope from the front of the boat to the rear of the boat.  When the viewer was looking towards the front of the boat they would see reality, in normal time and with no manipulations, as the viewer panned across the scenic landscape towards the rear of the boat, time distortions and images are manipulated to show the viewer past events during the cruise voyage.

What was the artist’s aim for this interactive installation?

Comments from Alex Davies’ website outline his aim for Drift to be a “fluid and elastic manipulation of time in a panoramic visual display [altering] individuals’ perceptions of the journey by being able to examine aspects of the past in significant detail” [2].

Did the viewer experience the artist’s intentional aims?

I think the viewer was able to fully experience the artist’s intentional aims, by allowing the viewer to control how they manipulated time by panning from the front to the rear of the boat whilst at the same time, being able to explore past events that they had never seen.  I think the telescope was a very good choice of objects, because when the viewer was looking through the telescope, they were limited by what they could see, when the viewer looked towards the rear of the boat, the telescope provided the viewer with events that had happened whilst the viewer was looking towards the front of the boat.

What areas does the artist explore in this interactive installation?

The artist explores areas on the “movement of both individuals and media [and how they] are mapped across time and space” [2].  The artist’s exploration and development of the telescopic time manipulating object stretches our mind and perceptions of the physical reality.  The installation is unique because the images and past events that are shown in the telescope are taken from the physical reality, rather than from a pre-made collection library of images and videos.  I enjoy this particular area of art through interactive installations because each time the viewer peers through the telescope they can experience something new and unique rather than a collection of scenarios, similar to the Dislocation installation.

Reflux

What is Reflux?

Reflux is an interactive installation where participants interact with a child’s doll.  Basically, the participant shares a memory with the doll, and the doll responds with a recorded memory that was recorded earlier by someone else, using time to distort and fragment the memories throughout the dolls’ life cycle (exhibition duration).

How does Reflux work?

Reflux works by installing a number of different sensory inputs into the interior of the doll, the artist installed 2 microphones, one in each ear of the doll to record the memories given by the participants, a mercury sensor was also installed to detect when a participant was interacting with the doll, which switched on the recording equipment.  Each time a participant recites a memory; the doll records the memory in a computer database and responds with a random memory that was recorded earlier by another participant, as the exhibition time progresses, the doll’s data memory begins to fragment, similar to the common medical condition amnesia.

What sorts of computing technologies are used in the installation?

Similar to Dislocation, Max/MSP is used to trigger and record the audio memories.  Using the installed mercury switch, Max/MSP is programmed to switch on recording software and log each incoming memory with a unique tag and store it in a database.  Alex Davies also used Max/MSP to control the choice of memories from the database that the doll would reply.

How did physical time affect the doll’s responses (audio recordings)?

The doll was programmed with a set of limitations, variables and statements.  The doll was created so that it would have a limited lifecycle similar to a human being, except the doll’s lifecycle would end at the end of the exhibition. Throughout the exhibition as time passed and as the doll recorded increasing amounts of memory data, the program started fragmenting and losing the data so that the doll’s responses became less fluent and sometimes just noises and tones.  This fragmenting and data loss explored the way in which human beings lose data and memory, as they get older.

What did I learn from Alex Davies’ interactive installations?

By studying these four interactive installations: Dislocation, Pugilist449, Drift and Reflux I gained a better understanding and awareness of what sorts of technologies and possibilities are available with regards to audio, visual and the process of interacting with it.  I particularly liked the Drift installation, whilst it was a site-specific installation, each time the participant interacted with the telescope, they had the freedom to choose whether they wanted to view a scene in real time or warp time and space together.  I found the Pugilist449 installation to be a lit bit ‘childish’ amongst all the other complex interactive installations Alex had produced, because the installation did not require much interaction other than sight and hearing.  Although the installation was relatively simple, I think this installation would need a specific audience.

I learnt that many of the programs were created using Max/MSP, which has also shown me what is possible for my teams’ own major project.

References

[1 + Alex Davies Profile Image] (1996). "Asialink." Retrieved 02,04, 2008, from http://www2.asialink.unimelb.edu.au/arts/residencies/VApastresThailand.html

[2 + Installation Images] Davies, A. (15/04/07). Retrieved 10,04, 2008, from http://schizophonia.com/installation/index.htm

Critical literature review - haptic interfaces (late submission)

Apologies for the lateness, I too find it difficult to juggle both employment and uni work effectively (but im working on it).

Haptics is the study of the touching behaviour in humans, as well as how this can be incorporated with computers and technology. “Haptic interfaces can be defined as the communication with a computer through a tactile method involving a device that senses body movement, such as a data glove [1]”. “Haptic technology refers to the technology which interfaces the user via the sense of touch by applying forces, vibrations and also other motions” [2]. The many uses for haptic interfaces include educational purposes, not only to teach people the many concepts associated with haptics, but to also give them virtual environments to help with the learning process.

An example of a multi-fingered haptic interface robot (image source: http://www.jamd.com/search?assettype=g&assetid=53040767&text=haptic+interface)

Haptic technologies work by providing sensory feedback through the use of specialised hardware devices, such as data gloves and joysticks. When used correctly this hardware senses the body’s movements, enabling the user to “not only feed information to the computer but to receive information from the computer in the form of a felt sensation on some part of the body” [3].

A 3d haptic mouse (image source: http://www.3pointd.com/20070109/novint-3d-haptic-ships-at-higher-price/)

Advantages

Haptic interfaces open a new avenue of communication between humans and computers. Previously, computers could only communicate to humans through two of the five senses – those of sound and sight, but now through the use of haptic interfaces this now includes touch (see video for some examples). This allows users to actively engage in learning a vast range of skills and ideas, literally putting the control of learning in their own hands.

Disadvantages

There are a few disadvantages associated with haptic interfaces, which include extremely complex applications which often require highly specialised hardware and loads of processing power, not to mention the high cost that may be associated with obtaining the necessary equipment.

[1]Webopedia Computer Dictionary, “Haptic Interface”,

http://www.webopedia.com/TERM/H/haptic_interface.html

[2]Wikipedia Online Encyclopaedia, “Haptic Interface”, http://en.wikipedia.org/wiki/Haptic_interface

[3]Webopedia Computer Dictionary, “Haptic”, http://www.webopedia.com/TERM/H/haptic.html

Critical Literature Review - Robotic Interaction and Cyborgism

Robotic interaction and cyborgism relates to extending the capabilities of humans, through artificially enhancing and the use of artificial limbs. Robotic interaction involves the use of humanoids, intelligent vacuums and other such devices, which are capable of being configured. They are therefore mechanical extensions which can possibly be interactive. Cyborgism is the extension rather than improvement of a human’s physical and mental impulses. This includes but is not limited to external limbs and walking robotic exoskeletons. This is usually accomplished by embedding responsive sensors into the body. As there is anatomical modification, ethical have been raised from the introduction of robotic interaction, and especially with cyborgism.

Robotic interaction has many sub categories. Robotic interaction can either be configurable to complete a set task, or work neutrally imitating the notions of the human brain.

Human enhancement refers to attempts to overcome limitations of the human body via an artificial means. Technological means are used to select or alter human characteristics, whether or not they exceed the existing range of human capabilities. This is particularly useful for giving people with disabilities a quality of life up to the standard of able bodied persons. An example of this would be attaching a prosthetic limb to a person who had their original limbs amputated, and then connecting it to respond to their brain’s impulses.

http://tardis-db.co.uk/2008/02/data-streaming-mind-controlled-human.html

Humanoids refer to any structure which resembles the structure of the human body. Humanoid robots are therefore robots with an overall appearance of a human. Humanoid robots are autonomous in that they have the ability to adapt to their environment and interact in its environment and have the ability to solve simple problems.

Because of these abilities, humanoid robots are usually more technologically advanced than other kinds or robots. They possess qualities which allow them to sense, actuate, plan and control; rather than having a human input directions controlling its actions.

Humanoid robots were initially created to imitate actions that humans would undertake on a regular basis. These days they are also used for research purposes, as a study of human body behaviour and structure. Global commercialism and marketing has also resulted in humanoid robots being sold as toys to children, and interact as ‘pets’. In countries such as Japan, humanoid robots are even used as a means of entertainment where people pay to watch robots interact in a controlled environment.

Intelligent vacuums such as iRobot’s Roomba have made life much more convenient as they do the job that humans generally do. They interact with their environment a large contact sensor on the front of the unit, and an infrared sensor at the front centre of it. Some models have capabilities such as dirt detection and a designated home base.

http://irobot.com/sp.cfm?pageid=122

IRobot have been a major force in intelligent robots, creating other robots which have capabilities such as:

Scooba – Floor washing

Dirt Dog – Shop Sweeping

Verro – Pool Cleaning

Looj – Gutter Cleaning

ConnectR – Virtual Visiting

Cyborgism is the extension of physical or mental impulses of human. This can be achieved in the form of human limbs, walking robots and electronic exoskeletons. Human characteristics are generally included by embedding sensors into the body. Because such features are trying to simulate humans and their actions, they simply extend human capabilities rather than improve them.

The term transhumanism has been used to refer to artificial human enhancements. It is an international, intellectual and cultural movement supporting new sciences and technologies which have the ability to enhance human physical and mental capabilities.

Prosthetic limbs have helped to enhance the quality of life of people who have had to have limbs such as arms or legs removed. Not only can they allow everyday tasks that people engage in, but it can also help in the rehabilitation process.

Electronic exoskeletons are mechanical structures which offer assistance in making movements, especially during the rehabilitation process of major accidents or surgery. For example, an exoskeleton of a leg can assist in leg movements in the forward direction and keeping lateral balance. They also help to support the physical load of the person they are supporting.

Radio-frequency identification (RFID tags) are used as a method of automatic identification, where data is stored and remotely retrieved via transponders. They can be embedded into persons or other beings with the purpose of identification and authentication, improving efficiency within a system. They are also used in supermarket chains and other stores as a means of security on stock, as they are contained within a small circuit connected to other receivers which are permanently in the store. However, people have begun to create ways to overpass this type of security, such as “RFID Blockr” where a faraday; cage which stops magnetic fields; protects RFID tags.

http://www.rfidblockr.com/

Although cyborgism devices are capable of extending human capabilities, they don’t necessarily improve them. This is because a computer cannot think for itself unless information is inputted into the device and it has been trained to adapt to a situation or an environment. Therefore, it is only capable of extending the abilities of what it already knows, without the ability to exceed these traits. For example, a hearing aid can assist a person’s ability to hear sounds, therefore extending their capabilities in that field. However, it does not improve their actual hearing ability beyond that of a regular person’s hearing ability.

Cyborgism and its reliance on fringe science has proven to become controversial as many ethical issues have been raised. People have argued that devices which support cyborgism are ‘playing god’ as they are not allowing the human body to exist the way that they were created to. Religious and conservative groups have disputed the ethical value of artificially enhancing the ability of the human body. It can also be argued that manipulating the capabilities of the human body will end up manipulating the future, therefore changing the way the world functions. Despite this, this field of artificial intelligence continues to grow at a rapid rate as there is clearly a high demand for it in many areas including medical and entertainment purposes.

Assignment Task 1ii - Critical Literature Review: Haptic Interfaces

Hahah took me a while to remember my login, I kept on typing my Hotmail account!

Assignment Task 1ii - Critical Literature Review: Haptic Interfaces

Introduction

Haptics changes the way we interact with computers. Consider haptic interfaces as the connection between “the human hand and manual sensing and manipulation” [i] with machine. Haptics are used often in sync with robotics to create information obtained by actively touching and exploration of an object – known as feedback. This feedback can come from interactions between humans and machines whether it be in real or virtual space. The keyboard, mouse and trackball are much known tools. However they are known to be “passive”[ii] as the users control everything. On the other hand, the “active” such as data gloves allow not only the user to control, but to feel objects.

Advantages

I believe that haptics will become everyday entities as it will engage humanity and improves our (virtual) lives. Haptic applications are in various streams such as in medicine, there are tasks which manipulate robotics for performing surgery, simulations for training medical staff, and help the disabled. A device can measure how an instrument moves, and when information is given to the computer, it can cause a slight push to simulate resistance.
Video games and simulators allow users to manoeuvre solid substances, liquid forms and different tools for a new type of entertainment. A sense often ignored in games, surprisingly it can improve a user’s senses and allow them to be more emerged into the virtual game dissolving the line of virtual and reality. Haptic features in games can involve joysticks vibrating as vehicles travel on rough terrain and signaling the speed by the frequency of vibrations. Humans and a controller would share the same control interface like a steering wheel, and by having authority over this, people can be more free to negotiate a task more freely.[iii]
For designers, manipulating objects in environments would be simple if we can incorporating haptic systems into CAD. Selecting, moving, and resizing can be a time consuming act, however with a haptic technology, the accuracy and speed of the ability to take and replace the object would be more efficient. You can execute tasks in 3D, rather than on normal 2D levels. Visual art displays bring about a whole new level of art with haptic features such as touching virtual musical instruments to play sound using exoskeletal gloves.

Technology

Much money has been put into the development of the industry. A recent example is a Maglev Haptic control unit at Carnegie Mellon University which uses electromagnetics (alone costing tens of thousands of dollars). The two yellow bowls contain a joystick which the user interacts with shapes that are displayed on a screen. You can move objects, get reaction from the collision, and artificially feel the weights from the magnets. “Hollis said that it's the fastest-moving haptic technology, and can be made to vibrate so fast it produces music.”[iv]
The mainstream consumer product that is commercially available of haptic interfaces is the wireless Wii remote. The best thing about the Wii is that you can use this anywhere within five meters of the sensor bar, allowing space for the user, and the capability to be everywhere else. The haptic ability allows you to easily navigate through the menus simply by making a ‘physical’ bump when you scroll over a button. In golf, you can feel the impact when you hit the ball; in boxing, when you get hit, the vibrations that shake your hands make you even try at it; and it even senses a tilt when you swing in bowling. So it translates our hand and body movements “into 3D spatial co-ordinates, and force/direction of acceleration data, which it then transmits to the Wii's processors for input data.”[v]



This short film shows how maybe people in the future would be too reliant on technology. This particular piece shows the extent of how haptic technology can evolve into real life.

Future

We are fed by our lust in creating things that were deemed impossible a decade ago. Where would the future of haptics interfaces lead us? Even now, I search for haptics, and there are articles which connect it to ‘virtual sex’ – and claiming it would be better in years ahead with the extra sensory responses placed in the application.
Would all our newly graduated doctors have only practiced on precision haptic applications achieving an accurate sense of touch? There is a $1.3 million Da Vinci system already from Intuitive Surgical which allows robotic arms to be controlled by doctors allowing them to operate.
And in gaming, would devices turn into a full body simulation where how you move in reality determines your movements in the game?

Summary

Pointing tools and GUI input devices require agility and motor capabilities to use. Haptic devices provide us with tactile clues such as simple bumps, giving us more sense in the task at hand. Thus, the human interaction is enhanced through the feedback giving us performance achieved quickly and easier. It is then possible to have haptic experiences that may not be available in the real world for it would definitely change computing.


[i] Kay M. Stanney, 2002, Handbook of Virtual Environments: Design, Implementation, and Applications, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 20
[ii] Biggs, J and Srinivasan, M, Laboratory for Human and Machine Haptics (The Touch Lab), Massachusetts Institute of Technology, Cambridge, MA, pp. 93
[iii] Griffiths, P and Gillespie, B, (2005) Sharing Control between Humans and Automation Using Haptic Interface: Primary and Secondary Task Performance Benefits, “Human Factors” Vol 47, iii, pp. 574
[iv] Beschizza, R, 2008, Article: Hands-On With Maglev Haptic Control Technology, Wired (Accessed via Firefox, 2nd April 2008) (Online) http://blog.wired.com/gadgets/2008/03/hands-on-with-m.html
[v] The Wii: Truly a Wanda for Everyone? (Accessed via Firefox, 2nd April 2008) (Online) http://www.virtualworldlets.net/Resources/Hosted/Resource.php?Name=WiiWandaForAll

Critical Literature Review: Human–Robot Interaction & Cyborgism.

The term ‘robot’ in the average person’s mind conjures up images of the Hollywood style movie depictions of shiny metal (gold in the case of Star Wars’ C3P0) humanoids with uncanny human-like characteristics and behaviours. When in reality the term ‘robot’ is used to describe any mechanical, artificial agent, which, by its appearance or movements, conveys a sense that it has intent or agency of its own [1]. Now this leaves scope for quite a variety of things that could be classed as ‘a robot’ and this ambiguity is a problem when it comes to actually deciding what is and what is not a robot. In general terms robots are said to have most but not necessarily all of the following properties [2]:

• Is not 'natural' i.e. artificially created
• Can sense its environment, and manipulate or interact with things in it
• Has some degree of intelligence or ability to make choices based on the environment, often using automatic control or a pre-programmed sequence
• Is programmable
• Moves with one or more axes of rotation or translation
• Makes dexterous coordinated movements
• Appears to have intent or agency

Major research in human – robot interaction (HRI) has been to create robots with a more natural intelligence and behaviour; to enable them to act and respond in a more life-like, human manner. The prime example of this can be seen in the development of humanoid robots such as Honda’s ASIMO [ASIMO Image]. According to Honda - ASIMO is a 120 cm tall people-friendly robot that is capable of operating freely in the human living space [3]. Honda has gone into some depth explaining the reason behind the physical design of ASIMO - revolving around the robots height. ASIMO’s size “allows the robot to operate light switches and door knobs, and work at tables and work benches. Its eyes are located at the level of an adult's eyes when the adult is sitting in a chair. A height of 120cm makes it easy to communicate with.”

ASIMO’s two eyes not only serve a purpose for the robot such as facial recognition, environment recognition etc but also serve a purpose in the human side of things. Humanoids are anthropomorphic robots. Duffy states that the ‘exterior design of the robot has therefore aimed to define its functionality and facilitate its assertion of its identity as distinct from that of a mere object’ [4]. When we talk to a human we talk to their face or more specifically we look at their eyes. What it feels like is for a genuine interaction between a robot and ourselves we feel the need to make them anthropomorphic – to give them a face or even a personality. This can be seen in other examples including SONY’s AIBO robotic dog toy and PINO the humanoid. As highlighted in the PINO research, “the aesthetic element [plays] a pivotal role in establishing harmonious co-existence between the consumer and the product” [5].

As the advancement in robot AI becomes more and more lifelike you have to wonder whether ethics has a role in robotics? If you give a robot a personality/identity should you think of them differently from other technical machines? Should we give robots the ability to ‘feel’? Roboethics is study surrounding this topic and has highlighted whether the early-envisaged ‘Three laws of Robotics’ by Isaac Asimov still have meaning today?

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey orders given to it by human beings, except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Discussions within roboethics have also asked what role will these intelligent autonomous humanoids play in our society? Duffy provides some examples such as a museum guide, a corporate building butler, or a post office clerk but more importantly states that the fundamental issue is to ascertain what the robot, as a machine, is inherently good at [6]. They’re really just more elaborate forms of what autonomous machines perform today – jobs that require high level of accuracy (the museum guide robot could have access to the internet to bring up endless information and facts); jobs that a menial and boring; jobs that require speeds not possible by humans.

While scientists are striving ahead with the advancement in humanoids others are turning the science on themselves. Kevin Warrick, a British scientist and professor of Cybernetics has successfully conducted experiments that by definition have made him a Cyborg. A Cyborg is a cybernetic organism (i.e., an organism that is a self-regulating integration of artificial and natural systems).[7] For an in depth reading I suggest viewing his personal website at www.kevinwarrick.com but his pioneering experiment involved the neuro-surgical implantation of a device into the median nerves of his left arm in order to link his nervous system directly to a computer in order to assess the latest technology for use with the disabled.

While Warrick is conducting experiments on a scientific basis, cyber artist Stelarc is exploring cyborgism in a more artistic sense. He has produced many cyber influenced art pieces including Fractal Flesh in which Stelarc plugged himself into muscle-stimulation circuitry controlled by a Mac. The Mac, in turn, was connected, via the Internet, to Paris's Centre Pompidou, Helsinki's Media Lab, and Amsterdam's Doors of Perception conference. By pressing a color-coded 3-D rendering of a human body on a touch screen, participants at the three sites jolted the artist's (literally) wired body into action [8].

Depending on where you stand in term of robotics depends on how you feel about cybernetics. Those who feel that robotics implemented into the body used to better understand the human body will have different opinions from those who feel that cyborgs are people who want to improve the human body as a tool.

Where do you stand on robotics/cyborgs? Should robotics be inspired by humans in order to realise sophisticated machines, or use machines to better understand humans?




[1] [2] Robots Wikipedia : http://en.wikipedia.org/wiki/Robot
[3] Honda ASIMO: http://world.honda.com/ASIMO/technology/concept.html
[4] [6] Anthropomorphism and Robotics , Brian R. Duffy; Available From: http://www.prism.ucd.ie/publications/pub2002/AISB02-Duffy.pdf
[5] PINO http://www.symbio.jst.go.jp/~yamasaki/
[7] Kevin Warrick Homepage www.kevinwarrick.com
[8] FRACTAL FLESH: STELARC'S AESTHETIC OF PROSTHETICS http://www.stelarc.va.com.au/fractal/index.html

Task 2 - Critical Literature Review: Tele-immersive Network Interaction

Tele-immersion could be the future of the world; it has the ability to revolutionize the way we communicate for work, education and day-to-day life. Tele-immersion is an innovative technology that allows people to communicate as though face-to-face when they may be geographically situated far from one another.

How does it work?
Tele-immersion requires several cameras, arranged in a semi-circular shape around the room, allowing a variety of camera angles to capture each individual movement of the participant. A kind of “living statue” is created from the camera scans and this is projected onto a large screen for the other participants in the tele-immersion network to see (Frauenfelder, 2001). To create a stereoscopic view and thus a totally believable scene, participants must wear polarized glasses so that the retina of each eye is able to see a slightly different image. Several other pieces of equipment are also required such as microphones, speakers and head mount displays.

Benefits of tele-immersion
The benefits of tele-immersion will not be fully recognised until its use is put into practice in a variety of fields such as education, business and general communication. However, we can envisage the kind of effect it will have on so many people just as video-conferencing has already proved to be a great success.

Video-conferencing is a tool much like tele-immersion used for collaboration between people who may not be situated within the same area. It uses real-time transmission of audio and video but it does not provide the same experience as tele-immersion as it does not give the effect of being in the same room as the other participants as only one camera angle is usually used thus not creating a three-dimensional effect. Nevertheless, video-conferencing has recently taken off and is now commonly being used for business, as well as in other fields. The first major benefit of video-conferencing and thus tele-immersion is that it considerably more economical for a group of people to communicate over the internet than for a number of people to have to travel to a particular place to conduct a meeting. Although there may be initial costs involved with tele-immersion such as computers, cameras, speakers etc. the costs involved with actual face-to face meetings are substantial (e.g. travel costs, accommodation) and also endless assuming that the meetings will take place on more than one occasion. Secondly, tele-immersion has major benefits as it alleviates participants from travelling to a meeting point to communicate the same message. This is of particular significance for people with disabilities or who are less mobile for the reason that with tele-immersion they can collaborate with others from the comfort of their own home. Similarly, the convenience that tele-immersion provides, allowing people to communicate from such different locations, encourages further collaboration because of its simplicity and accessibility.

So why isn’t everyone using it now?
While the benefits of using tele-immersion can be seen there are significant pitfalls that need to be overcome before it will become more prevalent. Firstly, tele-immersion requires a huge bandwidth because it generates such large amounts of data per second. While the large amounts of data is also quite taxing on computer processing power, the principal concern is that the data must be transmitted quickly to allow uninterrupted real time collaboration between the tele-immersion participants. Secondly, the costs involved with tele-immersion are quite high during initial set-up. Although the ongoing costs are minimal, at the current price of the equipment required it is unrealistic for individuals or groups that are not part of large companies to be expected to purchase it.

Uses for tele-immersion
As the use of tele-immersion becomes more widespread education will be one of the areas that will benefit significantly. Tele-immersion networks will be able to be used for schools where the students are not all located in the same area. For instance, students of the School of the Air would be able to keep in touch with fellow schoolmates face-to-face and with their teacher encouraging greater collaboration and thus further education. Tele-immersion could also be used to keep in touch with exchange schools overseas – becoming the modern day version of a pen pal which would promote further study of other cultures and languages.

Some businesses no doubt are already using tele-immersion to conduct business meetings with international offices however as tele-immersion advances are made and its use becomes more dominant a greater global marketplace would be created. Businesses around the world would become more accessible thus not restricting clients to companies within a certain range and this therefore would increase competition consequently benefiting the customer.

Everyday communication with family and friends will be completely revolutionised, as tele-immersion will further encourage social networking. After the initial costs, keeping in touch via tele-immersion will become as simple as sending an email today.


As bandwidths increase and computers become more powerful tele-immersion will become more prevalent in our society. Until then further research is required into such technologies to make networking more simple, attractive and less confronting to the everyday person.


Ditlea, S., 2001, Tele-immersion: Tomorrow’s Teleconferencing. (Accessed via Safari on 26th March 2008). (Online). http://www.cs.unc.edu/Research/stc/inthenews/pdf/CGW_2001_jan.pdf

Frauenfelder, M., 2001, The Year iin ideas: A to Z; Tele-immersion. (Accessed via Safari on 26th March 2008). (Online). http://query.nytimes.com/gst/fullpage.html?res=980DE4DD143CF93AA35751C1A9679C8B63

University Of Pennsylvania, (2002, November 20). Tele-Immersion System Is First "Network Computer," With Input, Processing And Output In Different Locations. ScienceDaily. (Access via Safari 26th March 2008). (Online). http://www.sciencedaily.com­/releases/2002/11/021120072242.htm

Image:
Fuchs, H., & Towles, H., 2005, Group Tele-Immersion. (Access via Safari 28th March 2008). (Online). http://www.cs.unc.edu/Research/ootf/Projects/groupti.html

Critical Literature Review: Interaction Design

Note: Apologies for the brevity of this. I fail at work/uni balance :(

Interaction Design
The field of Interaction Design (IxD) is defined by Wikipedia as:

The discipline of defining the behaviour of products and systems that a user can interact with. The practice typically centres around complex technology systems such as software, mobile devices, and other electronic devices. However, it can also apply to other types of products and services, and even organizations themselves. Interaction design defines the behaviour (the "interaction") of an artefact or system in response to its users. (Wikipedia, 2008a)

Interaction Design is an inherently multidisciplinary field having ties to Human Centred Computing Interaction (HCC), industrial design and graphic design. By using human factors – made up of the fields of usability and psychology – interaction designers can follow an informed, iterative process to improve the user experience and usability of interactions through research, observation, and qualitative and quantitative analysis of various metrics.

An Interaction Process

An Interaction Designer will begin this process by researching users to gain an understanding of their goals and needs. This is achieved through the use of questionnaires, interviews and observation.

Some questions which an interaction designer tries to answer include (adapted from Wikipedia, 2008b):

• Who are the users of the product?
• What are the users’ tasks and goals?
• What are the users’ experience levels with the product, and other products like it?
• What functions do the users need from the product?
• What information might the users need, and in what form do they need it?
• How do users think the product should work?

The results from these studies can then be compiled into qualitative analyses where themes and issues can be identified through comparisons of users; and also into quantitative analyses where these issues can be ranked and values assigned.

The second stage is the creation of an interaction which the interaction designer expects users to follow. This process is guided by the design research, and a collaboration of many different design, science and engineering fields – Human-Computer Interaction, User Experience Design and Psychology. This stage creates the structure of the interaction, the steps involved and the elements needed to create the interaction.

Finally, these elements are combined to create a prototype interface for the interaction. These may be low-fidelity prototypes such as wireframes, sketches and role-plays. This enables potential-users to test fully featured versions of the interaction, without being fixated on the final design. The first stage is then repeated on these designs to provide data on its effectiveness.
During the low-fidelity prototyping stage of this process, interaction designers can make informed changes quickly and easily. These changes are then retested to check for improvement in their effectiveness.

When satisfied with these results, the interaction designer can continue on to the creation of high-fidelity prototypes, which can appear to be working versions of the interaction. These can help test the product in a more complex way, providing more detailed information on the effectiveness of a design solution while also allowing iterative changes and testing to be made without massive expense.

Finally, the product is implemented as a final solution -although the task of testing and minor changes may continue later into the product-development cycle. These final-stage changes tend to be limited to any remaining issues remaining in the product due to the expense and difficulty of reimplementation.

Summary

Interaction Design has become a major part of modern technology. Interaction designers are employed to develop the interfaces on modern mobile phones, operating systems and applications. Interaction designers also progress the field by researching new ways of interacting through haptic interfaces such as Microsoft Surface, gestural machine-vision based interfaces and speech recognition.

Interaction design is also involved in taking the field of computing into new territory by creating physical interfaces to interactions – for example by manipulating physical objects.

References
Interaction Design, Wikipedia (2008a). http://en.wikipedia.org/wiki/Interaction_design
Usability, Wikipedia (2008b). http://en.wikipedia.org/wiki/Usability
Knemeyer, Dirk and Svoboda, E (2005). User Experience - UX. Interaction-Design.org http://www.interaction-design.org/encyclopedia/user_experience_or_ux.html
Apple Human Interface Guidelines, Apple Inc (2008) http://developer.apple.com/documentation/UserExperience/Conceptual/OSXHIGuidelines/XHIGIntro/chapter_1_section_1.html
User Centered Design, SAP AG (2006) http://www.sapdesignguild.org/resources/ucd_process.asp

Assignment 2 Critical Essay Robotics Interaction and Cyborgism

Robotic interaction + cyborgism

Cyborgism, Cyborgology and cyborgs are cybernetic organism which is an organism that is combining artificial and natural systems or it can be defined as “the melding of the organic and the mechanic, or the engineering of a union between separate organic systems” quoted by Gray, Mentor, and Figueroa-Sarriera. Robotic interaction is the study of a robotic interacting with a human or another robot. These robots will be able to have facial expressions, speech recognition, gestures and even a personality. [1]

Cyborgism is a human to have robotic features. Many movies such as The Terminator series, Bicentennial Man, AI and Star Wars have given the idea that cyborgism is a robot which has the brain of a human. They are able to think and have emotions like humans. However our technology has not advanced to that level. Cyborgism today can be recognised as the technology a quadriplegic patient would use or when an athlete has had to have a limb replaced, the artificial leg or arm is a cybrog. [2]

The relationship between cyborgism and robotic interaction is that the research to create robots which reflect and resemble humans in a physical and mental way is becoming more demanding. However, humans are very unique species; they have two distinct features; unique physical nature and emotions / feelings. To implement this into a robot is very difficult because to simply model the movement of a human is difficult, let alone to model the emotional features of humans. Another factor to consider is that when a robot is interacting with another human or object, it will keep a safe distance between the object and itself. This applies to humans as well. Two humans interacting together, they have a certain distance between them which makes them feel more comfortable. Robotic interaction is more than just navigating a human and interacting with them. The robot needs to understand and recognise human social behaviour and act in a way which will not cause conflict. Research in robotic-human interaction is developing in understanding how to make a human feel more comfortable with the presence of a robot. With this research, scientists are trying to develop robots which will be helpful in the workplace and at home. [3]

An example of how a robot can be used in at home is in the movie “The Bicentennial Man”, a robot was designed to do housework. It acted as a nanny or carer. During the movie, the robot started to become interested in how humans interacted together, hence his ambition to “turn” into human began. He first got human skin made for him and then his organs where modelled for him, in the end he became mortal and was the longest living human in history. Now, for that to be possible in the real world is still a question. But it does raise the issue that if these robots were to really become available to humans and they wanted to become a human and they did, are they really considered human?

Firstly, robotic-human interaction needs to be more than simple speech recognition. When a human is communicating with another human, they rely heavily on facial expression and body language. Non verbal gestures and eye contact play an important role in visual feedback. It will regulate the flow of conversation and provide a good interaction between the two people. Robots will need to be able to recognise and produce non-verbal communication cues to be an effective collaborative partner.

Common interactive robots which have been designed are robots which are placed in museums to guide people through and to answer any questions from the viewer. Sage, who was created by Nourbakhsh et al in 1999 was placed the Dinosaur Hall at the Carnegie Museum of Natural History. Visitors at that museum would interact with Sage through basic speech recognition and a LCD screen. His speeches include humour (which is basic communication tool which humans use to interactive with other humans) to interact and engage with the visitors. Sage also had facial expressions which would express his mood when confronted with different type of questions. Sage shows not only how speech affects communication, but also how the form of speech and non-verbal communication influences how well communication takes place.[3]

Another type of interactive robot which is going into development are robots which assist with the elderly and disabled. For example, reminding and helping the elderly to take their medicine, helping them get things off shelves or assist them with the refrigerator, getting in and out of bed and inform other people that there is an emergency. The reason of designing robots which take care of the elderly is that many aged people enjoy independency and to be put into a home can be intimidating and expensive. Having a robot which assists with all the day to day needs helps the family financially, and keeps the elderly happy. [4]

With the intense research to create robots to have a human resemblance in both physical and mental aspects, the problem is what will humans do when the robots know how we think and start to become smarter than us? Do we rely too much on technology that technology one day may take in control? The idea to have a robot which humans can interactive is a good idea. But we should always look at the situation in the long terms. Robotics and cyborgism should be only researched to benefit humans which cannot be benefited in any other way, for example, the loss of a limb or a hearing aid. Before we get carried away with all this research and development, there should be a time where we stop and think what it is we really need. There is a difference between making something which is something we need and something we want. The increase of robotic use in our homes and workplace means a decrease of employment for humans. There is a consequence to a rising unemployment rate which would instead affect humans.

References

[1] [2]“Cyborgology 101” 2006
http://www.sjsu.edu/faculty/butryn/whatisa.htm#Cyborgs%20on%20the%20Web

[3] Janaka B, & Chandrajith M, & Watanabe K “Adaptive Personal Space for Humanizing Mobile Robots” http://s.i-techonline.com/Book/Human-Robot-Interaction/ISBN978-3-902613-13-4-hri01.pdf

[4] Songmin,J &Kunikatsu, T “Development of Service Robot System With Multiple Human User Interface”
http://s.i-techonline.com/Book/Human-Robot-Interaction/ISBN978-3-902613-13-4-hri07.pdf

Critical Literature Review: Haptic Interfaces for the Blind

Haptics is the study of how to combine the human sense of touch with a computer-generated world. So why develop haptic interfaces? Ivan Sutherland, a founding father of virtual reality suggested, “human kinaesthetic sense is as yet another independent channel to the brain, a channel whose information is assimilated quite subconsciously” [1]. This inevitably led to the development of haptic interfaces.

The anatomy and physiology is very important in designing haptic interfaces for a human. The hands are most commonly used in haptic interfaces- humans constantly use their hands to explore environments that are poor in visibility, particularly for those visually impaired. The fingers are one of the most sensitive parts of the surface of the skin, with an amazing 135 sensors per square centimetre at the fingertip. While humans are very good at identifying 3D objects with their hands, we are not as capable at identifying 2D objects. However ways of exploring such spaces are through raised surfaces on a plane, where we are able to gather information by first identifying an edge and then following a contour, illustrating the importance of touch.

Why develop multi-finger tactual displays?

The motivation of design was to explore the sense of touch as an alternative communication channel. This will act as a sensory substitution for individuals who have hearing or visual disabilities. It also serves as a new haptic interface used as an exploration to human-computer interactions through a tactual channel

The potential to receive information tactually is been well demonstrated by natural methods of tactual speech. One particularly noteworthy is the ‘Tadoma’ method used by individuals who are both deaf and blind. The method of speech reading is based on vibrotactile reception of the articulatory movements and actions that occur during the creation of speech [2]. The hand of the individual is placed over the face and neck of the talker, so that the thumb is positioned lightly on the lips and the fingers fan out over the cheek and neck.

This simple method produces remarkable abilities to experienced Tadoma users. Individuals could understand every day speech, through observing a variety of actions associated with speech production at very high levels, allowing rich two-way conversations between both familiar and novel talkers.

What makes Tadoma successful is that it is received by hand. A talking face is eternally rich, concurrently displaying various stimulating qualities that engage both the kinaesthetic and cutaneous sensory systems. Trying to recreate this in tactual devices has been mostly unsuccessful, with one major problem being the nature of the output display. These displays are commonly composed of multiple stimulators that send a moderately homogenous cutaneous stimulation, with few distinguishing perceptual qualities [3].

Several studies conducted by Reed and Delhorne [4], show that- long term users of artificial tactile aids demonstrated limited improvements in their speech reading abilities, while normal subjects with imitated deafness and blindness can achieve a certain level of skill in discriminating and identifying speech through the Tadoma method after a limited amount of training.

The studies found that long-term users of artificial aids had little gain, as they were observed for the reception of words in isolated sentences, only averaged around 25%. The benefits of current wearable tactual devices to speech reading appear to be limited, even after subjects had worn these aids for several years.

Studies with inexperienced users of the Tadoma method suggest that some ability to receive speech through the tactual sense alone can be obtained through limited amounts of training. In an analytical study of the Tadoma method conducted by Reed et al, reported that with training of 100 hours each, two inexperienced Tadoma users performed in a way comparable with that of an experienced Tadoma user on consonant and vowel identification tasks. They also demonstrated the ability to receive sentences constructed from a limited vocabulary [4].

The difference between Tadoma users and users of artificial aids may reflect differences in the amount of information available in a talking face, as compared with that available from current artificial tactual aids.

The New Age: FEELit Mouse

Recognition of the need for richer tactual displays is now prevalent. Rarely new technologies have emerged that have the potential to greatly enhance the computing experience, particularly those visually impaired. The FEELit Mouse, designed by Immersion Cooperation is one such technology. Whilst most blind computer users have a screen reader combined with synthetic speech and or Braille display, giving them access to text on the screen, but not to the graphics. But with the FEELit Mouse, it grants the remarkable ability to enable computers to convey realistic sensations to users, representing an essential milestone in the evolution of human-computer interaction. It is a sophisticated ‘information technology’, allowing users to ‘touch’ their software, taking advantage of the same tactile sense and physical instinct that guide and inform us in real world interactions [5].

Traditionally, a mouse is confirmed only through direct visual feedback. As a result, cursor-targeting tasks demand significant visual attention, not only slowing execution, but is unusable for those visually disabled. The FEELit Mouse can manipulate the interface, feeling each encounter between the cursor and the items visually displayed on the screen e.g. the edge of a window feels like a groove carved into the desktop and when the cursor slides into the groove you feel a forceful physical engagement. Tasks such as resizing the window will be effortless.

The FEELit Mouse has Important Network Implications making it also a valuable Internet tool, allowing users to feel elements drawn within like HTML web pages. It enhances targeting, manipulation and exploration. One additional benefit of the FEELit Mouse that is exceptional is allowing multiple users to interact with each other through feel. Imagine exploring a shared environment, and being able to interact with other users through direct physically contact. Being able to feel the actions of others greatly enhances communication and understanding, extending the idea of computer access for blind persons, from being inaccessible to bringing it to a whole new level.

[1] Smith, C (2004). “Human Factors in Haptic Interfaces”, http://www.acm.org/crossroads/xrds3-3/haptic.html
[2] Reed, C M. “The implications of the Tadoma method of speechreading for spoken language processing”, http://www.asel.udel.edu/icslp/cdrom/vol3/1002/a1002.pdf
[3] Tan H Z and Rabinowitz W M (1996). “A Multi-finger Tactual Display”, http://www.ecn.purdue.edu/HIRL/projects/papers/tactuator/C19_Tan_ASME1996.pdf
[4] Tan H Z et al. (1999). “Information transmission with a multifinger tactual display”, http://www.ecn.purdue.edu/HIRL/projects/papers/tactuator/J08_Tan_PP1999.pdf
[5] (2005). “FEELit Mouse”, http://www.cwhonors.org/search/his_4a_detail.asp?id=592

Critical Literature Review - Tele-immersive Network Interactions

Tele-immersive network interaction refers to the collaboration of people, who are geographically distant, through the use of multiple cameras at each location, whose inputs are reconstructed into a 3D virtual reality. Participants from all locations are able to see themselves, as well as the others they are working with, actually moving and interacting with each other in real time in a shared virtual environment.

Participants are generally represented as realistic computer generated depictions of themselves. The environment they share can be anything from a realistic everyday room, such as an office, to a large endless plane. Tele-immersive interactions generally have real world objects and/or settings captured in real time rendered in the virtual environment. What makes tele-immersion different from the usual virtual reality is that “the environment transmits gestures as well as audio and video, so users have a greater sense of presence in the shared space than they would with other collaborative media” [1].

Tele-immersion creates a dramatic new medium for groups of people who are physically separated from each other to work together and share experiences collectively in an immersive 3D virtual environment, just as though they were all within the same physical room. The ultimate goal of tele-immersion is not to reproduce a real face-to-face meeting in every detail, but “to provide the "next generation" interface for collaborators, world-wide, to work together in a virtual environment that is seamlessly enhanced by computation and large databases” [2]. The potential inherent in tele-immersion will revolutionise the way we communicate and collaborate with one another.

Tele-immersive networks would bring many advantages in fields which require group collaborations and/or interactions. An already popular form of communication in the business world is videoconferencing, using internet telephone technology and video cameras to bring together people from widely spread destinations on video monitors. However, the ability for people to interact in these situations is restricted to a certain degree. Tele-immersion takes the concept of videoconferencing a step further. The interactive nature of tele-immersive networking would allow business men to physically interact with each other through their virtual representations, creating a stronger sense of a conference or meeting atmosphere. This technology would allow for them to meet more frequently as they would not have to go out of their way to travel to certain locations, there wouldn’t be extra expenses due to travel costs or to hire out a room if it were to be a big conference meeting. Videoconferencing would then be made a lot easier and efficient if done so using tele-immersion.

Also, as tele-immersion technology advances, the pace of surgical innovations has increased dramatically. The need for safer means of training surgeons in various forms of surgical practices has shown another advantage from the further research which has gone into tele-immersive network interactions. The conventional video recorded lessons, used for surgical training, have also long been available to help surgeons learn new procedures. However, these videos are generally seen as only slightly effective due to a number of reasons: the fixed point of view that is integral to the narration, lack of depth perception and interactivity, and the little bits of missing information which the camera does not always pick up but could end up being vital [3]. Overall, the experience of watching a video is just not as effective a learning method as being there and seeing the procedure done in person. “Tele-immersion would allow trainees to freely and naturally walk around a life-sized, high-fidelity, 3D graphical reconstruction of the original time-varying events, pausing or stepping forward and backward in time to satisfy curiosity or allay confusion” [4], thus making it a much more effective and safer means of training for surgical procedures.

Furthermore, the time zones and distance of researchers have caused a limit to the productivity and the efficiency of collaborative research projects. Using tele-immersive networking technology would prove to be more productive because it would eliminate the barriers of space and time so researchers will be able to interact with each other, synchronously or asynchronously, without the interruptions and expenses of travel or time-dependent studies. Researchers from one location would be able to work on and manipulate the project and then leave the environment, so that researchers in another location could enter the environment at their appropriate times and see and interact with the changes that have been made. “In tele-immersion, participants are not talking about a storm; they are standing inside it. They are not looking at a scale model of a car design; they are standing inside the engine block. They can then change parameters on a supercomputing simulation of the storm or engine block and collectively study the impact of those changes. With these enhanced capabilities, collaborators may choose to work virtually even if more traditional face-to-face meetings are possible” [5].

Tele-immersive network interaction would prove to be a major advantage in areas which require collaboration and interactions from participants who are geographically distant from one another because having physical body representations can be very helpful in aiding conversation and understanding in the virtual space as you can see where your collaborators are, and what they are looking at or pointing at. Tracking the user's head and hand position and orientation allows articulated avatars to transmit a decent amount of body language, and are very useful in task oriented situations [6].

However, in order for tele-immersion to be at its most efficient, more research needs to go into making it more commercial. At the moment, current bandwidth limits, camera quality and processing speeds of everyday computers are not strong enough to support the use of tele-immersion in everyday communicating activities. Nevertheless, when the technology is readily available and cheap enough to be made commercial, the way we communicate would be revolutionised, allowing newer and easier ways of communication to occur.

[1] [5] Leigh, J et. al. (1999). “Visualization in Teleimmersive Envinronments”, Scientific Visualization, http://www.evl.uic.edu/aej/papers/computer99.pdf
[2] [6] Imai, T. et. al. (2000). “Overcoming Time-Zone Differences and Time Management Problem with Tele-Immersion”, http://www.startap.net/startap/images/PDF/timezone.pdf
[3] [4] Brown University Computer Graphics Group - Applications, http://graphics.cs.brown.edu/research/telei/teleImmersionApplications.html

Critical Literature Review - Interacting in Virtual Reality

The virtual reality (VR) medium can be defined into three main characteristics (as stated in J. Whyte’s: Virtual Reality and the Built Environment):

• Interactive – users can interact with models
• Spatial – models are represented in three spatial dimensions
• Real-time – feedback from actions is given without noticeable pause

As defined above, interaction plays a key role in the virtual reality medium, just as in real life or the built environment humans interact with each other.  Imagine if there was no interactive element in the virtual reality medium, models and objects would be static and it would be a world with no life, thus the interaction between the user and the spatial environment created is most important to the success of the virtual reality medium.

There are two groups of things in a VR environment: the players and the characters.  The players are the actual human beings that sit at a computer or joystick pad and use controls to interact with the characters that co-exist inside the VR medium.  The characters could be objects, models or pieces of digital information that are designed and exist only within the VR medium.  One of the most important goals during the creation and implementation of a VR medium is to provide an interactive and virtual experience between the player and the character, and not the opposite way around (providing the experience for the character and not the player).

It is important to discuss how different types of players interact within the same VR medium.  Using R. Bartle’s: Designing Virtual Worlds, Bartle categorizes a player into:

• Achievers – These players interact with the VR medium expecting to achieve a defined goal that will allow their character to advance to another level within the game.  This will involve players interacting with menus and messages that alert the players of the defined goals that need to be achieved.
• Socialisers – These players seek enjoyment and satisfaction purely by interacting and communicating with other players within the VR medium.  Players may choose to create their own identity when describing themselves in the real world.  This will involve the players interacting with a keyboard or microphone via local and/or global chat systems.
• Explorers – These players will tend to interact with global navigational functions of the VR medium.  They will interact with menus and maps to select places and locations they wish to explore and discover.  These players want to be able to interact with anything they see so that they can learn anything new about how the VR environment works.
• Killers – These players are looking to dominate the world their character resides in, the players want to be able to interact with other players in a way that they can attack or make life in the VR medium difficult and stressful.  These players want to be able to interact with an object in a way that causes terror amongst other players.

All four types of players may or may not be found in every VR environment, for example: in Second Life, all four types of players can be found because the VR environment is designed so that the virtual world mirrors that of the real world, though in other VR environments such as online 3D first person shooter games, generally only killers and achievers can be found, as the general goal of the game is to dominate and achieve defined goals rather than to sit around and have a discussion.

Players interact in many different ways with the character in the VR medium depended on the type of game or virtual environment the player is using.  The range of peripherals varies from a general keyboard and mouse, to large room sized spaces with cameras, motion sensors, audio speakers and visual projectors.  For example: Players in the Second Life virtual 3D environment use directional keyboard keys and a mouse to interact, move and place their characters in the virtual environment.  However in a game such as a flight simulator (http://www.dimagemaker.com/ktml2/images/uploads/display/c6-iowa.jpg) a player interacts using a specifically designed headset and joystick to manoeuvre the plane and sight targets around a 3D space.

The most simple and widely used interactive purpose of a VR environment is to navigate the environment.  Commonly used in architectural based VR environments but also in other environments (e.g. second life), allowing the player to navigate their movement through rooms, up stairs and onto balconies that “can reveal aesthetic or functional problems present in the design but not readable from traditional or non-interactive architectural representations [e.g. scaled cardboard models]” (Bertol, D, 1997).

A new type of futuristic interaction peripheral was used in Stephen Spielberg’s movie: Minority Report (http://www.technovelgy.com/ct/content.asp?Bnum=1186).  This wired glove allowed the player to interact with certain windows, objects, and information on a display.  Using the wired glove the player can move, scale, manipulate and transform the object as if the player was physically touching the object.  A gaming accessories company: Mattel released the first commercially sold wired glove in 1989 called the Power Glove suitable for the Nintendo Entertainment System (http://en.wikipedia.org/wiki/Power_Glove).  Nintendo has been renowned for their focus on interactive controls (e.g. Wii remote) rather than graphics and game hardware (e.g. Playstation 3 and Xbox 360).  The general purpose of the Power Glove was to allow the user to interact using hand movements such as grabbing, pulling or punching.  Although over 100,000 Power Gloves had been sold in the USA alone, the product did not have a very long life cycle, as the interactive movements were very tedious and unreliable.

A current issue among developers and participants alike regards the interaction between two avatars and censorship.  It’s often difficult to police a general censorship rating inside a VR environment because the virtual world is constantly evolving and becoming larger as more players create and discover this new infinite cyber world.  In second life developers have created a censorship rating of: mature content and general content.  Players are made aware upon first login of this censorship system, and most ‘search’ related menus have a checkbox where the player can deselect search material that includes mature content (e.g. popular places: strip clubs, would not appear if the player selects to turn on the censorship filter).  I think this is one of the best censorship systems because whilst developers of second life would not be able to police every avatar created, they can however transfer the onus onto the players themselves 

The distance between technology and humans will continue to decrease as technological advancements in peripherals, will evidently invent a world where humans will no longer need to use a keyboard and mouse to navigate a computer screen, with web cams and audible systems allowing every player to interact directly, as the player waves his arm at the web cam, his created avatar waves his hand.  Whilst the future holds many possibilities regarding this area of virtuality, interaction will always be a major characteristic of the virtual reality medium.

References

Bartle, R. A. (2004). Designing Virtual Worlds, New York, New Riders Publishing.

Bertol, D. (1997). Designing Digital Space: An Architect’s Guide to Virtual Reality, New York, John Wiley & Sons, Inc.

Whyte, J. (2002). Virtual Reality and the Built Environment, Oxford, Architectural Press.

 

Drawn - Zachary Lieberman


Drawn takes a users drawn creations and gives them a life of their own. Originally a performance piece based on the same concept, audiences were so keen to try it themselves Zachary Lieberman turned it into an interactive installation. When a user draws on the page, a computer takes the data and then projects an image back in real-time. This projection also takes into account the users hand movements and the drawings and then manipulated from this.






Move – Andrew Hieronymi


Generally computer games involve sitting down and controlling an avatar, or representation that a user can control. In ‘Move’, there is no avatar and instead of sitting down and using controls, the users body movements in 3D space immerse them into the game. A graphic element represents the opposition and the user must go through 6 modules including Jumping, avoiding, chasing, throwing, hiding and collecting. If the user collides with the graphic element, the game is over. The game also includes varying degrees of difficulty.







KAGE – plaplax

A Shadow based interactive system; KAGE involves a number of cones placed on the floor with a projector above. While at first the cones appear to have natural shadows from the cast light, they begin to trembles as time pass