Overview

The EyeCon software is a camera-based motion recognition system especially designed for stage and installation art work. It was developed by Frieder Weiss and is an on-going project of the Palindrome Inter-Media Performance Group which is co-directed by Robert Wechsler and the program author. The first version of EyeCon appeared in 1995.

Applications

EyeCon permits movement to control or generate sounds, music, text, stage lighting or projectable art. It is adaptable to an enormous number of applications, lending itself to experimentation in genuinely new directions in performing and installation art. It offers a way to create interactive video environments without the need to get into graphical or script-based programming. This means, that people without special computer skills are able to use it.

See also >artistic considerations an essay by Robert Wechsler<.

EyeCon has been used by a number of theater and dance groups as well as singers and performance artists. The Palindrome IMPG piece 'Seine hohle Form' uses EyeCon to control an interactive MAX/MSP patch written by Butch Rovan The piece was awarded First Prize at the Berlin Transmediale in 2002 (category: "interactive art"). EyeCon was also the basis for various installation works developed during the EU workshop project 'Realtime Presence' which were presented at Ars Electronica 2002 and the Cynetart festival in Dresden.

How motion is sensed

A video signal is fed into the computer and the video image appears on the computer screen. Using the mouse, graphical elements like lines or fields are superimposed onto the video image in different colors . When a performer touches one of these imaginary lines, or moves within one of the fields, something happens, for example, a certain sound might be heard. This is the basic principle of Touchlines and Dynamic Fields. Position Trackers work somewhat differently since they are not in fixed positions, but rather move with the performers, following them as they go.

Credits

EyeCon has been used in various applications by media artists around the world*. These users and others have provided us with invaluable comments and suggestions:

Humatic (Christian Graupner, Niels Peters),
Theater der Klänge
Georg Hobmeier
Tenteki
Pablo Ventura
BlueLab Dresden
Orm Finnendahl
Dominik Rinnhofer
Butch Rovan
Bertrand Melier and Jean-Marc Duchenne
Margarita Bali

The authors would especially like to thank Bertrand Merlier for his assistance with this manual. (Bertrand Merlier and Jean-Marc Duchenne created the piece "4Hands", which is also EyeCon-driven).

* Please let us know if you are creating art using EyeCon and tell us your experiences. We will be happy to include your link.

Comments, Suggestions, Bugs

Palindrome IMPG

Johannisstr. 42

90419 Nürnberg

+49 911 397472

Germany

www.palindrome.de

eyecon@palindrome.de

EyeCon Help, this file last changed on 20. January 2004


What You Need

Home

What you need to work with EyeCon

System Requirements

  • Software

runs on Windows-PC (Windows 98, ME, Windows 2000, Windows XP), 800MHz or higher (recommended)

can be used with multiple (networked) computers

  • Hardware

There are two possible types of cameras and camera-computer connections which may be used

Directshow compatible video device (webcam, video board, firewire camera etc).

or

any standard video camera (black and white or color, NTSC or PAL). This, however, requires the purchase of a special frame grabber board for your computer. The board provides 2 Composite and 1 SVHS camera inputs and is compatible with the drivers we have written.

see "Purchasing EyeCon and accessories".

EyeCon Help, this file last changed on 14. Febv 2003


Introduction

Home

EyeCon

Introduction:

Human movement in general, and dance in particular, is of high complexity. As humans we perceive it subjectively and are able to interpret its functional or expressive content. As an engineer i have to look at it from a different perspective. I could try to break it down into a set of parameters. I could look at the overall speed of the movement or the exact position of the limbs in space or the body's position in the room. That's exactly what EyeCon is doing, it's breaking down movement into a set of abstracted features. In the end it allows you to individually assign digital media to each of those movement parameters.

The systems by which EyeCon senses movement are represented through graphic structures called Elements. The Elements are superimposed onto the live video image in the Video Window where they can be scaled and manipulated. The Elements are then assigned their individual properties such as the volume control of a particular sound or triggers for external media.

Types of Elements*

TOUCHLINES

Touchlines are lines drawn on the video image in the computer. these act as triggers for the presence or absence of body parts or objects, but can also be scaled, so that different places along the line can have different effects. In this sense they provide an easy way to track position along a line.

DYNAMIC FIELDS

Dynamic fields are boxes which represent fields and respond to total movement dynamic. They can be used to trigger sounds and images or to control the volume and pitch of sound files so that, for example, the faster the dancer gestures or moves, the louder the sound or the higher the pitch. As you might imagine, because they function so intuitively, Dynamic Fields are the most accesible and easy to use element.

FEATURE FIELDS

Feature Fields look at first like Dynamic Fields, but they function quite differently. For example, Feature Fields can measure the dancer's overall size (how expanded or contracted they are) or how close two dancers are to one another. They can be used to analyze shape (width compared to height), or topmost, leftmost, rightmost point on the body. Finally, there are a set of controls for sensing the direction of movement, so that a step to the right will sound differently than a step to the left, reaching up differently than reaching down, etc.

POLYGONAL FIELDS

The polygonal fields are acting like dynamic fields. The advantage is that they can have any shape and are not restricted to rectangles.

POSITION TRACKERS

Position trackers track the position of one or multiple persons as they move around the video image. This means, if you have an overhead camera, you can track the location of persons as they move within a room thus the environment can be made to respond to different persons differently. Touchlines and Dynamic fields can be attached to the tracker, so that a given array of controlers can move with the performer as they move around the space. Finally, this feature permits color-specific tracking, presenting the possibility of distinguishing between dancers by the color of their costume.

Types of output

Sounds

External synthesizers or other musical equipment
Internal synthesizer (MIDI). Virtually all computers have a built-in synthesizer chip.
*.wav files may be triggered, volume controlled, panned, pitched, etc.

Images

Still pictures (digital pictures, *.jpg, *.bmp, etc.)

Animations

FLASH animations

Movies

*.avi files (movies) may be played with control over starting and stopping points, direction (ie. forwards or backwards film motion) and speed.

Data link

By linking EyeCon to external systems -- either running on the same machine, or a different one -- it is possible to control a great variety of secondary media. This is done via data link or network. The additional computer does not need to be a PC. External software and hardware which can be controlled by EyeCon include software and hardware synthesizers, MIDI-compatible lighting boards, as well as programs like Director, MAX/msp, Reactor, and Isadora. The data link can be via USB or Ethernet and can use MIDI or OSC protocols.

Technical background

EyeCon works by comparing the individual pixels of two different video frames and analysing them for differences in brightness or color. The difference between these two frames is time. They are the same scene, but one is the present, and one is the past. In some cases, there may be only 0.04 seconds between, but to the body part or parts in motion, this is plenty to allow the determination that motion has occurred. Generally speaking, not all of the pixels in the two images are compared, but rather only those marked off by EyeCon's Elements.

* - A complete list of Elements, along with technical information about each one, can be found in our Table of the Elements.

EyeCon Help, this file last changed on 14. Feb 2005


First Steps

Home First Steps - How to start

How to start

In this chapter you will learn how to begin using EyeCon. You will be able to create interactive environments using midi and sound output. You will learn how to use EyeCons basic video analysis features like touchlines, dynamic fields, feature fields and position tracking.

Overview - Features

Installation and Setup

Installation for DirectX-video device (Webcam, TV board)

Installation for Falcon Framegrabber

Camera Connection

Creating EyeCon Video Sensing Elements

Touchlines

Dynamic Fields

Position Tracking

Media and System Control

Midi

Sound output

OSC Introduction OSC Implementation

Playing Video

Special features

How to organize scenes (Sequencer)

Autostart EyeCon

Artistic considerations, an essay written by Robert Wechsler

Introduction

The Nature of Interaction

Mapping

Conclusion

EyeCon Help, this file last changed on 14. Feb 2005


EyeCon Installation

Hardware Requirements

EyeCon was initially programmed for a special video framegrabber board called 'FALCON'. With current versions of EyeCon it's also possible to use other video input devices like Webcams, TV-cards, DV cameras. For optimal performance and flexibility we still recommend getting the FALCON board although there are some execellent webcams out now (i.e Philips ToUCam 840K) which allow good results with EyeCon.

Make sure your video device has a so called WDM drivers. This is the newer version of the video support in Windows. Older (Video for Windows) drivers lead to very low performance of the whole application.

http://eyecon.palindrome.de/equipment.html gives an overview of recommended hardware.

Go to the Falcon installation page for detailed information on how to install the EyeCon (Falcon) framegrabber board.

Software Installation

1. Install video hardware and software

Before you try to run EyeCon, make sure your video hardware is functioning well with the applications that come with the hardware.

2. Install EyeCon Software

Run SetupEyeCon1xx.exe. The setup program will guide you through the necessary steps. Make sure you select the right option for the video device support. You basically have tree choices:

1. Installation for DirectX-Devices (Webcam). This is the default and the option you should use if you didn't get the special framegrabber hardware from us.

2. Installation for Falcon-Framegrabber and PAL cameras.

3. Installation for Falcon-Framegrabber and NTSC cameras.

After completing the setup you can run the EyeCon software (without restarting Windows). If you are working with a DirectX-device (Webcam or other) you have to select and setup that video device. Look here for detailed information.

EyeCon Help, this file last changed on 16. April 2003


EyeCon Installation

Home

EyeCon Installation for Windows Video Devices

Hardware Requirements

In addition to the Eyecon software, you need a video camera. Maybe you already have a video device installed in your computer. Here are some remarks on the usability of this cameras and video devices in respect to EyeCon.

Digital Cameras

So called "webcam" cameras offer a simple and affordable solution since they do not require any addition hardware or adapters. They are usually connected via the computers USB port. Webcams tend not to have the picture quality of a normal video camera. This may and may not be a factor: it really depends on what you want to do with the system. If the camera is far away from the stage (such as in a big theater), or if the lighting is not very good (low level, low contrast), then you may run into trouble. We recommend USB webcams for tryout and experimenting. The Philips webcams are known for the high picture quality and light sensitivity.

Newer DV digital cameras generally have a firewire (IEEE1394) output and you might have a firewire connector on your notebook or desktop PC. DV cameras obviously have excellent picture quality. Their drawback is that they always use full TV resolution resulting in a high data rate and perceivable latency when capturing data with Eyecon. Also, some cheaper models don't allow you to turn off the autofocus or automatic aperture (Eyecon would interpret such camera setting changes as "motion" resulting in elements being mistakenly triggered). We can't really recommend the use of DV digital cameras with EyeCon.

Analog Cameras

The standard video camera today outputs an analog video signal. For Eyecon's purposes, the quality of these cameras is almost always adaquate . Black and white cameras generally have more-than-sufficient picture resolution (and indeed, often have less video noise than color cameras). Analog cameras are the recommended solution for flexible stage use, cables can be as long as 100m, you can select the appropriate lens for your needs and professional clamps are available.

They do, however, require some additional hardware to get their image into the computer (most computers do not come with a video-input capability).

Analog to Digital Converters

There are three possibilities: One is to install a video-input board into your computers PCI slot. The installation is easy, and this solution has the advantage that you can attach multiple cameras to your computer. Many so called TV-cards work with EyeCon.

Another option is to use a video-to-USB adapter unit (available from us) and then simply use the USB jack of your computer. With the USB solution you may experience a perceivable latency (less than 100ms) between the time when a person moves, and the time when the media is triggered.

There also adapter units for the computers firewire connector. The consumer versions (DV-Bridge and similar) have the same drawbacks we described for the DV cameras. They usually only convert full TV resolution resulting in a high data rate and long latency when capturing data with Eyecon. The latency is even worse than with the USB solutions. (Special drivers (Canopus) can minimize that problem.)

There's a special analog to firewire converter box available that doesn't have the described drawbacks. It's special industrial equipment and transmits the video data fast and without compression. This is the best solution for analog to digital conversion for firewire (and notebook computers) we know off. It's relatively high-prized.

Software requirements

There are two generations of windows drivers for video devices. The newer version is the 'WDM' driver (Windows Driver Model). This is the one you should check for. EyeCon also supports the older 'VfW' drivers (Video for Windows) but performance and setup possibilities are very limited.

In case you are using an older video device (the analog video editing board you installed years ago...) please check wether you find 'WDM' drivers for your board from the manufactorers website.

Software installation

Please make sure you select the right option during installation of EyeCon for DirectX-video support

After installation run the EyeCon program for the first time. First you get a selection box where you can choose between the installed video devices:

Select the video device you want to use. If you want to get the selection box everytime you start EyeCon, select 'Use' to proceed. If you always want to work with that specific video device, select 'Use as Default'. When you start EyeCon the next time, it's loading that driver without asking you again. You can later disable the default setting in the video setup menu.

For testing you can also select an AVI-file from the file menu. It depends on the the codec and compression of the video file, whether you can use this option. Not all files might work.

Now the program main screen should come on and you should see a video picture in the video window.

Setting up the video input parameters

First check your video settings. Go to menu Options-Videopara to setup your device. It depends on the video device how this menu will look. You can usually setup brightness and contrast and different other parameters.

For your start with EyeCon you should turn down the color saturation to 0 or turn off colors with the checkbox (if available) . This will result in better overall picture quality.

The checkbox 'use current video device as default' allows to select whether you are asked for selection of a video device after program start or the current device is used as default.

Custom video dialog

Also check the custom dialog for you video device. It looks different for each individual video device.

Here you see the custom dialog for the Philips ToUCam 740K.

Usually you should turn off all automatic settings, set a high framerate and switch to black and white. Not all options are always available. Sometimes the settings interfere between the main video setup and the custom settings. It just depends on the driver.

EyeCon Help, this file last changed on 16. Aprilt 2003


EyeCon Installation

Home Installation of Falcon PCI Framegrabber

EyeCon Installation for Falcon PCI Framegrabber

Hardware Installation

EyeCon runs best in combination with a special video input board (framegrabber). You can buy this board from palindrome.

Install the FALCON framegrabber board in a free PCI slot int your PC.

Software Installation

1. Install Framegrabber driver

After you installed the framegrabber board, Windows will detect new hardware. The driver is located on the EyeCon CD in the directory \driver\disk2. Windows might ask twice for the location of the board driver. You can also install some demo software for the framegrabber by running \driver\disk1\setup.exe from the CD. But this step is not really necessary.

If you don't have a CD you can download the driver from http://eyecon.palindrome.de in the download section.

2. Install EyeCon Software for Falcon PCI Framegrabber.

Run SetupEyeCon1xx.exe from the base directory on the CD or the path on your harddisk where you downloaded the setup-file to. When you are asked to select the setup type use the option 'EyeCon Framegrabber' Pal or NTSC depending on the camera you want to use.

Connecting a camera:

Connect a video camera to the video input. TheFALCON video framegrabber has two inputs for composite video and one for SVHS connections. Video Input #1 is the one that is further away from the SVHS connector.

On the control window you can select the used camera input.

You should see the camera picture in the Video window now. To change the Video-Input channel or the contrast/brightness settings choose Options/Videopara from the menu or press F6. Here you can switch the framegrabber inputs and tune the input parameters.

By default the color saturation is set to 0. That means that you don't see any colors in the video window even though you might have connected a color camera. The reason is, that neither touchlines nor dynamic fields need color information. Turning on the color usually leads to a noisier videopicture and makes controled interaction more difficult.

Check http://eyecon.palindrome.de for recent updates of the software, the driver or the helpfile.

You can keep multiple versions of the software in different directions. They don't interfere with each others.

EyeCon Help, this file last changed on 10. April 2003


Touchlines

Home Create Touchlines

creating touchlines

The first systems we used at Palindrome were simple sound triggers, the so called 'touchlines'. They detect the presence of a bodypart or object in a certain fixed location along a line.

In the leftmost column of the Element-Editor there are creation-buttons for the Eyecon elements. Press Line to create a new touchline. A new line entry in the element list is made. In the video-window the new line is displayed in the upper left corner. A touchline compares all the image pixels that are covered by that line with a stored image of the scenery, usually the empty stage, empty room etc.

Moving the touchline.

Move the touchline by moving the end points of the line. Click the end point with the left mouse button, hold the mouse button and move one end of the line. You can also move the whole line by grabbing the line between the endpoints.

Assigning a note

Now you can assign a midi-synthesizer note to the touchline. Select 'Single' in the Midi-Notes setup page. By default 'No note' is selected. By default the note 60 (C3) is selected. You can change that value in the combobox 'Note From'. If you want to assign a whole range of notes to the line you have to remove the 0 in the field 'Scale' (actually this is a confusing step). Now you can use the field 'Note to' to set the note or the endpoint of the line.

Testing the touchline in simulation mode

Below the video picture you find four tiny boxes. They can be moved around by dragging them with the right mouse button down. When you touch the line the assigned sound will play.

Background Picture

Before you may switch to run mode you have to take a snapshot picture of the empty stage. In 'run mode' the current video image is compared to that reference picture. Changes in the image are used to trigger the media elements. To capture this reference or background picture you have to click on the Backgound-Button in the control-window. Whenever you press this button, EyeCon switches through all the connected videocameras and stores individual backgound pictures for each channel.

Testing the touchline in Run mode (from Video)

Press [image] in the control window and try to physically reach the touchline in the camera picture.

Is the note playing? If not, check the Midi-Note page setting for the midi port. Usually a sound board has multiple midi ports. One or more for the internal sound section, one for the external midi interface. Make sure an internal synthesizer is selected.

Playing Scales

Now let's play more than one note with the touch line. And also use a different sound patch. The sound patch can be changed by opening the Patch combobox and selecting one of the General Midi Instruments. Maybe Marimba. Now go to the Edit field called 'Scale' It has a 0 in it. You can enter a list of numbers which stand for the semitone distance to the Base-Note you select in the field 'Note From'. The numbers are separated by commas. Entering 0,4,7,12 is laying out the notes of a major chord along the touch line. Try playing it. If you delete the whole 'Scale' entry, the Field 'Note To' becomes available. When you select 'C4' for example, than all chromatic notes between 'C3' (Note From) and 'C4' (Note To) are layed out along the line.

Possible Problems

- wrong or no trigger

Maybe the sensitivity of the touchline is too high (randomly playing even if not triggerd) or too low (just sometimes playing even if it should be triggered). One basic setting is the so called pixel threshold in the control window. When EyeCon compares a video picture with the stored background picture it looks at the brightness values of each individual pixel and calculates the brightness differences between the pixel and the same pixel in the background. If the difference on any of the lines pixels is higher than 'pixel threshold' than EyeCon assumes that the touchline is triggered. Depending on the light situation you have to adjust the pixel threshold. A value in the range from 60 to 100 is a good start. Under bad circumstances like low lighting and noisy camera picture you might have to use values up to 250. In some situation i was able to use values down to 30 which makes the system very sensitive.

Another cause for oversensitive behaviour of a touchline is if the line crosses very high contrasting object edges. In that case even the slightest movement of the camera or some signal jitter in the video-connection leads to high brightness changes in the pixels at the contrasting edges. Try to avoid these situations.

- no sound

If you don't hear any sound check which Midi-Port you are using. In the Midi output window you find a combobox which list all available port. There are usually ports for internal sound synthesizers and also ports for external midi equipment. Select one of the internal synthesizers.

Also check the computers volume mixer settings.

EyeCon Help, this file last changed on 14. Nov 2002


First Steps - Dynamic Fields

Home Dynamic Fields

Dynamic Fields

Dynamic fields are a very easy to use and intuitive way to create sound with movement. You select a rectangular field in the video picture and assign a sound. The volume of the sound is controlled by the amount of movement in the field. You can use multiple and overlayed fields to create interesting sound environments.

Creating Dynamic Fields:

Create a new Dynamic field by clicking on the Field button in the leftmost column of the Element-Editor. A new field entry shows up in the element list and also a graphical representation in the upper left corner of the video window. A dynamic field measures the changes in the video image between two consecutive video frames.

By dragging the editing boxes on the corners of the field you can resize and move the boxes. Grabbing the box between the editing dots moves the whole field without resizing it.

The main advantage of dynamic fields is that they not only can be used to trigger media events but also allow to modulate sound volume and pitch etc. Technically they calculate a floating point number between 0 and 1 that is proportional to the amount of movement in the field.

Audio-Files

Obviously it is possible to map midi parameters like midi volume controller to the amount of movement. In this example we want to use .WAV sample-files instead and control their volume by the dynamic of the movement.

Switch to the 'Samples' page in the Element-Editor properties page. With the New button you can load one or more sample files from the hard-drive and assign it to the selected field. As default the sample files are looped and the volume is mapped to the amount of movement.

Remark: Don't directly import sounds from CD because loading is too slow. Copy them to your local harddisk first. the assignment between the line and the sound is made by marking the sound-file in the sound-list.

Sample File options

- Use: Allows to turns sample playing on and off. This field is implemented because you might want to mute one sound temporarily without deleting the link between the line and the sound file.

- Loop makes the sound endlessly repeatable as long as you hit the touchline.

- Restart allways starts the sound from the beginning of the sample file whenever you trigger the line. If not checked the sample would play from the position you last stopped playing the sample.

- Permanent plays the whole sample file even if you trigger the touchline just shortly.

- Random selects one soundfile from a list whenever the field is triggered (at least a minimal movement is detected).

Remark: Be careful if you select neither Loop nor Restart always. After playing the sound once, the sound would not play again on subsequent trials to trigger the field.

EyeCon Help, this file last changed on 14. Nov 2002


Position Tracking

Introduction

The position tracking feature allows you to track the XY-coordinates and other properties of multiple persons. This is useful in many different stage and installation applications. In the following description we are assuming that we have the camera hanging above the tracking area. You can for example map coordinates to midi events and sound parameters. In many applications this feature has been used to send the tracked positions to external programs like Max/MSP via MIDI or OSC.

Position Tracking is not an easy task for a computer. Basically the computer doesn't have an understanding of a body. For the tracking function, Eyecon is looking at the 'static difference image'. it tries to find connected areas which are different to the stored background image, the so called 'blobs'. The EyeCon tracking feature is predictive and makes intelligent assumptions. Once the software has identified a blob, it tries to continuosly follow it by predicting the most likely position in subsequent videopictures. This makes tracking more robust. In case the tracked object is lost for a short time period due to lack of picture contrast or other circumstances, the last position is kept for a 'persistence' period.

If multiple persons have to be tracked new problems arise. In dance performances (and other situations) the tracked objects are sometimes getting really close. Technically that means two or more 'blobs' are visually merging in the video image, the computer actually just sees one object. If that happens in the central area of the tracked area EyeCon assumes that both objects are still present, as persons usually don't just dissappear. Now there's a problem! What happens if those merged blobs apart again? Well, the software doesn't know their identity, so we might loose track of who is who (which might also be fun).

Creating a position tracker

Creating a tracking object is as simple as creating a touchline or dynamic field. Just click on the tracker creation button in the element editor. As a visual representation of the tracker an eliptical object is created in the upper left corner of the video image. If you want to track more then one person, just keep creating as many trackers as you need. They are distinguished by an ID number which ranges from 1 to the number of objects you want to track.

What happens, when EyeCon is tracking people? While it searches for blobs, it is maintaining a list of tracked objects. First of all, Eyecon is trying to assign the currently found blobs to the already known persons. Their position information is then updated. If there are remaining unassigned blobs there might be a new object to track. EyeCon checks whether those blobs are big enough to form a new object and whether they are in the accepted creation area. This is usually the whole image, but in some cases we might also want to limit the space where new objects can be detected. New objects are added to the list of tracked objects. To keep a unique identifier, the list is storing the time of the creation of the object. This timestamp is a valueable tag to identify objects in the list. The last step in the list mangement will be the deletion of entries which are no longer present. If an entry in the object list is not supported by any detected blobs for more than a certain time it is most likely gone off and therefor deleted from the list.

Now it's easy to understand the meaning of the ID in the EyeCon tracker element properties. it's simply referring to the number in the tracked objects list. If you create the first tracker object it is by default related to the first entry in the list. Second tracker object related to second entry in the tracked objects list.

Mapping

How is a tracking object fitting into Eyecons simple mapping scheme? Remember, Eyecon is basically trying to generate a logical yes/no information and a continuos number, preferably in the value range 0..1? Well, yes/no decision is obvious: if a tracked object is visible that will be a yes. But what's the continues number: At the moment there are the follwing modes:

X and Y coordinate: gives the x or y position. Due to the straighforward mapping in Eyecon you have to choose one coordinate to be the main controlling parameter for that element (for example for the volume of a sound). This is bit dissapointing in a three dimensional world already reduced to the two dimensions in the video image. But you can create two trackers and set them to the same ID if you want at least to use both catersian coordinates simultaneous. [By the way: You can choose the origin of the cartesian coordinate system by moving the graphical represantations of the tracker objects to different quadrants. If you move them to the right half of the video image, the origin of the x coordinate is moved to the right side of the tracked area. ahhm, not really functional yet].

Distance: A better approach: The distance between the person and the graphical representation of the tracker object is measured. Move the graphical object and make it a hotspot. The volume of a sound could now easily be mapped to the closeness to the hotspot.

Size: Another choice: use the size of the tracked object as control parameter.

Tracking options:

The whole tracking process has a bunch of global settings which are made in the tracking window. This can be opened from the menu (Window/Tracking) or by pressing F8 or Ctrl-T.

Minimum size: the minimum size of a 'blob' to be detected as a valid object. Represented by the size of a circle in the tracking window.

Image filtering: The amount of image filtering that is done before analysing blobs. This filter is trying to smoothen the edges of blobs. see the filter view to check the effect.

Persistence: A parameter determining how long an object is kept in the list of tracked objects when it is temporarily not visible.

Range: A parameter which determines how far from the center of every tracked object EyeCon is looking when it is analysing the next video image.

Creation area: The area where new objects can be detected. Represented by a red rectangle which by default is almost the size of the whole camera image.

Fallback mode: Decides what happens if objects with low IDs dissapear before the ones with a higher ID. The IDs can be either kept or be renumbered (this is called fallback here).

EyeCon Help, this file last changed on 01. Jan 2004


Interface Description

In this chapter you will learn about EyeCons features and the function of the different windows and the meaning of the controls.

The EyeCon software can be divided in two completely different parts: motion detection and multimedia action(s). The concept of EyeCons is that you create motion sensing elements which are in many cases graphically represented as lines and rectangular fields. The Element Editor is the main mapping panel where you assign how movement controls media. Media output includes MIDI functionality, sound sample player control, ethernet UDP and OSC communication, picture and video control.

EyeCon has two operating modes, Simulation and RUN mode. In RUN mode that actual video analysis is performed and media is played. In simulation mode you can use tiny boxes and move them with the mouse to emulate moving objects in the video image. The Control window allows you to control the main parameters of the video processing like camera selection, threshold for movement detection, basic timing etc.

The Video window allows you to watch the live video and see how the motion sensing elements react.

To allow the creation of complex interactive setups, EyeCon offers a scene managment function. Each sensing element is functional in a selectable level. The Sequencer Window allows you to arrange these interactive setup levels.

Element Editor

Control window

video window

additional live views

Sequencer Window

EyeCon Help, this file last changed on 26. Nov 2002


Element Editor

The element window is the main window of the software. There, you can choose your detection objects, define their behaviour and their properties, that means the actions that they will trigger and control. Whenever you create an element it will show up in the Elementlist and the graphical representation will be displayed on top of the live video (in case of lines and fields).

Figure 2 : The left part of the elements window

Layout

The editor window is structured into different columns:

  • Create: A list of creation buttons for the different element-types; Lines, Fields, Communication, etc.
  • List: A list of all elements in that eyecon setup. Parameters for the highlighted entry are shown in the other fields of the editor window.
  • Options: Basic parameters, identifier, element group and scene level relevant to all object types
  • Tweak: Element type specific parameters.
  • Properties: The mapped output settings.

Element types

An element can be a:

  • Line (a touchline),
  • Field (a box),
  • Tracker object (a mobile object following motion),
  • Static object,
  • Sensor object
  • Midi object.

Each element has its own set of parameters that will latter be presented in detail. Some of the elements parameters are just valid for a specific element type while most of them are valid for all element types.

In principal each element is calculating a logical on/off result and one numerical value. In case of touchline the logical state is the presence or absence of an object along the line, while the numerical value is representing the point where the object cuts the line. For dynamic fields the logical state is calculated from the presence or absence of any movement while the numerical value represents the amount of movement in the field.

Creating Elements

Element List

Element Options

Type specific options

EyeCon Help, this file last changed on 26. Nov 2002


Creating Elements

Home Element Editor - Creating Elements

Element Editor

Creating new Elements

As already stated before, EyeCon software can be divided in two completely different parts: motion detection and multimedia action(s). The Element Editor is the main mapping panel where you assign how movement controls media.

First, you must create a new element (line or field or …) from the Create list. This element appears in 2 places: in the Element list and in the Display window. At first, the new element appears as a small object in the left top corner. You can graphically change this element size and position. Now, if you want the new element to trigger an action, you must define its options and behaviour in the Element window.

Element Types

  • Line When the picture of your gesture crosses or covers the line or touchline, actions are trigged.
  • Field When the picture of your gesture enters or moves in this area, actions are trigged.
  • Tracker Object position tracking feature.
  • Static This element works in relation with levels and groups. When the right levels and groups are selected (for example by a sequencer action), then actions are trigged. This element is not at all related to motion tracking, but it is used for configuration purposes.
  • Sensor This element allows to program actions trigged by external switches and sensors plugged into the COM port.
  • Midi This is a MIDI input.

Other functions

  • Copy This button allows to duplicate the currently selected element.
  • Import This button allows to load element parameters from an external file into the currently selected element.
  • Export This button allows to save the currently selected element parameters into an external file.
  • Delete This button allows to delete the currently selected element.

EyeCon Help, this file last changed on 26. Nov 2002


Elementlist

Home Element Editor - Elementlist

Element Editor - Elementlist

The second column in the Element-Editor hold a list with all the elements. If you select one element it is highlighted in the graphical view. The element specific settings are loaded into the parameter fields of the editor.

Clicking with the right mouse button on the element list opens a context menu.

  • Show all Show all elements in the project (default). If you turn this off, only the elements in the active levels are displayed.
  • Multiple Selection Allows to select more than one entry of the list. If multiple elements are selected, parameter changes affect all the se selected elements. (Not fully supported yet)


Midi Output

Home Element Editor - Midi Options

Previous page - Next page

Element Editor - Midi Options

Short overview over MIDI.

Midi was defined in the early 80s to make connections between keyboards and synthesizers. Til then every sound synthesizer needed its own keyboard. With MIDI it became possible to connect different synthesizers and play them from one physical keyboard.

Nowadays MIDI is used as an 'easy to understand' protocol for communication between interactive systems. It can not only be used to connect external midi synthesizers through a physical midi port. Each computer has a built-in midi synthesizer you can play from EyeCon. You can also install virtual MIDI drivers which allow to communicate between different programs on one computer without using any cables. Popular 'virtual' midi links are 'Hubi's loopback' or 'Midi-Yoke'. Both can be found on the internet. You can use them to play a software synthesizer or other software with midi input from EyeCon. There is also a midi-driver that lets you send midi information through a network or through the intenet ('Midi via Net'), although you probably will prefer to use the OSC protocol if supported by your application.

Basic MIDI

The midi protocol is sending abstracted information for MIDI instruments. That means, it is not sending recorded music through its wire. Instead it is only communicated when and which keys are pressed on a MIDI instument. The midi protocol is based on the concept of playing a musical keyboard. Eyerything you can control on a keyboard has it's representation in the midi protocol. The basic information that is sent is pitch and velocity of the key you press. There are also commands to switch to a certain sound patch, change the volume, use the pitch wheel and modulation controls you find on most keyboards. Veloctiy determines how hard you hit a key, depending on the sound it will affect the sound.

Imagine you want to play different instruments simultaneously over one MIDI connection. The midi protocol therefor uses so called channels to logically structure the information flow. The use of channels makes it possible to play different voices (sound patches) simultaneously. You might setup two voices on channel 1 and 2 with for example piano and flute. In the midi logic that means you're sending patch change commands on channel 1 and 2 to set them to the desired sound setting. Now you can send out note commands. when they are on channel 1 you will hear a piano sound, notes on channel 2 play the flute. Pitch bend or volume controllers are also just valid for the chanel they are sent on.

MIDI messages

Midi Notes: message to turn a note on or off. A midi note message carries information about the used channel, the key (pitch) and the velocity (how hard you hit the key) of the note. A velocity of 0 is used as 'note off' command. Key and velocity values are represented by numbers in the range 0..127.

Midi Control: message to change sound settings. A midi control message sends information about the used channel, a controller number to address effects (like volume, modulation, panning etc) and the controller number. There are 127 different controller values possible, some have predefined meaning (7 = volume), most of them don't. The controller value is represented by a number between 0..127.

Patch Change: select a sound patch for a specific channel. The General MIDI standard defines how Patch numbers are supposed to sound. 1 should be a Piano, 74 a flute etc.

Pitch Wheel: send information about the position the pitch wheel, which is used to quickly change pitch while you play a keyboard. It is useful in interactive applications because it has higher resolution than control messages.

EyeCon Help, this file last changed on 19. Jan 2006


Midi Output

Home Element Editor - Midi Output (1)

Previous page - Next page

Element Editor - Midi Output

Setting up Midi Output

Eyecon implements a complet set of midi commands. The midi section is distributed on two pages called 'Midi-Note' and 'Midi-Control'.

The port and Channel setting on the first page are also valid for the Midi-Control page.

Quick Start

  • Create a touchline, select the Midi note tab (default anyway)
  • Switch from No Note to Single Note
  • Select a Midi Port which can actually play through your speakers (usually you will find the Microsoft Software Synth)
  • On the video window, grab one of the little colored boxes (simulators) and move them through the touchline
  • You will (hopefully) hear a single piano note playing whenevery you hit the touchline with the simulator
  • Select different instruments, or notes from the selector boxes

But now a detailed explanation of the individual fields

Note Type (left column)

  • No Note No midi notes played
  • Single Plays the midi-notes when the Eyecon element (line or field) is hit, and turns the note off again, when play state is changed to off. In case of a touchline the note is playing as long as you touch the line, after leaving the line the note stops.
  • Repetitive As long as the element is triggered, the note is repeatedly re-played with the basic program timebase (set in the control panel). The speed is also affected by the debounce parameter, higher values of debounce slow the the playing rate down.
  • Permanent The midinote is triggered when the element is switched to play-state (touchline hit). The note is not turned off when the element is no longer triggered. Technically spoken, the midi note off command is not sent. This feature is used either to trigger continues sounds or to send trigger information to other programs that are not sound related.
  • Off If this checkbox is selected, eyecon is sending a 'all note off' command for the element specific midi channel before playing a note. This is often used in combination with Permanent notes. Setup two different lines with the permanent and the off option selected and set them to play different notes with for example an organ sound. The two lines will act as a toggle between the two sounds. One line is cutting out the sound of the other line before playing its own sound.
  • Velocity A midi note message always sends out a note pitch and a note velocity together. In EyeCon the normal procedure is to send out a note when you trigger an element. The note velocity is detemined by the elements current action value (i.e triger position on a touchline) mapped to a range of velocity values (Vel From..Vel To). If the action value is changed but the play state is not, there is no new command sent out

EyeCon Help, this file last changed on 19. Jan 2006


Midi Output

Home Element Editor - Midi Output (2)

Previous page - Next page

Element Editor - Midi Output

More details about setting up MIDI parametes

Port

The Port list shows all the available Midi connections on your computer. There is usually at least one, the Microsoft software synthesizer. Depending on external interfaces or installed virtual ports, there might me several more. The port selections determins where all MIDI messages for that Eyecon line or field are sent to.

Patch

Determines, which patch change message is sent before sending the actual MIDI note. The patch change selects a sound. Sounds will sound a little different on each synthesizer.

Bank

This selector allows to select alternative patch bank sets. With the patch chang you can select from up to 127 different instruments. Modern synthesizers often support more than 127 instruments. So somebody came up with the idea to have them on different 'pages' or 'banks'. To switch between banks actually a control change message is used. The bank change message is not supported by all MIDI equipment.

Channel

message to change general purpose settings. There are 127 different controller values possible, some have predefined meaning (7 = volume), most of them don't. The controller value is a number between 0..127.

Note From, Note To

Defines the Midi Note which is sent when you hit a touchline or field. If Note to is not selected, only one note is associated with the line (or field). In case you select a range of notes which are spread over the line. That means if you hit the touchline at the beginning of the line (end with a slightly bigger dot) the lowest note is played, the other end plays the other note. The notes are always spread chromatically, if you don't want that, you have to use the 'Scale' field.

Velocity

Note commands are always sent with a velocity value which indicates, how hard you hit the key, if midi is generated by a keyboard. The 'from' and 'to' settings describe how velocity values are spread over the range of the line (of field). The field 'Def' determines, what happens if you leave a line (or field). Then the note is sent out again with the Default (you could also call it fallback) value. A velocity of 0 is used as a 'note off' command.

Scales, Arpeggios and Chords

The last section on the Midi note are 3 fields which allow you to define variations of the note pitch which is output. You can type note offsets in those fields. Imagine you want to distribute the four notes of a major-7 chord along a touchline. With Note From-Note To you always get chromatic steps. So it's not the way you want. You select the base note in the Note-From field instead. Then you type the list of note offsets in the 'Scale' Field. For a major chord that would be 0 for the base note, 4, 7 and 11 for the chromatic offsets. You would type 0,4,7,11 into the Scale Field.

If you would move those numbers into the chord field, all the notes of the chord are played simulaneously.

In case you move the numbers into the arpeggio field, they are played one after the next, whenever you hit the touchline (or field) again. With the 'repetitive' mode you get a arpeggiated sequence of the selected notes.

EyeCon Help, this file last changed on 19. Jan 2006


Midi Output

Home Element Editor - Midi Output (3)

Previous page - Next page

Element Editor - Midi Output

Setting up MIDI control messages

Control

If you want to send out a MIDI control message, switch the output settings to the Midi-Control tab after checking that you selected the proper Midiport and channel on the 'Midi-Note' page. Enable the sending of control change messages by checking the check-box. Select the controller number (7 for example is the channel volume). The From and To values define how the midi control values are mapped to the parameters of the Eyecon element. The 'Def' value defines a 'Default' or 'Fallback' value, which is sent out, when the Eyecon element is not triggered anymore. A value of -1 indicates that there is no message sent at all when you leave the line or field.

Example: You have a touchline and one dynamic field in your setup. The touchline is set up for playing a midi chord on a sustaining instrument like a flute (74). The dynamic field is used to control the volume of the sound, mapped to the amount of movement. As you always want to hear a minimum volume, even though there is only very little movement, you select a value of 30 in the From box. You also don't want to reach the full volume of 127 ever. Set the 'to' value to the maximum volume value.

Pitch Change

Allows to send pitch wheel change message, which is used to quickly change pitch while you play a keyboard. It is useful in interactive applications because it has higher resolution than control messages. Although the indicated values imply a range of 0..127, those values are internally multiplied and interpolated to give the full range of 0..16383 which is supported by pitchwheel changes (14bit).

NRPN

Non registered parameter numbers (NRPN) are a rarely used set of additional configuration values. Sound Blaster cards used to make use of them for special settings like lowpass filter control etc.

After Touch

The After Touch message is used to send changes in how hard you mechanically press down a key on a keyboard, once you are playing it. Eyecon allows to send that command.

Sequencer control

Sequencer control are messages which are used to start, stop and pause sequencer programs or drum machines. You can select which of those messages are sent when you hit a line or field (On) or release it (Off)

EyeCon Help, this file last changed on 19. Jan 2006


Media Control - Visuals

Home First Steps - How to start

Controling Visual output

In this chapter you will learn how to use EyeCon for outputting still images or video. The implementation is restricted to pretty basic functions like switching picture files and triggering video files. Further releases will extend the functionality.

The related functions are distributed over two pages in the element editor. this is primarily due to space restrictions. One tab is called 'visual', the other one 'video'

Step 1: making the image canvas visible

First of all, you'll need a window to show the visuals. The so called canvas is opened by pressing 'F7' or via the menu entry 'window/graphic canvas/show-hide'

A black window immediately opens. You probably want to move that to a different place on your screen. To do so, either check the resize menu in the main menu or use the context menu by right-clicking on the canvas-window. You will get the typical windows window frames and can resize or drag the window now.

A typical situation will be that you want the canvas on a second VGA output of your laptop or computer. Use the 'Second Monitor' shortcut to automatically resize the canvas to the full size of the second monitor.

There are different visuals you can control. For this version the possibilities are pretty basic. You can trigger images, live video, video or Shockwave files. There is no control over transparency, meaning there are no soft crossfades or ways of layering yet. There is always only one visiual element displayed, if you trigger multiple visuals simultaneously you will still only see one (typical the last triggered one on the list of elements).

1. Displaying still images

2. Displaying live video

3. Displaying video files

4. Shockwave flash

EyeCon Help, this file last changed on 14. Feb 2005


Media Control - Still Images

Home Media Control - Images

Media Control - Images

EyeCon can trigger WIndows Bitmap (.BMP) and JPEG Files (.jpg). To setup the appropriate mapping, select the 'Visual' tab in the element editor.

The simplest situation is if you have one Eyecon element, for example a touchline, triggering one image. Just type the full pathname in the filename field or press the 'Browse' button to find your image file.

As soon as the Eyecon element is triggered, the image will appear on the visual canvas.

Options:

The 'Stretch' checkbox enables resizing your image to allways fit the size of your canvas

The 'Def' selector determines what happens, if the touchline is no longer triggered, you can select between 'no change', 'black' or an indexed filename, which will be explained below.

Indexed file names

You might have the idea to map different positions along your touchline with different images. You can use indexed filenames to achieve that. Lets assume your image files are called image2.jpg, image3.jpg through image6.jpg. Now you need to set up a placeholder in the filename field which represents the varying indices along your touchlines. In our example you would use '%d' as a placeholder. You also have to select the start and end index in the 'From' and 'To' boxes. Eyecon will calculate the displayed image index according to the triggered position along your touchline. Remember that you can use all the possible EyeCon element types (fields, trackers) in the same way.

With indexed file names you can also select a certain image index which would be triggered after the Eyecon element is no longer triggered. type the default index into the 'Def' field.

EyeCon elements

Option: Increment

A variation of the use of indexed file names is the increment feature. It simply means that the used index is not determined by the position along your touchline. Whenever you trigger the touchline an internal index counter would be incremented. This feature could be used to make slideshows being triggered by touchlines.

Other formatting options

Image files are often numbered using indeces with trailing zeros. To achieve this you have to use a different placeholder. Replace the '%d' with '%4.4d' if your numbers are formatted being allways 4 digits length (example DSC0002.jpg). Change the format placeholder according to your needs.

EyeCon Help, this file last changed on 14. Feb 2005


Media Control - Live Video

EyeCon can also display your current live image on the canvas. To setup the appropriate mapping, select the 'Visual' tab in the element editor and use the checkboxes in the 'Live-canvas' section

.

A typical use of the 'Live image' would be to display or project your live image with the eyecon elements. You could setup a static field with 'Live-canvas turned on'.

Options:

The 'Live canvas' checkbox enables the display of the current live image.

The 'Stretch' checkbox enables resizing your image to allways fit the size of your canvas

'Show objects' will also display the Eyecon elements.

'Negative' inverts the image

Remark:

Usually the live features work on the current camera image. In case you selected one of the special 'Views' like Background etc, the topmost of those windows will be used instead as image source.

EyeCon Help, this file last changed on 14. Feb 2005


Media Control - Video

EyeCon can trigger and control video files in AVI format. To setup the appropriate mapping, select the 'Video' tab in the element editor.

Basically everything you have to do is to browse for your video file or type it into the filename field. Different options determine, how the video is played. After loading the video file you will see a filelength info which is showing the playing time of the video clip in Milliseconds. You can setup a certain range in the clip you want to trigger with an EyeCon element. Input is in seconds. Play speed can be varied.

Options:

The 'Loop' checkbox enables looping of the selected loop. Use 'from' and 'to' fields in the 'Position' setup to mark the loop points.

'Permanent' indicates your video file as being played to the end of the selection even if the EyeCon element is only triggered shortly.

'Restart only' will make your clip jump to the beginning everytime you trigger it. Otherwise it's played from were it was last stopped.

'Mediapos' is changing the mapping to a video scrubbing. You can control the playing-position of the video with the position along a touchline or a position field. Reverse changes the begin-end mapping.

Important remark on video codecs:

AVI files can be based on different codecs. Not all codecs might be installed on your system. Unfortunately on windows there are different ways video codecs are used. As Eyecon video playing is not yet based on the Directshow interface, it might happen that you can play videos with other media players but not with EyeCon. Check back with us if you need help finding a codec (for example for playing DV-AVI files).

Not all codecs are appropriate for use in interactive environments. Some modern streaming video formats are high quality at low bitrates, but due to the way they work not practical. MPEG-2, MPEG-4 are codecs based on encoding changes in video images. In case you want to trigger a certain image in your videostream, the computer might have to decode a whole series of video images before it reaches the selected video frame.

For all interactive video triggering applications you would prefer formats based on individual frame compression and high performant decompression. (CINEPAK, Indeo).

The simplest situation is if you have one Eyecon element, for example a touchline, triggering one image. Just type the full pathname in the filename field or press the 'Browse' button to find your image file.

As soon as the Eyecon element is triggered, the image will appear on the visual canvas.

EyeCon Help, this file last changed on 14. Feb 2005


UDP-OSC

Home

UDP - OSC An Introduction

The UDP protocol

The UDP- and OSC protocols allow the use of normal ethernet network connection to send information between computers and other interactive media systems. Because of its speed and availability in almost every new computer it's a good replacement for the MIDI interface in many cases.

The UDP protocol is based on IP network connections. That means, you address other network clients with an IP address. Unlike the TCP protocol, UDP does not include a procedure for ensuring that individual messages are not lost in transit. Under normal network conditions, it is not uncommon for individual messages to be lost or damaged. By omitting the error checking that is a part of the TCP protocol, UDP can be faster in some circumstances.

UDP should only be used for sending frequent messages that are meant to update information, such as the position information from the motion tracker or data from dynamic fields etc. Because this kind of information is usually sent frequently or in every processing cycle, the interactive control in a host system is not adversely affected if an occasional message is lost in transit, since another update message will be sent soon. The UDP protocol should not be used for messages that must always succeed, such as server commands or other critical messages.

You have to come up with your own definitions of how the data is structured. In EyeCon you can send UDP strings that represent the on/off state of an element and the numeric information an element is representing.

The OSC protocol

OSC is the abbreviation of OpenSound Control. It is a protocol for communication among computers, sound synthesizers, and other multimedia devices that is optimized for modern networking technology like Ethernet, USB, Firewire etc. That means it is not necessarely transported via UDP network, but in most practical uses it is used as another layer on top of UDP connections to standardize the way in which information is transferred between media systems. UDP is protocol to transfer blocks of data via ethernet connections. How the data is organized in a block of data is not specified. This makes another protocol-layer like OSC necessary.

OSC has been developed by the 'Center for New Music and Audio Technologies' of the university of Berkeley. It is available for many media systems that are used nowadays like Max/MSP, Supercollider, CSound, Pd, Reaktor etc.

For further information look at http://cnmat.cnmat.berkeley.edu/OSC/

OSC and Max/MSP

You can download a MAX/MSP patch that demonstrates the use of the OSC protocol from http://cnmat.cnmat.berkeley.edu/OSC/Max

OSC and FLOSC

Flosc is a java server that allows the control of Macromedia Flash Animations via OSC. Basically it is a software that can receive OSC packets from EyeCon and transform the information in XML based files that can be accessed from inside Macromedia Flash. Look at http://www.benchun.net/flosc/ for details

EyeCon Help, this file last changed on 16. Oct 2002


EyeCon and OSC

Home

Previous page - Next page

EyeCon and the OSC protocol

A typical Eyecon setup now often involves controlling external media system like Max/MSP or pd, Macromedia Director and Flash, Reaktor, Isadora and similar. Communication between systems has become an essential step in creating interactive work. There are not many universal ways for doing this besides MIDI and DMX protocols, but a very universal and appropriate is using the OpenSoundControl (OSC) protocol. It allows you to transfer information between Eyecon and the other involved systems either on the same computer or on different ones. External communication is usually done via the computers built in network ports. It is even easy to communicate between different operating systems, be it on Windows, Mac-OS or Linux.

How to prepare for an OSC connection:

The main preparation is to get the involved computers connected. Let's imagine you want to send Eyecon motion sensing data to a Max/MSP patch, running on a Macintosh. If you already on an existing network with both machines, you probably don't need to change your setup. All you need to find out, which IP address the receiving computer has. In our example you would check for the IP address on the Mac-OS network control panel. IP addresses are in the form of a set of 4 numbers, separated with dots. Often they will start with the numbers 192.168.

If you are usually using your computers in a network, but now just having them setup to communicate between themselves, you will probably have to change the network settings on both of them. All you actually need is a network cable between the two computers. A so called 'cross-link' cable is good to have for that setup, but with many modern computers a normal network cable will also work.

What you need to do for the direct connection between two computers is assigning them fixed ip addresses. This needs to be done on both computers, on the Windows computer check and set the properties of the TCP/IP settings for your LAN-connection. Select manual addresses and type in 192.168.1.1 On the Macintosh side, select manual settings on the network control panel and assign it the address 192.168.1.2 These numbers are a usable example, although the actual rule is only that the first 3 numbers be the same, while the last number has to be different. You will also be asked to type in a subnet mask, which will always be 255.255.255.0

How to setup OSC communication in Eyecon:

After assigning and checking the computers IP addresses you have to tell Eyecon where to send the information to. Open the address setup dialog from the menu entry Window-UDP-Hosts, by pressing 'Ctrl-F9' or by clicking on the button above the UDP-host entry field on the 'OSC' page.

You can setup a list of computers and assign symbolic names for each of them, distinguished by the IP address and a port number. This information must reflect the settings in the connected computers and systems. The port is a number between 1 and 65535 which is used as an extension to the addressing with the IP-address. Imagine you would want to send information to two different programs on one computer. You would set them up for data exchange on different port numbers. The field 'Options' is usually preset with the option 'Bundle' which means, that multiple numeric informations are sent in one UDP datagram. You usually dont have to change this option.

The field 'Own IP Addresses' is for information only.

Connecting via OSC - step by step

To configure information exchange between EyeCon and a connected computer you have to perform the following steps:

1. Create an EyeCon element, for example a touchline

2. Switch to the OSC page in the properties section of the element.

3. Enable OSC 'Use' Checkbox

4. Select a Host from the dropdown combobox 'Host'. It holds all entries that have been defined in the 'UDP-Hosts' dialog box.

5. Enter an OSC parameter specification in the 'Value' field. The first part of the line is always an address string which addresses the specific function in the receiver.

6. Check the 'Each cycle' checkbox, if you want the information be sent in every processing cycle, even if the information is not changed.

The 'Compose' button opens a dialog box which assists you in setting up the message. The available Parameters depend on the type of the element. For touchlines you have Position (Pos) and Width (Wid) as possible parameters. The value range for an element parameter is usually 0.0 to 1.0 in floating point notation.

EyeCon Help, this file last changed on 28. Dec 2005


EyeCon, OSC and Director

Home

Previous page - Next page

EyeCon and the OSC protocol - Control a Macromedia Director application

Using the OSC-Active-X Control to send data from Eyecon to Macromedia Director (on Windows only)

Recently i wrote an Active-X control which allows you to receive OSC information in different programming environments like for example Macromedia Director. Active-X controls are acting as plugins into Windows, in this case enabling you to receive OSC data. Many programs allow you to access and use Active-X controls, this tutorial is about setting up communication between Eyecon and Director.

How to prepare for the use of the OSC-Active-X control

Download the OSC-ActiveX.zip to your computer and unpack it into a directory of your choice. The actual control is called OSCXControl.ocx You have to register that Control element in windows. That is easily done by double-clicking and executing the REG.BAT file. From now on, the OSC active-x control will be available on your Windows system.

Inserting the OSC-Active-X control in your Director application

On the 'Insert' menu of Macromedia Director you will find the 'Control' sub-menu which contains the selection for Active-X. (Windows only). It opens a dialog window which lists all the installed Active-X controls. You might be surprised, how many there are. You are looking for the 'OSCX Element'. Once you select it, it becomes a member of your control palette and can be put on the timeline as any other control.

Properties, Methods and Events:

An Active-X control is represented by it's properties, methods and events. Properties are settings you can check and change. In the case of the OSC Active-X, the most relevant property is the port number. It has to be the same setting you are using in Eyecon. Eyecon and the OSC Active-X use port 7000 as a default. Methods are function calls made to the control. In our case the most important methods are 'Open' and 'Close', which will start and stop the OSC function. A good place to call OPen and Close are the startmovie and endmovie scripts of your Director program. Have a look at the sample program i provide.

An event is an externally triggered moment, where you want your program to respond to. In case of the OSC control it would be the moment a message is received. You have to define an event handler in your program, which defines, how the message is treated. The name of the receiver event is OnOSCInput

Receiving a message

Once your OSCInput event is triggered, the OSC message is stored in an internal buffer until you release it. That allows you to access all the elements of the message as long as you want. Remember, each OSC-message has at least one address field. It might also have one or more numeric or string parameters. They are delivered as 'Variant' data types which gives you flexibility in the interpretation of the data.

In case you only want to transmit the on/off state of a touchline, you probably just use the address field to distinguish the lines and states. Create a touchline, select your Host/Ip settings, enable the OSC transmission and type some identification string in the On and Off fields. In this example the addressfield of the OSC-message would be LineOn when you just hit a touchline, LineOff, once you release the touchline. The 'value' field is left empty, which means, there are no numeric parameters appended. In this example the message is sent to 'localhost', which is the same computer which you are running Eyecon on.

How does it look in Director

The OSC control is represented by a little box which is showing the port it's set up for. There is also a little red indicator line. Whenever it is moving, there was a OSC message received. It helps you debug your application. After you tested your application you will probably want to make the OSC control invisible. Just set it's property 'visible' to false.

In the OSC-Director sample which can be downloaded here, there is a bunch of touchlines set up to allow you playing on a virtual piano. Director is used to visualize the keys you have hit, whereas Eyecon plays the sound.

EyeCon Help, this file last changed on 28. Dec 2005


OSC- ActiveX

Home

Previous page - Next page

The OSC-ActiveX reference

This Active-X control allows you to receive OSC data in applications which support ActiveX controls on Microsoft Windows.

Properties, Methods and Events:

An Active-X control is represented by it's properties, methods and events. Properties are settings you can check and change. In the case of the OSC Active-X, the most relevant property is the port number. It has to be the same setting you are using in Eyecon. Eyecon and the OSC Active-X use port 7000 as a default. Methods are function calls made to the control. In our case the most important methods are 'Open' and 'Close', which will start and stop the OSC function. A good place to call OPen and Close are the startmovie and endmovie scripts of your Director program. Have a look at the sample program i provide.

An event is an externally triggered moment, where you want your program to respond to. In case of the OSC control it would be the moment a message is received. You have to define an event handler in your program, which defines, how the message is treated. The name of the receiver event is OnOSCInput

OSC basics

Each OSC message consists at least out of one address string, which tells the application which subfunction is addressed. In the simplest case the address string is the only part of the whole message. Often you will have other parameters following though. The OSC active-X currently supports integer, floating point and string parameters.

How to use

1. Set the Listeningport to the desired port setting

2. OSCX.Open starts the OSC receiver

3. Define an even hander for the OnOscInput event

//a single OnOscInput event could deliver more than one message, it is recommended to process them in a loop.

for i := 1 to OSCX.MsgCnt do

begin

address := OSCX.Address;

//depending on Addressstring you implement the functionality

if address = ....

//access the messageparameters

if OSCX.MsgParamCnt > 0 then param1 = OSCX.Data1

//release current message

OSCX.Next

end

4. When closing your application, don't forget to call the OSCX.Close method

List of properties

A bunch of properties are defined to determine how the control is visually represented.

The only functionally important setting is the Listeningport.

Listeningport | The port number the control is listening to. Has to match with settings in sending program
Visible | turns visualization of the control on or off. Default On
BevelInner BevelOuter COlor Font Cursor | design, how the control looks in your application
Indicator | size of the progress indicator bar. This bar moves when there is OSC input activity.
Version | read only; returns an identifying string for the control

List of methods

Open | Iinitializes the OSC controls function.
Close | Ends the OSC controls function.
MsgCount | Indicates, how many OSC messages are ready and waiting for being processed in the controls buffer.
Address | Returns the address string for the current OSC message
MsgParamCnt | Returns how many parameters the current OSC message is carrying
Data1 | Returns the first OSC parameter of the current message
Data2, Data3, Data4 | Returns more OSC parameters.
Data(n:integer) | returns the OSCparameter with index n, more universal form of the preceding methods , n runs from 0 to MsgParamCnt-1
Next | Release the current message, make the next one current. Mandatory use after accessing elements of the message
TimeStamp | Bundled OSC messages carry a timestamp which is supposed to allow better synchronization.
Typetag | Returns the typetag of the current OSC message, only necessary for special applications.
BundleID | A running number created by the control. If multiple messages are bundled, they all share the same BundleID. Only needed for special applications. 0 if message is not bundled
AboutBox | Opens an identifying Copyright dialog

List of events

OnOscInput called whenever a new OSC message is received

OSC Active Help, this file last changed on 03. Jan 2006


Live Video Window

The display window is the viewing area of Eyecon. You can display live video picture an the graphical elements; You change the element position by moving the graphical elements with the mouse. By moving tiny colored blocks with the mouse you can simulate changes in the video picture.

When performing, you can see such a picture.When an element is activated, its colour changes and a small dot appears to display the "action value". On the picture, the hands triggers a touchline element (called intro). This green touchline becomes purple. A small green dot shows the average position of the hands. This dot position may be in relation with the action parameters.

Editing graphical elements

The live video window is where you position the graphical elements. Drag the endpoints of the lines and boxes to move the corners. If you select a graphical element between the endpoints you can move the element without changing size and orientation.

Monitoring element action

As described the element also responds to visual changes when in RUN mode. For lines the following changes are graphically represented:

If the line is touched at all than the color is changed to the complementary color of the line. The actual pixels which are triggered reamin at the original color. A little box appears to represent the action value of the line. This value is used to determine output settings like volume, pitch etc. (if line not only used as on/off trigger). Depending on the line settings this value is calculated from the average of all triggered pixels or the first or last triggerd pixel along the line.

For 'dynamic fields' there appears a short white line on the right side of the box which represent the action value for the box.

For 'feature fields' (right, left etc.) a full-length line is drawn to represent the action value.

Simulation mode

The four colour squares at the left bottom of the window are test spots. Use the right mouse button to pick a square and move it through the touchlines or the fields. You will be able to test the action of your elements.

Note: This function only works in Stop mode!

Note: The colour of the squares don't have a meaning for lines or fields! Use any colour for your own convenience. The color was used for color specific position tracking (no longer supported)

Button description

The Switches in the toolbar allow you to select the following functions:

- View

Opens a new window where you can proceed to several actions on the display, the background(s) and the analysis processes. Multiple Views can be opened to monitor different aspects of the analysis process.

The View window allows you to save Bitmap images to the harddisk.

- Edit / Lock

Edit is a toggle button linked with Lock (see below)

Enables/disables moving the graphical elements on screen. Lock mode prevents unwanted incidental element movement. When you are performing on stage, it is a good idea to lock the elements in order to avoid any unwanted trouble. Default value: Edit.

- Undo

Restores position of graphical element to the state before last change. (just one move for now)

- Show

Shows / hides the graphical elements on the screen (lines, boxes…). In hide mode, you only see the camera picture. Default value: On.

- Name

Shows / hides the name (specific text identifier) of all graphical elements.

- Track

Show outlines of tracked objects (Position tracking)

- Solo

Enable output only for currently selected element (not yet implemented)

Technical information

The bottom of the window displays some technical information: the RGB (Red Green Blue) values, the Y (chrominance) value under the mouse pointer. The Diff values are the differences between the previous values and the background values. If you work with a Black and White camera, the three RGB values should be the same (in fact, it is a grey level) .
The Dif value is the difference between the background memory and what the camera sees. This difference is measures at the mouse position.

EyeCon Help, this file last changed on 26. Nov 2002


Aditional live views

Home Interface Description

Additional View Window

A menu offers the following functions :

- Current

- Background1 2 3 display background

- Dyn Background

- Show static dif

- Show dynamic dif

- Show Filtered

- Mask static dif

- Mask dynamic dif

The View window allows you to save Bitmap images to the harddisk. They are stored in the current directory with an automtically produced name.


Control Panel

Home Interface Description - Control Panel

Control Window

The Control Window defines basic parameters and holds the important buttons for capturing the background image, start and stop the 'run'-mode.

Timebase

The underlaying timing, a new picture is fetched from the framegrabber after this time (Milliseconds). The PAL videostandard offers 25 full pictures per second, that means 40ms per picture. Therefor 40ms is the default value for the timebase.

Half of the lines in the picture are updated after 20ms, for timecritical setup like playing percussive instruments you might want to change this setting to 20ms.

Pixel-Threshold

This value determin the minimum brightness or color difference that is necessary to detect a change in the picture. This value depends on the quality of the camera, lighting situation etc. Start with a value of about 80. If using colorcameras start with a value of 160.

Camera

The currently selected camera.

Background

Takes a picture from all cameras and saves them as reference for comparison with current pictures. Only necessary for objects with static reference like touchlines. Dynamic fields don't use the background picture.

Auto-Background

Turns on the automatic background compensation. This is usually necessary if lighting conditions are not constant. The parameters for this function can be changed under Options - System Settings - Global - Autobackground

Run - Stop

Switches between Stop mode that allows simulation with the colored boxes and Rund Mode wher EyeCon actually responds to the live video picture.

Panic

Kills all played sounds and stuck midi notes.

EyeCon Help, this file last changed on 26. Nov 2002


Monitor Window

The monitor helps you to trace the different generated outputs. It displays selected Midi events, shows the audio files you are playing and helps testing your setup.

The switch buttons at the top of the window allow to select which outputs you want to monitor.

EyeCon Help, this file last changed on 26. Nov 2002


Sequencer

Home

Using the Sequencer

Overview

The sequencer is a feature that allows you to program your EyeCon Elements into a series of scenes so that not all Elements need to be visible at the same time (when an Element is not visible, its not active). That allows you to structure your interactive performance just like a theater production consisting of different scenes, light cues etc.

Every EyeCon Element has level and a color. These properties can be prescribed in Element Window>Options>Level and Group respectively and are displayed in the Elements List. These are the properties which allow different scenes to show different Elements.

The sequencer window

An element is active and visible if its Level property is equal to the globally active scene for its color which is set in the sequencer window. You can think of the color as a track or layer in your whole setup logic. The color setting is also setting the color of the element in the video window. The level setting acts as a selector.

In the bginning you will just have to use one color. When you create a new element its color setting is always green.

Let's make a test: Make a new Touchline. You will see that it has the property Green-1. Since the global setting for green elements is '1' the line is visible in the Video Window. Now change the level setting for your element to '2'. The line will disappear from the Video Window. Now do a scene change in the sequencer settings: Set the active level for green elements also to '2'. The line will reappear and again be active.

If you just use one color (or track) you will always switch all the active and visible elements. Sometimes you would want to keep some of the elements although you switch others. Maybe you want a dynamic field staying for a longer time and a few touchlines changing between scenes. For that task you need the color setting. There are five choices: Green, Yellow, Blue Red and White. In our example we leave the dynamic field on green and change the setting of all our touchlines to yellow.

Now you can for example switch the group property of the global elements in our example from Green to Yellow. Switching the current scene-level for 'Green' elements doesn't affect the visibilty of the 'Yellow' elements any more.

Scene setups can become pretty complex with that in mind. In the sequencer window you can type in a cue sheet with comments that allows you to switch between scenes. Instead of switching the scene-level-fields manually, you write commands in the Text-field of the sequencer window. Press the 'Edit' Button to switch to the input mode.

The basic switching commands consist of a charachter for the group-color (G-Y-B-W-R) and the scene level. You can group multiple commands in one line. After a semicolon you can add comments. (You should also add a blank before the semicolon.)

G1 Y1 B1 W1 ; Initialize all scene-levels to 1

G2 ; Change only green to scene level 2

G3 Y3 ; Change green and yellow elements to scene level 3

Press the 'Run' Button in the sequencer window and click on the sequencer line you want to activate. The scene changes will be performed immediately.

There are other program options you can change with sequencer commands.

Cn : switches to camera n (valid range 1..3)

Httt : Pixelthreshold set to value ttt (valid range 1...768, default 100)

Tiii : Time interval between processing of two pictures (valid range 10...1000, default 40)

A+, A- : turn automatic background feature on/off.

BK : do a new background

Those commands are also relevant if you want to control Eyecon functions with Eyecon elements or via OSC. The following additional commands are controlling the sequencer itself. This is especially useful in controlling Eyecon through touchlines and fields or from a remote computer.

Sn : jump to sequencer line n.

S+, S- : jump to the next/previous line in the sequencer

obviously it is not very convenient to use fix line numbers. A better approach is labeling lines like this:

:Label1 G1

is creating the line label 'Label1' changing to Green 1, a jump to that line can be performed by using the labeled jump

L:Label1

EyeCon allows you to create an unlimited number of different scenes in one setup-file.

EyeCon Help, this file last changed on 03. July 2004


Keyboard Shortcuts

Home background picture

The background (reference) picture

EyeCon Help, this file last changed on 27. May 2003


Autostart

Home program autostart

EyeCon Autostart:

Application

In some cases you might want to run your EyeCon Patch without any further operator interaction. You might want to use a desktop link to startup and run an EyeCon patch or add a link to the autostart menu.

EyeCon has two command line parameters, one is the filename of the configuration file. The second is an optional '1' which switches Eyecon to run-mode immediately after loading the patch.

Setting up command line parameters

The easiest way to use the command line options is to create a new program link to EyeCon on the desktop. You can either right-click on the desktop, create a new link and browse for the executable file in your installation folder. A quick way is to drag the Eyecon link from the startmenu to the desktop with the right mouse button held down. Windows is asking whether it should copy or move the program link, in our case you want to copy it.

Now right click the newly created icon and set the properties.

EyeCon Help, this file last changed on 25. May 2003


Keyboard Shortcuts

Home keyboard shortcuts

EyeCon keyboard shortcuts:

Predefined shortcuts

Key | Function
Esc | Panic - Stop all playing sounds, send Midi 'All Note Off' command
Enter | Rewind - Move all media files (AVI,WAV) to beginning
Insert | Start - Switch to Run-Mode
Pause | Stop - Switch to Stop-Mode
* (Num keyb) | Background
Page Up (Cursor field) | Sequencer: jump to first line
Page Down (Cursor field) | Sequencer: jump to last line
+ (Num Keyb) | Sequencer: jump to next line
- (Num Keyb) | Sequencer: jump to previous line

How to edit the keyboard shortcuts

You can redefine the keyboard shortcuts by editing the file 'eyecon.kbd' in the program folder. By default it looks like this

Panic=1
Rewind=284
Start=338
Stop=339
Background=55

SequencerHome=329
SequencerEnd=337
SequencerNext=78
SequencerPrev=74

The function names are followed by the assignment of a keyboard scancode and an optional modifier statement (Alt, Control, Shift). To find out the keyboard scancodes you can watch the EyeCon statusbar. The right-most field displays the code for the last pressed key.

Remark: The used scancodes represent the physical position of the key on the keyboard, not the logical and language dependent characters the keys normally produce.

To use an additional modifier key you can add one of the implemented modifier identifiers after the numeric key code:

AL | Left Alt key
AR | Right Alt key
SL | Left Shift key
SR | Right Shift key
CL | Left Control key
CR | Right Control key

One function can have multiple key assignments. Just add more lines with the same function names to you keyboard shortcut file.

A sample entry would look like this: (Ctrl-A) and the ESC key act as Panic button.

Panic=30 CL

Panic=1

It's also possible to use the modifier keys as function keys without the combination with another key.

EyeCon Help, this file last changed on 14. April 2003


Artistic Considerations

Home Robert Wechsler - Artistic Considerations - Forward

Artistic Considerations in the use of the EyeCon Motion Tracking System

Forward

EyeCon can be put to many uses: Maybe you want to control your room lighting with the movements of your gold fish. Maybe you want to count the cars driving past your bedroom window. I am assuming though that most of you are interested in tracking human movement and in an artistic context.

Even within this focus there is much room for quixotism. /1/ EyeCon may be used with pedestrians (persons with no prior instruction), laypersons with some training or preparation (as in a structured improv setting) and, of course, actors, musicians and dancers in rehearsed roles. The point is, you may have in mind uses for EyeCon which are very different than ours and your artistic tastes may be miles away from mine. Believe me, the last thing I want to do with my little essay is to limit your vision! We've seen things done with EyeCon which we wouldn't in our wildest dreams have imagined, some with surprising success. Lesson one: there are a lot of ways to make art.

Having said that, we have been making pieces with this system since 1995, and have learned a few things about how to implement it effectively (basically, we've just had time to make a lot of mistakes!). We've also learned some things about the nature of interaction. The purpose of this essay, then, is to save you some start-up time.

Introduction

The Nature of Interaction

Mapping

Conclusion

EyeCon Help, this file last changed on 26. Nov 2002


Artistic Considerations

Home Robert Wechsler - Artistic Considerations - Introduction

Artistic Considerations in the use of the EyeCon Motion Tracking System

Introduction

The human body in motion draws its expressive power in no small measure from the special sensitivity of the human eye to particular movement qualities or parameters. These are felt by the actor or dancer -- in her or his so-called "kinesthetic sense" -- but equally important, they are perceived by the audience. The point is, when it comes to human movement, there is a difference between what the objective eye -- in this case the video camera -- sees, and what the viewer perceives. We are not especially aware of it, but the human brain does quite a bit of "image processing" after the picture is taken. Thus, sometimes even the subtlest of movements can play a crucial role in our perception, influencing our impression of what we have seen and in the end determining what the dance says to us.

Computers, at least those of the current generation, lack this sensitivity. To them, human motion is just so many pixels changing light or color value. We are thus left with a huge discrepancy in data between what we humans experience as "dance" and what a computer, any computer, measures and has available to process. Understanding this discrepancy and learning to compensate for it is the key to using Eyecon effectively. One needs to find "work-arounds" -- ways to make the computer seem smarter than it actually is.

To put it another way: Sometimes a key quality or moment in a dance involves a relatively small, or localized change in the image (what to a computer means "few pixels have changes in light value"). Meanwhile, at the same moment, artistically unimportant movement may be large or fast (large pixel value changes) and yet these larger changes may not even be noteworthy or noticeable to an audience. Thus, we, as interactive piece designers, are faced with a situation in which we must "teach" the system what to pay attention to, otherwise we may end up with a piece which is technically interactive, and yet fails to seem this way for anyone other than perhaps the technicians and performers. For the audience, such a use of technology is a red herring; something the program notes tell them about, but which has no real relevance to the experience of the piece.

It is remarkable how often this occurs, even at performances of quite renowned companies! To me, this always seems perplexing; it seems so avoidable. There are different reasons why this happens:

One is simply that the artist wanted it that way. My taste in these things is not everyone's! One hears it argued that since the performers are aware of the interaction, and they report that it affects the way they dance, then the use of technology is justified for it has, if indirectly, altered the final outcome. So we can all go home having seen an "interactive performance". Artists, remember, do love to be obtuse. The last thing we want is for our work to be obvious or pedantic. Our tendency is to err on side of obscurity, then at least we can hide behind the ruse of artistic license and eccentricity. No one can accuse us of having used technology as a "special effect" just to wow an audience.

My reaction is: Come on guys -- we can do better than that! There was a time when contact improv (perhaps the most interactive of contemporary dance forms) was used on stage with the same justification. The problem in either case is that the audience is marginalized in the event. To me this seems to fly in face of what interaction is all about.

Interactive or not, art is ultimately something which should engage an audience -- challenge them, surprise them, touch them. Systems such as EyeCon allow artists possibilities for doing this in ways which were not available in the past and it seems a shame to miss out on the very quality that these systems are so well-suited to achieve just because of poor planning or poor understanding of how interaction functions.

And there is another issue which has made for slow going in this field. The possibilities of interactive art are simply too poorly understood by all of us. Collaboration of the sort required, between artists and engineers, can be slow going. Engineers present us with flashy state-of-the-art devices and choreographers (along with the rest of society) get unrealistic ideas of what they may be good for; where their true potential lies. Artists may lack the technical knowledge necessary to suggest modifications to the system -- the things which can be altered to "bend" the system to their special, sometimes "illogical" purposes. Thus, in the creation process a crucial play-ground phase is often missing, and pieces result which employ technical systems basically straight off the engineer's table without really having undergone an "application-oriented" developmental process.

I have two lines of thought that I hope might be helpful to those wishing to do interactive performing. One concerns the nature of interaction, i.e. that special quality -- what is really a psychological phenomenon -- one can achieve using systems like Eyecon. The other is the more technical issue of how you go about designing pieces to capitalize on a system's capabilities while hiding its weaknesses. This second point is referred to as mapping, i.e. the choices of input and output -- what characteristics of the body in motion the system should be focussed on, and then how it is we want to use the data. This second point, as we shall see, also concerns the "way" performers do their performing.

EyeCon Help, this file last changed on 14. Feb. 2003


Home Robert Wechsler - Artistic Considerations - The Nature of Interaction

Artistic Considerations in the use of the EyeCon Motion Tracking System

The Nature of Interaction

So what is this thing "interaction"? Its what people do with each other. Its what we are not doing now. If you were to write me back a letter, I would get your reaction, but we don't really start interacting until we sit down together and hash it out. It belongs to the most primitive of human instincts that when we are together with one another, exchanging on almost any level, we become animated and excited. We share this, of course, with most other animals.

Human beings have been dancing and making music for 10,000 years. During most of this long history performances were highly interactive, much more so than they are today. The distinctions of "performer" and "audience", and even those of "musician", "dancer", etc. were far less clear than they are today. There was no, "think I'll sit this one out". Everyone was part of the event. There are still today examples in African of traditions for which the same word is used for both dance and music. Participants feed off of each other's energy in a way which is seen today only in such settings as dance clubs (the good ones) and music/dance improv jams. Jazz music provides perhaps a last bastion of highly interactive performing in the West, which is also codified and highly disciplined.

Beginning with the predominance of the bourgeoisie, theater in Europe saw a closing off of interaction between performer and audience. With bright lights on one side of a proscenium, and a darken area with seats on the other, the audience's role was pretty much reduced to sitting quietly and then clapping before going home. This has not changed much in the last two hundred years.

Between artists, interactivity has also seen a great decline. Ironically, modern technology is a major culprit. Recording and sampling techniques have meant that musicians, for example, often work separately. Pop music relies heavily on sampling ("stealing with respect" I have heard it called) rather than creating from scratch or getting together and jamming. Dancers and musicians, meanwhile, rarely work directly with one another anymore. Only a tiny fraction of dances performed today (including those by big name companies) directly involve musicians in any part of the actual stage production: creation, rehearsal or performance.

But the biggest interactivity-buster of all is surely the projection screen. Not only did video further reduce the need for dancers and composers to work together creatively, but of course we don't even need to be part of an audience today to watch a performance: we just turn on the television.

And then there is the "group energy" thing. From disco dancing to Ashtanga yoga, we generally pump up better when we are part of a group. As much as we may admire the individualist, in our hearts (and in our genes) we're basically a pretty social bunch of monkeys.

Palindrome has a piece called "Publikumsstück" ("audience piece") in which 10 audience members are brought backstage during the intermission. We teach them ingredients for a structured improvisation as well as give them a crash-course in interactive performing. After the intermission the piece is performed within an interactive stage environment so that different audience members control different sounds with their movements. A woman once came up to me after the show and commented that she liked the part that "we were involved in". I looked at her for I did not recognize her as one of the ten. She wasn't. She meant "we" the audience.

This woman's reaction points out something quite fundamental about how interaction works. It is very much a "feeling thing" -- a subjective, rather than objective phenomenon. Small amounts of participation can have an enormous effect. On the other side, giving the audience many things to do may have little effect on their "interactivity" -- their sense that they were part of the piece. It depends heavily on how it is done; the issue is the context, getting that group-feeling thing going and the choices of mapping.

The same principle applies to performer-based interactive pieces. Sometimes a minor, or short-lived interactive event, with a painfully simple mapping scheme, may completely alter the nature of the piece.

You will notice, we can talk today of two kinds of interactivity: interaction between artists and interaction involving the audience and artists together. To my way of thinking, the same basic principles apply to both; they share the same psychological roots and in practice will function in a similar way. In both cases, it is highly dependent on the performer being loose and easy in their role so that they can respond honestly to the media and people around them.

Which is not the same as improvisation. Palindrome's work is probably 90% choreographed. Still, within a basic structure there must be a certain critical "play-room". If a piece is truly fixed, it can never be interactive.

When you first experience a system like Eyecon, there is the "aha!" effect. Oh, I can trigger sounds with movements, cool. I imagine this must have been something like what people experienced when the light switch was invented. People probably did shows where they turned lights on and off -- automatically! Notice: neither switching on a light, nor triggering sounds with movement has anything to do with interaction. Let's not get confused. Interaction entails a back-and-forth, not just a forth.

If you are dancing in a (now I have to say) "so-called" interactive system, you may choose to "interact" with it or not. If your movement controls, say, the sound you are dancing to, and your dancing is modulated by the real-time changes in the music, then there is a kind of interaction; a circle is closed, so to speak. You will notice, this does not strictly fit the definition I gave you earlier since there is no one else there! You might say the computer has taken over this role. But let's not split hairs, the point is that a particular spontaneous quality can be achieved -- a sense of liveliness -- in a performance of this kind. The dancer, through her movement, is empowered by having control over the music to which she is dancing. Whether you choose to call this "interactive" or not, the process by which it is created generally involves much more collaboration -- and a more intimate kind of collaboration -- than in a traditional dance work. The exchanges between dancer, composer, choreographer and perhaps engineer are highly interactive. There are not only many more interdisciplinary issues involved, but there may be more than one secondary media (Eyecon may also control stage lights, for example).

Now, there is one way to work with systems like EyeCon which holds "interactive" to its word. These are pieces which use technology as a kind of mediator between live performers. My partner, Frieder Weiß is a champion of this school of thought. Such a piece might would function like this: Dancer's movements are tracked and, instead of playing music electronically, the data are fed to live musicians (for example notes may appear on their electronic note stands). The musicians then interpret this information (i.e. play the notes as can, or see fit). The sound of their instruments is then fed again to a computer where it is translated to information which can in turn inform the dancer's movement. The system we have used, for example, involves projecting a light field onto the stage from above. The size and shape of this field is altered according to the pitch and dynamics of the music. The dancer must try to fit her movements into the size and shape of the projected field -- thus the dancer's stage (and thus her movements) are modified by the music. The circle is thus closed.

EyeCon Help, this file last changed on 14. Feb. 2003


Home Robert Wechsler - Artistic Considerations - Mapping

Artistic Considerations in the use of the EyeCon Motion Tracking System

Mapping

EyeCon allows a variety of movement parameters to be used as "input" to the system:

  • Position of body parts in space around you (i.e. the "touchline" feature, or triggers in space)
  • Dynamics (total body movement) within defined fields
  • Position of the body on the stage area (using overhead camera)
  • Height of body from the floor (using horizontal camera, and the "top" feature)
  • Width from left-most to right-most point on the body
  • Degree of expansion or contraction
  • Size of the body image (this is not quite the same as expansion-contraction. the former is relative, the later absolute.)
  • Degree of symmetry in the body (how much left resembles right)
  • Number of dancers on stage (tracking feature)
  • Relative closeness of dancers to each other
  • Individual tracking and differentiation of dancers (based on costume color)

In terms of output, the sky's the limit. A few examples which come to mind:

  • Audio

    musical notes (synthesizer)
    on-off
    volume control
    pitch bending
    panning
    samples (*.wav files)
    on-off
    volume control
    pitch bending
    panning
    real time signal processing (using, for example, MAX/msp)

  • Video Projection

    on-off
    play-freeze
    film play forward-film play in reverse
    real time signal processing (using, for example, NATO)

  • Stage Lighting
  • Mechanical Devices (mechanical hammers, wind, etc.)

Here then is a sample mapping:

Each choice of mapping (the arrows) may have two directions of compliance /2/. That is, "more movement" may mean "more sound" -- but it can also mean "less sound". This may sound counterintuitive, but there are cases when it feels exactly right. For example, holding a shape -- what we dancers call a suspension -- may require a lot of energy. So although not moving, this moment might anyway be well represented by mapping "less motion" to "more sound".

Must mapping be intuitive? Of course not. But in practice it won't work very well otherwise. What we have found is that straying, even a little, from what "makes sense" on a feeling level, results in an outsider quickly looses the connection. It is harder than you think to follow these mappings, any mappings, even when you've had them explained! Therefore, I would advise to stay as intuitive as possible. "Boring" you say? No way! Believe me, there is still ample room for experimentation and surprises in and around this basic framework.

Indeed, as I look at the diagram above, it occurs to me that if all those things were happening at once, no one would have any idea what was going on! I actually can't think of a single situation when we used more than two mapping at one time. In an entire piece, we rarely employ more than three or four! We have made pieces with upwards of 250 Eyecon elements in them; this is not what I am talking about. The number of mappings -- the kinds of the parameters used -- are far more limited. Not because it would be technically difficult to do so. It wouldn't. The reason is simply that one very quickly comes up against the limits of what the inexperienced viewer can follow.

The arrows in the diagram above should also not be confused with something like "tracks" in a composition. You might think, well, in an orchestra you are not "aware" of what every individual instrument is playing either, and yet the complexity can be marvelous. In the case of an orchestra, however, there is at least mutual support! If you use too many mappings they will likely fight one another, and the result will be that no single one will be evident at all. In all likelihood, no single audience member will know that anything interactive has occurred. But like I say: don't take my word for it. Try it!

Making good mapping choices is really the "art" of using Eyecon. There are no simple rules. Bizarre combinations of parameters have produced stunning results. As I have emphasized already, sufficient clarity is the most common problem. Ways to raise clarity include:

  • repeat crucial movements a number of times.
  • if you are a performer, move conscientiously. be aware of the affect you are having on your environment and let this "inform" your movement.
  • map to multiple outputs. for example, you may wish to link a sound element with a particular movement as well as a visual element (a stage lighting change or video projection element)
  • think about camera angles. choose one which helps the movements to be accurate and repeatable.
  • dancers tell choreographers or technicians what you need instead of the other way around.
  • "link" the media event to a particular gesture. the way a movement is executed may be the key to an audiences perception of the event. try to use memorable, movements -- those with character (even though technically there may be no advantage to doing it that way)
  • trigger or control the same events from the same stage positions or from the same body posture even though, again, this may be irrelevant to the system you are using.
  • look for intuitive mappings (higher body level-to-higher pitch, faster-to-louder, busier movement-to-busier sound, heavier movement-to-heavier sound, etc.)
  • Near the beginning of the piece, or at least at some time during the piece, use the system in a clear and transparent way. this way the system will "explain itself" to the viewer. once having done this, they will become sensitized to the "interactive experience" and will be naturally attuned to "subtler" mappings.

And finally here's one that creates a stir:

  • either before, or after the piece, explain to your audience how the technology works. There are as many good reasons to do this as there are not to -- but it is an option. To my thinking, it depends on the context (how formal the setting is, for example) as well as the artistic demands of the piece involved. Some pieces don't need it, some don't want it, and others simply love it. Either way, whether you do it or not, I will guarantee you one thing: people will come to you after the show and thank you from the bottom their hearts for your decision, just as the person beside him castigates you for it!

Mappings may of course also be too simple. To lower the clarity, I recommend:

  • if you are using Touchlines (triggers), try to avoid hand or foot pokes. Find ways to use your dancerly skills (there must be 50 ways to extend a body part).
  • avoid using physically easy ways to execute an event. Make it harder for yourself than it needs to be.
  • choose dynamic rather than passive body positions (use plié, relévé, spirals, etc. even when it is of no technical advantage).
  • use more than one mapping in a piece (though not necessarily simultaneously!)
  • think about the "direction" /2/ of the interactive relationship. positive, negative; presence, absence. You may wish to reverse it.

EyeCon Help, this file last changed on 14. Feb. 2003


Home Robert Wechsler - Artistic Considerations - Conclusion

Artistic Considerations in the use of the EyeCon Motion Tracking System

Conclusion

No matter how long my essay runs on, working interactively will require understandings which can only be gleaned from the physical experience itself. Interactive performance work has a learning curve all its own so don't get frustrated! And let's not take the easy way out! Let's re-tool and re-think.

The choreographer, composer and designer must all give up trusted paradigms. The traditional model, in which a choreographer engages a designer or composer to build a "setting" for their dance has little relevance to work of this kind. Interactive systems are not "a kind of stage set"! They require a new kind collaboration between artists -- it is more intense, involving more sacrifice. The composer, for example, may have to give up considerable control over how the music sounds as it may be played by a group of dancers through their movements in space. Remember, dancers are generally, not musicians. Speaking as a dancer: I have learned this! We are not used to hearing what we are doing and in fact the demands on our sense of timing are far greater than we are used to. When you "follow" music there a certain room for error -- a half second here or there is generally not noticeable. Not so when the music is following you!

And it is not only timing I am speaking about. Spacing -- being a few inches more to the left -- may be critical in a way reminiscent of film or television work. The dance must be modified in other ways as well. And more important than any of the technical details is getting a "feel" for its use; learning to give human qualities to the technical system (what, other than this, is dance?).

We must compensate for the "unintelligent" computer and this requires engineers and artists to communicate extremely well with one another. This, in turn, means gaining rudimentary knowledge of one another's field. There is much to learn when you start down the road of dance and technology. Every piece must be "premiered" three times as you will not get it right the first time. These are all things which dancers and the dance world as a whole are not used to. The point is, have patience; it will be rewarded.

EyeCon Help, this file last changed on 14. Feb. 2003


Control Panel

Home License

EyeCon License

The EyeCon software is distributed as try-before-you-buy. This means:

1. All copyrights to EyeCon are exclusively owned by Frieder Weiss and Palindrome.de.

2. Anyone may use this software during a test period of 30 days. Following this test period of 30 days or less, if you wish to continue to use EyeCon, you MUST register.

3. Once registered, the user is granted a non-exclusive license to use EyeCon on one computer (i.e. a single cpu), for any legal purpose, at a time. The registered EyeCon software may not be rented or leased independent from the computer it is registered to, but may be permanently transferred, if the person receiving it agrees to terms of this license. If the software is an update, the transfer must include the update and all previous versions.

4. The EyeCon unregistered trial version, may be freely distributed, and with exceptions noted below, provided the distribution package is not modified. No person or company may charge a fee for the distribution of EyeCon without written permission from the copyright holder. The EyeCon unregistered trial version may not be bundled or distributed with any other package without written permission of the copyright holder.

5. To register you must send registration fee, to one of the authorized registration sites, see REGISTER.TXT.

6. EyeCon IS DISTRIBUTED "AS IS". NO WARRANTY OF ANY KIND IS EXPRESSED OR IMPLIED. YOU USE AT YOUR OWN RISK. THE AUTHOR WILL NOT BE LIABLE FOR DATA LOSS, DAMAGES, LOSS OF PROFITS OR ANY OTHER KIND OF LOSS WHILE USING OR MISUSING THIS SOFTWARE.

7. You may not use, copy, emulate, clone, rent, lease, sell, modify, decompile, disassemble, otherwise reverse engineer, or transfer the licensed program, or any subset of the licensed program, except as provided for in this agreement. Any such unauthorized use shall result in immediate and automatic termination of this license and may result in criminal and/or civil prosecution.
All rights not expressly granted here are reserved by Palindrome.de.

8. Installing and using EyeCon signifies acceptance of these terms and conditions of the license.

9. If you do not agree with the terms of this license you must remove EyeCon files from your storage devices and cease to use the product.

Thank you for using EyeCon.

EyeCon Help, this file last changed on10. Mar 2003


;Element Editor

.Topic 2001
;Text
Defines a text identifier for  the
element. Select 'Show' in the Live Video Window,
and you will see all the identifier close to their
graphical representation

.Topic 2002
;Group Color
The color of the graphical element. Used to group
the elements. This makes it possible to change the
scene-level for one group (color) and leave other
elements active.

;Level
.Topic 2003

;Level Persistent
.Topic 13004
Level Persistent


;Popup-texte für ElementEditor

.Topic 13001
Defines a text identifier for  the
element. Select 'Show' in the Live Video Window,
and you will see all the identifier close to their
graphical representation

.Topic 13002
The color of the graphical element. Used to group
the elements. This makes it possible to change the
scene-level for one group (color) and leave other
elements active.

.Topic 13003

;Level Persistent
.Topic 13004
Level Persistent

.Topic 15401
Enable/disable picture
.Topic 15402
Filename, use %d as placeholder for current number
.Topic 15403
.Topic 15404
.Topic 15405
Index when element not triggered
-1 no change
-2 black
.Topic 15406
stretch to Canvas
.Topic 15407
autoincrement when triggered
.topic 15450
enable/disable live picture on canvas


Camera Connection

Home Camera Connection

Connecting a camera:

Connect a video camera to the video input. TheFALCON video framegrabber has two inputs for composite video and one for SVHS connections. Video Input #1 is the one that is further away from the SVHS connector.

On the control window you can select the used camera input.

You should see the camera picture in the Video window now. To change the Video-Input channel or the contrast/brightness settings choose Options/Videopara from the menu or press F6. Here you can switch the framegrabber inputs and tune the input parameters.

By default the color saturation is set to 0. That means that you don't see any colors in the video window even though you might have connected a color camera. The reason is, that neither touchlines nor dynamic fields need color information. Turning on the color usually leads to a noisier videopicture and makes controled interaction more difficult.

EyeCon Help, this file last changed on 14. Nov 2002