The Stanza Logo-Motoria
The Stanza Logo-Motoria (see the figures 1, 2 and 3) is a multimodal interactive system for learning and communication developed in collaboration with the University of Genova (InfomusLab, Department of Informatics, Systems and Telematics) and the University of Udine (AVIRES Lab, Department of Mathematics and Computer Science) by means of the EyesWeb XMI platform (www.infomus.org). It is permanently installed at the "E. Frinta" Primary School in Gorizia (Italy), where it is used as an alternative and/or additional tool to the traditional ways of teaching. The Stanza Logo-Motoria is used by all the pupils of the school - from the first to the fifth grade - including the children with disabilities.
Objectives
The Stanza Logo-Motoria aims to:
- enhance alternative intelligences and communication;
- promote learning motivation;
- develop pupils’ different cognitive styles;
- offer an alternative and/or additional teaching method for dyslexic children and disabled pupils;
- propose a new didactic and interactive tool for foreign language learning.
Fig.1: The Stanza Logo-Motoria
encourages the body movement.
Fig.2: Listening a story and
miming the characters
in the Stanza Logo-Motoria.
Fig.3: Learning English in the Stanza Logo-Motoria.
System Architecture
The Stanza Logo-Motoria system architecture (fig.4) consists of three major components:
- The input component, receiving the video stream captured by a webcam observing the space. This component is also responsible for data pre-processing.
- The feature extraction component, which analyzes the input data in order to get information about a) how the user occupies the space, b) the expressiveness of their gestures.
- The component for real-time processing of audiovisual content, which is responsible for the real-time control and processing of audio and video material and depends on the features extracted by the feature extraction component.
Fig.4: The system architecture of the Stanza Logo-Motoria.
Resonant Memory
The Stanza Logo-Motoria can be used in different ways, one of these is based on the Resonant Memory patch: the space captured by a webcam is divided into nine areas (fig.5), eight of these are peripheral whereas the ninth is central; the number of zones may vary depending on the didactic needs. Sound or visual information corresponds to each area. A child explores the resonant space in which he/she can freely move without using sensors.
Fig.5: Resonant Memory modality.
- Noises, sounds, and music are associated with the peripheral zones and are reproduced when the child reaches a peripheral zone.
- The audio reproduction of a story is associated with the central zone; the story provides references to the various sounds located in the peripheral areas. The child, listening to the story, enjoys searching for the sounds heard before and at the same time, he/she creates the soundtrack of the story.
The video analysis and sound rendering tasks are performed by an EyesWeb patch that can be subdivided into three states.
- The input stage: the signal from the camera is processed in order to extract several low-level features related to the user’s movements. The background subtraction is achieved via a statistical approach: the brightness/chromaticity distortion method. Extracted features include the trajectory of the center of mass, the Motion Index and the Contraction Index.
- In the mapping stage, the patch analyses the features and, according to the user's actions, it runs the transitions through the four phases: exploration, story, pause and reset.
- Finally, the output stage controls the playback of a set of pre-recorded audio files.
Publications
A complete list of group publications can be found here.
Physically based sound modeling
Audio in multimodal rendering
Music expression modeling
Sound processing
Industry funded research