Interactive Environments for Learning
The Stanza Logo-Motoria
The Stanza Logo-Motoria is a multimodal
interactive environment for learning. Since its first realization in 2008, it has been installed at
the "E. Frinta" Primary School in Gorizia ("Gorizia 1" Comprehensive
Institute, Italy) and other educational institutions where it has been used
as an innovative tool to the traditional ways of teaching and it has been
experimented with a large number of schoolchildren.
The Stanza Logo-Motoria (Stanza means room, Logo
derives from the Greek word logos, and Motoria means movement/gesture/motion)
is used by all the pupils of the school - from the first to the fifth
grade - including children with disabilities.
The Stanza Logo-Motoria was conceived with the idea to
pursue a specific educational objective: have a classroom become an
Interactive and Multimodal Environment in order to provide
schoolchildren with alternative learning methods that enable them to
meet their real educational needs.
The first prototype of the Stanza Logo-Motoria was designed
and developed in 2008 in collaboration with the InfoMus Lab of
Department of Communication, Computer, and System Science of Genova
University. The Stanza Logo-Motoria used in school from 2009
with scientific purposes, has been developed by the Centro di
Sonologia Computazionale (CSC) of the Department of Information Engineering
(DEI), University of Padova.
Video 1 shows an example of lesson in the Stanza Logo-Motoria.
Video 1: Learning in movement in the Stanza Logo-Motoria.
Objectives
The Stanza Logo-Motoria is used to develop a great deal of educational activities involving several school subjects:
- As a tool to activate the written production of a tale (Video 1).
- In order to study a school subject such as History.
- As a tool to develop the orientation and mobility skills of students with severe visual impairments.
- As a method to improve the communication skills of immigrant children who are learning Italian as their Second Language.
- In order to improve listening and comprehension skills of various populations of children having different learning difficulties or disabilities.
- As a listening tool for ESL (English as a Second Language) lessons.
The video 2 shows how the Stanza Logo-Motoria is used in ESL lessons.
Video 2: learning English in the Stanza Logo-Motoria.
The System
The Stanza Logo-Motoria is a modular system (Fig. 1 and 2) mainly composed by:
- Input Components, standard sensors (e.g a webcam or a microphone) which track the user's body movements, gestures, and sounds;
- Real-Time Processing Components, a specifically developed software for multimedia interaction which a) processes the input signal in order to calculate low-level motion features used to estimate how the user occupies the space and the quality of gestures that the user performs (Feature Extraction) and b) controls the interaction logic and the real-time rendering of audiovisual contents.
- Multimedia Output Components, conventional devices such as loudspeakers and video projectors which allow the multimedia reproduction.
Fig.1: The system architecture of the Stanza Logo-Motoria.
The figure below shows a basic configuration of the Stanza Logo-Motoria.
Fig.2: The Stanza Logo-Motoria basic configuration.
The Stanza Logo-Motoria Applications
The Stanza Logo-Motoria architecture, briefly described above, can be adapted to specific educational needs by using different applications:
- The Resonant Memory application is used to teach several school subjects.
- The Fiaba Magica application may be used to support a stregthening path of gestural intentionality of children with multi-disabilities.
- The PlayAndLearn application aims at teaching pupils new words and meanings.
Resonant Memory
The Stanza Logo-Motoria can be used in different ways, one of these is based on the Resonant Memory application: the space captured by a webcam is divided into nine areas (Fig.3), eight of these are peripheral, whereas the ninth is central. Multimedia contents correspond to each area. A child explores the "resonant" space in which he/she can freely move without using sensors. In particular:
Fig.3: Resonant Memory modality.
- Noises, sounds, and music are synchronized with the peripheral zones and are reproduced when the child reaches one of these.
- The audio reproduction of a story is synchronized with the central zone; the story provides references to the various sounds located in the peripheral areas. The child, listening to the story, enjoys searching for the sounds heard before and in real time creates the soundtrack of the story.
The video analysis and sound rendering tasks are performed by a specific software for multimodal interaction able to manage:
- The input stage: the signal from the webcam is processed in order to extract several low-level features related to the user's movements. Extracted features include the trajectory of the center of mass and the Contraction Index.
- In the mapping stage, the patch analyses the features and, according to the user's actions, it runs the transitions through the four phases: exploration, story, pause and reset.
- Finally, the output stage controls the playback of a set of pre-recorded audio files.
Fiaba Magica (Fairy Tale)
The Fiaba Magica is an application for the Stanza
Logo-Motoria with the main aim of improving the gesture
intentionality of pupils with multi-disabilities. Fiaba Magica
enables them to interact with a tale by performing simple gestures:
the users may reconstruct sounds and images of a tale by widening
their arms and moving within a specific space (Fig. 4).
With Fiaba Magica a child, entering a specific area of the Stanza
Logo-Motoria, triggers a) the audio reproduction of the first
part of the story and b) the projection on the screen of an image (for
example two characters). Once the audio reproduction of the first part
of the story is played, by raising their arms sideward, the user can
play with the characters projected, animating them. The second and
third advancement in the story is performed in the same way as the
first.
At school, all the schoolchildren can use Fiaba Magica;
teachers indeed could convey spatial concepts, personal awareness of
the different body parts, how the body moves in relation to other
bodies and objects and the understanding of where the body is in an
environment, to learn new words and meanings, to do interactive
listening.
Fig.4: The Fiaba Magica application.
PlayAndLearn
The PlayAndLearn application is an evolution
of Fiaba Magica. It is an interactive and multimodal
application for the Stanza Logo-Motoria with which pupils may
acquire new words and meanings. The user's movements and gestures are
used to trigger audio/visual contents.
The space of the Stanza Logo-Motoria is subdivided into three
areas; each area is synchronized with a) the audio reproduction of a
spoken word and its meaning and b) the projection of two animations.
The user listens to the audio reproduction of the definition of a word
and selects the animation which corresponds to the definition heard.
The selection is performed by widening their right or left arm
sideward according to the position of the animation on the screen. The
PlayAndLearn application tracks the user's movements and
gestures by means of the Microsoft Kinect sensor.
FINALIST OF START CUP VENETO 2012
The project Stanza Logo-Motoria, presented by the TechnoTale Group, made up by Antonio Rodà, Sergio Canazza, Leonardo Amico (DEI of Padova University), and Serena Zanolla (DIMI of Udine University), won the third prize of START CUP VENETO 2012 (Fig. 5), the business plan competition that awards the best ideas of enterprise and promotes technology transfer.
Fig.5: The TechnoTale group at the Auditorium Santa Margherita of Venezia, Italy
(October 19, 2012).
Research Threads
3D Audio
Audio in multimodal interfaces
Audio restoration
Interactive environments for learning
Music expression modeling
Physically-based sound modeling
Virtual rehabilitation
History of CSC research
Sub-Threads
Projects
A complete list of projects and industrial partners can be found here.