A Pelican Blog - Project

Scale-Up4Rehab

2024-06-14T00:00:00+00:00

This Interreg NWE project aims at creating a plateform that creates for rehabilitation practitionners the possibility to integrate XR games within their everyday work with patients. This plateform (that we also call "virtual clinic") would also potentially help patients pursue their rehabilitation process at home, while still allowing practitionner to monitore their practice through distance.

You can get further informations about the projet on this link

Objectives

Scale-Up4Rehab is a five year project which will provide a platform to take a major step forward in the digitization of the rehabilitation sector, by being the first to build an open virtual rehabilitation "clinic," which will scale up existing virtual rehabilitation therapies across north-west Europe.

Results

University of lille contributes to WP 1 (Virtual rehabilitation clinic framework development ) by sketching several technical services that virtual clinic may provide, either in its firt version or in future version, in order to facilitate software collaboration between virtual clininc and rehabilitation games, for achieving integrated activity for practitionners, as well as secure rehabilitation process for patients.
University of Lille also contributes to WP2, activity 1 (pre-pilots belgium and france) with EuraSanté partner through participative sessions with local medical structures, and 4 (Virtual Rehabilitation implementation support) through two points: first, the development of exploratory games, that helps sketching possible software services for the virtual clinic; second, PhD student work about large displays interaction, that can potentially be a specific practitionner tool for game configuration, for the aspects that deal with patient movement abilities.

Detailed information

Interreg NWE Funding : University of Lille total elligible budget 998.7 Keuros, ERDF 599.2 Keuros

Project start and end dates : 2023-2028

Partners : Sint-Martins Klinic (NL, lead), University of lille (FRA), University College Cork (IE), In4Care (BE), HealthValley (NL), Rehazenter (LU), University College Dublin (IE), Tilburg University (NL), EuraSanté (FRA), Radboud universitair medisch centrum (NL), University Hospital RWTH Aachen (DE), Roessingh center for rehabilitation (NL)

Local contacts

Laurent Grisoni (local coordinator)
Thomas Boute (research engineer, working on virtual clinic services)
Martin Birlouez (PhD student, working on large displays interaction techniques)

3D Musical Drawing

2023-10-01T00:00:00+00:00

Reverberations

2023-06-12T00:00:00+00:00

The MINT team has been collaborating since 2024 with musician Sébastien Beaumont and sound engineer Titouan Dumesnil on an immersive art installation in mixed-reality.

The installation was exhibited in 2025 at the XR Gallery of the IEEE Conference on Virtual Reality and 3D User Interfaces and relies on the IVMI builder framework, maintained by MINT to facilitate the design of immersive musical instruments and sonic installations.

Here is a video of this first version :

Following the scan of the abbey in Saint-Michel en Thierache with the help of PIRVI, a new version of the installation will be presented at the "Eclectic Campgnes" festival on May 22, 23 and 24.

The project has received support from the Fédération de Recherche Sciences et Cultures du Visuel.

More information on Sébastien Beaumont's website.

Tactile Feedback and 3D Interaction

2023-06-12T00:00:00+00:00

In this project, we combine tactile feedback based on friction modulation and 3D interaction.

One objective is to study cross-modal interaction between visual depth perception and tactile perception.

Get the paper

Vibrating Shapes

2023-06-12T00:00:00+00:00

Vibrating Shapes is a collaboration between musicians of the Muzzix collective and scientists from the Université de Lille and Junia/Isen on the use of Spatial Augmented Reality displays for actuated musical instruments.

Here is a performance that shows some results of the project.

More detail on the project is available on the official website.

Mavii

2023-03-24T00:00:00+00:00

MAVII is a multidisciplinary research-action project at the crossroads of artistic creation and computer science. MAVII's goal is to design simple tools and make them available to audiovisual creators, directors and exhibition designers so that they can quickly experiment with their ideas for interactive scenarios in the virtual space for immersive headsets. The tools are presented on a graph of nodes to be connected, some parameters of which are directly adjustable without writing code.

More information on https://mavii.univ-lille.fr/

Phone with the Flow

2021-11-29T00:00:00+00:00

Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and requires expensive computation. With Phone with the Flow (PwF), we propose combining camera based motion/gesture input with the touch input, in order to filter movement information both temporally and spatially, thus increasing expressiveness while reducing computation time.

PwF is currently implemented as an Android 9.0 App. You can download the apk file here.

How to

Every touch input activates a region of interest (ROI) in the camera image. The captured movements in the ROIs are then sonified. The sound synthesis can be done either directly on the mobile device, with restrictions on the complexity of the synthesis due to limited computing capabilities, or the features can be sent to external musical software via OpenSoundControl messages. For further information about the mappings used in the app, you can download the Pure Data patch and control it by enabling the OSC output of PwF.

Get the paper

Co-located Mixed-Reality Performances

2021-10-01T00:00:00+00:00

Arm Fatigue in VR

2021-06-12T00:00:00+00:00

In this thesis we studied the perceived arm fatigue of the users of mid-air interaction systems such like VR headsets through multiple experiments. We focused on the variation of arm fatigue due to:

the hand synchronisation (the use of one or two hands synchronised or not).
the rhythm and the use of a sound to highlight it.
the use of control/display ratio during musical interaction.

We contributed through some design guidelines and ideas for future work.

Effect of Control-Display Ratio in Immersive Musical Instruments

Get the paper

Revealed Interfaces

2021-03-01T00:00:00+00:00

In this project, we investigate the use of a particular type of Spatial Augmented Reality, named "Revealed Interfaces" in the context of musical expression.

Augmenting gestural instruments

Musical Exploration of Volumetric textures

Audience Experience

2021-02-01T00:00:00+00:00

In this project, we aim at understanding how the audience perceives and understands performances with Digital Musical Instruments, and how we can augment their experience with 3D displays.

Spectator Experience Augmentation Techniques

When the collaboration between humans and machines hap- pens in public, the audience can face difficulties in distin- guishing the actual human contribution from the contribution of autonomous processes. In music concerts involving digi- tal interfaces doubts about the performer’s contribution can drastically hinder the audience interest. The disappearing of the direct physical link between actions and effects is one of the reasons of this confusion. Consequently both artists and researchers have explored techniques to augment the ex- perience of spectators. However their respective impact on the multiple aspects of audience experience has not yet been formally compared. In this controlled study, we compare two techniques : pre-performance explanations and visual augmen- tations. Despite contradictory results on comprehension tasks, we show that contrary to pre-performance explanations, visual augmentations improve the audience experience, increase their subjective comprehension and restore the trust in performers by reversing the doubt in their favou

titre: Have a SEAT on Stage : Restoring Trust with Spectator Experience Augmentation Techniques
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: Designing Interactive Systems (DIS), Jul 2020, Eindhoven, Netherlands. ⟨10.1145/3357236.3395492⟩
Accès au texte intégral et bibtex

titre: A Taxonomy of Spectator Experience Augmentation Techniques
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: NIME 2020 - The 20th International Conference of New Interfaces for Musical Expression, Jul 2020, Royal Birmingham Conservatoire, United Kingdom
Accès au texte intégral et bibtex

Levels of Detail

Because they break the physical link between gestures and sound, Digital Musical Instruments offer countless opportunities for musical expression. For the same reason however, they may hinder the audience experience, making the musician contribution and expressiveness difficult to perceive. In order to cope with this issue without altering the instruments , researchers and artists have designed techniques to augment their performances with additional information, through audio, haptic or visual modalities. These techniques have however only been designed to offer a fixed level of information, without taking into account the variety of spectators expertise and preferences. In this paper, we investigate the design, implementation and effect on audience experience of visual augmentations with controllable level of detail (LOD). We conduct a controlled experiment with 18 participants, including novices and experts. Our results show contrasts in the impact of LOD on experience and comprehension for experts and novices, and highlight the diversity of usage of visual augmentations by spectators

titre: Levels of Detail in Visual Augmentations for the Novice and Expert Audiences
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: Computer Music Journal, In press
Accès au texte intégral et bibtex

Controllar

2021-02-01T00:00:00+00:00

ControllAR is a collaboration between Florent Berthaut (CRIStAL, Université de Lille) and Alex Jones (University of the West of England).

Despite the development of touchscreens, many expert systems for working with digital multimedia content, such as in music composition and performance, video editing or visual performance, still rely on control surfaces. This can be due to the accuracy and appropriateness of their sensors, the haptic feedback that they offer, and most importantly the way they can be adapted to the specific subset of gestures and tasks that users need to perform. On the other hand, visual feedback on controllers remains limited and/or fixed, preventing similar personalizing. In this paper, we propose ControllAR, a novel system that facilitates the appropriation of rich visual feedback on control surfaces through remixing of graphical user interfaces and augmented reality display. We then use our system to study current and potential appropriation of visual feedback in the case of digital musical instruments and derive guidelines for designers and developers.

Get the software

Get the paper

Revealable Volume Displays

2021-02-01T00:00:00+00:00

We present a class of mixed-reality displays which allow for the 3D exploration of content in public exhibitions, that we call Revealable Volume Displays (RVD). They allow visitors to reveal information placed freely inside or around protected artefacts, visible by all, using their reflection in the panel. We first discuss the implementation of RVDs, providing both projector-based and mobile versions. We then present a design space that describes the interaction possibilities that they offer. Drawing on insights from a field study during a first exhibition, we finally propose and evaluate techniques for facilitating 3D exploration with RVDs.

Get the paper

Boeuf

2021-01-01T00:00:00+00:00

The BOEUF Orchestra Unification Framework project aims to research and promote advanced musical collaboration between musicians using digital musical instruments (DMIs).

We have proposed a theoretical framework that can be used when analysing, designing and implementing networked ensembles with a variety of collaboration modes.

We have developed bf-pd, a set of software components running in Pure Data, which musicians and makers can integrate into their digital instruments and systems. Bf-pd allows musicians to share parameters and output data between instruments, to control each other’s instruments, to synchronize between instruments, to visualize each other’s activity, and to exchange messages.

We are conducting research on mixed-reality interfaces to improve non-mediated (i.e. in-person) collaboration between musicians, and to enable them access the advanced musical collaboration modes offered by our framework. These interfaces will allow musicians to preserve both the existing control interfaces of their instruments and the physicality of their performance, and would improve spectators' ability to perceive the musical exchanges (thereby improving visibility).

The Boeuf project is led by Dr Florent Berthaut at the Université de Lille, France, and by Dr Luke Dahl at the University of Virginia, USA.

More details at http://bf-collab.net/

titre: The Effect of Visualisation Level and Situational Visibility in Co-located Digital Musical Ensembles
auteur: Florent Berthaut, Luke Dahl
article: New Interfaces for Musical Expression, Jun 2022, Auckland, New Zealand. ⟨10.21428/92fbeb44.9d974714⟩
Accès au texte intégral et bibtex

titre: Adapting & Openness: Dynamics of Collaboration Interfaces for Heterogeneous Digital Orchestras
auteur: Florent Berthaut, Luke Dahl
article: New Interfaces for Musical Expression, Jul 2020, Birmingham, United Kingdom
Accès au texte intégral et bibtex