A Pelican Blog

Esther Gruy will present her paper Using Mixed Reality Trajectory Visualisation for the Musical Exploration of Sonic Spaces at the ACM Interactive Spaces and Surfaces International Conference.

More detail soon !

A collaboration between the team and Pierrick Uro (Munster Technological University) will be presented at the 3rd International Workshop on the Musical Metaverse (IEEE IWMM) in October. The paper is entitled A design space for viewpoint-decoupling in mixed-reality musical ensembles.

More detail soon !

A collaboration between MINT, Algomus and the musician Barbara Dang will be presented at the 7th Conference on AI Music Creativity. The paper is entitled Audience Experience of Visual Augmentations for Human-AI Co-Improvisations.

More detail soon !

Esther Gruy presented her paper Extended Reality Audio-Visual Instruments: Design Framework and Case Study.

Here is the video :

and here is the paper.

Barbara Dang will perform on the first concert of the Journées d'Informatique Musicale 2026 at la Malterie on June 15th.

The performance will make use of the MusIAug system developed as a collaboration between MINT and Algomus. More detail on the project soon !

Reverberations, a mixed-reality art installation created as a collaboration between MINT, musician Sébastien Beaumont, and sound engineer Titouan Dumesnil, and already presented at the IEEE VR 2025 XR Gallery, was presented at the Eclectic Campagne Festival.

Here is a video of what could be seen by participants :

Yichen Wang is starting a postdoctoral project on Mediating trust in musical collaboration through spatial communication with intelligent systems in collaboration with Ken Déguernel (CRIStAL/Algomus).

Two papers will be presented at the International Conference on New Interfaces for Musical Expression in June.

• Esther Gruy will present her paper Extended Reality Audio-Visual Instruments: Design Framework and Case Study.
• Nathan Salin will present his paper KinoGroove: Composing with Muscle and Motion in Extended Reality.

More detail soon …

Milad Jamalzadeh will present his paper Does It Matter Which Finger You Use? Investigating Finger Identity and Haptic Pattern Recognition for Stationary and Moving Fingers at ACM CHI in April. More information soon.

Esther Gruy presented her paper Exploring the Use of Auditory Feedback as a Guide for 3D Drawing in Extended Reality at ICAT-EGVE. The paper is available here.

Pierrick Uro successfully defended his PhD thesis titled Interaction, Space and Copresence in Colocated Mixed Reality .

Congratulations !!

Three new members just joined the team !

• Jennifer Bertrand is starting a postdoctoral project on Design and Understanding of Sensory Multimodality for Gesture and Tactile Interaction
• Alyssa Lemoine is starting a PhD in co-supervision with Tifanie Bouchara (LISN, Paris Saclay) on Personalised and Accessible extended Reality Musical Instruments for Practitioners-Patients …

Martin Birlouez presented his paper AirSpartOne: One-Handed Distal Pointing for Large Displays on Mobile Devices and in Midair at the 27th ACM International Conference on Multimodal Interaction (ICMI). The paper is available here.

Nathan Salin will present his paper Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with SPARK at ACM VRST in November. The paper is available here.

Yosra Rekik joins the team as associate professor !

Pierrick Uro will present his paper Strategies for the Reconciliation of Artistic Intent and Technical Constraints in Mixed Reality Performances at the ACM Conference on Designing Interactive Systems in July.

See https://dis.acm.org/2025/ for more informaion.

Florent Berthaut will present the paper GdPd / Ivmi-Builder : A Libre Software Framework for Extended Reality Musical Instruments and Immersive Sonic Installations at the Linux Audio Conference in June.

See https://jimlac25.inria.fr/lac/ for more informaion.

Pierrick Uro will present his paper Decoupling Physical and Virtual Spaces for New Collaboration Strategies in Co-Located Mixed Reality Instruments at the International Conference on New Interfaces for Musical Expression in June.

See https://nime2025.org/ for more information.

The team presented three projects at the IEEE VR Conference in March in Saint-Malo :

• "Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study" (Cagan Arslan, Florent Berthaut) can be accessed here

• "Behavioral Measures of Copresence in Co-located Mixed Reality" (Pierrick Uro, Florent …

MINT collaborated with the Éolie Songe theater company during 3 years for the creation of a Cyber Opera that relies on the Spatial Augmented Reality technology developped by the team.

Here is the project website : TerRev and the Terres Rares Cyber Opera

Presentation at IEEE VR 2025

titre

Examining the …

The team will present three projects at the IEEE VR Conference in March in Saint-Malo :

• the paper "Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study" (Cagan Arslan, Florent Berthaut)

• the paper "Behavioral Measures of Copresence in Co-located Mixed Reality" (Pierrick Uro …

We are developing a set of addons for the Godot 3D engine and audio programming languages, to facilitate the design of immersive musical instruments and sonic installations in extended reality, including :

GdPd: Design audiovisual applications in Godot with external or embedded Pure Data patches. https://gitlab.univ-lille.fr/ivmi/gdpd …

Two PhD students joined the team, in co-supervision with Valérie Gouranton and Ronan Gaugne @ IRISA, Rennes, and fully or partially funded by the PEPR eNSEMBLE :

Guillaume Vallet will investigate immersive participatory artistic performances, looking at new metaphors for collaboration between artists and audience members in virtual environments.

Nathan Salin will …

Esther Gruy will present a review paper on 2D and 3D Musical Drawing at the ACM AudioMostly Conference in Milan in September.

Esther Gruy presented a paper on a new approach and device for musical drawing in extended reality with a physical interaction loop at the NIME conference.

Here is the video :

and here is the paper

This Interreg NWE project aims at creating a plateform that creates for rehabilitation practitionners the possibility to integrate XR games within their everyday work with patients. This plateform (that we also call "virtual clinic") would also potentially help patients pursue their rehabilitation process at home, while still allowing practitionner to …

Esther Gruy will present a paper on a novel device for musical drawing in extended reality at the NIME conference in Utrecht in September.

More details soon ...

Two PhD students joined the team !

Esther Gruy will work on novel devices and techniques for expressive 3D interaction, in particular 3D musical drawing.

Martin Birlouez will work on semi-private proxemic interaction for co-located collaboration.

With advances in immersive displays and gesture tracking technologies, many novel interfaces for musical and visual expression have been developed, which often explore combinations of audio and visual productions. One category of such interfaces is musical drawing, in which artists simultaneously produce both visual and sonic content. Since it deals …

The MINT team has been collaborating since 2024 with musician Sébastien Beaumont and sound engineer Titouan Dumesnil on an immersive art installation in mixed-reality.

The installation was exhibited in 2025 at the XR Gallery of the IEEE Conference on Virtual Reality and 3D User Interfaces and relies on the IVMI …

In this project, we combine tactile feedback based on friction modulation and 3D interaction.

One objective is to study cross-modal interaction between visual depth perception and tactile perception.

Get the paper

Vibrating Shapes is a collaboration between musicians of the Muzzix collective and scientists from the Université de Lille and Junia/Isen on the use of Spatial Augmented Reality displays for actuated musical instruments.

Here is a performance that shows some results of the project.

More detail on the project is …

MAVII is a multidisciplinary research-action project at the crossroads of artistic creation and computer science. MAVII's goal is to design simple tools and make them available to audiovisual creators, directors and exhibition designers so that they can quickly experiment with their ideas for interactive scenarios in the virtual space for …

A collaboration with Detjon Brahimaj and colleagues of the L2EP was presented as a paper at the French national conference on Human Computer Interaction : IHM 2023.

• "Cross-modal interaction of stereoscopy, surface deformation and tactile feedback on the perception of texture roughness in an active touch condition" The paper is accessible …

A performance of the project Vibrating Shapes in collaboration with Sebastien Beaumont and Ivann Cruz, was selected to be presented at the Conference on New Interfaces for Musical Expression 2023.

It can be watched on this page !

Three artistic projects that use the revealed augmented reality technology created by the team :

• The Cyber-Opera Terres Rares, a collaboration with Éolie Songe, was presented at Grand Sud, Lille on September 23rd and 24th.
• Vibrating Shapes, a collaboration with Muzzix, was presented at the Forum des Sciences on September 25th …

The team will be presenting two papers at the International Conference on New Interfaces for Musical Expression :

• "Vibrating shapes : Design and evolution of a spatial augmented reality interface for actuated instruments"
• "The Effect of Visualisation Level and Situational Visibility in Co-located Digital Musical Ensembles".

More details soon !

The "revealed" augmented reality technology designed by the team (presented here and here ) will be used in the cyber-opera "Terres Rares" created by Éolie Songe.

It will premiere in Grenoble on April 5th.

More information on the collaboration here.

Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and …

Vincent Reynaert successfully defended his PhD on the "Study of Arm Fatigability during Mid-Air Gestual Interactions In Virtual Reality"

Congratulations Vincent !

Vincent Reynaert will be presenting a paper entitled The Effect of Rhythm in Mid-air Gestures on the User Experience in Virtual Reality at INTERACT 2021.

In this thesis we studied the perceived arm fatigue of the users of mid-air interaction systems such like VR headsets through multiple experiments. We focused on the variation of arm fatigue due to:

• the hand synchronisation (the use of one or two hands synchronised or not).
• the rhythm and the …

Vincent Reynaert and Florent Berthaut presented papers at the International Conference on New Interfaces for Musical Expression (NIME 2021).

• The Effect of Control-Display Ratio on User Experience in Immersive Virtual Musical Instruments (Read the paper)

• Musical Exploration of Volumetric Textures in Mixed and Virtual Reality (Read the paper)

Fatma Ben Guefrech presented the paper on "Revealable Volume Displays" at IEEE Virtual Reality and 3D User Interfaces. The paper also received a honorable mention.

You can read the paper here.

And here is the supporting video

In this project, we investigate the use of a particular type of Spatial Augmented Reality, named "Revealed Interfaces" in the context of musical expression.

Associated publications

titre

Revgest: Augmenting Gestural Musical Instruments with Revealed Virtual Objects

auteur

Florent Berthaut, Cagan Arslan, Laurent Grisoni

article

International Conference on New Interfaces for …

In this project, we aim at understanding how the audience perceives and understands performances with Digital Musical Instruments, and how we can augment their experience with 3D displays.

Spectator Experience Augmentation Techniques

When the collaboration between humans and machines hap- pens in public, the audience can face difficulties in distin- …

ControllAR is a collaboration between Florent Berthaut (CRIStAL, Université de Lille) and Alex Jones (University of the West of England).

Despite the development of touchscreens, many expert systems for working with digital multimedia content, such as in music composition and performance, video editing or visual performance, still rely on control …

The team will be presenting two papers at the International Conference on New Interfaces for Musical Expression :

• "The Effect of Control-Display Ratio on User Experience in Immersive Virtual Musical Instruments"
• "Musical Exploration of Volumetric Textures in Mixed and Virtual Reality".

More details soon !

We present a class of mixed-reality displays which allow for the 3D exploration of content in public exhibitions, that we call Revealable Volume Displays (RVD). They allow visitors to reveal information placed freely inside or around protected artefacts, visible by all, using their reflection in the panel. We first discuss …

Rivill is an application for building and manipulating 3D SceneGraphs for Mixed-Reality Artistic Performances and Digital Mediation. It is based on OpenGL (glfw, glm), OpenNI2 and is entirely controllable via OpenSoundControl messages. It relies on the approach of revealing virtual content in the physical space by intersecting it with performers …

The BOEUF Orchestra Unification Framework project aims to research and promote advanced musical collaboration between musicians using digital musical instruments (DMIs).

We have proposed a theoretical framework that can be used when analysing, designing and implementing networked ensembles with a variety of collaboration modes.

We have developed bf-pd, a set …

The team will be presenting a paper at IEEE Virtual Reality and 3D User Interfaces 2021 entitled : "Reveable Volume Displays : 3D Exploration of Mixed-Reality Public Exhibitions. More details soon !