MINT

MINT @ IWMM 2026

2026-08-31T00:00:00+02:00

A collaboration between the team and Pierrick Uro (Munster Technological University) will be presented at the 3rd International Workshop on the Musical Metaverse (IEEE IWMM) in October. The paper is entitled A design space for viewpoint-decoupling in mixed-reality musical ensembles.

More detail soon !

MINT @ AIMC 2026

2026-07-31T00:00:00+02:00

A collaboration between MINT, Algomus and the musician Barbara Dang will be presented at the 7th Conference on AI Music Creativity. The paper is entitled Audience Experience of Visual Augmentations for Human-AI Co-Improvisations.

More detail soon !

Extended Reality Audio-Visual Instruments @ NIME 2026

2026-06-25T00:00:00+02:00

Esther Gruy presented her paper Extended Reality Audio-Visual Instruments: Design Framework and Case Study.

Here is the video :

and here is the paper.

Barbara Dang + MINT + Algomus @ la Malterie

2026-06-08T00:00:00+02:00

Barbara Dang will perform on the first concert of the Journées d'Informatique Musicale 2026 at la Malterie on June 15th.

The performance will make use of the MusIAug system developed as a collaboration between MINT and Algomus. More detail on the project soon !

MINT @ Festival Eclectic Campagne

2026-05-28T00:00:00+02:00

Reverberations, a mixed-reality art installation created as a collaboration between MINT, musician Sébastien Beaumont, and sound engineer Titouan Dumesnil, and already presented at the IEEE VR 2025 XR Gallery, was presented at the Eclectic Campagne Festival.

Here is a video of what could be seen by participants :

Welcome Yichen !

2026-05-26T00:00:00+02:00

Yichen Wang is starting a postdoctoral project on Mediating trust in musical collaboration through spatial communication with intelligent systems in collaboration with Ken Déguernel (CRIStAL/Algomus).

MINT @ NIME 2026

2026-05-02T00:00:00+02:00

Two papers will be presented at the International Conference on New Interfaces for Musical Expression in June.

Esther Gruy will present her paper Extended Reality Audio-Visual Instruments: Design Framework and Case Study.
Nathan Salin will present his paper KinoGroove: Composing with Muscle and Motion in Extended Reality.

More detail soon …

MINT @ ACM CHI 2026

2026-03-02T00:00:00+01:00

Milad Jamalzadeh will present his paper Does It Matter Which Finger You Use? Investigating Finger Identity and Haptic Pattern Recognition for Stationary and Moving Fingers at ACM CHI in April. More information soon.

MINT @ ICAT-EGVE 2025

2025-12-03T00:00:00+01:00

Esther Gruy presented her paper Exploring the Use of Auditory Feedback as a Guide for 3D Drawing in Extended Reality at ICAT-EGVE. The paper is available here.

Pierrick Uro PhD Thesis

2025-11-18T00:00:00+01:00

Pierrick Uro successfully defended his PhD thesis titled Interaction, Space and Copresence in Colocated Mixed Reality .

Congratulations !!

New team members

2025-10-09T00:00:00+02:00

Three new members just joined the team !

Jennifer Bertrand is starting a postdoctoral project on Design and Understanding of Sensory Multimodality for Gesture and Tactile Interaction
Alyssa Lemoine is starting a PhD in co-supervision with Tifanie Bouchara (LISN, Paris Saclay) on Personalised and Accessible extended Reality Musical Instruments for Practitioners-Patients Collaboration in Music Therapy
Ziyun Liu is starting a PhD in co-supervision with Ken Déguernel (CRIStAL/Algomus) on Long-term and Curiosity-Driven Learning for mixed initiative musical co-creativity

MINT @ ACM ICMI 2025

2025-10-01T00:00:00+02:00

Martin Birlouez presented his paper AirSpartOne: One-Handed Distal Pointing for Large Displays on Mobile Devices and in Midair at the 27th ACM International Conference on Multimodal Interaction (ICMI). The paper is available here.

MINT/Seamless @ ACM VRST 2025

2025-09-23T00:00:00+02:00

Nathan Salin will present his paper Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with SPARK at ACM VRST in November. The paper is available here.

New team member

2025-09-01T00:00:00+02:00

Yosra Rekik joins the team as associate professor !

MINT @ ACM DIS 2025

2025-05-10T00:00:00+02:00

Pierrick Uro will present his paper Strategies for the Reconciliation of Artistic Intent and Technical Constraints in Mixed Reality Performances at the ACM Conference on Designing Interactive Systems in July.

See https://dis.acm.org/2025/ for more informaion.

MINT @ Linux Audio Conference

2025-05-05T00:00:00+02:00

Florent Berthaut will present the paper GdPd / Ivmi-Builder : A Libre Software Framework for Extended Reality Musical Instruments and Immersive Sonic Installations at the Linux Audio Conference in June.

See https://jimlac25.inria.fr/lac/ for more informaion.

MINT @ NIME 2025

2025-05-01T00:00:00+02:00

Pierrick Uro will present his paper Decoupling Physical and Virtual Spaces for New Collaboration Strategies in Co-Located Mixed Reality Instruments at the International Conference on New Interfaces for Musical Expression in June.

See https://nime2025.org/ for more information.

MINT was at IEEE VR 2025

2025-03-20T00:00:00+01:00

The team presented three projects at the IEEE VR Conference in March in Saint-Malo :

"Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study" (Cagan Arslan, Florent Berthaut) can be accessed here
"Behavioral Measures of Copresence in Co-located Mixed Reality" (Pierrick Uro, Florent Berthaut, Thomas Pietrzak, Marcelo Wanderley), it can be accessed here
the artistic installation "Reverberations" (Sebastien Beaumont, Florent Berthaut). Here is the video :

Terres Rares

2025-02-01T00:00:00+01:00

MINT collaborated with the Éolie Songe theater company during 3 years for the creation of a Cyber Opera that relies on the Spatial Augmented Reality technology developped by the team.

Here is the project website : TerRev and the Terres Rares Cyber Opera

Presentation at IEEE VR 2025

titre: Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study
auteur: Cagan Arslan, Florent Berthaut
article: IEEE Transactions on Visualization and Computer Graphics, 2025, pp.1-10. ⟨10.1109/TVCG.2025.3549194⟩
Accès au texte intégral et bibtex

MINT @ IEEE VR 2025

2025-01-21T00:00:00+01:00

The team will present three projects at the IEEE VR Conference in March in Saint-Malo :

the paper "Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study" (Cagan Arslan, Florent Berthaut)
the paper "Behavioral Measures of Copresence in Co-located Mixed Reality" (Pierrick Uro, Florent Berthaut, Thomas Pietrzak, Marcelo Wanderley)
the artistic installation "Reverberations" (Sebastien Beaumont, Florent Berthaut)

IVMI

2025-01-06T00:00:00+01:00

We are developing a set of addons for the Godot 3D engine and audio programming languages, to facilitate the design of immersive musical instruments and sonic installations in extended reality, including :

GdPd: Design audiovisual applications in Godot with external or embedded Pure Data patches. https://gitlab.univ-lille.fr/ivmi/gdpd.

IVMI-builder: Design immersive musical instruments and sonic installations in XR, building on GdPd, and adding 3D interaction and visualisation techniques for musical expression, networking for collaborative applications. https://gitlab.univ-lille.fr/ivmi/ivmi-builder.

The full list is available at : https://gitlab.univ-lille.fr/ivmi.

The following paper describes the general approach and design workflow : https://hal.science/hal-05095979v1.

Here are some projects that rely on them :

New PhD students 2024

2025-01-01T00:00:00+01:00

Two PhD students joined the team, in co-supervision with Valérie Gouranton and Ronan Gaugne @ IRISA, Rennes, and fully or partially funded by the PEPR eNSEMBLE :

Guillaume Vallet will investigate immersive participatory artistic performances, looking at new metaphors for collaboration between artists and audience members in virtual environments.

Nathan Salin will work on co-creation in virtual environments between different modalities (music, dance, visual art ...)

MINT @ ACM AudioMostly 2024

2024-08-29T00:00:00+02:00

Esther Gruy will present a review paper on 2D and 3D Musical Drawing at the ACM AudioMostly Conference in Milan in September.

MagneTip @ NIME 2024 - Video

2024-07-09T00:00:00+02:00

Esther Gruy presented a paper on a new approach and device for musical drawing in extended reality with a physical interaction loop at the NIME conference.

Here is the video :

and here is the paper

Scale-Up4Rehab

2024-06-14T00:00:00+02:00

This Interreg NWE project aims at creating a plateform that creates for rehabilitation practitionners the possibility to integrate XR games within their everyday work with patients. This plateform (that we also call "virtual clinic") would also potentially help patients pursue their rehabilitation process at home, while still allowing practitionner to monitore their practice through distance.

You can get further informations about the projet on this link

Objectives

Scale-Up4Rehab is a five year project which will provide a platform to take a major step forward in the digitization of the rehabilitation sector, by being the first to build an open virtual rehabilitation "clinic," which will scale up existing virtual rehabilitation therapies across north-west Europe.

Results

University of lille contributes to WP 1 (Virtual rehabilitation clinic framework development ) by sketching several technical services that virtual clinic may provide, either in its firt version or in future version, in order to facilitate software collaboration between virtual clininc and rehabilitation games, for achieving integrated activity for practitionners, as well as secure rehabilitation process for patients.
University of Lille also contributes to WP2, activity 1 (pre-pilots belgium and france) with EuraSanté partner through participative sessions with local medical structures, and 4 (Virtual Rehabilitation implementation support) through two points: first, the development of exploratory games, that helps sketching possible software services for the virtual clinic; second, PhD student work about large displays interaction, that can potentially be a specific practitionner tool for game configuration, for the aspects that deal with patient movement abilities.

Detailed information

Interreg NWE Funding : University of Lille total elligible budget 998.7 Keuros, ERDF 599.2 Keuros

Project start and end dates : 2023-2028

Partners : Sint-Martins Klinic (NL, lead), University of lille (FRA), University College Cork (IE), In4Care (BE), HealthValley (NL), Rehazenter (LU), University College Dublin (IE), Tilburg University (NL), EuraSanté (FRA), Radboud universitair medisch centrum (NL), University Hospital RWTH Aachen (DE), Roessingh center for rehabilitation (NL)

Local contacts

Laurent Grisoni (local coordinator)
Thomas Boute (research engineer, working on virtual clinic services)
Martin Birlouez (PhD student, working on large displays interaction techniques)

MagneTip @ NIME 2024

2024-04-17T00:00:00+02:00

Esther Gruy will present a paper on a novel device for musical drawing in extended reality at the NIME conference in Utrecht in September.

More details soon ...

New PhD students

2023-11-17T00:00:00+01:00

Two PhD students joined the team !

Esther Gruy will work on novel devices and techniques for expressive 3D interaction, in particular 3D musical drawing.

Martin Birlouez will work on semi-private proxemic interaction for co-located collaboration.

3D Musical Drawing

2023-10-01T00:00:00+02:00

With advances in immersive displays and gesture tracking technologies, many novel interfaces for musical and visual expression have been developed, which often explore combinations of audio and visual productions. One category of such interfaces is musical drawing, in which artists simultaneously produce both visual and sonic content. Since it deals with two different sensory channels, one of the main problematic is to find the right balance between the sonic and the visual aspects of a project.

In her Phd thesis, Esther Gruy explores physical interaction devices and audiovisual mappings that open creative opportunities for musical and visual expression in extended reality.

Videos

Papers

titre: Extended Reality Audio-Visual Instruments: Design Framework and Case Study
auteur: Esther Gruy, Florent Berthaut
article: New Interfaces for Musical Expression (NIME), Jun 2026, London, United Kingdom
Accès au texte intégral et bibtex

titre: Exploring the Use of Auditory Feedback as a Guide for 3D Drawing in Extended Reality
auteur: Esther Gruy, Florent Berthaut
article: International Conference on Artificial Reality and Telexistence & Eurographics Symposium on Virtual Environments (ICAT-EGVE 2025), Dec 2025, Karlskrona, Sweden
Accès au texte intégral et bibtex

titre: Musical Drawing in 2D and 3D: Dimensions and Perspectives
auteur: Esther Gruy, Florent Berthaut
article: Audio Mostly, Sep 2024, Milan (Italie), France. pp.181-188, ⟨10.1145/3678299.3678317⟩
Accès au texte intégral et bibtex

titre: MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in Extended Reality
auteur: Esther Gruy, Florent Berthaut
article: New Interfaces for Musical Expression, Sep 2024, Utrecht (Netherlands), France. ⟨10.5281/zenodo.13904763⟩
Accès au texte intégral et bibtex

Reverberations

2023-06-12T00:00:00+02:00

The MINT team has been collaborating since 2024 with musician Sébastien Beaumont and sound engineer Titouan Dumesnil on an immersive art installation in mixed-reality.

The installation was exhibited in 2025 at the XR Gallery of the IEEE Conference on Virtual Reality and 3D User Interfaces and relies on the IVMI builder framework, maintained by MINT to facilitate the design of immersive musical instruments and sonic installations.

Here is a video of this first version :

Following the scan of the abbey in Saint-Michel en Thierache with the help of PIRVI, a new version of the installation was presented at the "Eclectic Campagnes" festival on May 22, 23 and 24.

The project has received support from the Fédération de Recherche Sciences et Cultures du Visuel.

More information on Sébastien Beaumont's website.

Tactile Feedback and 3D Interaction

2023-06-12T00:00:00+02:00

In this project, we combine tactile feedback based on friction modulation and 3D interaction.

One objective is to study cross-modal interaction between visual depth perception and tactile perception.

Get the paper

Vibrating Shapes

2023-06-12T00:00:00+02:00

Vibrating Shapes is a collaboration between musicians of the Muzzix collective and scientists from the Université de Lille and Junia/Isen on the use of Spatial Augmented Reality displays for actuated musical instruments.

Here is a performance that shows some results of the project.

More detail on the project is available on the official website.

Mavii

2023-03-24T00:00:00+01:00

MAVII is a multidisciplinary research-action project at the crossroads of artistic creation and computer science. MAVII's goal is to design simple tools and make them available to audiovisual creators, directors and exhibition designers so that they can quickly experiment with their ideas for interactive scenarios in the virtual space for immersive headsets. The tools are presented on a graph of nodes to be connected, some parameters of which are directly adjustable without writing code.

More information on https://mavii.univ-lille.fr/

MINT @ IHM 2023

2023-03-24T00:00:00+01:00

A collaboration with Detjon Brahimaj and colleagues of the L2EP was presented as a paper at the French national conference on Human Computer Interaction : IHM 2023.

"Cross-modal interaction of stereoscopy, surface deformation and tactile feedback on the perception of texture roughness in an active touch condition" The paper is accessible here.

MINT @ NIME 2023

2023-03-23T00:00:00+01:00

A performance of the project Vibrating Shapes in collaboration with Sebastien Beaumont and Ivann Cruz, was selected to be presented at the Conference on New Interfaces for Musical Expression 2023.

It can be watched on this page !

Artistic collaborations

2022-09-26T00:00:00+02:00

Three artistic projects that use the revealed augmented reality technology created by the team :

The Cyber-Opera Terres Rares, a collaboration with Éolie Songe, was presented at Grand Sud, Lille on September 23rd and 24th.
Vibrating Shapes, a collaboration with Muzzix, was presented at the Forum des Sciences on September 25th. Below a glance at the new prototype :
La Terre Rêve Encore, an installation based on scenes from Terres Rares, will be presented at Experimenta 2022, on October 20-21-22.

MINT @ NIME 2022

2022-03-21T00:00:00+01:00

The team will be presenting two papers at the International Conference on New Interfaces for Musical Expression :

"Vibrating shapes : Design and evolution of a spatial augmented reality interface for actuated instruments"
"The Effect of Visualisation Level and Situational Visibility in Co-located Digital Musical Ensembles".

More details soon !

Revealed Augmented Reality in the "Terres Rares" cyber-opera

2022-02-22T00:00:00+01:00

The "revealed" augmented reality technology designed by the team (presented here and here ) will be used in the cyber-opera "Terres Rares" created by Éolie Songe.

It will premiere in Grenoble on April 5th.

More information on the collaboration here.

Phone with the Flow

2021-11-29T00:00:00+01:00

Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and requires expensive computation. With Phone with the Flow (PwF), we propose combining camera based motion/gesture input with the touch input, in order to filter movement information both temporally and spatially, thus increasing expressiveness while reducing computation time.

PwF is currently implemented as an Android 9.0 App. You can download the apk file here.

How to

Every touch input activates a region of interest (ROI) in the camera image. The captured movements in the ROIs are then sonified. The sound synthesis can be done either directly on the mobile device, with restrictions on the complexity of the synthesis due to limited computing capabilities, or the features can be sent to external musical software via OpenSoundControl messages. For further information about the mappings used in the app, you can download the Pure Data patch and control it by enabling the OSC output of PwF.

Get the paper

Vincent Reynaert Phd Defense

2021-11-25T00:00:00+01:00

Vincent Reynaert successfully defended his PhD on the "Study of Arm Fatigability during Mid-Air Gestual Interactions In Virtual Reality"

Congratulations Vincent !

Co-located Mixed-Reality Performances

2021-10-01T00:00:00+02:00

MINT @ Interact 2021

2021-10-01T00:00:00+02:00

Vincent Reynaert will be presenting a paper entitled The Effect of Rhythm in Mid-air Gestures on the User Experience in Virtual Reality at INTERACT 2021.

Arm Fatigue in VR

2021-06-12T00:00:00+02:00

In this thesis we studied the perceived arm fatigue of the users of mid-air interaction systems such like VR headsets through multiple experiments. We focused on the variation of arm fatigue due to:

the hand synchronisation (the use of one or two hands synchronised or not).
the rhythm and the use of a sound to highlight it.
the use of control/display ratio during musical interaction.

We contributed through some design guidelines and ideas for future work.

Effect of Control-Display Ratio in Immersive Musical Instruments

Get the paper

MINT @ New Interfaces for Musical Expression 2021

2021-05-01T00:00:00+02:00

Vincent Reynaert and Florent Berthaut presented papers at the International Conference on New Interfaces for Musical Expression (NIME 2021).

The Effect of Control-Display Ratio on User Experience in Immersive Virtual Musical Instruments (Read the paper)
Musical Exploration of Volumetric Textures in Mixed and Virtual Reality (Read the paper)

Revealable Volume Displays @ IEEE VR 2021

2021-03-01T00:00:00+01:00

Fatma Ben Guefrech presented the paper on "Revealable Volume Displays" at IEEE Virtual Reality and 3D User Interfaces. The paper also received a honorable mention.

You can read the paper here.

And here is the supporting video

Revealed Interfaces

2021-03-01T00:00:00+01:00

In this project, we investigate the use of a particular type of Spatial Augmented Reality, named "Revealed Interfaces" in the context of musical expression.

Associated publications

titre: Revgest: Augmenting Gestural Musical Instruments with Revealed Virtual Objects
auteur: Florent Berthaut, Cagan Arslan, Laurent Grisoni
article: International Conference on New Interfaces for Musical Expression, May 2017, Copenhagen, Denmark
Accès au texte intégral et bibtex

titre: Musical Exploration of Volumetric Textures in Mixed and Virtual Reality
auteur: Florent Berthaut
article: International Conference on New Interfaces for Musical Expression (NIME 2021), Jun 2021, Shangai, China. ⟨10.21428/92fbeb44.6607d04f⟩
Accès au texte intégral et bibtex

titre: Vibrating shapes : Design and evolution of a spatial augmented reality interface for actuated instruments
auteur: Cagan Arslan, Florent Berthaut, Anthony Beuchey, Paul Cambourian, Arthur Paté
article: New Interfaces for Musical Expression, Jun 2022, Auckland, New Zealand. ⟨10.21428/92fbeb44.c28dd323⟩
Accès au texte intégral et bibtex

titre: Examining the Design Process for 3D Interactions in Performing Arts : A Spatial Augmented Reality Cyber-Opera Case Study
auteur: Cagan Arslan, Florent Berthaut
article: IEEE Transactions on Visualization and Computer Graphics, 2025, pp.1-10. ⟨10.1109/TVCG.2025.3549194⟩
Accès au texte intégral et bibtex

Software

These projects rely on the Rivill software.

Audience Experience

2021-02-01T00:00:00+01:00

In this project, we aim at understanding how the audience perceives and understands performances with Digital Musical Instruments, and how we can augment their experience with 3D displays.

Spectator Experience Augmentation Techniques

When the collaboration between humans and machines hap- pens in public, the audience can face difficulties in distin- guishing the actual human contribution from the contribution of autonomous processes. In music concerts involving digi- tal interfaces doubts about the performer’s contribution can drastically hinder the audience interest. The disappearing of the direct physical link between actions and effects is one of the reasons of this confusion. Consequently both artists and researchers have explored techniques to augment the ex- perience of spectators. However their respective impact on the multiple aspects of audience experience has not yet been formally compared. In this controlled study, we compare two techniques : pre-performance explanations and visual augmen- tations. Despite contradictory results on comprehension tasks, we show that contrary to pre-performance explanations, visual augmentations improve the audience experience, increase their subjective comprehension and restore the trust in performers by reversing the doubt in their favou

titre: Have a SEAT on Stage : Restoring Trust with Spectator Experience Augmentation Techniques
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: Designing Interactive Systems (DIS), Jul 2020, Eindhoven, Netherlands. ⟨10.1145/3357236.3395492⟩
Accès au texte intégral et bibtex

titre: A Taxonomy of Spectator Experience Augmentation Techniques
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: NIME 2020 - The 20th International Conference of New Interfaces for Musical Expression, Jul 2020, Royal Birmingham Conservatoire, United Kingdom
Accès au texte intégral et bibtex

Levels of Detail

Because they break the physical link between gestures and sound, Digital Musical Instruments offer countless opportunities for musical expression. For the same reason however, they may hinder the audience experience, making the musician contribution and expressiveness difficult to perceive. In order to cope with this issue without altering the instruments , researchers and artists have designed techniques to augment their performances with additional information, through audio, haptic or visual modalities. These techniques have however only been designed to offer a fixed level of information, without taking into account the variety of spectators expertise and preferences. In this paper, we investigate the design, implementation and effect on audience experience of visual augmentations with controllable level of detail (LOD). We conduct a controlled experiment with 18 participants, including novices and experts. Our results show contrasts in the impact of LOD on experience and comprehension for experts and novices, and highlight the diversity of usage of visual augmentations by spectators

titre: Levels of Detail in Visual Augmentations for the Novice and Expert Audiences
auteur: Olivier Capra, Florent Berthaut, Laurent Grisoni
article: Computer Music Journal, In press
Accès au texte intégral et bibtex

Controllar

2021-02-01T00:00:00+01:00

ControllAR is a collaboration between Florent Berthaut (CRIStAL, Université de Lille) and Alex Jones (University of the West of England).

Despite the development of touchscreens, many expert systems for working with digital multimedia content, such as in music composition and performance, video editing or visual performance, still rely on control surfaces. This can be due to the accuracy and appropriateness of their sensors, the haptic feedback that they offer, and most importantly the way they can be adapted to the specific subset of gestures and tasks that users need to perform. On the other hand, visual feedback on controllers remains limited and/or fixed, preventing similar personalizing. In this paper, we propose ControllAR, a novel system that facilitates the appropriation of rich visual feedback on control surfaces through remixing of graphical user interfaces and augmented reality display. We then use our system to study current and potential appropriation of visual feedback in the case of digital musical instruments and derive guidelines for designers and developers.

Get the software

Get the paper

MINT@NIME 2021

2021-02-01T00:00:00+01:00

The team will be presenting two papers at the International Conference on New Interfaces for Musical Expression :

"The Effect of Control-Display Ratio on User Experience in Immersive Virtual Musical Instruments"
"Musical Exploration of Volumetric Textures in Mixed and Virtual Reality".

More details soon !

Revealable Volume Displays

2021-02-01T00:00:00+01:00

We present a class of mixed-reality displays which allow for the 3D exploration of content in public exhibitions, that we call Revealable Volume Displays (RVD). They allow visitors to reveal information placed freely inside or around protected artefacts, visible by all, using their reflection in the panel. We first discuss the implementation of RVDs, providing both projector-based and mobile versions. We then present a design space that describes the interaction possibilities that they offer. Drawing on insights from a field study during a first exhibition, we finally propose and evaluate techniques for facilitating 3D exploration with RVDs.

Get the paper

Rivill

2021-02-01T00:00:00+01:00

Rivill is an application for building and manipulating 3D SceneGraphs for Mixed-Reality Artistic Performances and Digital Mediation. It is based on OpenGL (glfw, glm), OpenNI2 and is entirely controllable via OpenSoundControl messages. It relies on the approach of revealing virtual content in the physical space by intersecting it with performers and spectator's bodies and props. This content can be used for visualisation, e.g. to augment digital performances or museum cabinets, or for interaction, e.g. controlling sound with 3D widgets.

Rivill can be downloaded from https://gitlab.univ-lille.fr/mint/rivill

Principle

Projects

Rivill (previously named Revil) has been / is being used in the following projects :

Boeuf

2021-01-01T00:00:00+01:00

The BOEUF Orchestra Unification Framework project aims to research and promote advanced musical collaboration between musicians using digital musical instruments (DMIs).

We have proposed a theoretical framework that can be used when analysing, designing and implementing networked ensembles with a variety of collaboration modes.

We have developed bf-pd, a set of software components running in Pure Data, which musicians and makers can integrate into their digital instruments and systems. Bf-pd allows musicians to share parameters and output data between instruments, to control each other’s instruments, to synchronize between instruments, to visualize each other’s activity, and to exchange messages.

We are conducting research on mixed-reality interfaces to improve non-mediated (i.e. in-person) collaboration between musicians, and to enable them access the advanced musical collaboration modes offered by our framework. These interfaces will allow musicians to preserve both the existing control interfaces of their instruments and the physicality of their performance, and would improve spectators' ability to perceive the musical exchanges (thereby improving visibility).

The Boeuf project is led by Dr Florent Berthaut at the Université de Lille, France, and by Dr Luke Dahl at the University of Virginia, USA.

More details at http://bf-collab.net/

titre: The Effect of Visualisation Level and Situational Visibility in Co-located Digital Musical Ensembles
auteur: Florent Berthaut, Luke Dahl
article: New Interfaces for Musical Expression, Jun 2022, Auckland, New Zealand. ⟨10.21428/92fbeb44.9d974714⟩
Accès au texte intégral et bibtex

titre: Adapting & Openness: Dynamics of Collaboration Interfaces for Heterogeneous Digital Orchestras
auteur: Florent Berthaut, Luke Dahl
article: New Interfaces for Musical Expression, Jul 2020, Birmingham, United Kingdom
Accès au texte intégral et bibtex

MINT @ IEEE VR 2021

2021-01-01T00:00:00+01:00

The team will be presenting a paper at IEEE Virtual Reality and 3D User Interfaces 2021 entitled : "Reveable Volume Displays : 3D Exploration of Mixed-Reality Public Exhibitions. More details soon !