Question: how to mix music when you are not a sound engineer ?

The original goal of the MusicSpace project was to answer this question by introducing a constraint solver to control the location and movements of sound sources. Movements initiated by the user trigger the solver which then moves automatically other sound sources to satisfy a set of “constraints”. We have shown that a limited set of constraints (available in MusicSpaces’s constraint palette) suffice to ensure, for instance, that every movement of user would always produce a “good” mix. Other constraints can be set to produce automatic trajectories of sound sources. The original MusicSpace system and technology are described in several papers, including Olivier Delerue’s Ph.D thesis.
MusicSpace is now integrated into Max/MSP (as an mxj object). As a consequence, MusicSpace’s original constraint solver can be used to manipulate not only spatialisation parameters, but also arbitrary Max/MSP data. This integration opens up exciting new possibilities which we are only starting to explore, notably for controlling sound synthesis engines (see the videos for examples).

Selected Papers:

Delerue, O. Pachet, F. and Roy, P. A new MusicSpace integrated in Max, to appear.

Delerue, O. Spatialisation du son et programmation par contraintes: le système MusicSpace. Université Paris 6, Paris, January 2004.

Pachet, F. and Delerue, O. On-The-Fly Multi-Track Mixing. Proceedings of AES 109th Convention, Los Angeles, USA, 2000. AES.

Pachet, F. and Delerue, O. MusicSpace: a Constraint-based Control System for Music Spatialization. Proceedings of ICMC 1999, pages 272-275, Beijing, China, 1999. ICMA.

Pachet, F. and Delerue, O. Constraint-Based Spatialization. First COST-G6 Workshop on Digital Audio Effects (DAFX98), pages 71-75, Barcelona, Spain, November 1998.

Pachet, F. and Delerue, O. MidiSpace: a Constraint-based Temporal Music Spatializer. ACM Multimedia Conference, pages 351-359, Bristol, UK, 1998.

Question: “Can we build systems that interact meaningfully with animals ?”

As a followup of our work on audio feature generation and classification, we have become interested in animal audio communication. The initial idea was to test whether our techniques developped for music applications would also fit with animal vocalizations. We have shown (together with the team of Csaba Molnar) that feature generation could improve dog bark classification to reach performance that exceed the performance of human experts. This work led to the stuy of other animals, such as parrots and canaries. Now we are interested in establishing complete close-loop audio interactions between an animal and an artificial system. The goal of such a study is twofold. First, we want to trigger interactions and behaviors that are otherwise difficult if not impossible to observe. These behavior can help ethologists to understand animal behavior and cognition.. The second goal is to study the mechanisms of reflexive interaction in non human species to study the relationships between perception and action.

Selected papers:

Skandrani, Z. Mise en place d’interactions acoustiques chez les canaris. Juin 2009. Rapport de stage effectué au Laboratoire d’Ethologie et Cognition Comparées, Université Paris Ouest - Nanterre La Défense / Sony

Giret, N., Roy, P., Albert, A., Pachet, F., Kreutzer, M. and Bovet, D. Evolving Acoustic Features for Parrot Vocalizations. submitted, 2009.

Molnár, Csaba, Kaplan, Frédéric, Roy, Pierre, Pachet, Francois, Pongrácz, Péter, Dóka, Antal and Miklósi, Ádám Classification of dog barks: a machine learning approach. Animal Cognition, 11(3):389-400 2008.

Al Ain, S., Giret, N., Roy, P., Pachet, F., Kreutzer, M. and Bovet, D. Different Acoustic Analysis Methods Yield Different Results. Vocal communication in birds and Mammals, St Andrews, Scotland, July 2008.

Question: “Can we push ourselves to be more creative” ?

The Continuator project started in 2000, and pionnered musical reflexive interaction through interactive style modeling. The system was experiences with many musicians at that time, including Bernard Lubat (Uzeste festival, Ircam) and György Kurtag Jr and Sr. Branch of this research activity was devoted to the study of children musical interaction, with the collaboration of Anna-Rita Addessi from the University of Bologna. In particular, we have shown that Continuator could be considered as a particular Flow machine, in the sense of Csikszentmihalyi.
This activity has then developped in other currently active branches: modeling of Bebop improvization, and music interaction games.

Selected Papers:

Pachet, F. Bebop explained. submitted, 2009.

Pachet, F. Description-Based Design of Melodies. Computer Music Journal, 33(4), Winter 2009.

Pachet, F The future of content is in ourselves. ACM Journal of Computers in Entertainment, 6(3), 2008.

Benghi, D., Addessi, A.-R. and Pachet, F. Teaching to Improvise with the Continuator. Proceedings of the 28th International Society for Music Education World Conference, Bologna, Italy, 2008.

Pachet, F. Creativity Studies and Musical Interaction. In Deliège, I. and Wiggins, G., editor, Musical Creativity: Multidisciplinary Research in Theory And Practice, Psychology Press. 2006.

Ferrari, L. Addessi, A.-R. and Pachet F. New technologies for new music education: The Continuator in a classroom setting. In Baroni et al., editor, Proceedings of ICMPC 06, Bologna, Italy, 2006.

Addessi, A.-R. and Pachet, F. Experiments with a Musical Machine: Musical Style Replication in 3/5 year old Children.. British Journal of Music Education, 22(1):21-46 March 2005.

Pachet, F. On the Design of Flow Machines. In Tokoro, M. and Steels, L, editor, A Learning Zone of One’s Own, The Future of Learning , IOS Press. Amsterdam, 2004.

Pachet, Francois Beyond the Cybernetic Jam fantasy: The Continuator. IEEE Computers Graphics and Applications, 4(1):31-35 January/February 2004. special issue on Emerging Technologies

Pachet, F., Addessi, Anna-Rita When Children Reflect on Their Playing Style: The Continuator. ACM Computers in Entertainment, 1(2), 2004.

Pachet, Francois The Continuator: Musical Interaction with Style. Journal of New Music Research, 32(3):333-341 2003.

Pachet, Francois Interacting with a musical learning system: the continuator. In C. Anagnostopoulou, M. Ferrand, A. Smaill, editor, Music and Artificial Intelligence, Lecture Notes in Artificial Intelligence (vol. 2445 ), pages 119-132, Springer Verlag. September 2002.

Pachet, Francois The Continuator: Musical Interaction with Style. In ICMA, editor, Proceedings of ICMC, pages 211-218, Göteborg, Sweden, September 2002. ICMA. best paper award

Pachet, F. Playing with Virtual Musicians: the Continuator in practice. IEEE Multimedia, 9(3):77-82 2002.

Question: How to program content-based management systems efficiently and easily? How to capitalize and reuse programming and design know-how for this new class of systems?

More precisely we propose three working hypothesis for building an environment that integrates smoothly all the content-management techniques covered by the Music Group research in an integrated manner, so as to propose novel applications that can expand the possibilities of music access.

• Integration of activities in a single environment. The different applications envisaged will necessarily share many information, data, metadata and also software components; It is therefore crucial that they can communicate with each other smoothly.
• Need for managing efficiently large databases. Metadata is interesting, by definition, only for managing large databases, which in turn creates issues of efficiency. Compilers which create efficient Sql queries are mandatory to create systems useable by non professionals.
• Need for vertical languages to develop these new systems. The development of a content-based music application requires the handling of many different layers of software development, from the design of audio acoustic descriptors to the development of graphical interfaces (Matlab, C++, Sql, Php, Java, etc.).

Managing these different levels and their interconnections is time consuming and acrobatic. Vertical languages reduce the difficulty by packaging vertically services, thereby freeing the developer to handle manually all these levels.

To implement the hypothesis proposed above, we have developed an object-oriented framework (in the sense of (Fayad et al. 1999)) called MCM (standing for Multimedia Content Management). This framework contains all the important services needed to build content-based music applications, from the design of perceptive descriptors (using the EDS system) to the creation of specific ontologies such as genre and the creation of user interfaces.

Question: How to extract features from audio signals (such as music titles or sound samples) that are efficient for a given classification task ?

Our team has pionneered the development of so-called “feature generation” techniques, in the audio domain. Feature generation consists in letting a system evolve features automatically, for a given classification problem, rather than relying on existing feature sets. We have introduced the notion of analytical features, as audio features consisting in mathematical compositions of elementary operators. The EDS system was designed to generate billions of these features and test them, to evolve efficient features and feature sets. We have shown that analytical features perform better than standard features on a series of well-known audio classification problems. We now focus on the mathematical study of the analytical feature space (e.g. is fitness continuous ?) using tools borrowed from complex system theory.

Selected Papers:

Pachet, F. and Roy, P Analytical Features: a Knowledge-Based Approach to Audio Feature Generation. EURASIP Journal on Audio, Speech, and Music Processing, 2009(1), February 2009.

Molnár, Csaba, Kaplan, Frédéric, Roy, Pierre, Pachet, Francois, Pongrácz, Péter, Dóka, Antal and Miklósi, Ádám Classification of dog barks: a machine learning approach. Animal Cognition, 11(3):389-400 2008.

Roy, P., Pachet, F. and Krakowski, S. Improving the Classification of Percussive Sounds with Analytical Features: a Case Study. Proceedings of Ismir 07, pages 229-232, Vienna, Austria, 2007.

Pachet, F. and Roy, P. Exploring billions of audio features. In Eurasip, editor, Proceedings of CBMI 07, pages 227-235, Bordeaux, France, 2007.

Defréville, B. Roy, P., Rosin, C. and Pachet, F. Automatic Recognition of Urban Sound Sources. Proceedings of the 120th AES Conference, Paris, France, 2006.

Monceaux Jérôme; Pachet, François; Amadu, Frédéric; Roy, Pierre and Aymeric Zils Descriptor-based spatialization. Proceedings of AES Conference 2005, Barcelona, Spain, 2005.

Giordano Cabral, François Pachet, and Jean-Pierre Briot. Automatic X traditional descriptor extraction: The case of chord recognition.. Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR’2005), pages 444-449, London, U.K., September 2005.

Zils, A. and Pachet, F. Automatic Extraction of Music Descriptors from Acoustic Signals using EDS. Proceedings of the 116th AES Convention, Berlin, Germany, May 2004.

Zils, A. & Pachet, F. Extracting Automatically the Perceived Intensity of Music Titles. Proceedings of the 6th COST-G6 Conference on Digital Audio Effects (DAFX03), pages 180-183, London, U.K., September 2003. Queen Mary University

Pachet, F. and Zils, A. Evolving Automatically High-Level Music Descriptors From Acoustic Signals. Springer Verlag LNCS, 2771:42-53 2003.

Pachet F., Zils A. Evolving Automatically High-Level Music Descriptors From Acoustic Signals. Sony CSL, 2003.

Question: How to find “interesting” music in large and unknown music collections? How to structure and index automatically music collections?

The intentionally ambiguous expression “Popular Music Browser” reflects the two main goals of this project, which started in 1998, at Sony CSL laboratory. First, we are interested in human-centered issues related to browsing “Popular Music”. Popular here means that the music accessed to is widely distributed, and known to many listeners. Second, we consider “popular browsing” of music, i.e. making music accessible to non specialists (music lovers), and allowing sharing of musical tastes and information within communities, departing from the usual, single user view of digital libraries.

The MusicBrowser is the first music content management tool able to handle large music catalogues, and offer users many novel content-based access methods in an integrated environment. It integrates all aspects of the music-to-listener chain, from music description - descriptor extraction from the music signal, or data mining techniques -, similarity based access and novel music retrieval methods such as automatic sequence generation, and user interface issues.

We are currently conducting a series of user-studies, first a workshop at University of Bologna in May 2004, then a week-long user study at Cité des Sciences, Paris in June 2004, and finally a 2-week “atelier” at Cité des Sciences during the “Villette Numérique” biennale in Septembre 2004, where communities of 20+ people extensively use the Browser, to validate existing descriptors, and design their own descriptors, using the automatic learning from EDS. To this effect we have introduced the notion of “music game”, in which users have to localize a particular song they hear, using the various search methods at their disposal.

Media

• Music Browser demo: Music Browser demo movie
• Music Browser interface:
  
• Pictures of a 3-day seminar at the Facolta di Scienze della Formazione, University of Bologna, Italy:
  
• Pictures of a week-long public workshop at the Multimedia Library, Cité des Sciences, Paris.
  

Question: How to interact with a style learning machine ? Can such an interaction pass the musical Turing test ? How to create and sustain musical excitement?

The Continuator project involves a real time interaction with a system that learns musical styles. With the Continuator, users can play music as they wish, and the system will automatically engage in a dialogue by producing musical phrases designed as continuations of user input. Gradually, the dialogue becomes more and more interesting and challenging as the system continuously learns from all the previous interactions. We are interested in the forms of excitement produced, and the links with, e.g. the theory of Flow.

See also: Duets with the Continuator

Media

Bernard Lubat and the Continuator at Ircam during the Résonances Festival.

Claude Barthélémy, one of the fastest and most original guitarist in the world.

György Kurtag, performer of improvised contemporaneous music.

The Continuator and children between 3 and 5 years old.