Good question. You can find a first formalization here : https://hal.archives-ouvertes.fr/hal-00854719 based on parametric timed automata.
This approach is attractive because it relies on a well known formalism widely used in temporal model-checking for instance.
However this approach is not satisfactory : timed automata provide useful tool if they use only one clock (which is the standard use : this clock represents the passing of physical time). In order to formalize the synchronization the various strategies, or to reason with both realtive time (driven by the performer) and physical time, several clocs are required.
The language's constructions have evolved and the up-to-date presentation is given in the refrence manual: http://antescofo-doc.ircam.fr/ Temporal constructions are detailed here: http://antescofo-doc.ircam.fr/Reference/5-synchro/ but are not formalized. Look also at the conference paper: https://hal.archives-ouvertes.fr/hal-01638115
Most of the publications related to this work are accessible from my publication page at: http://repmus.ircam.fr/giavitto/publications
Thank you Jean Louis- like Susan I am sorry that we didn't have you talking through the ideas in person. Slightly tangentially, but as you may have anticipated –your presentation also evoked memories of my youthful fascination with Henri Bergson on time, and that then sparked off some random speculation about using your schema for thinking about various types of information/lived experience interactions.
Yes! I have been influenced by a beautiful short text of composer Bernd Alois Zimmermann: “Intervall und Zeit.” In Frankfurter Allgemeine Zeitung, 1957 (I don't have access to the original in German, nor to an english translation, but to a French version).
Zimmermann introduces a very Bergsonian notion of inner time embedded in performers. He is a reader of Bergson, Husserl and Saint Augustin. This text was very influential, despite being only one page long, and is discussed for instance in https://doi.org/10.21476/PP.2019.42226
Zimmermann sees in music the possibility of reconciling two notions of time: objective time – the chronometric and measurable time of clocks – and the lived time of teh subjective experience – that of the inner perceptive consciousness where, thanks to memory and imagination, past, present and future merge.
This is precisely the problem of Antescofo: can we share time between a machine and a human. The approach of Antescofo relies on the model of Edward Large et Mari Riess Jones (« The dynamics of attending: How we track time varying events » Psychological Review, 106 (1), 119-159 (1999)) used to track the inner tempo of the performer through the audible events he or she produces.
Thanks Jean-Louis, this looks like a really interesting project. I wonder if there has been any related work incorporating bio-feedback directly into performances? I ask as I recently had a conversation with the a psychologist in UoY where he told me that the heart rates of people working effectively in teams synchronised. Raising interesting questions about what the synchronicity means for the team work, and also for people working closely with machines perhaps?
The COSMOS project, an ERC leaded by Elaine Chew, which joined our lab last year, is on pianistic interpretation. COSMOS aims to use data science, optimization / data analytics, and citizen science to study musical structures as they are created in music performances and in unusual sources such as cardiac arrhythmias.
Elaine want to use EEC data to characterize the "impact" of the play during a performance, and to use methods able to classify musical performance, to categorize or diagnose cardiac arrhythmias.
In another area, John McCallum, who did his PhD at the CNMAT in Berkeley, has developed several pieces where a musical generative system is controlled in real time by the heartbeat of a dancer reacting to the music produced.
Professor Eduardo Miranda, University of Plymouth, has also used EEG to drive musical generative systems. And also from other 'bio' data (like slime mold).
thanks everybody for the discussion! reminds me of some completely different kind a lack of synchronous perception(s) -- if that is the right way to phrase it. it is about the subjective perceived severity of some people with (non-neurological) speech impairments - in often stark contrast with any objective measures, of say speech intelligibility. have been pondering over this contrast for a range of impairments for a while now, but there seems to be not too much understanding of the background (at least i haven't found any robust such information).
Hi, "unknown" -- Please remember to comment under your (academic) name -- if for some reason you can't, do please add your name into / after your comment.
I admit that this is beyond my area of expertise. Specifically on speech impairment, I cannot point any meaningful references.
I know that there are studies on the intelligibility of speech from an acoustic point of view. Hearing aids, for example, use filters to increase the intelligibility of speech, much like we have transformations to increase the contrast of an image.
My lab conducts studies to try to characterize the voice, for example by determining descriptors related to an emotional state (the roughness of the voice for example, or are an assertive character). These are difficult studies, carried out with psycho-acousticians, and are based on many psychoacoustic tests. See for example the CREAM project (cracking the emotional code of music): http://cream.ircam.fr/
angelika again (just in case...). thanks -- interesting point about psychological element! some anecdotal evidence -- people fear problems with telephones (a known'enemy'- quality of microphones, etc, but also body language missing) -- so speech even worse than normal when having to use a phone. thanks for the link -- will have a look!
Hi Jean-Louis -- thanks for fascinating presentation -- I just wish I could have heard it in person!
ReplyDeleteYou mention various programming paradigms for various time models (pdf slide 11) -- do you have a formal model of your musical model of time?
Good question. You can find a first formalization here :
Deletehttps://hal.archives-ouvertes.fr/hal-00854719
based on parametric timed automata.
This approach is attractive because it relies on a well known formalism widely used in temporal model-checking for instance.
However this approach is not satisfactory : timed automata provide useful tool if they use only one clock (which is the standard use : this clock represents the passing of physical time). In order to formalize the synchronization the various strategies, or to reason with both realtive time (driven by the performer) and physical time, several clocs are required.
Another approach, which may handle multiple clock and all the machinery of synchronization strategies, is presented in the PhD of José Echeveste:
https://hal.inria.fr/tel-01196248
but the thesis is in French, which is not handy. The language is described here:
https://hal.archives-ouvertes.fr/hal-00917469
The conference paper
https://hal.archives-ouvertes.fr/hal-01054667
propose a denotational semantics but restricted to the modeling of temporal pattern matching and does not handle synchronization strategies. However, it gives a good feeling of the formalization of a timed notion of streams, the basic ingredient to model the underlying hybrid time needed for antescofo (continuous passing of multiples partially related times with discrete events).
The language's constructions have evolved and the up-to-date presentation is given in the refrence manual:
http://antescofo-doc.ircam.fr/
Temporal constructions are detailed here:
http://antescofo-doc.ircam.fr/Reference/5-synchro/
but are not formalized. Look also at the conference paper:
https://hal.archives-ouvertes.fr/hal-01638115
Most of the publications related to this work are accessible from my publication page at: http://repmus.ircam.fr/giavitto/publications
Thank you Jean Louis- like Susan I am sorry that we didn't have you talking through the ideas in person. Slightly tangentially, but as you may have anticipated –your presentation also evoked memories of my youthful fascination with Henri Bergson on time, and that then sparked off some random speculation about using your schema for thinking about various types of information/lived experience interactions.
ReplyDeleteYes! I have been influenced by a beautiful short text of composer Bernd Alois Zimmermann: “Intervall und Zeit.” In Frankfurter Allgemeine Zeitung, 1957 (I don't have access to the original in German, nor to an english translation, but to a French version).
ReplyDeleteZimmermann introduces a very Bergsonian notion of inner time embedded in performers. He is a reader of Bergson, Husserl and Saint Augustin. This text was very influential, despite being only one page long, and is discussed for instance in https://doi.org/10.21476/PP.2019.42226
Zimmermann sees in music the possibility of reconciling two notions of time: objective time – the chronometric and measurable time of clocks – and the lived time of teh subjective experience – that of the inner perceptive consciousness where, thanks to memory and imagination, past, present and future merge.
This is precisely the problem of Antescofo: can we share time between a machine and a human. The approach of Antescofo relies on the model of Edward Large et Mari Riess Jones (« The dynamics of attending: How we track time varying events » Psychological Review, 106 (1), 119-159 (1999)) used to track the inner tempo of the performer through the audible events he or she produces.
Thanks Jean-Louis, this looks like a really interesting project. I wonder if there has been any related work incorporating bio-feedback directly into performances? I ask as I recently had a conversation with the a psychologist in UoY where he told me that the heart rates of people working effectively in teams synchronised. Raising interesting questions about what the synchronicity means for the team work, and also for people working closely with machines perhaps?
ReplyDeleteThe COSMOS project, an ERC leaded by Elaine Chew, which joined our lab last year, is on pianistic interpretation. COSMOS aims to use data science, optimization / data analytics, and citizen science to study musical structures as they are created in music performances and in unusual sources such as cardiac arrhythmias.
DeleteElaine want to use EEC data to characterize the "impact" of the play during a performance, and to use methods able to classify musical performance, to categorize or diagnose cardiac arrhythmias.
In another area, John McCallum, who did his PhD at the CNMAT in Berkeley, has developed several pieces where a musical generative system is controlled in real time by the heartbeat of a dancer reacting to the music produced.
Professor Eduardo Miranda, University of Plymouth, has also used EEG to drive musical generative systems. And also from other 'bio' data (like slime mold).
thanks everybody for the discussion! reminds me of some completely different kind a lack of synchronous perception(s) -- if that is the right way to phrase it. it is about the subjective perceived severity of some people with (non-neurological) speech impairments - in often stark contrast with any objective measures, of say speech intelligibility. have been pondering over this contrast for a range of impairments for a while now, but there seems to be not too much understanding of the background (at least i haven't found any robust such information).
ReplyDeleteHi, "unknown" -- Please remember to comment under your (academic) name -- if for some reason you can't, do please add your name into / after your comment.
Deletei thought i was commenting as angelika sebald -- as that is what the template indicated i was doing (pierre seesm to have (had) the same problen?
DeleteI admit that this is beyond my area of expertise. Specifically on speech impairment, I cannot point any meaningful references.
ReplyDeleteI know that there are studies on the intelligibility of speech from an acoustic point of view. Hearing aids, for example, use filters to increase the intelligibility of speech, much like we have transformations to increase the contrast of an image.
My lab conducts studies to try to characterize the voice, for example by determining descriptors related to an emotional state (the roughness of the voice for example, or are an assertive character). These are difficult studies, carried out with psycho-acousticians, and are based on many psychoacoustic tests. See for example the CREAM project (cracking the emotional code of music): http://cream.ircam.fr/
angelika again (just in case...). thanks -- interesting point about psychological element! some anecdotal evidence -- people fear problems with telephones (a known'enemy'- quality of microphones, etc, but also body language missing) -- so speech even worse than normal when having to use a phone. thanks for the link -- will have a look!
ReplyDelete