type of project: HIDA-fellow research project

published: 2021

by:Martin Hennecke

website(s):

maintainer(s)/contact: martin.hennecke@theater.digital martin@percussionunderconstruction.de

repository:

The (Un)Answered Question – A Data Science Powered Music Experiment (Martin Hennecke)

Within this research project, specific methods and techniques from the field of Data Science will be investigated in order to make the uniqueness of each live performance tangible in a special way.

Based on the poem 'The Sphynx' by Ralph Waldo Emmerson and the piece 'The Unanswered Question' by composer Charles Ives, a prototype performance is being developed, in which humanoid data of the participants (audience and performers) is being collected through biomedical imaging, put in context by artificial intelligence and then processed into video projections and a live orchestra remix of the original composition. The proof of concept will be within a smaller scaled format with string quartet instead of a full orchestra and reduced test audience.

Martin Hennecke is a German percussionist, composer and since 2009 the Associate Principal Timpanist at Saarland State Orchestra.

With his ensemble 'Percussion Under Construction', he has been creating opportunities to perform the best percussion repertoire through the development of a series of full length live shows and mixing solo with ensemble percussion literature extending from the classic masterpieces to contemporary works including his own compositions. His music (mostly percussion, orchestra and electronic pieces) have been performed at venues like Philharmonie Luxemburg, the UNESCO World Heritage Site Völklinger Hütte and festivals like Euroclassic, SubsTanz, Zappanale etc.

He is a sought after timpanist and percussionist and has been invited to work with about 1/5th of the 129 professional orchestras in Germany, e.g. the Bamberg Symphonic, Bremen Philharmonic, Frankfurt Opera, Gürzenich-Orchestra Cologne, Hannover State Orchestra, Baden State Orchestra Karlsruhe, Nuremberg State Orchestra and the radio orchestras of WDR, SWR and SR, as well as orchestras outside Germany like the Orchestre Philharmonique du Luxemburg or the Qatar Philharmonic Orchestra.

At the 2013 German Music Competition, he was awarded with the coveted scholarship and invited to participate in the concert series 'National Selection of Young Artists'. Martin Hennecke also is a member of the committee for theater and concert orchestras at the German Musician’s Union and is teaching at University of Music Saar.

This research project has been carried out within a joint fellowship program between the Helmholtz Information and Data Science Academy (HIDA) and the Academy for Theater and Digitality (ATD). The program aims to artistically make the field of Data Science accessible to public, as well as seeking to ensure that cutting-edge research into artificial intelligence serves as a source of ideas and inspiration for the next generation of theater and cultural professionals.

For this project, not only one but two research groups from different Helmholtz Centers are collaborating:

Prof. Thoralf Niendorf's group at Berlin Ultrahigh Field Facility, Max Delbrück Center for Molecular Medicine (MDC)



High Resolution Cardiac MRI at 7.0 T, two chamber view, (1 x 1) mm2 in-plane spatial resolution, © MDC

Andreas Schreiber's group at the Institute for Software Technology, Department Intelligent and Distributed Systems, German Aerospace Center (DLR)

IslandViz: A tool for visualizing modular software systems in virtual reality © DLR

Phase I - Establishment of concept and research questions at Academy for Theater and Digitality Dortmund (five weeks, 06.09.21-08.10.21), audio and video recording of strings, winds and trumpet (featuring musicians of Dortmund Philharmonic Orchestra, 01.10.21), MRT (featuring a actor of Saarland State Theater) at Berlin Ultrahigh Field Facility, Max Delbrück Center for Molekulare Medicine, research group
 Prof. Thoralf Niendorf (04./05.10.21).

Phase II and Phase III - Further research (four weeks) and development of technical solutions (eight weeks) at German Aerospace Center (DLR), Institute for Software Technology, Department Intelligent and Distributed Systems, reseach group Andreas Schreiber (total of twelve weeks, 17.01.22-10.04.22).

Phase IV - Dortmund (section has yet to be written)… (three weeks, 01.05.22-23.05.22).

Is it possible to create pieces for theater/opera/orchestra, that will be different every night, depending on who is performing and watching?

Different humans are attending a performance every night (performer and audience). Can we utilise this resources of “varying data”?

Can this data be processed by approaches we know from the field of Data Science?

Could this change a performance, e.g. by varying musical parameters (“remix”) or visuals?

What kind of (biomedical) data? How can these be processed artistically?
 How does the data go into the DAW/video program? How is it made performable for the musicians?

To what extent can one build on projects already carried out by the Helmholtz partners?

Can biomedical imaging and AI be used to find connections between vital signs and poetry/music and can that be used artistically?

Is it possible to find connections between “inspiration” and “subordinate work” through Deep Data, and again use that artistically?

The research goal is the development of a three part prototype performance.

Part I: An actor recites the poem 'The Sphynx' by Ralph Waldo Emerson (1803-1882). 
While reciting, an MRI-video of the actor’s heart, which has been taken beforehand (also while reciting) at Berlin Ultrahigh Field Facility, will be shown in sync.
In the context of the Oedipus myth, the Sphinx is said to have guarded the entrance to the Greek city of Thebes, asking a riddle to travellers to allow them passage. If they could not answer, she would devour them (Edmunds, Lowell. The Sphinx in the Oedipus Legend, Hain: 1981).

“Thou art the unanswered question;

Couldst see they proper eye,

Alway it asketh, asketh;

And each answer is a lie

So take thy quest through nature,

It through thousand natures ply

Ask on, thou clothed eternity;

Time is the false reply.”

Excerpt of 'The Sphynx', Ralph Waldo Emerson

Prof. Niendorf and the seven tesla MRI at BUFF © MDC

Part II: An orchestra performs 'The Unanswered Question (revised version 1930/1935) by Charles Ives, which features compositional techniques particular to the twentieth century (such as “reordered chronologies and the layering of seemingly independent material”) and has been inspired by 'The Sphynx' - in particular in terms of “the language, imagery, structure, and worldview” (McDonald, Matthew. Silent Narration? Elements of Narrative in Ives's The Unanswered Question. 19th century music Vol. 27, Issue 3: 2004). 
Meanwhile, humanoid data of the audience is being collected, e.g. heartbeat, ECG, or data of the 
emotional state via 
facial recognition.

“The strings play ppp thoughout with no change in tempo.

They are to represent “The Silence of the Druids -

Who Know, See and Hear Nothing”.

The trumpet intones “The Perennial Question

of Existence”, and states it in the same tone of

voice each time. But the hunt for “The Invisible

Answer” undertaken by the flutes [the woodwinds]

and other human beings, becomes gradually

more active, faster and louder (…)”

Foreword to the 1953 print edition by Charles Ives

Part III: The sets of (humanoid) data from part I and II are being contextualised by AI and processed into a live orchestra remix and visuals. The Data is being converted into the OSC protocol so the output of the AI can be used to trigger and manipulate events in Ableton Live (by Ableton AG) and other software. The live musicians get their… (section has yet to be written)

The proof of concept has been ….scaled down… problems… (section has yet to be written).

I acknowledge the Helmholtz Information & Data Science Academy (HIDA) for providing financial support enabling a short-term research stay at Max Delbrück Center (MDC) to learn about biomedical imaging and provide the MRI and a three month research stay at the German Aerospace Center (DLR) to develop technical solutions for this project and create the required software.

The research stay as part of the fellowship program at Academy for Theater and Digitality (ATD) in Dortmund is funded by the Wilo-Foundation.

Project developed @AKADEMIE FÜR_THEATER_UND_DIGITALITÄT.

  • projects/theunansweredquestion/start.txt
  • Last modified: 06.10.2021 11:00
  • by Martin Hennecke