When Ludwig von Beethoven died in 1827, he was three years from the completion of his Ninth Symphony, a work hailed by many as his magnum opus. He had started working on his 10th Symphony but, due to deteriorating health, could not make much progress: he left only a few musical sketches.
Since then, Beethoven fans and musicologists have been perplexed and complained about what could have been. His notes made fun of a magnificent reward, even if it seemed forever out of reach.
Now, thanks to the work of a team of music historians, musicologists, composers and computer scientists, Beethoven’s vision will come to life.
I chaired the artificial intelligence part of the project, at the head of a group of scientists from the creative AI startup AI form of play who taught a machine both the whole of Beethoven’s work and his creative process.
A full recording of Beethoven’s 10th Symphony is scheduled for release on October 9, 2021, the same day as the scheduled world premiere in Bonn, Germany – the culmination of an effort spanning more than two years.
Past attempts hit a wall
Around 1817, the Royal Philharmonic Society of London commissioned Beethoven to write his ninth and tenth symphonies. Written for an orchestra, symphonies often contain four movements: The first is played at a fast tempo, the second at a slower tempo, the third at a medium or fast tempo, and the last at a fast tempo.
But when it comes to the 10th Symphony, Beethoven didn’t leave much behind except a few musical notes and a handful of ideas that he wrote down.
There have been some past attempts to reconstruct parts of Beethoven’s 10th Symphony. Most famous, in 1988, musicologist Barry Cooper ventured to complete the first and second movements. He wove together 250 bars of music from the skits to create what he thought was a production of the first movement who was true to Beethoven’s vision.
However, the rarity of Beethoven’s sketches prevented symphonic experts from going beyond this first movement.
Assemble the team
At the start of 2019, Dr Matthias Röder, director of the Karajan Institute, an organization in Salzburg, Austria that promotes music technology, contacted me. He explained that he was forming a team to complete Beethoven’s 10th Symphony on the occasion of the composer’s 250th birthday. Aware of my work on AI generated art, he wanted to know if the AI would be able to help fill in the blanks left by Beethoven.
The challenge seemed sizeable. To achieve this, the AI would have to do something it had never done before. But I said I was going to give it a go.
Röder then put together a team that included Austrian composer Walter Werzowa. Famous for writing Intel signature bang jingle, Werzowa was tasked with implementing a new type of composition that would integrate what Beethoven left behind with what the AI would generate. Mark Gotham, an expert in computer music, led the effort to transcribe Beethoven’s sketches and process all of his work so that AI could be properly trained.
The team also included Robert levin, a musicologist at Harvard University who also happens to be an incredible pianist. Wine had already finished a number of incomplete 18th century works by Mozart and Johann Sebastian Bach.
The project takes shape
In June 2019, the group reunited for a two-day workshop at the Harvard Music Library. In a large room with a piano, blackboard and a stack of Beethoven sketchbooks covering most of his known works, we explained how fragments can be turned into a complete piece of music and how AI can help. to solve this puzzle, while remaining faithful. to Beethoven’s process and vision.
Music experts in the room were eager to learn more about the kind of music AI has created in the past. I explained to them how the AI managed to generate music in the manner of Bach. However, it was only a harmonization of an input melody that sounded like Bach. It didn’t come close to what we needed to do: build an entire symphony from a handful of phrases.
Meanwhile, the scientists in the room – myself included – wanted to know more about the type of materials available and how the experts plan to use them to complete the symphony.
The task at hand finally crystallized. We would need to use full notes and compositions from all of Beethoven’s work – as well as the available sketches of the 10th Symphony – to create something that Beethoven himself could have written.
It was a huge challenge. We didn’t have a machine to which we could send sketches, push a button, and spit out a symphony. Most AIs available at the time couldn’t continue an unfinished piece of music beyond a few extra seconds.
We would need to push the boundaries of what creative AI could do by machine-teaching Beethoven’s creative process – how he would take a few bars of music and painstakingly develop them into soulful symphonies, quartets, and sonatas.
Reconstructing Beethoven’s Creative Process
As the project progressed, the human side and the machine side of the collaboration evolved. Werzowa, Gotham, Levin and Röder deciphered and transcribed the sketches of the 10th Symphony, trying to understand Beethoven’s intentions.
Using his completed symphonies as a model, they attempted to piece together the puzzle of where the sketch fragments should go – what movement, what part of the movement.
They had to make decisions, such as determining whether a sketch indicated the starting point of a scherzo, which is a very lively part of the symphony, typically in the third movement. Or they could determine that a line of music was probably the basis of a fugue, which is a melody created by interweaving parts that all echo a central theme.
The AI side of the project – my side – found itself struggling with a series of difficult tasks.
First, and more fundamentally, we had to figure out how to take a short phrase, or even just a pattern, and use it to develop a longer, more complicated musical structure, just like Beethoven would have done. For example, the machine must have learned how Beethoven constructed the Fifth Symphony from a basic four-note pattern.