Sony has already promised that the PS5 with its built-in 3D audio processing unit will revolutionize gaming sound. Now it turns out that Sony’s next step is to develop AI-powered game soundtracks that respond to how you play, or to your emotions, and adapt the music accordingly.
The ‘Dynamic Music Creation in Gaming’ patent was first discovered by Game Rant, and although the PlayStation 5 is not specifically mentioned, it has all the features of a feature that Sony is considering for the upcoming console.
In the patent, the technicians of Sony describe how they intend to use machine learning to analyze how different ‘musical components’ such as rhythm, melody and harmony map to specific emotions, and then create different emotional variations on music scores that will evoke specific feelings in gamers as they play.
First, Sony lets its composers create musical ‘motifs’ that capture a specific character, in-game activity, locations, or even the ‘personality’ of the player. Then, these motifs are encoded to play based on Sony’s data about what the player feels or should feel at any given time.
For example, on the PS5 Sony can take over the world music from Assassin’s Creed Valhalla and lower the tempo and percussion for a quieter atmosphere when the player has stopped moving for a while, increase the tension when they have fixed themselves on a target, or add some soulful violin when a beloved character turns up.
Specific emotions that this dynamic system would try to evoke are Tension, Power, Joy, Miracle, Tenderness, Transcendence, Peace, Nostalgia, Grief, Sensuality, and Fear. The engineers plan to analyze tons of popular music in order to transform emotion-inducing music into a science. Their machine learning system will convert music into electronic scores, tune the scores with reviews from critics and fans, and then timestamp specific pieces of music corresponding to specific emotions.
Sony then plans to experiment with ‘mapping emotions to scenarios’. During a stressful moment, instead of playing predictably tense music, they claimed that an undercurrent of sensuality could have a ‘better and more interesting effect’ on your experience.
Although we do not know when the ‘Compositional Engine’ will be considered viable, it is certain that the end of 2020-bound PS5 could work as an attempt to differentiate the console from the Xbox Series X. Many of the current video games already adapt their music based on the actions or events of players in the game, but not with this level of subtlety and personalization.
Sony wants to customize the soundtrack based on who you are, with categories such as “young, old, male, female, introverted, extroverted”. But the patent explores how their musical system can go much further than such basic descriptions. Gamers could opt-in to provide their social media data to Sony for personality and behavioral analysis. Gamers could also use ‘biometric devices’ to provide data such as ‘electrodermal activity, pulse, and respiration, body temperature, blood pressure, brain wave activity, genetic predisposition’.
In order to keep track of this information, the engineers listed some potential biometric devices that could track users’ data: a ‘thermal or IR camera’, a ‘portable activity tracker’, or a ‘smartwatch with bio-sensing’. From there, the system would take your body data and adjust the soundtrack. Imagine you’ve died several times from a troublesome boss and the PS5 could feel your increased heart rate, interpret this as frustration, and adjust the soundtrack to play something soothing.
Many PS5 owners will feel uncomfortable when Sony knows so much about their health and personality. But if Sony’s machine learning can create profiles of a subset of gamers, they can probably predict average emotional responses to certain game segments. Then they can extrapolate how each gamer might react to a particular scene and adjust the music accordingly. Regardless of privacy concerns, this patent indicates that Sony is interested in tracking gamers’ health in other contexts. Nintendo has been researching sleep-tracking technology for years and the exercise game Ring Fit Adventure was a big hit.
It is possible that Sony has similar aspirations for the PlayStation 5, either to create its own tracking tech or to connect existing smartwatches and fitness trackers of gamers to the PS5 to deliver real-time data.