Spotify working on algorithm that knows when you’re sad or angry and recommends music by listening to your voice

The algorithm would be able to tell emotional state, age, gender, and accents from background sounds

(Getty Images for Spotify)

Spotify has patented a new technology that will let it use speech recognition to determine users  “emotional state, gender, age, or accent” so that it can better recommend content to them.

The patent, called  “Identification of taste attributes from an audio signal”, would use information about “intonation, stress, rhythm and the likes of units of speech” that Spotify could use to tell whether a user is “happy, angry, afraid, sad or neutral”, reports MusicBusinessWorldwide.

Spotify is developing this technology because asking users to provide personal information, such as their likes and dislikes, requires them “to tediously input answers to multiple queries in order for the system to identify the user’s tastes”.

“What is needed is an entirely different approach to collecting taste attributes of a user”, Spotify’s patent reads, so that human activity is “at least partially eliminated”.  

Numerous features “such as power spectrum density of frequency at maximum power can carry speaker information”, the patent continues, while age can be determined by pitch, as well as language or accent.

“Numerous other charactisations and classifications can be used”, meaning that Spotify could listen for more identifying features, and would separate those out from background noises such as vehicles outside or other people talking.  

This metadata would be taken with other information that Spotify has already gathered about users, such as their music library or friends’ listening history, which would then suggest actions.

“In one example, the output might simply be to play the next content,” the patent states. “In another example, the output might be a recommendation on a visual display.

“Accordingly, in one aspect, the output is audio output of music corresponding to the preferences. In another example aspect, the output is a display of recommended next music tracks corresponding to the preferences”.

The patent was filed in February 2018, and was granted on 12 January 2021, but the company is also working on numerous other technologies.

Spotify has also patented a method that would use artificial intelligence to detect plagiarism in music. It would use an algorithm to calculate how similar the song is and define which part of the song could be violating another artist’s copyright.  

Spotify imagines that the plagiarism risk detector could annotate songs “in real-time”, showing where in the track it might have occurred, annotating the lead sheet “in the form of a plugin” to existing software.