2024-01-10 07:15:43
Sounds are a fundamental part of any film or television series. If it is necessary to create them, those responsible can incorporate room effects, also known as Foley effects. These are sounds obtained in a studio using everyday materials, so that they sound like real sounds, fit the image and are perfectly integrated into the scene. Productions with a lower budget usually use sound libraries, a cheaper resource but which offers less natural results.
Now, however, artificial intelligence (AI) provides creators with a third option. Based on cutting-edge techniques in this field, researchers from the Polytechnic University of Madrid (UPM) in Spain have developed a system for the synthesis of room effects. And it already has to its credit the first short film made in Spain that has used artificial intelligence for this purpose.
The application is called Foley-VAE. The acronym, corresponding in English to variational autoencoder, refers to an artificial intelligence architecture that has the ability to generate completely new sounds. “After feeding it with sounds downloaded from a database, our software extracts the latent characteristics, which synthesize the fundamental aspects of the sounds,” says Mateo Cámara, assistant professor at the Higher Technical School of Telecommunications Engineers (ETSIT). “Extracting latent features involves exposing them, so that a user can modify them and observe how the sounds vary. It is at this point where we open the door for a Foley artist to [como se denomina al creador de efectos de sala] act.”
What does each variable represent? It cannot be known a priori, admits Cámara. “However, our software allows you to modify it in real time and hear how the original sound has changed.” She illustrates this with a couple of examples: “If modifying a variable makes the audio seem more watery, we can label it as amount of water. Another variable could produce a more electrifying, lightsaber-like sound, which we would call lightsaber quantity. Thus, the creator can put his personal name to each variable and generate effects based on the variations that he decides.
Mateo Cámara. (Photo: UPM)
“It seems like magic, but our system is capable, from normal and ordinary natural sounds, of generating infinite new mixtures, just as a Foley artist would do with a material,” continues Cámara, who is developing his doctoral thesis on the use of artificial intelligence in this field within the Signal Processing Applications Group. “Suppose we want to create a step. A Foley artist would take the ground he wants to step on and record himself doing it. We would take recordings of footprints on that floor and generate new steps, with the same quality with which the artist would do it and with virtually infinite variations, or new details added.”
In addition to this potential, the researcher highlights the ease of use of Foley-VAE, which does not require any technical knowledge to create the effects. All it takes is for the user to move a few sliding buttons in a simple interface. But feeding artificial intelligence “is still a very technical task,” he acknowledges. “We are working to simplify it.”
The short film with which the tool was tested is titled “The Witness” and its director is Alberto Kampmann, a graduate in telecommunications engineering from the ETSIT. Its protagonists carry a fantastic object, in the shape of a cylindrical bar, for which a sound had to be imagined. “I fed the artificial intelligence with all kinds of elements that could fit: sounds of water, tides, electromagnetic noises recorded in gadgets in my laboratory, sonar from submarines, matches lighting, birds chirping…” explains Cámara. “After a while playing with the software, almost as if we were mixing clay, we found several sounds that we liked. “We recorded them and sent them to post-production so they could integrate them into the short film.”
Foley-VAE was presented at the 54th Spanish Congress of Acoustics (Tecniacústica 2023), last October. (Source: UPM)
#Generate #sound #effects #artificial #intelligence