A new software called Lipdub has been released by Captions, an AI-powered video editing business, to translate footage into 28 other languages.
French, Hindi, Spanish, Italian, Portuguese, Japanese, and more languages are supported by Lipdub, which is a free download from the App Store. Users of the software may even convert films into baby babble, Gen Z, pirate, and Texas lingo. The demonstration video demonstrates how the software may alter lip movement by the chosen target language. There is occasionally a slight delay between the audio and the lip movement, though.
Users can interpret and post videos of one person speaking for up to one minute on other social networking sites.
According to Captions’ website, the self-titled video editing program has been used by more than 3 million creators. According to the startup, it has more than 100,000 daily users. The Captions app has several AI-powered video editing tools, including removing “hums” and “ahs,” lowering background noise, and improving pronunciation. Additionally, the app contains an “AI Lipdub” tool that may alter lip movement during post-production editing if the transcript is changed.
Gaurav Misra, who had previously served as Snap’s head of design engineering, started Captions in 2021. The business received $25 million in a Series B investment in June that was co-led by Kleiner Perkins and included Sequoia Capital, Andreessen Horowitz (a16z), and SV Angel. Funding for Captions has totaled $40 million so far.
A rising trend is using AI dubbing and translation to reach more people. YouTube said in June that it is testing a machine learning-based solution that would enable users to subtitle their videos in other languages automatically. Even better lip-syncing is being worked on, according to the business. The business said last month that it is integrating AI-powered dubbing into YouTube Studio for customers wishing to translate videos into other languages for simpler access.
ElevenLabs’ dubbing tool, which supports 29 languages, was also published on the AI-powered voice-generating platform earlier this month. As previously reported by Rest of the World, organizations that provide dubbing services make millions of dollars by translating videos for well-known YouTubers like MrBeast.
Startups that use AI for dubbing have drawn significant investor attention, with companies like UK-based Papercup and Israel-based Deepdub raising millions of dollars.
