Convi - AI Personalized Videos

See video below for a full walkthrough of the project! I built a complete software as a service and deployed it on a Next.js web-app, which allows users to generate personalized videos at scale using deep fake technology. Users record samples of their voice once, records one video, and then thousands of videos can be generated with specific word replacement in each video (e.g. first name, company name, job title). This is combined with a recording of the receiver's website in the background of each video, making for a "Loom-like" feel. The web-app part of the project included designing the UI (using Tailwind CSS), and building the backend and dynamic functionality on the webapp with Next.js (deployed on Vercel). Here, I styled everything and made custom components for most things (used Headless UI for some parts). I set up the MySQL database on PlanetScale, and used Prisma ORM to interface with it. On the cloud side of the project, I created a video generation pipeline that can generate videos every 2 minutes in parallel and in batches of 25 (meaning, an entire batch of videos would be completed in about 1 hour). In order to do this, I set up the following. 1. Set up a script that uses Chrome to connect to the receiver's website on a Linux virtual machine on AWS Batch. This records the website, moves the mouse around, and scrolls up and down. 2. Once recorded, a series of different AWS Lambda functions will be invoked to synthesize the speech, perform lip synchronization using AI, and edit the video together using ffmpeg commands. 3. Finally, I built a custom script and set it up on AWS Lambda that uploads the video to Loom, and that's the pipeline. All AI inference is set up on Beam.cloud serverless GPUs for more throughput and cheaper cloud costs at the same time. For the AI and deepfake technology, I researched and read world-class voice cloning research paper and implemented the best cutting-edge AI tech into our software system, and set it up on AWS SageMaker to fine tune to new voices at scale. Of course, the audio samples from Convi users requires preprocessing and labelling, so I also had to build something that transcribes the audio, normalizes the volume, and removes noise in order to get the best dataset to train the models on. Once the deep learning models are trained using AWS SageMaker, I set up Serverless GPU inference with Beam (because it's cheaper than GPUs on AWS Batch). This allows for both mass production of voice clones, as well as scalable audio generation on the cloud at a low cost due to it being serverless. Since all environments had to be tightly controlled, I set up all the cloud and AI parts using Docker images and hosted them on private AWS ECR repositories (including the AWS Lambda functions).