How to make an AI character sit next to you and talk (full local setup guide)
Imagine an AI character not just talking to you on screen, but actually sitting beside you in the same room, reacting, gesturing, and holding a full conversation. With today’s local AI tools, that’s no longer sci-fi – you can build it yourself.
From AI images to AI characters in your room
Most people’s first “wow” moment with AI comes from image generators: you type a prompt, and suddenly a realistic person appears on screen. The next natural question is: how do you pull that character off the screen and into real life?
The goal here is exactly that: take an AI-generated character, make them sit on a chair next to you, and have them talk and move as if they were really there. The whole process can be done with local tools, combining motion transfer, character swapping, and AI-driven lip-sync and gestures.
Planning and recording your base footage
The most challenging part of this effect is the opening shot where you and the AI character share the same frame. To pull this off, you first need clean base footage.
Option 1: Use a helper for motion capture
The easiest way is to ask someone to sit where the AI character will be and act out the motions and timing of the conversation. This person becomes your “driver” for motion transfer later.
Record a shot where you sit on one chair and your helper sits on the other. You talk to them as if they were the AI character. This gives you natural body language, eye lines, and timing.
Option 2: Solo method with two takes
If you don’t have a helper, you can still do it solo:
Set up your camera on a tripod so it doesn’t move.
Place two chairs in frame.
Record yourself on chair A delivering your lines.
Without moving the camera, record yourself again on chair B delivering the other side of the conversation.
In your video editor, place both clips on top of each other and mask or crop so the left side shows one version of you and the right side shows the other. This gives you a clean performance and a second body you can later replace with the AI character using motion transfer.
Cutting and preparing the motion transfer clip
For realistic compositing, you don’t want to run motion transfer on the entire frame. Instead, you crop out just the area where the character sits. This keeps the resolution high and the processing efficient.
The motion transfer model used here is Skel 2 (often referred to as “Scale 2” in the transcript), which currently works best around 720p. If you try to process the whole frame at that resolution, your character will look blurry and low detail. The solution is:
Crop a vertical slice around the character (for example, 768×1088 or 960×1080).
Use that cropped clip as the input to the motion transfer model.
You can do this cropping in a video editor, or by using a small custom tool that lets you select an exact resolution and region. The key is that the cropped video resolution must be known and consistent, because you’ll need to match it exactly later.
Swapping in your AI character with ComfyUI + FLUX/FLATTEN client
Once you have the cropped motion clip, the next step is to replace the human in that crop with your AI character. This is a character swap (or face/body swap) step.
For this, a ComfyUI workflow with a FLUX/FLATTEN-based character swap node (referred to as “Flats Two client” in the transcript) is used. The crucial detail: you must generate the swapped image at the exact same resolution as your cropped video (for example, 768×1088). That’s why ComfyUI is preferred here over some one-click tools – it lets you set custom resolutions precisely.
The process looks like this:
Take a reference frame from your cropped video where your helper (or second version of you) is clearly visible.
Feed that frame into your ComfyUI character swap workflow.
Use your AI character image as the target appearance.
Generate a swapped image that keeps the pose, lighting, and framing, but replaces the person with your AI character.
Try multiple generations if needed. You want a version where the outfit, hair, and hands look clean, because this single frame will guide the motion transfer model.
Applying motion transfer with Skel 2
Now you combine the two pieces: the cropped motion clip and the swapped AI character image.
In the Skel 2 motion transfer workflow:
Load your cropped driving video (the original human performance).
Load the swapped AI character image from ComfyUI as the target.
Set the output resolution to match the input exactly (e.g., 768×1088).
In the node that resizes images/masks, choose “match size” and link it to your source image so nothing gets stretched or rescaled unexpectedly.
Skel 2 will now generate a new video where your AI character mimics the motion of the original person – sitting, turning, gesturing – while roughly preserving the background.
In practice, the background is never perfect. There are often seams, color shifts, or slight misalignments. That’s expected and will be fixed in the compositing stage.
Blending the AI character back into the original shot
With the motion-transferred AI clip ready, you go back to your video editor to blend it into the original wide shot.
Here’s a simple compositing method:
Place the original full-frame shot on the bottom layer.
Place the Skel 2 output (the cropped AI character clip) on a layer above it.
Line up the timing so the actions match as closely as possible.
Apply a rectangular mask to the AI layer, covering only the character and a bit of surrounding area.
Add feathering to the mask edges so the two layers blend smoothly.
Round the corners of the mask to avoid visible straight edges.
By focusing the mask around the character and feathering the edges, you largely hide any background inconsistencies. The viewer’s eye is drawn to the character, and small differences in the wall or furniture become almost invisible.
This 5-second shot can easily take hours of experimentation to get right, but once you understand the workflow, you can reuse it for future scenes.
Creating simpler talking shots (single-person framing)
After the complex two-person shot, the rest becomes much easier. For close-up talking shots where only the AI character is on screen, you don’t need to worry about compositing two people together.
The workflow is similar but simpler:
Record a clean shot of a person sitting in the right lighting and position.
Use that frame as a reference for character swap in ComfyUI (again with the FLUX/FLATTEN client).
Try several swaps until you get a version where the outfit, hair, and hands look good.
Use this swapped frame as the starting image for your text-to-video model.
Because only one character is in frame, you don’t have to match two layers perfectly, and you can let the AI handle more of the background.
Cloning the perfect voice with Omni Voice
For believable AI characters, the voice is just as important as the visuals. In this setup, the voice is generated first, and the video is then created to match it.
Using a model like Omni Voice inside the 12GP environment, you can either design a new voice or clone one from a short sample:
Find or record a sample voice that fits your character’s personality.
Use the voice cloning feature to create a custom voice model.
Write your dialogue lines as text.
Generate audio clips for each line using the cloned voice.
This gives you clean, consistent audio files like “line1.wav”, “line2.wav”, etc. The emotional tone, pacing, and personality in these clips will drive the character’s performance in the next step.
If you’re new to AI tools and want a broader foundation before diving into this workflow, you might find our ChatGPT tutorial for beginners helpful for understanding prompt-based control in general.
Driving lip-sync and acting with LTX 2.3 and prompt relay
With the voice ready, it’s time to animate the character’s face and body. This is where the LTX 2.3 distill 1.1 model (inside 12GP or via a ComfyUI workflow) comes in.
Basic setup for talking-head generation
The configuration used here focuses on quality and control:
Model: LTX 2.3 distill 1.1.
Start with image: use the best character swap frame as the starting frame.
Generate video based on soundtrack and text prompt: upload the corresponding voice file.
Resolution: 1080p for crisp detail.
Frame rate: 30 fps for smooth motion.
Use the Omni LoRA to better match expressions and lip-sync to the Omni Voice output.
The model listens to the audio and tries to match mouth shapes and expressions, but the real magic comes from the prompt relay system.
What is prompt relay and why it’s a game-changer
Prompt relay lets you control what the character does at specific times in the video. Instead of one static prompt, you define a sequence of prompts tied to timestamps.
For example, a relay prompt might look like this conceptually:
0–2 seconds: “Neutral expression, looking at the interviewer, slight smile.”
2–4 seconds: “Laughs lightly, tilts head back, eyes wide.”
4–6 seconds: “Points at the camera, mischievous smile.”
6–8 seconds: “Waves goodbye, friendly but slightly creepy grin.”
By pairing this relay prompt with the Omni Voice audio, you’re effectively directing the AI actor: you control when they point, when they look at the camera, when they wave, and how their expression changes over time.
The result is a performance that feels far more intentional and “alive” than a generic talking head. The character doesn’t just move randomly; they react in sync with the line delivery.
If you’re interested in how different video models compare for this kind of work, you may also want to read our deep dive on major AI video generators.
Putting it all together: a local AI character pipeline
To recap, here’s the full pipeline to make an AI character sit next to you and talk, using only local tools:
Record base footage: Wide shot with two chairs; either use a helper or film yourself twice.
Crop the character area: Extract a vertical slice around the future AI character at a known resolution.
Character swap with ComfyUI: Use a FLUX/FLATTEN client workflow to replace the human with your AI character in a still frame, matching the crop resolution.
Motion transfer with Skel 2: Drive the AI character using the original human motion, keeping the same resolution.
Composite in your editor: Mask and feather the AI clip over the original shot to blend backgrounds.
Clone the voice with Omni Voice: Generate high-quality dialogue audio that fits your character.
Animate talking heads with LTX 2.3: Use the voice files, starting frame, Omni LoRA, and prompt relay to create expressive close-up shots.
The most time-consuming part is the initial experimentation: figuring out resolutions, matching outputs, and dialing in masks. But once you’ve built and tested this pipeline, you can reuse it to create entire conversations with AI characters who feel present, expressive, and surprisingly human.
As tools evolve, the core idea remains the same: your creativity and direction matter more than chasing the latest model. With a solid workflow and a bit of patience, you can bring your AI characters off the screen and into your world.
Comments
No comments yet. Be the first to share your thoughts!