Edit talking-head videos with Claude Code: the complete toolkit

I want to set up a workspace for "AI talking-head video editing" on this Mac (Apple silicon). Please install and verify each of the tools below. For each one, check whether it's already installed and only install it if it's missing. Before downloading anything over 100MB, tell me the file name, source and size, and wait for my approval.

1. Homebrew (if missing, install it with the official command from https://brew.sh)
2. Xcode Command Line Tools: xcode-select --install (needed for git and swift)
3. FFmpeg: brew install ffmpeg
4. Node.js: brew install node
5. uv: brew install uv
6. Python packages: pip3 install pillow mlx-whisper faster-whisper
7. Create a project folder at ~/video-studio/_build and run npm init -y, npm i playwright, npx playwright install chromium
8. Confirm swift can call Apple Vision: write a minimal swift script that runs VNRecognizeTextRequest on a test image and prints the text

When everything is done, give me a checklist: tool name, version and whether it works. If anything fails, explain why and how to fix it. Don't skip anything.

The environment is ready. Now help me prepare free assets for editing. Before each download, tell me the source, licence and size, and wait for my approval.

1. Sound effects (all CC0: commercial use allowed, no attribution required):
   - In ~/ai-tools/sfx, git clone https://github.com/Calinou/kenney-interface-sounds.git
   - In ~/ai-tools/sfx/elements, use curl to download these 17 files from https://tryelements.dev/sfx/<name>.mp3: boom click drone error glitch keyboard magic-reveal notification pop reverse-whoosh riser shutter success swoosh whoosh whoosh-alt1 whoosh-alt2. Then check each one opens with ffprobe.
2. ACE-Step 1.5 music generator (MIT licence, commercial use allowed):
   - git clone https://github.com/ACE-Step/ACE-Step-1.5.git into ~/ai-tools, then run uv sync
   - Start the server with the macOS API script (start_api_server_macos.sh, port 8001). The first start downloads 10–17GB of models; confirm with me first.
   - Following the official API docs (docs/en/API.md), generate a 30-second test track: tech electronic, instrumental, suitable as background for a talking-head video. Copy it to ~/ai-tools/test-bgm.wav and tell me its loudness in LUFS.
3. Transcription model: load mlx-community/whisper-large-v3-turbo with mlx-whisper and run it once on any short audio clip so the model downloads and caches.

Finally, list where each asset lives, its licence and how much space it uses.

I want to edit a talking-head video. The footage is in: [your video folder path]
My website is: [your URL]. Use its colours and overall style as the reference.
For animation ideas, look for pieces that fit this video on https://prompt-motion.com/.

If there is an editing-SOP.md in the project folder, read it first and follow its rules.
Follow this process and wait for my approval at each step:
1. Understand the content first: transcribe with the transcription tool installed in this environment (mlx-whisper on Mac, faster-whisper on Windows, both with the large-v3-turbo model) and read the whole script. Use ffprobe to check the footage specs, and extract a frame every few seconds into a contact sheet to understand the layout.
2. Then look on prompt-motion.com and GitHub for new tools and animation ideas for this video, and propose 1–3 concrete "breakthroughs" for it.
3. Write an edit plan: how to cut (remove pauses, drop repeated lines), which animation goes with each line (landing on specific words), caption style, output format (1080×1920 by default), music and sound effects. For any product or platform mentioned in the video, ask me for the official logo first.
4. After I approve, make a sample of about 10 seconds:
   - Write the animation in HTML, capture it frame by frame with Playwright as transparent PNGs, and layer it over the video with FFmpeg
   - Generate two background music options (A/B) with ACE-Step; use the CC0 sound effects in ~/ai-tools/sfx
   - The music should sit about 15 LU below the voice: audible, but never competing with it
5. Once the sample is approved, make the full version, then the cover.
6. Before each delivery, check it yourself: contact sheets of frames, loudness measurements. Then tell me what you checked.
7. Before mechanical steps like rendering and compositing, stop and remind me to switch the model to Sonnet to save allowance; remind me to switch back to Opus when design decisions are needed.

No company logo or watermark on screen (some platforms throttle them). Snap every cut and segment length to whole frames so the animation never drifts from the picture.

I want to set up a workspace for "AI talking-head video editing" on this Windows computer. Please use PowerShell and winget to install and verify each item. For each one, check whether it's already installed and only install it if it's missing. Before downloading anything over 100MB, tell me the file name, source and size, and wait for my approval.

0. First check whether this computer has an NVIDIA GPU and how much VRAM it has (nvidia-smi). Tell me the result; the steps below depend on it.
1. Git for Windows: winget install Git.Git
2. FFmpeg: winget install Gyan.FFmpeg
3. Node.js: winget install OpenJS.NodeJS.LTS
4. Python 3.12: winget install Python.Python.3.12
5. uv: winget install astral-sh.uv
6. Python packages: pip install pillow faster-whisper rapidocr onnxruntime mediapipe
   - With an NVIDIA GPU: follow the Windows instructions in the official faster-whisper README to add the CUDA 12 cuBLAS and cuDNN 9 libraries, and confirm faster-whisper runs with device="cuda"
7. Create a project folder at C:\video-studio\_build and run npm init -y, npm i playwright, npx playwright install chromium
8. Run one small test of each:
   - RapidOCR: read a screenshot containing Chinese and English text, and print the text and positions
   - MediaPipe: run person segmentation on a portrait photo and save a black-and-white mask
   - faster-whisper: transcribe a short Chinese or English clip with the turbo model and print per-word timestamps

If a newly installed command isn't found, remind me to reopen PowerShell before continuing.
When everything is done, give me a checklist: tool name, version, whether it works, and whether it uses the GPU or CPU. If anything fails, explain why and how to fix it. Don't skip anything.

The environment is ready. Now help me prepare free assets for editing. Before each download, tell me the source, licence and size, and wait for my approval.

1. Sound effects (all CC0: commercial use allowed, no attribution required):
   - In C:\ai-tools\sfx, git clone https://github.com/Calinou/kenney-interface-sounds.git
   - In C:\ai-tools\sfx\elements, download these 17 files from https://tryelements.dev/sfx/<name>.mp3: boom click drone error glitch keyboard magic-reveal notification pop reverse-whoosh riser shutter success swoosh whoosh whoosh-alt1 whoosh-alt2. Then check each one opens with ffprobe.
2. Background music:
   - With an NVIDIA GPU (6GB+ VRAM): git clone https://github.com/ACE-Step/ACE-Step-1.5.git into C:\ai-tools, install it following the Windows instructions in the official README (or use the official portable package), and run start_api_server.bat to start the API. The first start downloads over 10GB of models; confirm with me first. Then, following docs/en/API.md, generate a 30-second test track: tech electronic, instrumental, suitable as background for a talking-head video.
   - Without an NVIDIA GPU: don't install ACE-Step yet. Find 2–3 commercially licensed royalty-free or CC0 music sources, summarise each licence, and wait for my decision.
3. Transcription model: load the turbo model with faster-whisper and run it once on any short audio clip so the model downloads and caches.

Finally, list where each asset lives, its licence and how much space it uses.

# Talking-Head Video Editing SOP (student edition)

> How to use: save this file as `editing-SOP.md` in your project folder. Each time you start, tell Claude Code: "Read editing-SOP.md first and follow it."
> Compiled by WINS AI CLASS from the real editing of three videos (EP.29–31, SOP v1.6). Works on Mac and Windows.
> After each video, ask Claude Code to add a row to the change log at the end and fold new lessons into the rules.

---

## 0. Principles

> **Top rule: every video should be a little better than the last.**
> This SOP is the floor, not the ceiling. Before each video, understand the content first, then look online for new tools, styles and animation ideas, and decide what will make this one better than the previous one.

1. **Plan first, build after approval.** The order is fixed: edit plan → sample (about 10 seconds) → full version → cover. Wait for my approval at each step.
2. **Missing resources are not a reason to stop.** If music, sound effects, footage or a tool is missing, search the web and GitHub for free or open-source options and list the exact files, licences and sizes for me to choose from.
3. **Ask before downloading.** State the file name, source, size and licence, and download only after I agree.
4. **Never touch keys or passwords.** Do not read or print any API key or password.
5. **Save usage: one new conversation per video.** Do not edit several videos in one conversation. The longer a conversation gets, the more it re-reads on every step, and usage grows several times over.
6. **Save usage: before the mechanical steps, stop and remind me to switch to Sonnet** (see section 4).
7. **Check before you deliver.** Check the picture with contact sheets of frames and the audio with loudness measurements. Nothing counts as "done" until it has been checked. Whether the music sounds good is my call, so say so.

---

## 1. Confirm before starting (ask in your first reply)

| Item | Notes |
|---|---|
| Output format | Vertical video, 1080×1920 by default |
| Platform | On platforms that throttle branded posts (for example Xiaohongshu/RedNote), **no company logo and no watermark** on the cover or the video |
| Background music | Generate original music locally, two versions (A/B) for me to choose |
| Ending | Whether to add an end card or a "next video" teaser |
| Repeated lines and slips | List them and suggest cuts; I decide |
| Products or platforms mentioned | **Ask me for the official logos first**; no home-made placeholder icons |
| Series tag | My series name and episode number (for example "XXX · EP.01"), used in the opening and on the cover |
| Cover text | A main title and one subtitle; you propose, I confirm |

---

## 2. Brand style

- First read my website: [your URL]. Extract its colours, fonts and overall style into a set of colour and font variables, and build every animation, caption and cover from them.
- Captions: placed where they don't cover the face, on a dark translucent box, revealed word by word, with key words in the accent colour.
- Chinese fonts: PingFang on Mac. Windows has no PingFang, so specify Noto Sans SC (Google Fonts). For English, Geist or another clean sans-serif works well.
- Not allowed: company logo and watermark.

---

## 3. Tools

| Purpose | Mac | Windows |
|---|---|---|
| Cutting, compositing, mixing, loudness | FFmpeg | FFmpeg |
| Rendering animation (HTML captured frame by frame as transparent PNGs) | Node.js + Playwright | Node.js + Playwright |
| Transcription with per-word timestamps | mlx-whisper + large-v3-turbo | faster-whisper + turbo |
| On-screen text recognition (annotations that follow the footage) | Apple Vision (via swift) | RapidOCR |
| Person cut-out (text behind the speaker) | Apple Vision (via swift) | MediaPipe or rembg |
| Background music | ACE-Step 1.5 | ACE-Step 1.5 (NVIDIA GPU recommended) |
| Sound effects | Kenney Interface Sounds, Elements (CC0) | Same |
| Animation ideas | prompt-motion.com | Same |

Write each function as a **small reusable script** in the project's `_build/` folder: transcription, pause detection, caption draft, frame rendering, compositing, still checks, cover capture. For the next video, copy `_build/` as the starting point.

---

## 4. Workflow

### Step 0 — Understand the content, then find what's new (required)
1. **Understand the content**: transcribe and read the whole script. Work out the core message, which lines deserve visual emphasis, and what viewers should remember.
2. **Look for something new online** (every video, not just reusing the last one):
   - prompt-motion.com: browse kinetic type, diagrams, charts, particles, product UI and similar categories for animation ideas that fit this video, and note the links.
   - GitHub: look for new open-source tools (animation, captions, music, sound, cut-outs, tracking, 3D) and confirm the licence allows commercial use.
3. **Set 1–3 "breakthroughs" for this video** at the top of the plan. They must be concrete, doable and serve the content, and say how this video improves on the last. Test a new tool on a small sample first; if it fails, fall back to the usual method without delaying delivery.

### Step 1 — Analyse the footage
- Check resolution, frame rate and duration with ffprobe. **Phone footage is often variable frame rate**, so always conform to 30fps when compositing.
- Extract a frame every 3–5 seconds into a contact sheet to understand the layout (for example, screen on top and face below). Find split lines programmatically, not by eye.

### Step 2 — Transcribe
- Use the most accurate model (large-v3-turbo) and add this video's proper nouns (product names, technical terms) to the transcription prompt.
- **Judge homophones from context.** Where every model mishears the same word, ask me to confirm at delivery.
- The last word of a sentence is often given too long a duration, so set out-points from the actual pause.

### Step 3 — Edit plan (for my approval)
- Start with "breakthroughs for this video", with reference links.
- Remove pauses between sentences (keep 0.05–0.1 s). Keep pauses shorter than about 0.45 s, to avoid too many jump cuts.
- Find lines that repeat the same point and suggest cutting them.
- List the animations as a table: time / exact words / animation. **Every animation lands on a specific word.**
- List what I need to decide (see section 1).

### Step 4 — Edit timeline
- ⚠️ **Snap every segment's in-point and length to whole frames** (multiples of 1/30 s at 30fps) and force the exact frame count when compositing. Otherwise small errors add up across many segments and the animation drifts away from the picture.
- Alternate segments between normal framing and a ~10% push-in to hide jump cuts.
- Record caption and animation times in **source time** and convert them to output time in code, so animations stay aligned when cuts change.

### Step 5 — Build the animation
- Write animation in HTML. Every frame is drawn by `seek(t)`, and the same time always draws the same frame (use seeded randomness).
- Each scene checks its time range and hides outside it. Anything that appears "at a moment" defaults to `opacity: 0` in CSS.
- Give each scene's CSS classes a unique prefix so styles don't collide.
- **No text wider than the frame minus margins** (on a 1080-wide frame, about 1000 px max). Entrance scale-ups max out at 1.35.
- **Composite in layers**, bottom to top: source video → behind layer (big text and background behind the speaker) → cut-out of the speaker → picture-in-picture → front layer (captions, annotations).
- Text behind the speaker needs headroom. If the head touches the top edge, shrink the speaker into a bust (edges fade out) and paint an opaque brand background in the behind layer.
- Picture-in-picture clips (real videos, screenshots) must be padded with transparent frames from 0 s, or they drift out of sync with the other layers.
- When re-creating conversations or reports on screen, **quote them word for word; never invent lines**.
- Videos over 1 minute: add a progress bar and chapter cards (one chapter every 30–40 s), and give viewers a new visual change at least every 5–8 seconds.

### Step 6 — Check stills first (the biggest time saver)
- Pick 7–14 key moments, render one frame each, composite them over the video into a contact sheet, and only then render the whole thing.
- Look for: text running off-frame or wrapping, elements that never appear or never leave, wrong colours or fonts.

### 🔁 Model switch reminder (every video)
Use **Opus** for design and judgement; use **Sonnet** for mechanical work like running commands and waiting on renders (Sonnet uses roughly one fifth of Opus's allowance).

| Stage | Model |
|---|---|
| Steps 0–6: content, research, plan, animation, still checks | Opus |
| **Steps 7–8: rendering, compositing, loudness, frame checks** | **Sonnet** |
| Big design changes after review | Back to Opus |
| Step 9: cover (frame choice and layout) | Opus |

When the still checks are approved and rendering is next, **stop** and send a separate reminder:
> "Next up is rendering and compositing, which is mechanical work. Switch to Sonnet in the model picker to save about 4/5 of the allowance, then tell me when you're done."

Continue after I switch, and remind me to switch back to Opus when design work resumes.

### Step 7 — Sample (about 10 seconds)
- Render only the first few segments. With two music candidates, make A/B versions with identical picture.

### Step 8 — Full version
- Before delivery, check a contact sheet of the whole video sampled every few seconds, and verify that total frames in the base video equal total frames in the timeline.
- Overall loudness about **−15 LUFS**, true peak no higher than −1 dBFS.
- Keep intermediate files (frames, base video) in the project's `_build/cache/`, never in the system temp folder (it gets cleaned). Delete the cache only after I confirm the video is final.
- Deliver the full version only. I'll ask if I need a compressed copy.

### Step 9 — Cover (1080×1920)
- Fixed elements: series tag, big main title, one subtitle, a photo frame with a glowing border, logos of products mentioned in the video (tilted, overlapping the frame edge), and a closing line at the bottom (ideally my own words from the video). No company logo.
- Photo: build a contact sheet of candidate frames and pick one with **eyes open and an expression that fits the topic**.
- Cut-out method: inside the photo frame, from bottom to top: brand background → grid → a huge outlined keyword (2–4 characters or one short word) → the cut-out speaker. The head should overlap the lower half of the keyword for depth. If the speaker is far from the camera, scale the cut-out up (at most 1:1 with the source).
- **Only scale down, never up.** Upscaling blurs.
- Check: title fits, logos don't cover the face, tags stay inside the frame. Open the finished file and look at it before delivering.

---

## 5. Sound effects

- Give each animation a sound effect, with trigger times recorded in source time.
- Common pairings: glitch for the opening; boom when the title lands; whoosh family for transitions; whoosh-alt for cards flying in; magic-reveal plus confirmation for reveals; keyboard for typing; success for completion; drop for nodes appearing; tick for progress and levels.
- Volume guide: transitions 0.35–0.55, UI sounds 0.4–0.5, typing about 0.15–0.18, ambient beds about 0.15–0.2.

---

## 6. Background music (ACE-Step)

1. Start the ACE-Step API server (Mac: `start_api_server_macos.sh`; Windows: `start_api_server.bat`). The first start downloads over 10 GB of models.
2. Describe the style in words, for example: tech electronic, instrumental, leaves room for a voiceover. Generate two versions (A/B).
3. Copy generated files into the project's `_build/music/` or they will be cleaned up.
4. Tracks often open with a quiet intro: offset the music so the beat lands exactly when the opening hook ends.
5. For long videos, use section markers (Intro / Build / Drop / Breakdown / Outro) to generate sectioned music, and align the peak with the most important chapter.
6. **Mixing standard**: duck the music under speech (ratio about 3). Before delivery, measure it separately: the music should sit **about 15 LU below the voice**. More than 18 LU below is close to inaudible, so measure each track and adjust its volume.

---

## 7. Common pitfalls

| Pitfall | Fix |
|---|---|
| Stopping because there's no music | Find open-source and CC0 resources, or generate with ACE-Step |
| Proper nouns misheard (Claude → "Cloud") | Add proper nouns to the transcription prompt and proofread |
| Captions unreadable on a white wall | Add a dark translucent box |
| Text or cards running off-frame or wrapping | Cap text width and add `nowrap` |
| Placeholder icons instead of real logos | Ask for official logos before starting |
| Blurry cover from upscaling, or a mid-blink frame | Pick from a contact sheet; scale down only |
| Company logo or watermark added | Never add them |
| CSS class names colliding and overriding sizes | Prefix class names per scene |
| Flash-type elements fully visible by default, washing out the video | Default to `opacity: 0` |
| Frame errors adding up across segments, misaligning at joins | Snap to whole frames and verify total frame count |
| Picture-in-picture starting mid-way and shifting timing | Pad with transparent frames from 0 s |
| Original footage showing between re-created scenes | Keep a continuous background in the behind layer |
| Music ducked so hard it's inaudible | About 15 LU below the voice; measure every track |
| Intermediate files in the temp folder get wiped | Keep them in the project's `_build/cache/` |
| Several videos in one conversation burn through usage | New conversation per video |

---

## 8. Delivery checklist

- [ ] Full video: 1080×1920, 30fps, H.264, about −15 LUFS
- [ ] Cover: 1080×1920, no company logo
- [ ] `_build/` project kept for later edits
- [ ] Reminder to switch to Sonnet given before rendering and compositing
- [ ] Report covers: whether each breakthrough worked, what was checked, and what I need to confirm myself (such as how the music sounds)

---

## Change log

After each video, add a row and fold what you learned into the rules above; delete anything outdated.

| Date | Video | Breakthroughs | Changes / lessons |
|---|---|---|---|
| | | | |