From video to scroll animation – DIY with ffmpeg, or free with Scrollsequence vid2ssq beta.scrollsequence.com


Whether the clip came from Kling, Veo, Seedance, After Effects, or a phone, the web problem is the same: **scroll wants random access to frames**. An MP4 wants a decoder and keyframes. That is why `video.currentTime = progress * duration` jumps – especially scrolling back, especially on iOS.

The industry pattern (Apple product pages included) is:

1. Turn the video into numbered stills.
2. Draw the current still on a `<canvas>` from scroll progress.
3. Serve a lighter set on phones.

You can do that by hand. Or you can generate a production bundle with Scrollsequence’s free local pipeline.

## Part 1 – The generic DIY way

### 1. Extract frames with ffmpeg

“`
# Example: ~24 fps stills, 1600px wide WebP
ffmpeg -i clip.mp4 -vf "fps=24,scale=1600:-1" frame_%04d.webp
“`

Then **thin the set**. You rarely need every frame. A practical hero is often ~100–150 frames on desktop, fewer on mobile.

If you insist on scrubbing the MP4 anyway, re-encode with frequent keyframes first (`-g 1` or `-g 2`). That helps seeking; it still will not match true frame sequences for reverse scrub and memory control.

### 2. Scrub on a canvas (vanilla idea)

“`
<section id="pin" style="height: 300vh;">
<canvas id="seq" style="position: sticky; top: 0; width: 100%; height: 100vh;"></canvas>
</section>
“`

“`
const frames = []; // preload Image() / createImageBitmap for each frame_%04d.webp
const canvas = document.getElementById('seq');
const ctx = canvas.getContext('2d');
let current = -1;

function draw(i) {
const img = frames[i];
if (!img) return;
// size canvas to CSS * devicePixelRatio (cap at 2 on phones)
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
}

function onScroll() {
const el = document.getElementById('pin');
const rect = el.getBoundingClientRect();
const total = el.offsetHeight – window.innerHeight;
const scrolled = Math.min(total, Math.max(0, -rect.top));
const progress = total > 0 ? scrolled / total : 0;
const i = Math.round(progress * (frames.length – 1));
if (i === current) return;
current = i;
requestAnimationFrame(() => draw(i));
}

window.addEventListener('scroll', onScroll, { passive: true });
“`

That is the whole “Apple-style” trick in spirit: **progress → index → draw**, only when the index changes.

### 3. What DIY usually forgets (and why demos lag)

| Concern | DIY habit that breaks | Better default |
| — | — | — |
| File weight | One 1920px set for everyone | Separate ~1280 and ~1920 tiers |
| Memory (iOS) | Decode 150 full-HD frames | Cap DPR at 2; fewer/narrower frames on phones |
| Loading | Wait for all frames | Load every Nth frame first, fill gaps |
| Layout | Fixed-height parent fights sticky | Tall scroll section + sticky canvas |
| SEO | Text baked into frames | Real HTML overlays beside/over the canvas |
| Window Resize | When window resizes, the canvas overflows or underflows | Resize canvas and redraw on resize |

You can wire GSAP ScrollTrigger instead of the raw scroll math – same idea: scrub a number from 0→1, map to `frameIndex`.

—

## Part 2 – Free Scrollsequence path: `npx scrollsequence vid2ssq`

Scrollsequence v2 is built around the same model on every stack: **Scene → Layer → Canvas**. The asset side is a pipeline that turns raw `.mp4` / `.webm` into multi-resolution folders plus a `scrollsequence-manifest.json` that React, vanilla JS, and WordPress all consume the same way.

### Free vs optional AI (important)

From the [Asset Pipeline docs](https://docs.scrollsequence.com/v2/asset-pipeline) and [credits docs](https://docs.scrollsequence.com/v2/wordpress/credits):

* **100% free / local:** frame extraction, responsive resizing, format/quality, frame step, manifest generation (CLI uses local FFmpeg via Node / Wasm – no credits).
* **Optional / API key + credits:** `–depth` (3D depth maps), `–track "sports car"` (subject tracking), `–vision` (scene / camera / subject metadata for agents and automation).

So the default “video → scroll-ready bundle” path is free. Depth maps and vision are extras when you want WebGL tilt or agent-readable metadata.

### Run the wizard

“`
# Interactive wizard (beta package URL – see docs for current .tgz)
npx https://docs.scrollsequence.com/scrollsequence-2.0.0-beta.52.tgz vid2ssq

# Or if already installed:
npx scrollsequence vid2ssq
“`

Non-interactive example (free local processing):

“`
npx scrollsequence vid2ssq input.mp4 \
–name hero-sequence \
–sizes 1280,1920 \
–format jpg \
–quality 80
“`

Optional AI (needs `SSQ_API`):

“`
npx scrollsequence vid2ssq input.mp4 \
–name car-reveal \
–sizes 1280,1920 \
–depth \
–track "sports car" \
–vision
“`

### What you get that hand-rolled ffmpeg usually skips

1. **Multiple size tiers** (e.g. `1280` + `1920`) so phones are not force-fed desktop frames. The runtime picks a tier from canvas CSS width × devicePixelRatio.
2. **`scrollsequence-manifest.json`** – one file describing paths, frame counts, optional `files.d` depth maps, `tracking`, and `vision`.
3. **Depth maps (optional)** – grayscale per frame for WebGL 2.5D mouse / gyro tilt (shared lower-res depth across color tiers).
4. **Tracking + vision (optional)** – bounding data and scene/camera/subject JSON so an AI agent (or your own UI) can pin labels to a moving subject or reason about the shot.
5. **Same output for WordPress, React, and vanilla** – no separate “Elementor-only” asset format.

Inside Next.js / Vite, if a `public/` folder exists, the CLI writes straight to `public/<name>/` so the dev server can serve frames immediately.

### Drop the manifest into the page

**Vanilla (free engine pattern):**

“`
<div class="scrollsequence-scene" data-ssq-duration="1500" data-ssq-scrub="1">
<div class="scrollsequence-scene_inner">
<div class="scrollsequence-layer">
<div
class="scrollsequence-canvas2d"
data-ssq-media-groups='[{
"type": "manifest",
"rootDir": "/hero-sequence/",
"imageSizes": { /* from scrollsequence-manifest.json */ }
}]'
>
<canvas></canvas>
</div>
</div>
</div>
</div>

<script type="module">
import ScrollsequenceManager from 'scrollsequence';
import 'scrollsequence/scrollsequence.css';
ScrollsequenceManager.initialize().preloadAll();
</script>
“`

**React / Next.js:**

“`
'use client';
import { Scene, Layer, Canvas2d } from 'scrollsequence/react';
import manifest from '@/public/hero-sequence/scrollsequence-manifest.json';

export default function Hero() {
return (
<Scene duration={1500}>
<Layer>
<Canvas2d
mediaGroups={[{
type: 'manifest',
rootDir: '/hero-sequence/',
imageSizes: manifest.imageSizes,
}]}
/>
</Layer>
</Scene>
);
}
“`

WordPress v2 uses the same Canvas 2D + Media Tools `vid2ssq` flow (basic extraction free in-browser; AI toggles credit-based).

Install note (public beta): package currently installs from the docs `.tgz` until the npm name settles — see [docs.scrollsequence.com/v2](https://docs.scrollsequence.com/v2).

—

## Which path should you use?

| | DIY ffmpeg + custom JS | Free `vid2ssq` + Scrollsequence runtime |
| — | — | — |
| Learning / one-off | Great | Optional |
| Multi-size + manifest | You build it | Included, free |
| Depth / track / vision | DIY or other tools | Optional flags |
| WP + React + vanilla same assets | Manual | Native |
| Cost for basic convert | Your time | Free local |

DIY teaches the mechanism. `vid2ssq` is the same mechanism packaged so you are not reinventing responsive tiers and a manifest every project — and the basic convert stays free.

Docs: [Asset Pipeline & CLI](https://docs.scrollsequence.com/v2/asset-pipeline) · [Vanilla](https://docs.scrollsequence.com/v2/vanilla) · [React](https://docs.scrollsequence.com/v2/react) · [WordPress](https://docs.scrollsequence.com/v2/wordpress)

—

*Originally published on [Scrollsequence](https://beta.scrollsequence.com/blog/ai-video-scroll-animation-wordpress-elementor).*