feat(media): add image / video / audio surfaces with unified od media generate dispatcher
Extends Open Design from web-only to a multi-modal creation tool. The unifying contract is one code-agent loop driven by skills + project metadata + prompt constraints; for non-web surfaces the agent shells out to a single dispatcher (`od media generate`) that the daemon routes per (surface, model). - Types: new Surface union, MediaAspect / AudioKind, image/video/audio ProjectKind + ProjectMetadata fields, video/audio ProjectFileKind. - NewProjectPanel: top-level surface picker + Image / Video / Audio forms with model, aspect, length, duration, voice, audio-kind pickers. - ExamplesTab + DesignSystemsTab: surface filter row that scopes before mode / scenario / category filters. - FileViewer / FileWorkspace: native <video> and <audio> previews and matching tab icons. - Daemon: parses `od.surface` and `> Surface:` blockquotes; recognises mp4 / webm / mov / mp3 / wav / ogg / m4a / flac extensions; spawns agents with OD_BIN / OD_DAEMON_URL / OD_PROJECT_ID / OD_PROJECT_DIR env so any code-agent CLI with shell access can call the dispatcher. - daemon/media.js + daemon/media-models.js: surface-agnostic dispatcher with stub providers that emit deterministic placeholder bytes (1x1 PNG, valid mp4 ftyp, mp3 frame / silent WAV) so the framework works without API keys; real provider integrations slot in later. - daemon/cli.js: `od media generate --surface ... --model ...` subcommand routes to POST /api/projects/:id/media/generate and prints one JSON line for the agent to parse. - prompts/media-contract.ts: hard contract pinned LAST in the system prompt for image/video/audio surfaces — env vars, exact invocation, registered model IDs per surface, six workflow rules. system.ts metadata block updated to point at the contract. - Seed skills: image-poster, video-shortform, audio-jingle each ship a SKILL.md with `mode/surface: image|video|audio` and a stylized example.html preview, and instruct the agent to dispatch via the contract. Made-with: Cursor
This commit is contained in:
pftom
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: video-shortform
|
||||
description: |
|
||||
Short-form video generation skill — 3-10 second clips for product
|
||||
reveals, motion teasers, ambient loops. Defaults to Seedance 2 but
|
||||
works the same with Kling 3 / 4, Veo 3 or Sora 2. Output is one MP4
|
||||
saved to the project folder. When the workspace also ships an
|
||||
interactive-video / hyperframes skill, prefer composing several short
|
||||
shots into a single timeline rather than one long monolithic clip.
|
||||
triggers:
|
||||
- "video"
|
||||
- "clip"
|
||||
- "shortform"
|
||||
- "reel"
|
||||
- "短视频"
|
||||
- "动效"
|
||||
od:
|
||||
mode: video
|
||||
surface: video
|
||||
scenario: marketing
|
||||
preview:
|
||||
type: html
|
||||
entry: example.html
|
||||
design_system:
|
||||
requires: false
|
||||
example_prompt: |
|
||||
5-second product reveal — ceramic coffee mug rotating on a soft
|
||||
paper backdrop, warm side-light from camera-left, micro dust motes
|
||||
drifting through the beam. Cinematic, 16:9, slow drift on the camera.
|
||||
---
|
||||
|
||||
# Video Shortform Skill
|
||||
|
||||
Short-form (≤ 10s) is the sweet spot for current text-to-video models —
|
||||
they're great at one **shot** with one **idea**, weaker at multi-cut
|
||||
narratives. Plan one shot per call.
|
||||
|
||||
## Resource map
|
||||
|
||||
```
|
||||
video-shortform/
|
||||
├── SKILL.md
|
||||
└── example.html
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 0 — Read the project metadata
|
||||
|
||||
`videoModel`, `videoLength` (seconds), `videoAspect`. These are
|
||||
hard-locks — clamp the prompt to whatever the chosen model supports
|
||||
(Seedance 2 caps at 10s; Kling 4 supports up to 10s + image-to-video;
|
||||
Veo 3 supports 8s with audio).
|
||||
|
||||
### Step 1 — Plan the shot
|
||||
|
||||
Write the shotlist BEFORE calling the model:
|
||||
|
||||
| Slot | Content |
|
||||
|---|---|
|
||||
| Subject | What's in frame? |
|
||||
| Camera | Static / pan / push-in / orbit? |
|
||||
| Lighting | Key direction + temperature |
|
||||
| Motion | What moves, at what pace? Subject motion vs camera motion. |
|
||||
| Sound | Ambient bed? (only if the model supports audio) |
|
||||
|
||||
Show this to the user as a one-sentence plan before dispatching — they
|
||||
can redirect cheaply.
|
||||
|
||||
### Step 2 — Compose the prompt
|
||||
|
||||
Use the format the upstream model prefers (Seedance: motion + camera +
|
||||
mood; Kling: subject + camera + style; Veo: subject + cinematography +
|
||||
sound). Bind the project's `videoAspect` and `videoLength` directly to
|
||||
the API parameters; never put them in prose.
|
||||
|
||||
### Step 3 — Dispatch via the media contract
|
||||
|
||||
Use the unified dispatcher — do **not** call provider APIs by hand:
|
||||
|
||||
```bash
|
||||
node "$OD_BIN" media generate \
|
||||
--project "$OD_PROJECT_ID" \
|
||||
--surface video \
|
||||
--model "<videoModel from metadata>" \
|
||||
--aspect "<videoAspect from metadata>" \
|
||||
--length <videoLength seconds> \
|
||||
--output "<short-slug>-<seconds>s.mp4" \
|
||||
--prompt "<assembled shot prompt from Step 2>"
|
||||
```
|
||||
|
||||
The command prints one line of JSON: `{"file": {"name": "...", ...}}`.
|
||||
The bytes land in the project; the FileViewer plays it automatically.
|
||||
|
||||
### Step 4 — Hand off
|
||||
|
||||
Reply with: shot summary, the filename returned by the dispatcher, and
|
||||
one sentence on what to try if the user wants a variation.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- One shot per turn. Multi-shot timelines belong in a hyperframes /
|
||||
interactive-video skill, not here.
|
||||
- Match `videoAspect` exactly — re-renders are slow.
|
||||
- Never ship a video without saving the file — the user expects
|
||||
something to play in the file viewer.
|
||||
- When the underlying model fails (NSFW filter, content policy,
|
||||
timeout), report the error verbatim. Don't silently retry.
|
||||
@@ -0,0 +1,90 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Short-form video — example</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0e0d0c;
|
||||
--panel: #1a1816;
|
||||
--ink: #f5efe5;
|
||||
--muted: #8b8579;
|
||||
--accent: #c96442;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { margin: 0; padding: 0; background: var(--bg); color: var(--ink);
|
||||
font-family: 'Iowan Old Style', 'Charter', Georgia, serif; }
|
||||
body { min-height: 100dvh; display: grid; place-items: center; padding: 32px; }
|
||||
.stage {
|
||||
width: min(720px, 92vw);
|
||||
background: var(--panel);
|
||||
border-radius: 8px;
|
||||
padding: 22px;
|
||||
box-shadow: 0 24px 60px rgba(0,0,0,0.45);
|
||||
}
|
||||
.frame {
|
||||
position: relative;
|
||||
aspect-ratio: 16 / 9;
|
||||
border-radius: 6px;
|
||||
overflow: hidden;
|
||||
background:
|
||||
radial-gradient(circle at 30% 35%, #d8b08b 0%, #6f4a35 40%, #1a120c 80%);
|
||||
}
|
||||
.frame::after {
|
||||
content: ''; position: absolute; inset: 0;
|
||||
background: repeating-linear-gradient(0deg, rgba(0,0,0,0.18) 0 1px, transparent 1px 4px);
|
||||
pointer-events: none;
|
||||
animation: scan 12s linear infinite;
|
||||
}
|
||||
@keyframes scan { from { background-position-y: 0; } to { background-position-y: 200px; } }
|
||||
.frame .mug {
|
||||
position: absolute; left: 50%; top: 56%; transform: translate(-50%, -50%);
|
||||
width: 28%; aspect-ratio: 1 / 1;
|
||||
background: radial-gradient(ellipse at 35% 35%, #f5efe5 0%, #c2b8a7 50%, #6f6757 100%);
|
||||
border-radius: 18% 18% 22% 22% / 28% 28% 18% 18%;
|
||||
box-shadow: 18px 6px 30px rgba(0,0,0,0.45);
|
||||
animation: turn 6s ease-in-out infinite alternate;
|
||||
}
|
||||
.frame .mug::after {
|
||||
content: ''; position: absolute; right: -14%; top: 28%;
|
||||
width: 18%; height: 44%;
|
||||
border: 6px solid #c2b8a7; border-left: none; border-radius: 0 100% 100% 0 / 0 50% 50% 0;
|
||||
}
|
||||
@keyframes turn { from { transform: translate(-50%, -50%) rotate(-6deg); } to { transform: translate(-50%, -50%) rotate(6deg); } }
|
||||
.frame .timecode {
|
||||
position: absolute; left: 14px; bottom: 12px;
|
||||
font-family: ui-monospace, 'SF Mono', Menlo, monospace;
|
||||
font-size: 11px; letter-spacing: 0.16em;
|
||||
color: var(--muted);
|
||||
background: rgba(0,0,0,0.4);
|
||||
padding: 4px 8px; border-radius: 999px;
|
||||
}
|
||||
.frame .badge {
|
||||
position: absolute; left: 14px; top: 12px;
|
||||
font-family: ui-monospace, 'SF Mono', Menlo, monospace;
|
||||
font-size: 10.5px; letter-spacing: 0.2em; text-transform: uppercase;
|
||||
color: var(--accent);
|
||||
}
|
||||
.meta {
|
||||
display: grid; grid-template-columns: 1fr auto; gap: 10px;
|
||||
align-items: end; margin-top: 18px;
|
||||
}
|
||||
.title { font-size: 22px; line-height: 1.1; margin: 0; }
|
||||
.sub { font-family: ui-monospace, 'SF Mono', Menlo, monospace; font-size: 11px; color: var(--muted); letter-spacing: 0.14em; text-transform: uppercase; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="stage">
|
||||
<div class="frame">
|
||||
<span class="badge">● REC</span>
|
||||
<div class="mug" aria-hidden></div>
|
||||
<span class="timecode">00:05 · 16:9 · seedance-2</span>
|
||||
</div>
|
||||
<div class="meta">
|
||||
<h1 class="title">A 5-second product reveal — saved as MP4.</h1>
|
||||
<span class="sub">Open Design · Video</span>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user