feat(media): add image / video / audio surfaces with unified od media generate dispatcher

Extends Open Design from web-only to a multi-modal creation tool. The
unifying contract is one code-agent loop driven by skills + project
metadata + prompt constraints; for non-web surfaces the agent shells
out to a single dispatcher (`od media generate`) that the daemon
routes per (surface, model).

- Types: new Surface union, MediaAspect / AudioKind, image/video/audio
  ProjectKind + ProjectMetadata fields, video/audio ProjectFileKind.
- NewProjectPanel: top-level surface picker + Image / Video / Audio
  forms with model, aspect, length, duration, voice, audio-kind pickers.
- ExamplesTab + DesignSystemsTab: surface filter row that scopes
  before mode / scenario / category filters.
- FileViewer / FileWorkspace: native <video> and <audio> previews and
  matching tab icons.
- Daemon: parses `od.surface` and `> Surface:` blockquotes; recognises
  mp4 / webm / mov / mp3 / wav / ogg / m4a / flac extensions; spawns
  agents with OD_BIN / OD_DAEMON_URL / OD_PROJECT_ID / OD_PROJECT_DIR
  env so any code-agent CLI with shell access can call the dispatcher.
- daemon/media.js + daemon/media-models.js: surface-agnostic dispatcher
  with stub providers that emit deterministic placeholder bytes
  (1x1 PNG, valid mp4 ftyp, mp3 frame / silent WAV) so the framework
  works without API keys; real provider integrations slot in later.
- daemon/cli.js: `od media generate --surface ... --model ...`
  subcommand routes to POST /api/projects/:id/media/generate and
  prints one JSON line for the agent to parse.
- prompts/media-contract.ts: hard contract pinned LAST in the system
  prompt for image/video/audio surfaces — env vars, exact invocation,
  registered model IDs per surface, six workflow rules. system.ts
  metadata block updated to point at the contract.
- Seed skills: image-poster, video-shortform, audio-jingle each ship a
  SKILL.md with `mode/surface: image|video|audio` and a stylized
  example.html preview, and instruct the agent to dispatch via the
  contract.

Made-with: Cursor
This commit is contained in:
pftom
2026-04-28 22:40:58 +08:00
parent bc7c057216
commit ac70719d4d
28 changed files with 2902 additions and 78 deletions
+121
View File
@@ -0,0 +1,121 @@
---
name: audio-jingle
description: |
Audio generation skill — jingles, beds, voiceover, and sound effects.
Routes music requests to Suno V5 / Udio / Lyria, speech to MiniMax
TTS / FishAudio / ElevenLabs V3, and SFX to ElevenLabs SFX or
AudioCraft. Output is one MP3/WAV file saved to the project folder.
triggers:
- "music"
- "jingle"
- "bed"
- "voiceover"
- "tts"
- "sound effect"
- "音乐"
- "配音"
- "音效"
od:
mode: audio
surface: audio
scenario: marketing
preview:
type: html
entry: example.html
design_system:
requires: false
example_prompt: |
A 30-second upbeat indie-pop jingle for a coffee shop launch — warm
electric piano lead, brushed drums, gentle bass, a single sun-soaked
"ahhh" choir on the chorus. No vocals. Loop-friendly tail.
---
# Audio Jingle Skill
Three sub-modes. The active project's `audioKind` decides which one
runs:
| `audioKind` | Models we route to | Plan focus |
|---|---|---|
| `music` | Suno V5 (default), Udio, Lyria 2 | genre + tempo + instrumentation |
| `speech` | MiniMax TTS (default), Fish, ElevenLabs V3 | script + voice + pacing |
| `sfx` | ElevenLabs SFX (default), AudioCraft | texture + impact + duration |
## Resource map
```
audio-jingle/
├── SKILL.md
└── example.html
```
## Workflow
### Step 0 — Read the project metadata
`audioKind`, `audioModel`, `audioDuration` (seconds), and (for speech)
`voice`. Branch by `audioKind` and use the values verbatim — no
clarifying form unless something is marked `(unknown — ask)`.
### Step 1 — Plan
**Music**
- Genre + reference artists (1-2)
- Tempo (BPM) + key
- Instrumentation (3-5 instruments max)
- Vocals: yes / no / hummed / choir
- Mood arc (intro → chorus → outro)
**Speech**
- Script (final, not draft — TTS runs verbatim)
- Voice description (warmth, age, accent, pacing)
- Pronunciation hints for proper nouns / acronyms
**SFX**
- Texture (impact / whoosh / ambience / foley)
- Duration + envelope (sharp attack vs. gentle swell)
- Layering note (single hit vs. stacked)
State the plan in 2-3 sentences before dispatching.
### Step 2 — Compose the prompt
Use the format the upstream model prefers. Bind `audioDuration` to the
API parameter directly; never put "make it 30 seconds" in prose.
### Step 3 — Dispatch via the media contract
Use the unified dispatcher — do **not** call provider APIs by hand:
```bash
node "$OD_BIN" media generate \
--project "$OD_PROJECT_ID" \
--surface audio \
--audio-kind "<music|speech|sfx>" \
--model "<audioModel from metadata>" \
--duration <audioDuration seconds> \
--voice "<voice (speech only)>" \
--output "<short-slug>-<duration>s.mp3" \
--prompt "<assembled prompt from Step 2 — for speech, the literal script>"
```
The command prints one line of JSON: `{"file": {"name": "...", ...}}`.
The bytes land in the project; the FileViewer renders the audio
transport controls automatically.
### Step 4 — Hand off
Reply with: plan summary, the filename returned by the dispatcher, and
one sentence on what to try if the user wants a variation (e.g. "swap
tempo from 92 to 108 BPM" rather than "make it different").
## Hard rules
- TTS runs your script **literally**. Proof it before dispatching —
even one stray comma changes the cadence.
- Music: under 30s = single section; 3090s = intro + body; 90s+ =
full arc. Don't try to fit a 3-act song into 15 seconds.
- SFX: prefer one well-described layer over a paragraph of "make it
cool" — generators reward specific texture words.
- Save the file every turn. The audio viewer shows transport controls
the moment the file lands.