Files

T

pftom ac70719d4d feat(media): add image / video / audio surfaces with unified od media generate dispatcher

Extends Open Design from web-only to a multi-modal creation tool. The
unifying contract is one code-agent loop driven by skills + project
metadata + prompt constraints; for non-web surfaces the agent shells
out to a single dispatcher (`od media generate`) that the daemon
routes per (surface, model).

- Types: new Surface union, MediaAspect / AudioKind, image/video/audio
  ProjectKind + ProjectMetadata fields, video/audio ProjectFileKind.
- NewProjectPanel: top-level surface picker + Image / Video / Audio
  forms with model, aspect, length, duration, voice, audio-kind pickers.
- ExamplesTab + DesignSystemsTab: surface filter row that scopes
  before mode / scenario / category filters.
- FileViewer / FileWorkspace: native <video> and <audio> previews and
  matching tab icons.
- Daemon: parses `od.surface` and `> Surface:` blockquotes; recognises
  mp4 / webm / mov / mp3 / wav / ogg / m4a / flac extensions; spawns
  agents with OD_BIN / OD_DAEMON_URL / OD_PROJECT_ID / OD_PROJECT_DIR
  env so any code-agent CLI with shell access can call the dispatcher.
- daemon/media.js + daemon/media-models.js: surface-agnostic dispatcher
  with stub providers that emit deterministic placeholder bytes
  (1x1 PNG, valid mp4 ftyp, mp3 frame / silent WAV) so the framework
  works without API keys; real provider integrations slot in later.
- daemon/cli.js: `od media generate --surface ... --model ...`
  subcommand routes to POST /api/projects/:id/media/generate and
  prints one JSON line for the agent to parse.
- prompts/media-contract.ts: hard contract pinned LAST in the system
  prompt for image/video/audio surfaces — env vars, exact invocation,
  registered model IDs per surface, six workflow rules. system.ts
  metadata block updated to point at the contract.
- Seed skills: image-poster, video-shortform, audio-jingle each ship a
  SKILL.md with `mode/surface: image|video|audio` and a stylized
  example.html preview, and instruct the agent to dispatch via the
  contract.

Made-with: Cursor

2026-04-28 22:40:58 +08:00

3.6 KiB

Raw Blame History

name, description, triggers, od

name

description

triggers

audio-jingle

Audio generation skill — jingles, beds, voiceover, and sound effects. Routes music requests to Suno V5 / Udio / Lyria, speech to MiniMax TTS / FishAudio / ElevenLabs V3, and SFX to ElevenLabs SFX or AudioCraft. Output is one MP3/WAV file saved to the project folder.

music

jingle

bed

voiceover

tts

sound effect

音乐

配音

音效

mode

surface

scenario

preview

design_system

example_prompt

audio

marketing

type	entry
html	example.html

requires
false

A 30-second upbeat indie-pop jingle for a coffee shop launch — warm electric piano lead, brushed drums, gentle bass, a single sun-soaked "ahhh" choir on the chorus. No vocals. Loop-friendly tail.

Audio Jingle Skill

Three sub-modes. The active project's audioKind decides which one runs:

`audioKind`	Models we route to	Plan focus
`music`	Suno V5 (default), Udio, Lyria 2	genre + tempo + instrumentation
`speech`	MiniMax TTS (default), Fish, ElevenLabs V3	script + voice + pacing
`sfx`	ElevenLabs SFX (default), AudioCraft	texture + impact + duration

Resource map

audio-jingle/
├── SKILL.md
└── example.html

Workflow

Step 0 — Read the project metadata

audioKind, audioModel, audioDuration (seconds), and (for speech) voice. Branch by audioKind and use the values verbatim — no clarifying form unless something is marked (unknown — ask).

Step 1 — Plan

Music

Genre + reference artists (1-2)
Tempo (BPM) + key
Instrumentation (3-5 instruments max)
Vocals: yes / no / hummed / choir
Mood arc (intro → chorus → outro)

Speech

Script (final, not draft — TTS runs verbatim)
Voice description (warmth, age, accent, pacing)
Pronunciation hints for proper nouns / acronyms

SFX

Texture (impact / whoosh / ambience / foley)
Duration + envelope (sharp attack vs. gentle swell)
Layering note (single hit vs. stacked)

State the plan in 2-3 sentences before dispatching.

Step 2 — Compose the prompt

Use the format the upstream model prefers. Bind audioDuration to the API parameter directly; never put "make it 30 seconds" in prose.

Step 3 — Dispatch via the media contract

Use the unified dispatcher — do not call provider APIs by hand:

node "$OD_BIN" media generate \
  --project "$OD_PROJECT_ID" \
  --surface audio \
  --audio-kind "<music|speech|sfx>" \
  --model "<audioModel from metadata>" \
  --duration <audioDuration seconds> \
  --voice "<voice (speech only)>" \
  --output "<short-slug>-<duration>s.mp3" \
  --prompt "<assembled prompt from Step 2 — for speech, the literal script>"

The command prints one line of JSON: {"file": {"name": "...", ...}}. The bytes land in the project; the FileViewer renders the audio transport controls automatically.

Step 4 — Hand off

Reply with: plan summary, the filename returned by the dispatcher, and one sentence on what to try if the user wants a variation (e.g. "swap tempo from 92 to 108 BPM" rather than "make it different").

Hard rules

TTS runs your script literally. Proof it before dispatching — even one stray comma changes the cadence.
Music: under 30s = single section; 30–90s = intro + body; 90s+ = full arc. Don't try to fit a 3-act song into 15 seconds.
SFX: prefer one well-described layer over a paragraph of "make it cool" — generators reward specific texture words.
Save the file every turn. The audio viewer shows transport controls the moment the file lands.

3.6 KiB Raw Blame History Unescape Escape