We built an eight-agent Python system that takes long-form video and produces posted short-form clips: Whisper transcription, model-driven moment detection, an FFmpeg cut-caption-and-B-roll pipeline, and multi-account publishing — with dedicated auditor and security agents watching the other six.
Short-form clip production is a genuinely mechanical task that consumes enormous amounts of human time: watch an hour of video, find the six moments that stand alone, cut them to vertical, caption them, and post each one to several accounts on several platforms at the right times. Every step is judgement-light and tedium-heavy, which is the exact profile of work that should be automated and almost never is, because the pipeline crosses four unrelated technical domains. The reason most attempts fail is that they treat it as one program. Transcription, moment selection, video rendering and publishing have completely different failure modes, different retry semantics, different cost profiles and different rate limits. Rendering fails on a codec; publishing fails on an expired token; moment detection fails on a bad completion. A monolith handles none of these gracefully. The second reason attempts fail is account safety. Posting to multiple accounts on platforms with aggressive automation detection is how you lose the accounts. That is a first-class engineering constraint, not an afterthought.
`agents/` contains eight packages: `scout` (source discovery), `clip_ai` (moment detection), `editor` (the render pipeline), `publisher` (multi-platform posting), `growth` (performance feedback), `account_mgr` (credential and account state), `auditor` and `security`. The last two are the ones worth pointing at. Most agent systems have N agents that do work; this one has six that do work and two whose job is to watch. An `auditor` that independently checks what the pipeline produced, and a `security` agent separate from `account_mgr`, is the structure you arrive at after an automated system does something you did not intend and you realise nothing was positioned to notice. Beyond the agents there are `core`, `integrations`, `discord_hub`, `tg_hub`, `mac_relay` and `admin` — the last four being control surfaces. Chat clients as the operator interface is a pragmatic choice for a pipeline like this: you get authentication, mobile access, push notification and an audit trail of operator commands without building any of it.
Transcription runs Whisper with word-level timestamps, which is the enabling detail for everything downstream. Word timestamps are what make it possible to cut on a clause boundary rather than mid-word, and they are what make word-by-word 'pop' captions — where each word appears in time with the speech — possible at all. Moment detection sends the transcript to Claude and asks for the segments that stand alone. This is the one genuinely judgement-based step, and it is also the cheapest to get wrong in an interesting way: a model asked to find 'good moments' will happily return six mediocre ones rather than admitting there are two. Rendering is FFmpeg: cut to the segment, reframe to 9:16 at 1080x1920, burn captions, and composite B-roll. The caption path is worth a specific note because it is the kind of constraint real-world media work is full of — the FFmpeg build in use has no libass, so subtitle burning through the normal subtitles filter is unavailable, and captions are instead rendered as images and composited via the overlay filter. That is slower and uglier and it is what the environment permits. Documenting the constraint in the repository, rather than leaving the next person to rediscover it, is the actual craft.
`publisher` handles multi-platform, multi-account posting, with `account_mgr` owning credential and account state separately. Splitting those is deliberate: the thing that posts should not also be the thing that decides whether an account is healthy enough to post from, because when an account is rate-limited or flagged, the posting agent is the least trustworthy component to make that judgement. The deployment surface is small and portable — a Dockerfile, a Procfile, a `fly.toml`, a VPS bootstrap script, and SQLite for state. For a pipeline whose expensive resources are GPU-free CPU time and API quota, that is right-sized. There is no orchestration platform here, and there does not need to be one. One further component deserves naming: `mac_relay`. Some steps in a pipeline like this cannot run on a Linux host — a platform's official client may be macOS-only, or a codec or a hardware encoder may only be available there. A relay that lets a cloud-hosted pipeline hand specific steps to a machine sitting on a desk is an unglamorous but genuinely practical piece of architecture, and it is the kind of thing that only appears in a codebase after somebody has hit the wall it exists to get around.
The split into agents is not organisational tidiness; it is a response to the fact that the four stages have incompatible operational characteristics. Transcription is CPU-bound, slow and idempotent — you can retry it freely. Moment detection is a paid API call that is cheap, fast and non-deterministic, so retrying gives you a different answer rather than the same one. Rendering is CPU-bound, very slow, and produces large files that need somewhere to live. Publishing is fast, rate-limited, and irreversible. Sharing one retry policy across those four is impossible. Retrying a publish is a duplicate post; retrying a render wastes twenty minutes; not retrying a transcription throws away work that would have succeeded. Once each stage owns its own retry semantics and its own failure handling, the components stop fighting each other, and the state that has to be shared collapses to a small set of rows in SQLite. `core` and `integrations` carry what genuinely is shared — the database access, the configuration, the external clients — and `config` holds the tuning that an operator changes without touching code. That last boundary is the one most agent systems get wrong: anything an operator will want to adjust at three in the morning belongs in data, not in a constant.
FFmpeg in production is a category of difficulty that people who have not done it consistently underestimate. The command that works on your machine fails on the server because the build was compiled without a library; the filter graph that works for one input fails on a variable frame rate source; audio and video drift apart on a container change. The libass constraint in this pipeline is a perfect miniature of the whole problem: the correct, fast, standard way to burn subtitles was unavailable, so the pipeline uses a slower composite path that works everywhere. Media engineering is mostly this. Moment detection is a quality problem disguised as an inference problem. The model will always return something, and something is almost always worse than nothing when the output goes to a public account. The useful framing is not 'find the best moments' but 'find moments meeting a bar, and return none if none qualify' — and getting a language model to reliably return an empty result is meaningfully harder than getting it to return a full one. Multi-account publishing at any volume is an adversarial environment. Platforms actively detect automation, and the penalty is account loss rather than a failed request. That is why account state is a separate agent with its own view of health, and why the pipeline has a security agent at all. The engineering constraint is not throughput; it is not tripping a detector.
Built with Python, Whisper (transcription, word timestamps), Claude (moment detection), FFmpeg, yt-dlp, SQLite, Docker + Fly.io and Discord + Telegram control surfaces. Relationship: Internal build. Stage: Prototype. Period: 2026. Our role: Agent architecture, media pipeline, publishing, operational tooling Figures re-derived on 2026-08-23.