One night, pass by pass.
The same sequence runs every night, one process per project, each pass writing rows the next one reads. Here it is in order, with what every pass takes in, what it leaves behind, and when it fires.
Which projects run tonight is a database question.
Cron calls one dispatcher with a pass name. It asks the database which projects qualify, then fans out a process for each, four at a time, with a fifteen minute ceiling on any one. No project name is hardcoded, so joining the pipeline is a flag rather than a deploy.
Each pass asks a different question. Content passes want a project flagged for them. Listening passes want an active topic. Answer engine tracking wants its own config row, since it costs a call per question per engine per day. The repost scout wants two facts at once: a client who posts socially, and a topic already producing daily snapshots.
Collecting, from four in the morning
Weekly lines fire only on their day. A source a project has not configured is skipped and logged, and the pass carries on.
vault_pullevery hour, at :45
Lists the project research folder in storage and downloads only what changed, keyed on each file's updated_at. No model runs here. Everything later in the night reads this local copy.
market04:00 UTC, daily
An end-of-day price and volume pull for any tracked topic carrying a ticker, from a public feed needing no key, so a chart of chatter gets a price line underneath. A failed fetch leaves the previous rows and the night continues.
volume04:15 UTC, Mondays
Absolute monthly search volume for terms that already have a trend curve, batched into one task per project. The source restates the last twelve months on every call, so a skipped week is recoverable.
sentiment04:30 UTC, daily
Pulls recent items from whichever sources a topic configured (Reddit, X, search results, Yelp, Google Maps, Tripadvisor), scores each with a model, and appends them. The rollup the dashboard charts gets rebuilt from those rows. Two days back, a hundred items, since collection is billed per item fetched.
brand_posts04:40 UTC, Mondays
Pulls the brand's own Instagram and TikTok history, one row per post, updated in place, so a report can mark owned content against the sentiment line. Owner replies already sitting in review payloads get lifted out too, at no provider cost.
viral04:50 UTC, daily
The same posts, filed differently. Rather than overwrite a post with its latest size, this appends one row per post per day. Day one seeds the series, velocity shows up on day two, acceleration on day three.
aeo05:10 UTC, daily
Puts the project's configured questions to the answer engines and records whether the brand was cited, by alias and domain matching rather than a model. A skipped day is gone for good: no engine can be asked what it would have said last March.
Then the night has to produce something.
Everything past this point reads what the collectors wrote minutes earlier, which is why the clock times are what they are. The scout runs after the snapshot it ranks, and ahead of the pass that would otherwise draft over the top of it.
synopsis05:00 UTC, Mondays
One narrative per brand topic: a verdict, chapters, the events behind them, recommendations. It runs after Monday's collectors, so the week it describes is in the database before it starts writing.
repost_scout05:20 UTC, daily
Reads the viral snapshots through the velocity function and turns the top movers into repost cards with a suggested caption. It buys nothing from a provider, and sits ahead of the drafting pass so the cheapest content of the night counts toward the social target first.
imagine_actions05:30 UTC, daily
The drafting pass. It reads the cadence rows, works out how far behind the period is, and writes only the shortfall, capped at two content types a night. A piece citing a number absent from the source it was shown gets dropped. What survives lands as a suggested card with its goal, its documents and its concepts attached.
learnings05:00 UTC, Sundays
Reads the week of board activity: what was accepted, what was rejected and for what stated reason, and the edits people made to drafts. It writes one short note out of that. A week nobody touched produces no note.
email_weekly06:00 UTC, Sundays
Builds the digest and sends it, only to projects whose configured send hour matches the run. Trends are ordered by the size of the week-over-week move, so a small term climbing outranks a large one sliding.
The clock, in the order it fires
Every line is a cron entry, in UTC. The weekly and hourly passes are left out so the daily spine stays readable; the hourly two (vault pull at :45, capture triage at :25) run all day.
Which of these fires for you depends on what your project has configured.
- 04:00
- market price overlay written for tracked tickers
- 04:30
- sentiment topics swept, items scored, observations appended
- 04:50
- viral one row per post per day, velocity and acceleration
- 05:10
- aeo answer engines asked, citations recorded
- 05:20
- repost_scout top movers turned into repost cards
- 05:30
- imagine_actions cadence shortfall drafted, evidence attached
- 06:00
- email_weekly digest sent to the projects on this hour
- now
- your turn
What does a starved pass do?
The failure worth worrying about is the one that looks like success. A mistyped handle returns an empty dataset with no error attached. A provider over its monthly cap returns a rejection that gets swallowed. Both read as a healthy zero, and a healthy zero hid three real faults here for days.
So the rule is blunt. A pass configured to fetch something that fetched nothing exits with an error code. Green has to mean it worked.
| Pass | The situation | What it does about it |
|---|---|---|
| viral, brand_posts | Handles configured, zero rows written | Exits non-zero. The dispatcher records it as failed and it goes into the alert. |
| triage_captures | Captures waiting, nothing classified and nothing discarded | Exits non-zero, so an empty hour and a broken hour look different. |
| repost_scout | A second consecutive night with no candidate, on a project set up for it | Exits non-zero. One flat night is legitimate, so it takes two. |
| learnings | Reviews sitting in the window and no note written | Exits non-zero. Activity producing nothing is a fault. |
| imagine_actions | Clean exit, zero cards inserted | Recorded as empty rather than ok, unless it printed a line saying zero was correct. |
| typed graph | More than half the documents fail extraction | Marked failed despite the clean exit, so the graph cannot rot quietly. |
| any pass | Still running after 900 seconds | Killed, and logged as a timeout carrying whatever it printed first. |
All of it runs while the office is closed, and none of it publishes.
Each pass ends in a row in a table or a card in a queue. Cards arrive marked suggested and wait there. Approve one and it lands as a draft in the tool you publish from, research still attached, and the publish button stays with a person. Where a draft can land is a separate page.
spot-bench-empty.png1280 × 832If you want the shorter version, the homepage tells it in four steps.
Request access