Ten days ago I filed stablyai/orca#30114: the full sharded E2E run
on Orca’s main failed 8 specs on every run. None of them were regressions. Each one still expected behavior the
product had changed on purpose, so a red nightly run said nothing about whether anything had actually broken. The
fix went up the same day as #30115, test files only. Today it merged.
What the four specs that merged check now:
| Spec | What the product changed | What the spec checks now |
|---|---|---|
linear-url-workspace-entry, workspace-emoji-picker | Cmd+J also searches tasks (#28617) | the placeholder Search chats, terminals, workspaces, tasks, settings, and actions... |
workspace-emoji-picker | the worktree details dialog was renamed | a dialog named Workspace details |
os-opened-documents | CSV/TSV files opened from the OS are editable | a grid named CSV editor with gridcells, not a table |
profile-state-save-delay | the save-delay warning can be closed (#28900) | the toast’s only button is its close button |
Half of it got fixed while it waited
The PR sat unreviewed for ten days. In that time main fixed four of the eight specs itself, in a test PR, a
notification fix and a repository-wide comment cleanup, and my branch picked up conflicts. Before my own rebase
reached the branch, a maintainer had done it: rebased onto main, dropped the hunks main already covered, tightened
two assertions, and merged it with my authorship and a co-author line.
Fixes for stale tests race the code that made them stale. Next time I’d send one small PR per spec, so the ones nobody else touches can land without waiting for the rest.
“Fails on main too” is not a diagnosis
After the rebase, one more test in a touched file still failed: pasting a Linear URL into the composer never looked
the issue up. It failed the same way in main’s nightly run, so I called it pre-existing and out of scope.
The maintainer traced it to a change from three days earlier, #33270, which made Create worktree read Linear’s connection status from the host over IPC instead of the app’s store. The test only stubbed the store, so Linear looked disconnected and the URL was never looked up. One more stub, and the spec passed in the same PR.
A failure that also happens on main tells you your diff didn’t cause it. It doesn’t tell you it has nothing to do
with the files you’re in.
If you contribute from a fork
- Orca’s E2E workflow runs on third-party runners a fork can’t use, so runs on a fork queue forever. I ran the specs
from a fork-only branch with
runs-onswitched toubuntu-24.04. - In my fork, with Actions enabled, upstream’s scheduled workflows started running after a Sync fork and queued up the same way. I disabled them all and enable one only for the run I want.