Sixty synchronous @Test bodies that could construct WebKit off the main thread are now async, and a static guard makes the rule a build step instead of a paragraph in a report. The guard is the durability argument, so this review attacked it by construction rather than by reading it — writing each evading test first and seeing whether it was caught. Three that were not are now fixed.
origin/main checkout: 60 violations, same 9 files, same per-file counts. Cross-checked against a diff-side count of added async keywords.func helper (makeController() ×45, makeUnreadyController() ×5, makeWebController() ×3). Neither the multi-line-@Test nor the raw-string fix touches that path.async alone was accepted as safe. async buys a hop to the enclosing actor, so a test in a suite with no @MainActor hops nowhere. Verified by construction — a non-isolated suite constructing a WebPage in an async test scanned clean while reproducing the whole defect.is_async was a substring test. func trap() { // make this async later exempted a synchronous body, as did a parameter typed (Int) async -> Void. Now read from the signature's effects clause.@Test counter — that counter sees a lost test, and what goes missing here is a helper. The shape is idiomatic in this target.@autoclosure arguments and local func/let inside a body are each caught. I wrote the evading test for each./* */ did not desynchronise in any shape tried. The author is more pessimistic than the code warrants.#""" whose content holds a bare """), string interpolation containing closure braces, #if blocks, multi-line strings containing //, escaped quotes, and one-line raw strings. A Swift regex literal does desync — but it fails loudly, and there are none in the target.func member walker is wholly prophylactic. Defensible for a guard that is the entire durability argument — and notably, two of the three real bugs were living inside that speculative machinery.Makefile and CHANGELOG.md. Both are pure insertions into the same region; keep both. Nothing semantic.Ready to push
The fix is correct and the count is right. I re-derived the 60 independently — running the guard against a fresh git archive origin/main tree gives exactly 60 violations in exactly the same 9 files, and a diff-side count of added async keywords matches the report's table row for row. No removed line in any of the nine Swift files is anything but a func signature, so nothing was disabled, renamed, or quietly dropped alongside the conversion.
Three real holes were found in the guard and fixed, all in the direction that marks a test safe. Two are in the exemption itself: async was accepted without requiring main-actor isolation (an async test in an unannotated suite hops nowhere and reproduces the entire defect), and is_async was a bare " async" substring test that a trailing comment could satisfy. The third is a parser loss — a type written on one line was stepped over, taking every member with it, which is precisely the loss the author's @Test cross-check is structurally unable to see.
Everything else the author left open checks out, and then some: all three "needs no rule" claims hold, and both documented residuals turn out to be caught too. After the fixes the repository is still clean and the pre-fix tree still reports the same 60 in the same 9 files, so reachability widened without moving the historical count. Build, lint and both make guards are clean.
77d1581a T-2219/T-2096: Stop live-WebKit suites aborting the shared test host 5e591bc9 T-2219: Record verification status and the T-2146 execution blocker in the report 7fe19fe1 T-2219: Seed the shared test harnesses by qualified name, and pin why a508a79d T-2219: Record the iOS Simulator control run and the T-2146 macOS blocker c3666f01 T-2219: Add the second control group to the report 10465f4d T-2219: Close the guard's non-func evasion family and check both seed lists 52f58498 T-2219: Record a one-line type declaration instead of stepping over it 8c77beaf T-2219: Require main-actor isolation, and read `async` from the effects clause 90c31e5e T-2219: Cover the guard's last two untested edges, and state two limits Prism's unit tests all run inside a single process. If any one test crashes that process, every test still waiting in the queue gets reported as failed — even though it never ran. That is how a test run ends up reporting 189 or 233 failures that are entirely fictional, hiding whatever real problem you were actually looking for.
The crash has one cause. Some tests build a web view (WebKit). WebKit insists on being created on the app's main thread and deliberately kills the process if it isn't. Because of how this particular test target is compiled, a test written the ordinary way can quietly end up running on a background thread, and then the web view it builds takes the whole process down.
Marking such a test async fixes it — provided the test's suite is also marked @MainActor, which supplies the main thread for it to move to. Sixty tests across nine files needed that keyword, and now have it. Nothing else about them changed — no assertion, no test body.
The interesting half is the second change. A previous ticket diagnosed this exact problem a month ago, fixed one file, and wrote the rule down in a report. Nothing enforced it, so sixty violations stayed put. This PR turns the rule into a build step: a small Python script scans the test sources and fails the build if any test could reach a web view unsafely. A rule nobody checks is not a rule.
Because that script is now the only thing preventing a sixty-first violation, this review tried to sneak past it — writing tests specifically designed to evade it. Three got through, and all three are now closed.
A neat detail: the suite the bug reports blamed turned out to be innocent. It is simply the slowest web-view test, so it is usually the one on screen when some other test kills the process — and it got the blame every time.
The mechanism is precise and worth internalising. A synchronous @MainActor function has no hop-on-entry — its isolation is only realised if the caller hops first. The callers of Swift Testing test bodies are macro-generated thunks compiled inside prismTests, which builds in Swift 5 language mode with SWIFT_APPROACHABLE_CONCURRENCY = YES and — unlike the app target — no SWIFT_DEFAULT_ACTOR_ISOLATION. Those thunks are nonisolated(nonsending), so they run on the caller's executor: the runner's cooperative pool. Swift 5 mode has no runtime enforcement to catch the mis-hop.
An async member of a @MainActor type is immune because the executor switch is emitted in the callee and is part of the ABI. Both halves of that sentence carry weight, which is the substance of one of this review's findings: async supplies the hop, @MainActor supplies the actor to hop to, and neither works alone.
The guard (Tools/check-webkit-test-isolation.py) is a taint analysis over a hand-rolled brace-structure walker. Seeds are direct WebKit constructors plus a short list of production types and test harnesses that build them; taint propagates to a fixpoint through a per-file helper graph, and a test is a violation if it reaches a seed without being both async and main-actor isolated.
Two design choices carry weight. The fixpoint is deliberately per file, because tainting by bare function name across the target made KeyboardScrollControllerTests.makeController() — which builds no web view at all — inherit the guilt of the identically-named helper in WebDocumentControllerTests, for 26 bogus violations. Cross-file harnesses are therefore seeded by qualified name, which cannot collide. And that hand-maintained seed list is checked in both directions: a dead entry fails the run, and so does an undeclared cross-file harness that does reach WebKit. That second direction earned its keep immediately, discovering WebNavigationPrecedenceHarness.
A static check was chosen over a runtime test because the failure is a load-dependent scheduling race — a suite capable of aborting the host passes in isolation, every time. No runnable test can pin it. What is decidable from source is whether any body is capable of the off-main touch, and that is what gets enforced.
The alternative that would have fixed this mechanically — setting SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor on the test target — was rejected as too broad to land while the suite is itself the thing under repair: it changes default isolation for ~3,900 tests and pushes every body onto the main actor, a scheduling change of unknown blast radius for live-WebKit suites that poll across await points. That is the right call, and it is recorded as revisitable alongside a Swift 6 migration.
This deserved scrutiny, because the author's own @Test-count self-check exposed two blind spots the shipped guard already had — a multi-line @Test(...) attribute made every parameterised test parse as an ordinary function, and a raw string closing mid-expression (""")) desynchronised bracket depth for the remainder of the file, leaving MermaidCSPSpikeTests parsing 0 of 3 tests. A count that survives such a fix is ambiguous: it could mean the fix was orthogonal, or that something is still unparsed.
It is the former, and it is checkable. The reason breakdown of the historical 60 is 7 direct constructions and 53 through a same-file func helper — three helper names, all told. Neither blind spot lay on that path: the raw-string desync affected files whose violations are direct, and the multi-line-attribute bug affected parameterised tests, of which none are among the 60. Separately, parse coverage across the target is 4407/4407 @Test declarations, recomputed independently rather than taken on report.
Two defects, both marking a test safe. First, scan exempted on member.is_async alone. But async buys a hop to the enclosing actor; a suite with no global-actor attribute offers none, so the body runs on the cooperative pool exactly as a synchronous one would. A struct NotIsolatedTests with an async test constructing a WebPage scanned clean while reproducing the entire defect. Worse, the rule as written in CLAUDE.md and the agent notes — "touch WebKit only from async tests" — is the rule someone would have followed straight into that shape. The walker now threads main-actor isolation from the enclosing type to its members and accepts it on the test itself; _is_test generalised into _has_attribute, since the depth-aware look-back over multi-line attributes is the same problem for both.
Second, is_async was " async" in signature — a substring test over raw source. func trap() { // make this async once the harness lands exempted a synchronous body, as did any parameter typed (Int) async -> Void. It is now read from the effects clause, past the ) closing the parameter list, with comments and string literals stripped per line during accumulation. That last detail matters in the opposite direction: the accumulator joins with a space, so stripping once from the joined result would let a // on an early line swallow the async that follows and turn a correct test into a false positive. It needed its own fixture before the mutation test could see it.
The author's anti-blindness argument is stated generally — "a parser that silently loses a declaration would look exactly like a clean repository, so scan cross-checks" — but the cross-check is asymmetric. It counts @Test attributes. A lost helper is equally silent and has no counter behind it, and a lost helper is exactly what renders a violation invisible.
That gap was reachable. In parse_members.walk, the TYPE branch recursed only when brace + 1 < stop - 1 — only when the type had interior lines. A type written entirely on its declaration line failed that test and was skipped with index = stop, discarding every member. enum Fixture { static func page() -> WebPage { WebPage() } } plus a test calling Fixture.page() reported clean. The shape is idiomatic in this very target (actor CallCounter { var count = 0; func next() -> Int { … } }, five occurrences), and the docstring, report and PR body all claimed nested types were covered by construction. Such a type is now recorded whole as a read-triggered member, gated on its text reaching a seed so that the equally common struct DummyError: Error {} is unaffected.
Beyond those, the parser holds up better than advertised. Extended raw-string delimiters survive even when the content contains a bare """; string interpolation containing closure braces, #if blocks, multi-line strings containing //, escaped quotes, and single-line raw strings all parse correctly. A Swift regex literal does desynchronise bracket depth — but it trips the @Test counter and fails loudly, and the target contains none. Failing safe on an unwritten construct is the right posture; teaching a brace counter to lex regex literals is not proportionate.
Both residuals the report concedes are in fact caught. A closure rebound to a differently-named local is caught because the read rule triggers on any mention of the closure's name, and the rebinding line mentions it. Unbalanced braces inside /* */ did not desynchronise in any shape tried.
Measured rather than asserted: of the original thirteen detection rules, three (direct seed, same-file func fixpoint, cross-file harness seeding) account for all 60 historical violations and all zero current ones. The entire non-func member walker — computed properties, observers, lazy vars, stored closures, ambient init/deinit/subscript, nested types — catches nothing that has ever existed in this repository. The report states this plainly, so it is not an overclaim; but it is the honest cost accounting, and it cuts both ways. The prophylactic machinery is where two of the three real bugs were hiding, which is an argument both for having reviewed it hard and for not growing it further without cause.
The abort was never reproduced, and the author says so rather than implying otherwise. The macOS destination cannot launch a test host at all (T-2146, testmanagerd wedged, six attempts at ~705 s each executing zero tests). The iOS Simulator evidence is weak on counts — T-2236 records that live-WebPage tests are broadly flaky there in an unchanged tree, which makes failure-count comparisons noise in both directions — but it is meaningful on population: across eight runs no failure ever landed on a converted test, and the failing set on WebContentTerminationWiringTests was identical pre- and post-fix. The fix rests on the structural argument, which is the correct place for it to rest given a probabilistic crash; T-1541 already recorded a green baseline run on a tree that demonstrably could crash.
Tools/check-webkit-test-isolation.py
Why it matters. The exemption is the one part of a guard that must not be approximate, and this one was wrong in the direction that marks a test safe. `async` buys a hop to the ENCLOSING actor; a suite with no `@MainActor` offers none, so the body runs on the cooperative pool exactly as a synchronous one would. A non-isolated suite constructing a `WebPage` in an `async` test scanned clean while reproducing the entire defect.
What to look at. Tools/check-webkit-test-isolation.py — scan's exemption test, plus `_has_attribute` / `is_main_actor` threading in parse_members
Tools/check-webkit-test-isolation.py
Why it matters. `" async" in signature` over raw source. `func trap() { // make this async once the harness lands` exempted a synchronous body, and so did a parameter typed `(Int) async -> Void`. For a guard whose stated premise is that losing detection looks exactly like a clean repository, this was a one-line hole.
What to look at. Tools/check-webkit-test-isolation.py — `_is_async_signature`, `_code_only`, and per-line stripping in the signature accumulator
Tools/check-webkit-test-isolation.py
Why it matters. The TYPE branch recursed only when the type had interior lines; a type written on one line was skipped with `index = stop`, discarding every member it declared. A test calling into it reported clean. This is the one parser loss the author's `@Test` cross-check is structurally unable to catch.
What to look at. Tools/check-webkit-test-isolation.py — walk's TYPE branch
prismTests/WebRendering/WebDocumentControllerTests.swift
Why it matters. This is the fix. It is also the change most at risk of hiding something — sixty hand edits across nine files is exactly where a test gets quietly disabled or an assertion softened. It did not happen: no removed line in any of the nine files is anything but a `func` signature, and per-file added-`async` counts match the report's table row for row.
What to look at. 27 signatures in WebDocumentControllerTests; 11/5/4/4/3/2/2/2 across the other eight files
Tools/check-webkit-test-isolation.py
Why it matters. The helper fixpoint is per-file, so a cross-file harness is covered only because someone remembered to name it in a hand-maintained list — and hand-maintained lists are what go stale. The discovery direction is the single thing standing between this guard and new test infrastructure nobody registered.
What to look at. Tools/check-webkit-test-isolation.py — verify_test_harnesses / discover_test_harnesses
prismTests/MainActorHopContractTests.swift
Why it matters. Worth checking for tautology, since a test asserting `Thread.isMainThread` inside a `@MainActor` suite could easily be vacuous. It is not: it covers the one failure mode the static guard is blind to. If a future toolchain stopped emitting the entry hop, the guard would still pass — `async` is still written there — and the host aborts would return with everything else green.
What to look at. prismTests/MainActorHopContractTests.swift:41-58
The abort is a load-dependent scheduling race: a suite capable of killing the host passes in isolation, every time. No runnable test can pin it. What is decidable from source is whether any body is capable of the off-main touch. Correct call, and it is also the only check that can find a suite nobody has named — it found MermaidRendererTests and WebDocumentControllerTests, 31 tests between them, neither ever on a ticket.
Fixing only the named suites would have left 46 of 60 standing, including the 27-test suite that is the most likely true culprit. The ticket explicitly asked for a remedy structural enough to cover an unidentified third case.
Mechanically the strongest fix, and named as "plausibly the real fix" on T-1541. Rejected because it changes default isolation for ~3,900 tests at once and pushes every body onto the main actor — a scheduling change of unknown blast radius for live-WebKit suites that poll across await points. Deferred to a Swift 6 migration. I agree: this is not the change to make while the suite is itself the thing being repaired.
Forced by measurement, not taste: target-wide bare-name tainting produced 26 false positives in KeyboardScrollControllerTests. Qualified names cannot collide, so the cross-file cases are seeded instead, and the seed list self-invalidates in both directions.
Not a tightening for its own sake — the looser rule admitted a working reproduction of the original defect. Verified clean against the whole target afterwards, which also establishes something the PR had only asserted: every existing async WebKit test really is main-actor isolated.
A file-scope func makeLivePage() or an instance-method harness used across files is discovered by neither the harness scan nor the per-file fixpoint. Closing it needs construction tracking the walker does not do, and seeding such helpers by bare name is exactly the collision that produced 26 false positives. Neither shape exists in the target, so the limit is stated in the code where the stronger claim sits.
A regex literal desynchronises bracket depth, but the @Test counter catches the consequence and fails with "Fix the parser, not the file." The target contains none. Failing safe on an unwritten construct is proportionate; teaching a brace counter to lex regex literals is not. My assessment of an untested construct, not the author's.
| Severity | Area | Finding | Resolution |
|---|---|---|---|
| major | Tools/check-webkit-test-isolation.py — scan, exemption test | `async` alone was accepted as safe. `async` buys a hop to the ENCLOSING actor, so an `async` test in a suite carrying no `@MainActor` hops nowhere and runs on the cooperative pool exactly as a synchronous body does. Verified by construction: `struct NotIsolatedTests` with an `async` test constructing a `WebPage` scanned clean while reproducing the whole defect. The rule as written in CLAUDE.md and `docs/agent-notes/development-tooling.md` ("touch WebKit only from `async` tests") is the rule a reader would have followed into that shape. | The walker now threads main-actor isolation from the enclosing type to its members and accepts it on the test itself; `_is_test` generalised into `_has_attribute` since the depth-aware look-back over multi-line attributes is the same problem for both. Violation message distinguishes the two shapes. Rule statement corrected in the guard docstring, CLAUDE.md, the agent notes and the report's Prevention section. Mutation-tested. Repository still clean afterwards, which independently confirms every existing `async` WebKit test is genuinely isolated. |
| major | Tools/check-webkit-test-isolation.py — is_async | `is_async=" async" in signature` — a substring test over raw source, failing in the direction that EXEMPTS a test. Verified: `func trap() { // TODO: make this async once the harness lands` with a `WebPage()` body scanned clean, as did a signature carrying a parameter typed `(Int) async -> Void`. | Read from the signature's effects clause — past the `)` that closes the parameter list — with comments and string literals stripped per line during accumulation. A signature truncated before its closing paren answers False, erring toward reporting a violation rather than hiding one. Three tests added; per-line stripping needed a dedicated false-positive fixture before it became mutation-detectable. |
| major | Tools/check-webkit-test-isolation.py — parse_members.walk | A type declaration whose body sits entirely on its declaration line was skipped, discarding every member it declares. `enum Fixture { static func page() -> WebPage { WebPage() } }` plus a test calling `Fixture.page()` reported clean. This is the one parser loss the `@Test` cross-check is structurally incapable of catching — it counts lost tests, and what goes missing is a helper. The docstring, report and PR body all claimed nested types were covered by construction. The shape is idiomatic in this target (five existing uses of `actor CallCounter { var count = 0; func next() -> Int { … } }`). | Record the collapsed type whole as a read-triggered member, gated on its text reaching a seed. Two tests added covering both directions. Mutation-tested. Pre-fix tree still reports exactly the same 60 in the same 9 files. |
| minor | Tools/Tests/test_webkit_test_isolation.py — coverage | Three guard branches were reachable only through `test_this_repository_is_clean`, the least specific signal available: `PRODUCTION_WEBKIT_FACTORIES` had no positive fixture at all, and the ambient-owner edge in `reaches` (a helper tainted because it NAMES a type whose stored property builds WebKit) was exercised by nothing. This made the report's "each mutation-tested" an overclaim. | Added `test_flags_construction_through_a_production_factory` and `test_follows_taint_through_an_ambient_owner`. Both mutation-tested: removing the branch turns its own test red and, for the factory, nothing else that was not already red. |
| minor | Tools/check-webkit-test-isolation.py — docstrings | Three claims read stronger than the code delivers. (1) The anti-blindness claim is stated generally — "a parser that silently loses a declaration would look exactly like a clean repository, so `scan` cross-checks" — but the cross-check counts `@Test` attributes only and cannot see a lost helper. (2) `verify_seeds` is described as failing when a listed type "stops being a WebKit constructor", but it only checks the production FILE still contains one anywhere. (3) `discover_test_harnesses` is framed as "checked in both directions" but finds only STATIC members of named types — a file-scope `func makeLivePage()` or an instance-method harness used across files is invisible to it and to the per-file fixpoint. | All three limits stated where the claim is made, in the guard, the report and the agent notes. The third was left open rather than closed: seeding such helpers by bare name is precisely the collision that produced 26 false positives, and neither shape exists in the target today. |
| minor | specs/bugfixes/live-webkit-test-host-aborts/report.md — live runs | The report says the simulator flakiness is "worth its own ticket" without naming it, though T-2236 was filed from these very runs and contains the measurements. A reader cannot follow the thread, and the report does not state how much T-2236 bounds its own evidence. | Cited T-2236 at the point the evidence is read and in Related, with the caveat that on a destination whose live-`WebPage` population is flaky in an unchanged tree, "no converted test failed" is a strong signal about which tests fail and a weak one about counts. |
| minor | Tools/check-webkit-test-isolation.py — _depth_deltas | A Swift regex literal (`let re = /[a-z]{2,3}\{/`) desynchronises bracket depth. Confirmed by construction: the guard then parses 0 of 1 `@Test` declarations in the file. | Skipped deliberately. The failure is loud, not silent — the `@Test` counter fires and the message says "Fix the parser, not the file" — and the target contains no regex literals. Teaching a brace counter to lex regex literals is disproportionate to a construct nobody has written. Recorded here so the next person meeting the error knows the cause immediately. |
| minor | prismTests/MainActorHopContractTests.swift — parameterisation rationale | The header justifies 64 cases on the grounds that "Swift Testing runs the cases concurrently, which is the condition under which a missing hop actually lands off-main". All 64 cases are `@MainActor`-isolated and therefore serialise on the main actor's executor, so they generate no contention; and were the hop absent, one case would fail as deterministically as 64. The header also claims the test pins a caller-independent, cross-module ABI property, while it observes exactly one caller compiled in the same module. | Left alone. The test itself is sound and worth keeping — it is falsifiable, it self-protects (dropping the suite's `@MainActor` makes it fail rather than silently pass), and it covers the one failure mode the static guard is blind to. Only the stated justification is weaker than written, and rewriting a comment on an otherwise correct canary is not worth a commit here. Flagged for the author. |
| nit | PR body / CHANGELOG.md | The PR body's "The 31 largest contributors have never been named on any ticket" reads as 31 contributors; it means two suites totalling 31 tests. Both it and the CHANGELOG's "two suites no ticket had ever mentioned" also understate the finding — 7 of the 9 suites, covering 49 of the 60 tests, were never named on any ticket. | Left alone. The error is in the conservative direction and the report's own prose states it correctly. |
| nit | Build warnings | The iOS Simulator build emits 2 warnings, both in `PrismDocSchemeHandlerTests.swift`, which this PR does not touch: "main actor-isolated conformance of 'LocalFileValidation' to 'Equatable' cannot be used in nonisolated context; this is an error in the Swift 6 language mode". | Pre-existing, not this PR's. Worth noting that they corroborate the root-cause story from a second direction — the compiler is warning about exactly the nonisolated context that Swift Testing's macro expansion creates. Not a host-abort risk (no WebKit construction), but relevant to the deferred Swift 6 migration. |
| nit | Tools/ — reuse | `Tools/validate-localisation.py` already contains a character-level Swift lexer that handles `//`, `/* */`, `"""…"""`, `#"…"#` and interpolation. The new guard re-implements comment/string skipping line-wise with `STRING_RE`/`COMMENT_RE` and a `"""` toggle, handling neither block comments nor extended raw delimiters explicitly. | Not consolidated. The two produce different outputs (string extraction vs per-line bracket deltas) so it is not a drop-in reuse, and no false negative could be produced from the difference in practice. Worth a shared `Tools/swift_source.py` if a third consumer appears. |
Click to expand.
diff --git a/Tools/Tests/test_webkit_test_isolation.py b/Tools/Tests/test_webkit_test_isolation.pynew file mode 100644index 00000000..b795f74d--- /dev/null+++ b/Tools/Tests/test_webkit_test_isolation.py@@ -0,0 +1,962 @@+"""Unit tests for Tools/check-webkit-test-isolation.py.++The guard replaces a test that cannot exist: the host abort it prevents is a+load-dependent scheduling race, so nothing runnable pins it. That makes the+guard itself the only thing standing between the repository and another+four-figure fictional failure count, and a guard that quietly stopped detecting+anything would look exactly like a clean repository (T-1983's lesson, applied to+a different check). So each detection rule is asserted against a fixture here.++The script has a hyphen in its filename, so it is loaded via importlib.+"""++import importlib.util+import sys+import tempfile+import unittest+from pathlib import Path+from unittest import mock++TOOLS_DIR = Path(__file__).resolve().parent.parent+SCRIPT_PATH = TOOLS_DIR / "check-webkit-test-isolation.py"+++def _load_script():+ spec = importlib.util.spec_from_file_location("check_webkit_test_isolation", SCRIPT_PATH)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++guard = _load_script()+++class FakeRepo:+ """A throwaway tree with the production files the guard's seed check requires."""++ def __init__(self, sources: dict):+ self._dir = tempfile.TemporaryDirectory()+ self.root = Path(self._dir.name)+ for symbol, rel in guard.PRODUCTION_WEBKIT_TYPES.items():+ path = self.root / rel+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(f"// {symbol}\nlet view = WKWebView(frame: .zero)\n", encoding="utf-8")+ tests = self.root / guard.TEST_ROOT+ tests.mkdir(parents=True, exist_ok=True)+ for name, source in sources.items():+ (tests / name).write_text(source, encoding="utf-8")++ def cleanup(self):+ self._dir.cleanup()+++def scan_sources(sources: dict) -> list[str]:+ repo = FakeRepo(sources)+ try:+ return guard.scan(repo.root)+ finally:+ repo.cleanup()+++def scan_source(source: str) -> list[str]:+ return scan_sources({"SampleTests.swift": source})+++SYNC_DIRECT = """+@MainActor+struct SampleTests {+ @Test("direct")+ func directConstruction() {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++ASYNC_DIRECT = """+@MainActor+struct SampleTests {+ @Test("direct")+ func directConstruction() async {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++SYNC_VIA_HELPER = """+@MainActor+struct SampleTests {+ private func makeController() -> WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("via helper")+ func viaHelper() {+ #expect(makeController().isReady == false)+ }+}+"""++SYNC_VIA_NESTED_HELPER = """+@MainActor+struct SampleTests {+ private func makeController() -> WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ private func makeReadyController() -> WebDocumentController {+ let controller = makeController()+ return controller+ }++ @Test("via nested helper")+ func viaNestedHelper() {+ #expect(makeReadyController().isReady == false)+ }+}+"""++SYNC_NO_WEBKIT = """+@MainActor+struct SampleTests {+ @Test("pure")+ func pureAssertion() {+ #expect(1 + 1 == 2)+ }++ @Test("pure factory helper")+ func pureFactoryHelper() {+ #expect(WebDocumentControllerFactory.typographyVariables(from: .init()).isEmpty == false)+ }+}+"""++SYNC_NON_TEST_HELPER_ONLY = """+@MainActor+struct SampleTests {+ private func makeController() -> WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("does not call it")+ func unrelated() {+ #expect(true)+ }+}+"""+++HARNESS_DEFINITION = """+@MainActor+enum SpikeWebPageHarness {+ static func makePage(html: String) throws -> WebPage {+ WebPage(configuration: .init())+ }+}+"""++SYNC_VIA_SHARED_HARNESS = """+@MainActor+struct SpikeUserTests {+ @Test("uses the shared harness")+ func usesTheHarness() throws {+ let page = try SpikeWebPageHarness.makePage(html: "<p>hi</p>")+ #expect(page != nil)+ }+}+"""++INNOCENT_SAME_NAMED_HELPER = """+@MainActor+struct UnrelatedTests {+ private func makeController() -> KeyboardScrollController {+ KeyboardScrollController()+ }++ @Test("no web view anywhere near this")+ func unrelated() {+ #expect(makeController().canScroll == false)+ }+}+"""+++# ---------------------------------------------------------------------------+# Declaration forms that are not `func`. Every one of these runs Swift code on+# behalf of the test body, and a guard that only parses `func` sees none of them.+# ---------------------------------------------------------------------------++SYNC_VIA_COMPUTED_PROPERTY = """+@MainActor+struct SampleTests {+ private var controller: WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("via computed property")+ func viaComputedProperty() {+ #expect(controller.isReady == false)+ }+}+"""++SYNC_VIA_STORED_CLOSURE = """+@MainActor+struct SampleTests {+ private let make: () -> WebDocumentController = {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("via stored closure")+ func viaClosure() {+ #expect(make().isReady == false)+ }+}+"""++SYNC_VIA_LAZY_VAR = """+@MainActor+final class SampleTests {+ private lazy var controller = WebDocumentController(sessionID: "x", parseRevision: 1)++ @Test("via lazy var")+ func viaLazyVar() {+ #expect(controller.isReady == false)+ }+}+"""++SYNC_VIA_PROPERTY_OBSERVER = """+@MainActor+final class SampleTests {+ private var cached: WebDocumentController?++ private var sessionID: String = "seed" {+ didSet {+ cached = WebDocumentController(sessionID: sessionID, parseRevision: 1)+ }+ }++ @Test("assigning fires the observer")+ func viaObserver() {+ sessionID = "next"+ #expect(cached != nil)+ }++ @Test("never touches the observed property")+ func untouched() {+ #expect(1 + 1 == 2)+ }+}+"""++SYNC_VIA_STORED_PROPERTY = """+@MainActor+struct SampleTests {+ private let controller = WebDocumentController(sessionID: "x", parseRevision: 1)++ @Test("never names the property")+ func neverNamesIt() {+ #expect(1 + 1 == 2)+ }+}+"""++SYNC_VIA_SUITE_INIT = """+@MainActor+final class SampleTests {+ private let controller: WebDocumentController++ init() {+ controller = WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("never names the property")+ func neverNamesIt() {+ #expect(1 + 1 == 2)+ }+}+"""++SYNC_VIA_SUBSCRIPT = """+@MainActor+struct SampleTests {+ subscript(index: Int) -> WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: index)+ }++ @Test("uses a subscript the guard cannot spell")+ func viaSubscript() {+ #expect(self[1].isReady == false)+ }+}+"""++SYNC_VIA_NESTED_TYPE = """+@MainActor+struct SampleTests {+ private struct Fixture {+ let controller = WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ @Test("builds a nested fixture")+ func viaNestedType() {+ #expect(Fixture().controller.isReady == false)+ }+}+"""++# `async` is only an exemption when there is an actor to hop TO. Without a global+# actor on the suite, an `async` test runs on the cooperative pool exactly like a+# synchronous one — the whole defect, wearing the keyword that is supposed to fix it.+ASYNC_WITHOUT_MAIN_ACTOR = """+struct NotIsolatedTests {+ @Test("direct")+ func directConstruction() async {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++# The same suite, isolated on the test itself rather than on the type.+ASYNC_WITH_MAIN_ACTOR_ON_THE_TEST = """+struct NotIsolatedTests {+ @Test("direct")+ @MainActor+ func directConstruction() async {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++# Exempting on a bare `" async"` substring fails in the DANGEROUS direction: it+# lets a synchronous body through. Both of these scan clean under that test.+SYNC_WITH_ASYNC_IN_A_COMMENT = """+@MainActor+struct SampleTests {+ @Test("direct")+ func directConstruction() { // TODO: make this async once the harness lands+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++SYNC_WITH_AN_ASYNC_CLOSURE_PARAMETER = """+@MainActor+struct SampleTests {+ @Test("direct", arguments: [1])+ func directConstruction(+ _ index: Int,+ transform: (Int) async -> Void = { _ in }+ ) {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++# `PRODUCTION_WEBKIT_FACTORIES` had no positive fixture: only the whole-repository+# cleanliness test went red when it was removed, which is the least specific signal+# available and would not survive the factory being renamed.+SYNC_VIA_PRODUCTION_FACTORY = """+@MainActor+struct SampleTests {+ @Test("via the production factory")+ func viaProductionFactory() {+ let assembly = WebDocumentControllerFactory.make(sessionID: "x")+ #expect(assembly != nil)+ }+}+"""++# Taint reaching a member through an AMBIENT OWNER rather than through a call or a+# read: `makeFixture` names a nested type whose stored property builds WebKit, so+# `makeFixture` is itself tainted and the test that calls it is a violation.+SYNC_VIA_HELPER_BUILDING_AN_AMBIENT_TYPE = """+@MainActor+struct SampleTests {+ private struct Fixture {+ let controller = WebDocumentController(sessionID: "x", parseRevision: 1)+ }++ func makeFixture() -> Fixture { Fixture() }++ @Test("via a helper that builds a nested type")+ func viaHelperBuildingANestedType() {+ #expect(makeFixture().controller.isReady == false)+ }+}+"""++# Comments must be stripped PER LINE while the signature is accumulated, not once+# from the joined result: the accumulator joins with a space, so a `//` on any line+# but the last would comment out every line after it, hiding the effects clause and+# reporting a genuinely `async` test as a violation.+ASYNC_WITH_A_COMMENT_MID_SIGNATURE = """+@MainActor+struct SampleTests {+ @Test("direct", arguments: [1])+ func directConstruction( // one case per index+ _ index: Int+ ) async {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++# The same hazard with the type's body collapsed onto its declaration line, which+# is a shape this target already writes (`actor CallCounter { var n = 0; … }`).+# The walker is line-based, so before it handled this the whole type — members and+# all — was dropped and the test read as clean. Nothing else catches that: the+# `@Test` cross-check only notices a lost TEST, and what goes missing here is a+# helper.+SYNC_VIA_ONE_LINE_NESTED_TYPE = """+@MainActor+struct SampleTests {+ private enum Fixture { static func page() -> WebPage { WebPage() } }++ @Test("builds a one-line nested fixture")+ func viaOneLineNestedType() {+ #expect(Fixture.page() != nil)+ }+}+"""++INNOCENT_ONE_LINE_NESTED_TYPE = """+@MainActor+struct SampleTests {+ private struct DummyError: Error {}++ @Test("throws a dummy")+ func viaDummy() {+ #expect(throws: DummyError.self) { throw DummyError() }+ }+}+"""++SYNC_VIA_DEFAULT_ARGUMENT = """+@MainActor+struct SampleTests {+ private func assertReady(_ controller: WebDocumentController = WebDocumentController(+ sessionID: "x", parseRevision: 1+ )) {+ #expect(controller.isReady == false)+ }++ @Test("the default argument builds it at the call site")+ func viaDefaultArgument() {+ assertReady()+ }+}+"""++SYNC_VIA_AUTOCLOSURE = """+@MainActor+struct SampleTests {+ private func assertNotNil(_ value: @autoclosure () -> Any?) {+ #expect(value() != nil)+ }++ @Test("the argument expression runs in the test body")+ func viaAutoclosure() {+ assertNotNil(WebDocumentController(sessionID: "x", parseRevision: 1))+ }+}+"""++SYNC_VIA_LOCAL_FUNCTION = """+@MainActor+struct SampleTests {+ @Test("a local func is still this body")+ func viaLocalFunction() {+ func build() -> WebDocumentController {+ WebDocumentController(sessionID: "x", parseRevision: 1)+ }+ #expect(build().isReady == false)+ }++ @Test("an innocent sibling")+ func sibling() {+ #expect(1 + 1 == 2)+ }+}+"""++# --- The false-positive direction for each of the above. -------------------++INNOCENT_COMPUTED_PROPERTY = """+@MainActor+struct SampleTests {+ private var settings: RenderSettings {+ RenderSettings(theme: "dark")+ }++ private var summary: String {+ WebDocumentControllerFactory.typographyVariables(from: settings).description+ }++ @Test("reads a property that builds nothing")+ func readsAProperty() {+ #expect(summary.isEmpty == false)+ }+}+"""++INNOCENT_AMBIENT_IN_ANOTHER_SUITE = """+@MainActor+struct WebSuite {+ private let controller = WebDocumentController(sessionID: "x", parseRevision: 1)++ @Test("its own test")+ func ownTest() async {+ #expect(controller.isReady == false)+ }+}++@MainActor+struct PureSuite {+ @Test("a different suite in the same file")+ func pureTest() {+ #expect(1 + 1 == 2)+ }+}+"""++INNOCENT_LOCAL_DECLARATION = """+@MainActor+struct SampleTests {+ @Test("builds one locally")+ func buildsLocally() async {+ let page = WebPage()+ #expect(page != nil)+ }++ @Test("a sibling that shares the local's name")+ func sibling() {+ let page = "not a web page"+ #expect(page.isEmpty == false)+ }+}+"""++# --- Parser robustness. ----------------------------------------------------++MULTILINE_TEST_ATTRIBUTE = """+@MainActor+struct SampleTests {+ @Test(+ "a parameterised test whose attribute spans lines",+ arguments: 0..<8+ )+ func viaParameterisedTest(_ index: Int) {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""++RAW_STRING_WITH_TRAILING_BRACKETS = """+@MainActor+struct SampleTests {+ private func makeMarkdown() -> String {+ render(markdown: \"\"\"+ # Heading { not a brace }+ \"\"\")+ }++ @Test("declared after a raw string that closes mid-expression")+ func afterTheRawString() {+ let page = WebPage()+ #expect(page != nil)+ }+}+"""+++class ScanTests(unittest.TestCase):+ def test_flags_synchronous_direct_construction(self):+ violations = scan_source(SYNC_DIRECT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("directConstruction()", violations[0])+ self.assertIn("constructs WebKit directly", violations[0])++ def test_accepts_async_construction(self):+ # The whole point of the fix: an async member of a @MainActor suite hops+ # to the main actor on entry, so WebKit is safe there.+ self.assertEqual(scan_source(ASYNC_DIRECT), [])++ def test_flags_construction_through_a_helper(self):+ violations = scan_source(SYNC_VIA_HELPER)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("makeController()", violations[0])++ def test_follows_helpers_transitively(self):+ # `viaNestedHelper` never names a WebKit type; two hops away it builds one.+ # A one-level check would call this file clean, which is how+ # WebScrollabilityReportingTests would have been missed.+ violations = scan_source(SYNC_VIA_NESTED_HELPER)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaNestedHelper()", violations[0])++ def test_flags_construction_through_a_production_factory(self):+ # PRODUCTION_WEBKIT_FACTORIES, which no fixture exercised directly.+ violations = scan_source(SYNC_VIA_PRODUCTION_FACTORY)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaProductionFactory()", violations[0])+ self.assertIn("directly", violations[0])++ def test_follows_taint_through_an_ambient_owner(self):+ # The third propagation edge in `reaches`: a helper is tainted because it+ # names a TYPE whose ambient members build WebKit, not because it calls a+ # tainted function or reads a tainted property.+ violations = scan_source(SYNC_VIA_HELPER_BUILDING_AN_AMBIENT_TYPE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaHelperBuildingANestedType()", violations[0])++ def test_ignores_tests_that_touch_no_webkit(self):+ # `WebDocumentControllerFactory.typographyVariables` is a pure function on+ # a type whose `make` is not: matching the type name alone would flag ~20+ # innocent typography tests and make the guard something people disable.+ self.assertEqual(scan_source(SYNC_NO_WEBKIT), [])++ def test_ignores_an_unused_helper(self):+ self.assertEqual(scan_source(SYNC_NON_TEST_HELPER_ONLY), [])++ def test_flags_a_shared_harness_used_from_another_file(self):+ # The helper fixpoint is per-file, so the shared harnesses have to be+ # seeded by qualified name or a suite that builds its page through+ # `SpikeWebPageHarness` would be invisible to the guard.+ violations = scan_sources(+ {+ "SharedHarness.swift": HARNESS_DEFINITION,+ "SpikeUserTests.swift": SYNC_VIA_SHARED_HARNESS,+ }+ )+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("usesTheHarness()", violations[0])++ def test_a_same_named_helper_in_another_file_is_not_contagious(self):+ # Why the fixpoint is per-file rather than target-wide. Tainting by bare+ # name made `KeyboardScrollControllerTests.makeController()` — which+ # builds no web view at all — inherit the guilt of+ # `WebDocumentControllerTests.makeController()`, and the guard reported+ # 26 tests that construct nothing. A guard people have to disbelieve is+ # a guard people disable.+ violations = scan_sources(+ {+ "WebTests.swift": SYNC_VIA_HELPER,+ "UnrelatedTests.swift": INNOCENT_SAME_NAMED_HELPER,+ }+ )+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("WebTests.swift", violations[0])+++class AsyncExemptionTests(unittest.TestCase):+ """`async` is the exemption, so getting the exemption wrong reopens the defect.++ Both failures here are in the dangerous direction — they mark a test SAFE.+ """++ def test_flags_an_async_test_whose_suite_is_not_main_actor(self):+ # `async` buys a hop to the enclosing actor. With no global actor on the+ # suite there is no actor to hop to, and the body runs on the cooperative+ # pool exactly as a synchronous one would.+ violations = scan_source(ASYNC_WITHOUT_MAIN_ACTOR)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("directConstruction()", violations[0])+ self.assertIn("@MainActor", violations[0])++ def test_accepts_an_async_test_isolated_on_the_test_itself(self):+ # The isolation need not come from the suite; `@MainActor` on the test is+ # equally sufficient, and flagging it would be a false positive.+ self.assertEqual(scan_source(ASYNC_WITH_MAIN_ACTOR_ON_THE_TEST), [])++ def test_a_comment_mentioning_async_does_not_exempt_a_synchronous_test(self):+ # A bare `" async" in signature` substring test exempts this.+ violations = scan_source(SYNC_WITH_ASYNC_IN_A_COMMENT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("directConstruction()", violations[0])+ self.assertIn("synchronous", violations[0])++ def test_a_comment_mid_signature_does_not_hide_the_effects_clause(self):+ # The other direction, and a false POSITIVE: comments are stripped per+ # line during accumulation, because the accumulator joins with a space and+ # a `//` on an early line would otherwise swallow the `async` that follows.+ self.assertEqual(scan_source(ASYNC_WITH_A_COMMENT_MID_SIGNATURE), [])++ def test_an_async_closure_parameter_does_not_exempt_a_synchronous_test(self):+ # The `async` belongs to the parameter's type, not to this function.+ violations = scan_source(SYNC_WITH_AN_ASYNC_CLOSURE_PARAMETER)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("directConstruction()", violations[0])+ self.assertIn("synchronous", violations[0])+++class NonFunctionDeclarationTests(unittest.TestCase):+ """The evasion family: every declaration form that is not a `func`.++ A guard that parses only `func` reports an empty violation list for each of+ these while the test body still constructs WebKit synchronously — the exact+ shape the fix converted sixty times by hand.+ """++ def test_flags_a_computed_property(self):+ violations = scan_source(SYNC_VIA_COMPUTED_PROPERTY)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaComputedProperty()", violations[0])+ self.assertIn("controller", violations[0])++ def test_flags_a_stored_closure(self):+ violations = scan_source(SYNC_VIA_STORED_CLOSURE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaClosure()", violations[0])++ def test_flags_a_lazy_var(self):+ violations = scan_source(SYNC_VIA_LAZY_VAR)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaLazyVar()", violations[0])++ def test_flags_the_test_that_fires_a_property_observer(self):+ # And only that one: a `didSet` runs on assignment, so a sibling that+ # never names the property is not exposed to it.+ violations = scan_source(SYNC_VIA_PROPERTY_OBSERVER)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaObserver()", violations[0])++ def test_flags_every_test_in_a_suite_with_a_webkit_stored_property(self):+ # Swift Testing builds a fresh suite instance per test, so the stored+ # property's initialiser runs for a test that never mentions it.+ violations = scan_source(SYNC_VIA_STORED_PROPERTY)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("neverNamesIt()", violations[0])+ self.assertIn("outside", violations[0])++ def test_flags_every_test_in_a_suite_whose_init_builds_webkit(self):+ violations = scan_source(SYNC_VIA_SUITE_INIT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("neverNamesIt()", violations[0])++ def test_flags_a_subscript(self):+ # The guard does not model `[]` call syntax, so a subscript taints its+ # whole type rather than being missed.+ violations = scan_source(SYNC_VIA_SUBSCRIPT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaSubscript()", violations[0])++ def test_flags_a_nested_types_initialiser(self):+ violations = scan_source(SYNC_VIA_NESTED_TYPE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaNestedType()", violations[0])+ self.assertIn("Fixture()", violations[0])++ def test_flags_a_nested_type_written_on_one_line(self):+ # The walker consumes a declaration's whole extent in one step, so a type+ # whose body shares its declaration line has no interior lines to descend+ # into. Dropping it loses a helper, and a lost helper is invisible to the+ # `@Test` cross-check — the one parser loss that still looks like a clean+ # repository.+ violations = scan_source(SYNC_VIA_ONE_LINE_NESTED_TYPE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaOneLineNestedType()", violations[0])+ self.assertIn("Fixture", violations[0])++ def test_flags_a_default_argument_value(self):+ # The expression is evaluated at the call site, and lives in the helper's+ # signature — which is part of the declaration text the guard reads.+ violations = scan_source(SYNC_VIA_DEFAULT_ARGUMENT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaDefaultArgument()", violations[0])++ def test_flags_an_autoclosure_argument(self):+ violations = scan_source(SYNC_VIA_AUTOCLOSURE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaAutoclosure()", violations[0])+ self.assertIn("directly", violations[0])++ def test_flags_a_local_function_without_tainting_siblings(self):+ violations = scan_source(SYNC_VIA_LOCAL_FUNCTION)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaLocalFunction()", violations[0])+++class FalsePositiveTests(unittest.TestCase):+ """The other direction. A guard people have to disbelieve is a guard people disable."""++ def test_a_computed_property_that_builds_nothing_is_not_flagged(self):+ self.assertEqual(scan_source(INNOCENT_COMPUTED_PROPERTY), [])++ def test_an_ambient_member_does_not_taint_another_suite_in_the_same_file(self):+ # Tainting by file rather than by enclosing type would flag `pureTest`,+ # which instantiates nothing.+ self.assertEqual(scan_source(INNOCENT_AMBIENT_IN_ANOTHER_SUITE), [])++ def test_a_one_line_nested_type_that_builds_nothing_is_not_flagged(self):+ # One-line types are READ-tainted as a whole, which is deliberately broad.+ # It must still be gated on the type's text reaching WebKit, or every+ # `struct DummyError: Error {}` in the target becomes a violation.+ self.assertEqual(scan_source(INNOCENT_ONE_LINE_NESTED_TYPE), [])++ def test_a_local_declaration_does_not_taint_a_sibling_test(self):+ # `let page = WebPage()` inside one test body must not make `page` a+ # tainted member name for the whole suite.+ self.assertEqual(scan_source(INNOCENT_LOCAL_DECLARATION), [])+++class ParserRobustnessTests(unittest.TestCase):+ def test_recognises_a_multi_line_test_attribute(self):+ # `@Test(` / name / `arguments:` / `)` — the closing lines look nothing+ # like an attribute, so a prefix-only look-back reads the declaration as+ # an ordinary function and never checks it.+ violations = scan_source(MULTILINE_TEST_ATTRIBUTE)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("viaParameterisedTest()", violations[0])++ def test_a_raw_string_closing_mid_expression_does_not_desynchronise(self):+ # `\"\"\")` closes both the literal and a call. Dropping the `)` leaves the+ # parser inside the helper for the rest of the file, and every later test+ # silently vanishes.+ violations = scan_source(RAW_STRING_WITH_TRAILING_BRACKETS)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("afterTheRawString()", violations[0])++ def test_reports_when_a_test_declaration_could_not_be_parsed(self):+ # The anti-blindness rule: losing a declaration must look like a failure,+ # not like a clean file.+ with mock.patch.object(guard, "FUNC_RE", guard.re.compile(r"^\s*never matches\b")):+ violations = scan_source(SYNC_DIRECT)+ self.assertEqual(len(violations), 1, violations)+ self.assertIn("parsed 0 of 1 @Test", violations[0])+++class SeedTests(unittest.TestCase):+ def test_seed_check_passes_on_a_tree_that_still_constructs_webkit(self):+ repo = FakeRepo({"SampleTests.swift": SYNC_NO_WEBKIT})+ try:+ self.assertEqual(guard.verify_seeds(repo.root), [])+ finally:+ repo.cleanup()++ def test_seed_check_fails_when_a_listed_type_stops_constructing_webkit(self):+ # Otherwise the guard silently goes blind: it would keep passing while the+ # symbol it is watching no longer means what the list says it means.+ repo = FakeRepo({"SampleTests.swift": SYNC_NO_WEBKIT})+ try:+ symbol, rel = next(iter(guard.PRODUCTION_WEBKIT_TYPES.items()))+ (repo.root / rel).write_text("// no WebKit here any more\n", encoding="utf-8")+ problems = guard.verify_seeds(repo.root)+ self.assertEqual(len(problems), 1, problems)+ self.assertIn(symbol, problems[0])+ finally:+ repo.cleanup()++ def test_seed_check_fails_when_a_listed_file_disappears(self):+ repo = FakeRepo({"SampleTests.swift": SYNC_NO_WEBKIT})+ try:+ symbol, rel = next(iter(guard.PRODUCTION_WEBKIT_TYPES.items()))+ (repo.root / rel).unlink()+ problems = guard.verify_seeds(repo.root)+ self.assertEqual(len(problems), 1, problems)+ self.assertIn(symbol, problems[0])+ finally:+ repo.cleanup()+++NEW_HARNESS_DEFINITION = """+@MainActor+enum NewLiveHarness {+ static func makePage() -> WebPage {+ WebPage(configuration: .init())+ }+}+"""++NEW_HARNESS_CALLER = """+@MainActor+struct NewHarnessUserTests {+ @Test("uses the unlisted harness")+ func usesIt() {+ #expect(NewLiveHarness.makePage() != nil)+ }+}+"""+++class TestHarnessSeedTests(unittest.TestCase):+ """`TEST_HARNESS_FACTORIES` must stay honest in both directions.++ The helper fixpoint is per-file, so a shared harness is only covered because+ it is named here. Without a check, new cross-file test infrastructure — the+ thing most likely to be added by someone who has never read this script —+ narrows the guard silently, which is the exact failure mode `verify_seeds`+ exists to prevent for the production list.+ """++ def _problems(self, sources, factories):+ repo = FakeRepo(sources)+ try:+ with mock.patch.object(guard, "TEST_HARNESS_FACTORIES", factories):+ return guard.verify_test_harnesses(repo.root)+ finally:+ repo.cleanup()++ def test_accepts_a_harness_that_is_listed(self):+ problems = self._problems(+ {"SharedHarness.swift": HARNESS_DEFINITION, "User.swift": SYNC_VIA_SHARED_HARNESS},+ {"SpikeWebPageHarness": ("makePage",)},+ )+ self.assertEqual(problems, [])++ def test_fails_on_a_new_cross_file_harness_that_is_not_listed(self):+ problems = self._problems(+ {"NewHarness.swift": NEW_HARNESS_DEFINITION, "User.swift": NEW_HARNESS_CALLER},+ {},+ )+ self.assertEqual(len(problems), 1, problems)+ self.assertIn("NewLiveHarness.makePage", problems[0])+ self.assertIn("not in TEST_HARNESS_FACTORIES", problems[0])++ def test_ignores_a_webkit_harness_used_only_inside_its_own_file(self):+ # The per-file fixpoint already covers it; demanding a list entry would+ # be noise, and noise is how a guard gets switched off.+ problems = self._problems(+ {"NewHarness.swift": NEW_HARNESS_DEFINITION + NEW_HARNESS_CALLER}, {}+ )+ self.assertEqual(problems, [])++ def test_fails_when_a_listed_harness_no_longer_exists(self):+ problems = self._problems(+ {"SharedHarness.swift": HARNESS_DEFINITION, "User.swift": SYNC_VIA_SHARED_HARNESS},+ {"SpikeWebPageHarness": ("makePage", "renamedAway")},+ )+ self.assertEqual(len(problems), 1, problems)+ self.assertIn("renamedAway", problems[0])+ self.assertIn("dead", problems[0])+++class RepositoryTests(unittest.TestCase):+ def test_this_repository_is_clean(self):+ # The rule is only worth anything if it is actually true here.+ root = TOOLS_DIR.parent+ self.assertEqual(guard.verify_seeds(root), [])+ self.assertEqual(guard.verify_test_harnesses(root), [])+ self.assertEqual(guard.scan(root), [])++ def test_every_test_declaration_in_the_target_is_parsed(self):+ # `scan` folds this into its violation list; asserting it separately makes+ # a parser regression legible as a parser regression.+ root = TOOLS_DIR.parent+ for path in sorted((root / guard.TEST_ROOT).rglob("*.swift")):+ lines = path.read_text(encoding="utf-8").split("\n")+ declared = sum(1 for line in lines if guard.TEST_ATTR_RE.match(line))+ parsed = sum(1 for member in guard.parse_members(lines) if member.is_test)+ self.assertEqual(parsed, declared, path.relative_to(root))+++if __name__ == "__main__":+ sys.exit(0 if unittest.main(exit=False).result.wasSuccessful() else 1)
diff --git a/Tools/check-webkit-test-isolation.py b/Tools/check-webkit-test-isolation.pynew file mode 100755index 00000000..36d6e3d6--- /dev/null+++ b/Tools/check-webkit-test-isolation.py@@ -0,0 +1,725 @@+#!/usr/bin/env python3+"""Fail when a Swift Testing test in prismTests can construct WebKit off the main thread.++Why this guard exists+---------------------+The unit-test target shares ONE host process. When that process aborts, every+test still queued is reported failed with no recorded duration, so a run that+executed a handful of real tests reports a four-figure fictional failure count+(T-1983 scaffolding, T-1541, T-2096, T-2219).++The abort has one mechanism, established on T-1541+(`specs/bugfixes/svgwebviewtests-offmain-crash/report.md`): a SYNCHRONOUS+`@MainActor` function has no hop-on-entry. Its isolation is realised only if the+caller hops, and the callers of test bodies are Swift Testing's macro-generated+thunks — compiled in `prismTests`, which builds in Swift 5 language mode with+`SWIFT_APPROACHABLE_CONCURRENCY = YES` and no default actor isolation. Those+thunks are `nonisolated(nonsending)` and run on the *caller's* executor, i.e. the+runner's cooperative pool. Swift 5 mode has no runtime enforcement to catch the+mis-hop, so under load a synchronous `@MainActor` test body executes off the main+thread. WebKit's initialisers assert they are on the main thread and kill the+process when they are not.++`async` members of a `@MainActor` type are immune: the hop to the actor's+executor is emitted in the callee and is part of the ABI, independent of the+caller's module or language mode.++Hence the rule this script enforces, target-wide:++ A `@Test` function that can reach a WebKit constructor must be `async`+ AND main-actor isolated.++Both halves are load-bearing, and the second is easy to lose. `async` buys a hop+to the enclosing actor — so with no `@MainActor` on the suite (or on the test)+there is no actor to hop to, and the body runs on the cooperative pool exactly as+a synchronous one would. Exempting on `async` alone would let the entire defect+back in through any suite nobody remembered to annotate.++It is deliberately a static rule rather than a test. The abort is a+load-dependent scheduling race — a suite that can crash the host passes in+isolation, every time — so no runnable test can pin it. What CAN be pinned is+that no test body is capable of the off-main touch in the first place, and that+is decidable by reading the source. It is also the only check that covers a+suite nobody has identified yet, which is what T-2219 asks for: `MermaidRendererTests`+and `WebDocumentControllerTests` were both found by this scan, and neither was+named on any ticket.++What "can reach" means+----------------------+The guard is the entire durability argument for the fix: sixty test bodies were+converted by hand, and the only thing stopping the sixty-first from being+written the same way is this script. So the reachability model covers every+*declaration form* that can run WebKit code on behalf of a synchronous test+body, not just `func`:++* `func` — call-triggered, transitively, to a fixpoint.+* computed `var`, property observers (`didSet`/`willSet`), `lazy var` — read+ triggered: mentioning the name evaluates the body.+* a stored closure (`let make: () -> T = { … }`) — mentioning the name hands+ around something that runs the body.+* stored-property initialisers, `init`, `deinit`, `subscript` — *ambient*: they+ run when the suite type is instantiated (Swift Testing builds a fresh suite+ instance per test) or on an access whose syntax the guard does not model. Any+ synchronous test in that type is a violation, and constructing the type from+ elsewhere is call-triggered.+* nested types — constructing one runs its ambient members. A type whose body is+ written on its declaration line cannot be split by a line-based walker, so it is+ taken whole and read-triggered instead of descended into.+* default argument values and `@autoclosure` arguments need no separate rule:+ the expression is textually inside the declaration (respectively the caller's+ body) that the guard already reads. Same for a local `func` or `let` written+ inside a test body.++Declarations *inside* a body are not members — the parser consumes a member's+whole extent in one step — so a local `let page = WebPage()` taints only the+test that writes it, never its siblings.++Because a parser that silently loses a declaration would look exactly like a+clean repository, `scan` cross-checks the number of `@Test` attributes it parsed+against the number written in the file and fails the run if any went missing.+Note what that does and does not cover: it catches a lost TEST, not a lost+HELPER. A dropped helper is equally silent and has no comparable counter, so+every declaration form the walker meets must be recorded rather than skipped —+which is why a one-line type is taken whole above instead of being stepped over.++Usage: Tools/check-webkit-test-isolation.py [repo-root]+Exit 0 when clean, 1 on any violation (or if a seed list has gone stale).+"""++from __future__ import annotations++import re+import sys+from pathlib import Path++# Direct WebKit constructions. Reaching any of these off the main thread is what+# kills the host.+WEBKIT_CONSTRUCTORS = [+ r"\bWKWebView\s*\(",+ r"\bWKWebViewConfiguration\s*\(",+ r"\bWKUserContentController\s*\(",+ r"\bWebPage\s*\(",+]++# Prism types whose initialiser constructs one of the above. Kept as an explicit+# list because following Swift initialisers across the production target properly+# would need a real parser; `verify_seeds` below fails the run if one of these+# stops being a WebKit constructor (renamed, or the construction moved), so the+# list cannot silently go blind.+PRODUCTION_WEBKIT_TYPES = {+ # symbol: production file that must still construct WebKit+ "MermaidRenderer": "prism/Services/MermaidRenderer.swift",+ "WebDocumentController": "prism/ViewModels/WebDocumentController.swift",+ "FootnotePopoverWebPage": "prism/ViewModels/FootnotePopoverWebPage.swift",+}++# Production factories that hand back one of the types above.+PRODUCTION_WEBKIT_FACTORIES = [+ r"\bWebDocumentControllerFactory\.make\s*\(",+ r"\bWebDocumentStateSynchronizer\.makeAssembly\s*\(",+]++# Shared test harnesses that build a live page for suites in OTHER files. The+# helper fixpoint in `scan` is per-file, deliberately: tainting by bare function+# name across the whole target makes `KeyboardScrollControllerTests`'+# `makeController()` — which builds no web view at all — collide with+# `WebDocumentControllerTests`', and reports 26 tests that construct nothing.+# Qualified names cannot collide, so the cross-file cases are seeded instead.+#+# Stored as {type: (member, …)} rather than as regexes so `verify_test_harnesses`+# can check each named member in BOTH directions: a listed member that no longer+# reaches WebKit is stale, and a static test-harness member that DOES reach+# WebKit and is called from another file but is not listed here fails the run.+# Without that second direction the list is the one place the guard's "covers a+# suite nobody has identified yet" premise breaks down — new cross-file test+# infrastructure is exactly what gets added without anyone recalling this file.+TEST_HARNESS_FACTORIES = {+ "SpikeWebPageHarness": ("makePage", "loadedPage"),+ "WebDocumentLiveHarness": ("make", "makeStyled"),+ # Found by `verify_test_harnesses` the first time it ran — it mounts the real+ # production assembly (and with it a `WebDocumentController`) for two suites in+ # other files, and the hand-written list had missed it. Exactly the omission the+ # discovery direction exists to catch.+ "WebNavigationPrecedenceHarness": ("makeAssembly", "remount"),+}++TEST_ROOT = "prismTests"+++def test_harness_patterns() -> list[str]:+ return [+ r"\b" + owner + r"\.(?:" + "|".join(members) + r")\s*\("+ for owner, members in TEST_HARNESS_FACTORIES.items()+ ]+++def seed_pattern() -> re.Pattern[str]:+ parts = (+ list(WEBKIT_CONSTRUCTORS)+ + list(PRODUCTION_WEBKIT_FACTORIES)+ + test_harness_patterns()+ )+ parts += [r"\b" + name + r"\s*\(" for name in PRODUCTION_WEBKIT_TYPES]+ return re.compile("|".join(parts))+++def verify_seeds(root: Path) -> list[str]:+ """Each named production type must still construct WebKit, or the list is stale.++ Granularity worth knowing: this checks the FILE still contains a WebKit+ construction, not that the named type's initialiser is what performs it. A+ type that stopped building a page while some other declaration in the same+ file kept one keeps this green. It catches the renames and relocations that+ actually happen; it is not a proof of the seed's semantics.+ """+ ctor = re.compile("|".join(WEBKIT_CONSTRUCTORS))+ problems = []+ for symbol, rel in PRODUCTION_WEBKIT_TYPES.items():+ path = root / rel+ if not path.exists():+ problems.append(+ f"{rel} is gone, so the guard no longer knows whether {symbol} "+ f"constructs WebKit. Update PRODUCTION_WEBKIT_TYPES."+ )+ continue+ if not ctor.search(path.read_text(encoding="utf-8")):+ problems.append(+ f"{rel} no longer constructs WebKit directly, so listing {symbol} "+ f"as a WebKit constructor is stale. Update PRODUCTION_WEBKIT_TYPES."+ )+ return problems+++def discover_test_harnesses(root: Path) -> dict[tuple[str, str], Path]:+ """{(type, static member): defining file} for every test harness reaching WebKit.++ The shape a shared harness always has *here*: a static member of a named type+ whose body reaches a WebKit constructor. Discovering them is what keeps+ TEST_HARNESS_FACTORIES honest in the "somebody added a new one" direction.++ Two cross-file shapes it does NOT discover, stated because the surrounding+ claim ("checked in both directions") reads stronger than this is: a file-scope+ `func makeLivePage() -> WebPage` called from another file, and an INSTANCE+ method used as `LiveHarness().make()`. Both would need construction tracking+ the walker does not do, and seeding them by bare name is the collision that+ produced 26 false positives. Neither exists in the target today; if one is+ added, it is invisible here and the per-file fixpoint will not see it either.+ """+ seeds = seed_pattern()+ found: dict[tuple[str, str], Path] = {}+ for path in sorted((root / TEST_ROOT).rglob("*.swift")):+ members = parse_members(path.read_text(encoding="utf-8").split("\n"))+ call_tainted, read_tainted, _ = taint(members, seeds)+ for member in members:+ if member.is_test or not member.is_static or not member.owner:+ continue+ if member.name in call_tainted or member.name in read_tainted:+ found[(member.owner.rsplit(".", 1)[-1], member.name)] = path+ return found+++def verify_test_harnesses(root: Path) -> list[str]:+ """TEST_HARNESS_FACTORIES must list exactly the cross-file harnesses that exist."""+ discovered = discover_test_harnesses(root)+ texts = {+ path: path.read_text(encoding="utf-8")+ for path in sorted((root / TEST_ROOT).rglob("*.swift"))+ }+ problems = []+ for owner, members in TEST_HARNESS_FACTORIES.items():+ for member in members:+ if (owner, member) not in discovered:+ problems.append(+ f"{owner}.{member} is listed in TEST_HARNESS_FACTORIES but no test "+ f"harness by that name reaches WebKit any more (renamed, moved, or "+ f"it stopped building a page). The seed is dead — update the list."+ )+ for (owner, member), definition in sorted(discovered.items()):+ if member in TEST_HARNESS_FACTORIES.get(owner, ()):+ continue+ call = re.compile(r"\b" + re.escape(owner) + r"\." + re.escape(member) + r"\s*\(")+ callers = [path for path, text in texts.items() if path != definition and call.search(text)]+ if not callers:+ continue # Same-file only: the per-file fixpoint in `scan` already covers it.+ problems.append(+ f"{owner}.{member} ({definition.relative_to(root)}) builds WebKit and is "+ f"called from {callers[0].relative_to(root)}, but is not in "+ f"TEST_HARNESS_FACTORIES. The helper fixpoint is per-file, so every "+ f"synchronous test using it is currently invisible to this guard. Add it."+ )+ return problems+++MODIFIER = (+ r"(?:private|fileprivate|internal|public|package|open|static|class|final|lazy|weak|"+ r"unowned|nonisolated|isolated|override|dynamic|mutating|nonmutating|convenience|"+ r"required|indirect|@\w+(?:\([^()]*(?:\([^()]*\)[^()]*)*\))?)"+)+MODIFIERS = rf"(?:{MODIFIER}\s+)*"++FUNC_RE = re.compile(rf"^\s*{MODIFIERS}func\s+(`?\w+`?)\s*[(<]")+INIT_RE = re.compile(rf"^\s*{MODIFIERS}init\s*[?!]?\s*[(<]")+DEINIT_RE = re.compile(rf"^\s*{MODIFIERS}deinit\b")+SUBSCRIPT_RE = re.compile(rf"^\s*{MODIFIERS}subscript\s*[(<]")+TYPE_RE = re.compile(rf"^\s*{MODIFIERS}(?:struct|class|enum|actor|protocol)\s+(\w+)")+EXTENSION_RE = re.compile(rf"^\s*{MODIFIERS}extension\s+([\w.]+)")+VAR_RE = re.compile(rf"^\s*{MODIFIERS}(?:var|let)\s+(`?\w+`?)\s*(.*)$")+TEST_ATTR_RE = re.compile(r"^\s*@Test\b")+# `async` only buys the hop when there is an actor to hop TO. A `@MainActor`+# global-actor attribute on the suite (or on the test itself) is what supplies it.+MAIN_ACTOR_ATTR_RE = re.compile(r"^\s*@MainActor\b|@MainActor\b")++# A declaration continues onto the next line when it ends mid-expression. Brace,+# paren and bracket depth cover almost everything; these are the rest.+CONTINUATIONS = ("=", ",", "->", "&&", "||", "+")++STRING_RE = re.compile(r'"(?:\\.|[^"\\])*"')+COMMENT_RE = re.compile(r"//.*$")+++def _code_only(line: str) -> str:+ """`line` with string literals blanked and any `//` comment removed."""+ code = STRING_RE.sub('""', line)+ comment = COMMENT_RE.search(code)+ return code[: comment.start()] if comment else code+++def _is_async_signature(signature: str) -> bool:+ """True when `async` appears in the EFFECTS clause of this signature.++ A substring test for `" async"` is not good enough, and it fails in the+ DANGEROUS direction — it EXEMPTS a test. `func trap() { // make this async+ later` exempts a synchronous body, and so does any parameter typed+ `(Int) async -> Void`. Both scan clean while constructing WebKit off-main.++ The effects clause is whatever follows the parameter list, so: strip comments+ and string literals, then read only past the `)` that closes the parameter+ list. When the accumulated signature never gets that far — it was truncated+ at a closure default value's `{` — there is no visible effects clause and the+ answer is False, which errs toward reporting a violation rather than hiding+ one.+ """+ code = _code_only(signature)+ open_at = code.find("(")+ if open_at < 0:+ return bool(re.search(r"\basync\b", code))+ depth = 0+ for position in range(open_at, len(code)):+ if code[position] == "(":+ depth += 1+ elif code[position] == ")":+ depth -= 1+ if depth == 0:+ return bool(re.search(r"\basync\b", code[position + 1 :]))+ return False++# Declaration kinds. FUNC is call-triggered (`name(`), READ is triggered by any+# mention of the name (a computed property, observer, lazy var or stored closure+# runs, or is handed to something that runs it, on a bare reference), AMBIENT+# runs without being named at all so it taints its whole enclosing type.+FUNC, READ, AMBIENT, TYPE = "func", "read", "ambient", "type"+++class Member:+ """One type-level declaration. A plain class, not a dataclass: the unit tests+ load this script through importlib, and dataclasses cannot resolve annotations+ for a module that is not registered in `sys.modules` on Python 3.9."""++ __slots__ = (+ "name", "kind", "owner", "body", "line",+ "is_test", "is_async", "is_static", "is_main_actor",+ )++ def __init__(+ self, name, kind, owner, body, line,+ is_test, is_async, is_static, is_main_actor=False,+ ):+ self.name = name+ self.kind = kind+ self.owner = owner+ self.body = body+ self.line = line+ self.is_test = is_test+ self.is_async = is_async+ self.is_static = is_static+ self.is_main_actor = is_main_actor+++def _depth_deltas(lines: list[str]) -> list[int]:+ """Bracket-depth change per line, ignoring comments and string literals.++ Test files for a web renderer are full of HTML and JS fixtures in multi-line+ string literals; counting their braces would desynchronise the parser and+ silently swallow whole suites.+ """+ deltas = []+ in_raw_string = False+ for line in lines:+ delta = 0+ rest = line+ while True:+ if in_raw_string:+ _, fence, tail = rest.partition('"""')+ if not fence:+ break+ # Brackets AFTER the closing fence are code: the `)` in `"""),`+ # closes a call, and dropping it desynchronises the rest of the file.+ in_raw_string = False+ rest = tail+ continue+ head, fence, tail = rest.partition('"""')+ code = STRING_RE.sub('""', head)+ comment = COMMENT_RE.search(code)+ if comment:+ code = code[: comment.start()]+ delta += sum(code.count(char) for char in "([{") - sum(+ code.count(char) for char in ")]}"+ )+ if comment or not fence:+ break+ in_raw_string = True+ rest = tail+ deltas.append(delta)+ return deltas+++def _property_kind(rest: str, follow: str, is_lazy: bool) -> str | None:+ """Classify what follows `var|let NAME` — or None when nothing ever runs."""+ depth = 0+ marker = None+ for char in rest:+ if char in "([":+ depth += 1+ elif char in ")]":+ depth -= 1+ elif depth == 0 and char in "={":+ marker = char+ break+ if marker is None:+ return None # `var x: T` — a declaration with no body and no initialiser.+ if marker == "{":+ return READ # Computed property, or observers on a property with no initialiser.+ value = rest.split("=", 1)[1].strip() or follow.strip()+ if value.startswith("{"):+ return READ # Stored closure: the body runs when the name is used, not at init.+ return READ if is_lazy else AMBIENT+++def _match_declaration(lines: list[str], index: int) -> tuple[str, str, bool] | None:+ """(kind, name, is_static) for a declaration starting at `index`, else None."""+ line = lines[index]+ prefix = line.split("func", 1)[0]+ is_static = bool(re.search(r"\b(?:static|class)\b", prefix))+ match = FUNC_RE.match(line)+ if match:+ return FUNC, match.group(1).strip("`"), is_static+ for pattern, name in ((INIT_RE, "init"), (DEINIT_RE, "deinit"), (SUBSCRIPT_RE, "subscript")):+ if pattern.match(line):+ return AMBIENT, name, False+ match = TYPE_RE.match(line)+ if match:+ return TYPE, match.group(1), False+ match = EXTENSION_RE.match(line)+ if match:+ return TYPE, match.group(1).split(".")[-1], False+ match = VAR_RE.match(line)+ if match:+ follow = next((text for text in lines[index + 1 : index + 3] if text.strip()), "")+ keyword_prefix = re.split(r"\b(?:var|let)\b", line, maxsplit=1)[0]+ kind = _property_kind(+ match.group(2), follow, bool(re.search(r"\blazy\b", keyword_prefix))+ )+ if kind is None:+ return None+ return kind, match.group(1).strip("`"), bool(+ re.search(r"\b(?:static|class)\b", keyword_prefix)+ )+ return None+++def parse_members(lines: list[str]) -> list[Member]:+ """Every type-level declaration in the file, with its enclosing type path.++ Declarations nested inside a member's body are deliberately not members: the+ walker consumes a member's whole extent in one step, so a local+ `let page = WebPage()` stays attributed to the one test that writes it.+ """+ deltas = _depth_deltas(lines)+ members: list[Member] = []++ def extent(start: int, limit: int) -> int:+ depth = 0+ index = start+ while index < limit:+ depth += deltas[index]+ follows = lines[index + 1].lstrip() if index + 1 < limit else ""+ if depth <= 0 and not lines[index].rstrip().endswith(CONTINUATIONS):+ if not follows.startswith("."):+ return index + 1+ index += 1+ return limit++ def walk(start: int, end: int, owner: str, owner_main_actor: bool = False) -> None:+ index = start+ while index < end:+ declaration = _match_declaration(lines, index)+ if declaration is None:+ index += 1+ continue+ kind, name, is_static = declaration+ stop = extent(index, end)+ if kind == TYPE:+ brace = next((cursor for cursor in range(index, stop) if "{" in lines[cursor]), None)+ if brace is not None and brace + 1 < stop - 1:+ walk(+ brace + 1, stop - 1,+ f"{owner}.{name}" if owner else name,+ _is_main_actor(lines, deltas, index),+ )+ else:+ # A type whose whole body sits on its declaration line — the+ # shape `actor CallCounter { var n = 0; func next() { … } }`,+ # which this target already writes five times. The walker is+ # line-based, so it cannot split those members out; without+ # this branch they are dropped ENTIRELY and every test using+ # the type reads as clean. That is the one loss the `@Test`+ # cross-check in `scan` cannot see, because what went missing+ # is a helper rather than a test. Attribute the whole line to+ # the type itself and make it READ-triggered: naming the type+ # is what runs any of it, and over-tainting a rare one-liner+ # is the safe direction.+ members.append(+ Member(+ name, READ, owner, "\n".join(lines[index:stop]),+ index + 1, False, False, False,+ )+ )+ index = stop+ continue+ signature = _code_only(lines[index])+ cursor = index+ while "{" not in signature and cursor + 1 < stop:+ cursor += 1+ signature += " " + _code_only(lines[cursor])+ body = "\n".join(lines[index:stop])+ if kind == AMBIENT and name not in ("init", "deinit", "subscript") and "{" in body:+ # `var x: T = value { didSet { … } }` is two hazards with different+ # triggers: the initialiser runs when the suite is instantiated,+ # the observer only when something assigns to `x`. Splitting them+ # keeps an observer from tainting tests that never touch `x`.+ members.append(+ Member(name, READ, owner, body, index + 1, False, False, is_static)+ )+ body = body.split("{", 1)[0]+ members.append(+ Member(+ name=name,+ kind=kind,+ owner=owner,+ body=body,+ line=index + 1,+ is_test=_is_test(lines, deltas, index),+ is_async=_is_async_signature(signature),+ is_static=is_static,+ is_main_actor=owner_main_actor+ or _is_main_actor(lines, deltas, index),+ )+ )+ index = stop++ walk(0, len(lines), "")+ return members+++def _has_attribute(+ lines: list[str], deltas: list[int], index: int, attribute: re.Pattern[str]+) -> bool:+ """The attribute sits on the declaration line or on the lines above it.++ An attribute is frequently spread over several lines — `@Test(` / display+ name / `arguments:` / `)` — and the closing lines look nothing like an+ attribute. Walking back through them needs bracket depth, not a prefix test;+ without it every parameterised test in the target reads as an ordinary+ function and is never checked at all.+ """+ if attribute.search(_code_only(lines[index])):+ return True+ back = index - 1+ depth = 0+ while back >= 0:+ stripped = lines[back].strip()+ if depth < 0: # Inside the brackets of a multi-line attribute.+ if attribute.match(lines[back]) and depth + deltas[back] >= 0:+ return True+ depth += deltas[back]+ back -= 1+ continue+ if stripped == "" or stripped.startswith("//"):+ back -= 1+ continue+ if attribute.match(lines[back]):+ return True+ # A closing bracket continues an attribute; a closing BRACE is the end of+ # the previous declaration, and walking past it would inherit its `@Test`.+ if stripped.startswith("@") or (deltas[back] < 0 and "}" not in stripped):+ depth += deltas[back]+ back -= 1+ continue+ return False+ return False+++def _is_test(lines: list[str], deltas: list[int], index: int) -> bool:+ return _has_attribute(lines, deltas, index, TEST_ATTR_RE)+++def _is_main_actor(lines: list[str], deltas: list[int], index: int) -> bool:+ return _has_attribute(lines, deltas, index, MAIN_ACTOR_ATTR_RE)+++def _mentions(body: str, name: str, call_only: bool) -> bool:+ suffix = r"\s*\(" if call_only else r"\b"+ return bool(re.search(r"\b" + re.escape(name) + suffix, body))+++def taint(+ members: list[Member], seeds: re.Pattern[str]+) -> tuple[set[str], set[str], set[str]]:+ """(call-triggered names, read-triggered names, ambient owners) reaching WebKit.++ To a fixpoint: a helper that calls a tainted helper is itself tainted, and so+ on. Test bodies are never taint sources — they are what the taint is measured+ against.+ """+ calls: set[str] = set()+ reads: set[str] = set()+ owners: set[str] = set()++ def reaches(member: Member) -> bool:+ if seeds.search(member.body):+ return True+ if any(name != member.name and _mentions(member.body, name, True) for name in calls):+ return True+ if any(name != member.name and _mentions(member.body, name, False) for name in reads):+ return True+ return any(+ owner != member.owner+ and owner.rsplit(".", 1)[-1]+ and _mentions(member.body, owner.rsplit(".", 1)[-1], True)+ for owner in owners+ )++ for _ in range(len(members) + 2):+ changed = False+ for member in members:+ if member.is_test:+ continue+ if member.kind == FUNC and member.name in calls:+ continue+ if member.kind == READ and member.name in reads:+ continue+ if member.kind == AMBIENT and member.owner in owners:+ continue+ if not reaches(member):+ continue+ changed = True+ if member.kind == FUNC:+ calls.add(member.name)+ elif member.kind == READ:+ reads.add(member.name)+ else:+ owners.add(member.owner)+ if not changed:+ break+ return calls, reads, owners+++def scan(root: Path) -> list[str]:+ seeds = seed_pattern()+ violations: list[str] = []+ for path in sorted((root / TEST_ROOT).rglob("*.swift")):+ lines = path.read_text(encoding="utf-8").split("\n")+ members = parse_members(lines)+ declared = sum(1 for line in lines if TEST_ATTR_RE.match(line))+ parsed = sum(1 for member in members if member.is_test)+ if parsed < declared:+ # A parser that loses declarations looks exactly like a clean file.+ violations.append(+ f"{path.relative_to(root)}: the guard parsed {parsed} of {declared} @Test "+ f"declarations, so the rest are unchecked. Fix the parser, not the file."+ )+ continue+ calls, reads, owners = taint(members, seeds)+ for member in members:+ # `async` is only an exemption when there is an actor to hop TO.+ # An `async` test in a suite with no global-actor attribute runs on+ # the cooperative pool exactly like a synchronous one, so exempting+ # on `async` alone lets the whole defect back in through a suite+ # nobody remembered to annotate.+ if not member.is_test or (member.is_async and member.is_main_actor):+ continue+ if seeds.search(member.body):+ reason = "constructs WebKit directly"+ elif member.owner in owners:+ reason = (+ f"belongs to {member.owner or 'file scope'}, which builds WebKit outside "+ f"any test body (a stored property, init, deinit or subscript), so every "+ f"test in it runs that code"+ )+ else:+ via = [f"{name}()" for name in sorted(calls) if _mentions(member.body, name, True)]+ via += [name for name in sorted(reads) if _mentions(member.body, name, False)]+ via += [+ f"{owner.rsplit('.', 1)[-1]}()"+ for owner in sorted(owners)+ if owner.rsplit(".", 1)[-1]+ and _mentions(member.body, owner.rsplit(".", 1)[-1], True)+ ]+ if not via:+ continue+ reason = "constructs WebKit via " + ", ".join(dict.fromkeys(via))+ posture = (+ "is `async` but neither it nor its suite is @MainActor, so the "+ "hop it relies on is never emitted, and it"+ if member.is_async+ else "is synchronous and"+ )+ violations.append(+ f"{path.relative_to(root)}:{member.line}: {member.name}() {posture} {reason}."+ )+ return violations+++def main(argv: list[str]) -> int:+ root = Path(argv[1]).resolve() if len(argv) > 1 else Path(__file__).resolve().parent.parent+ stale = verify_seeds(root) + verify_test_harnesses(root)+ if stale:+ print("FAIL [webkit-test-isolation]: the guard's seed lists are stale:", file=sys.stderr)+ for problem in stale:+ print(f" {problem}", file=sys.stderr)+ return 1+ violations = scan(root)+ if violations:+ print(+ f"FAIL [webkit-test-isolation]: {len(violations)} test(s) can construct "+ f"WebKit from a synchronous body.",+ file=sys.stderr,+ )+ for violation in violations:+ print(f" {violation}", file=sys.stderr)+ print(+ "\nA synchronous @MainActor test body has no hop-on-entry, so under load it "+ "runs on the\ncooperative pool and WebKit's main-thread assertion aborts the "+ "shared test host —\nwhich reports every still-queued test as a failure it "+ "never ran (T-1541/T-2096/T-2219).\nMark the test `async` AND make sure its "+ "suite is @MainActor: an async member of a @MainActor\nsuite hops on entry as "+ "part of the ABI, but `async` alone hops nowhere.",+ file=sys.stderr,+ )+ return 1+ print("[webkit-test-isolation] OK: no synchronous test can construct WebKit.")+ return 0+++if __name__ == "__main__":+ sys.exit(main(sys.argv))
diff --git a/specs/bugfixes/live-webkit-test-host-aborts/report.md b/specs/bugfixes/live-webkit-test-host-aborts/report.mdnew file mode 100644index 00000000..09987bab--- /dev/null+++ b/specs/bugfixes/live-webkit-test-host-aborts/report.md@@ -0,0 +1,398 @@+# Bugfix Report: Live-WebKit Test Suites Abort the Shared Test Host++**Date:** 2026-08-22+**Status:** Fixed++## Description of the Issue++Transit tickets: **T-2219** (`MermaidCSPSpikeTests` aborts the test host) and+**T-2096** (`WebContentTerminationWiringTests` + `WebScrollabilityReportingTests`+SIGABRT when they share one xcodebuild invocation). Filed separately, fixed+together, because they are one defect wearing two suite names.++The unit-test target shares one host process. When a suite aborts it with signal+`abrt`, every test still queued is reported as a failure despite never having+executed. Observed cascades: 190 "failures" of which 189 had no recorded+duration (T-2219), and 234 of which 233 had none (T-2219 comment). While that+stands, `make test-quick` reports a mostly fictional four-figure failure count,+so a real regression is indistinguishable from the cascade without opening the+result bundle by hand and filtering on recorded duration.++**Reproduction steps (T-2096, the more repeatable half):**+1. `xcodebuild build-for-testing ... -testPlan prism -destination 'platform=macOS'`+2. `xcodebuild test-without-building ... -only-testing:prismTests/WebContentTerminationWiringTests -only-testing:prismTests/WebScrollabilityReportingTests`+3. The host aborts; each suite is green in isolation.++**Impact:** every full local run of the suite has had to be triaged by hand, at a+measured cost of 10-20 minutes per change (recorded on T-1541, the same defect's+first appearance). CI's per-locale sweep is 150 minutes of macOS runner time that+a single abort turns into noise.++## Investigation Summary++- **Symptoms examined:** both tickets and their comments; the T-2219 comment+ recording a cascade in which *both* named crashers were verified absent from+ the result bundle; `Tools/check-test-results.sh` output on wedged and cascaded+ bundles.+- **Prior art read in full:** `specs/bugfixes/svgwebviewtests-offmain-crash/report.md`+ (T-1541) and `docs/agent-notes/development-tooling.md`.+- **Code inspected:** `prismTests/**` (a full static scan for `@Test` bodies that+ reach a WebKit constructor), `prism/ViewModels/WebDocumentController.swift`,+ `prism/ViewModels/FootnotePopoverWebPage.swift`,+ `prism/Services/MermaidRenderer.swift`, `prism.xcodeproj` concurrency build+ settings, the Makefile test recipes and `prism.xctestplan`.+- **Hypotheses tested:**+ - *"Any two live-WebKit suites in one host process can abort it"* (the ticket's+ own hypothesis) - **refuted as stated, and replaced by something stronger.**+ It is not the pairing and not "live WebKit" that matters. What matters is+ whether a suite contains a test that can construct WebKit **synchronously**.+ Both suites T-2096 names do. Pairing only raises the probability, because+ Swift Testing runs suites concurrently inside one host.+ - *"`MermaidCSPSpikeTests` is guilty"* - **refuted by inspection.** It owns one+ WebKit test and that test is already `async`; it constructs nothing+ synchronously. It is, however, the single longest-running live-WebKit test in+ the target (26 diagram renders in one page), which makes it the most likely+ test to be *in flight* when some other suite aborts the host - and the abort+ is attributed to what was in flight.+ - *"`@MainActor` on the suite is sufficient"* - refuted on T-1541 already;+ re-confirmed here, since every suite involved is already `@MainActor`.+ - *"`.serialized` / `-parallel-testing-worker-count 1` bound the concurrency"* -+ refuted: the first orders tests only within one suite (stated in+ `SystemColorSchemeObserverTests`' own header), the second bounds test HOST+ processes, not Swift Testing's in-process concurrency.++## Discovered Root Cause++**Defect type:** Race condition / unenforced actor isolation (test-only).+**Same defect as T-1541**, which was fixed for exactly one suite.++A synchronous `@MainActor` function has no hop-on-entry: its isolation is+realised only if the *caller* hops before the call. The callers of test bodies+are Swift Testing's macro-generated thunks, compiled in `prismTests` - Swift 5+language mode, `SWIFT_APPROACHABLE_CONCURRENCY = YES`, and **no**+`SWIFT_DEFAULT_ACTOR_ISOLATION` (the app target sets it; the test target does+not). Those thunks are `nonisolated(nonsending)`, so they run on the caller's+executor - the runner's cooperative pool - and Swift 5 mode has no runtime+enforcement to catch the mis-hop. The synchronous body then constructs a+`WebPage` / `WKWebView`, WebKit asserts it is on the main thread, and the host+dies.++`async` members of a `@MainActor` type are immune: the switch to the actor's+executor on entry is emitted in the callee and is part of the ABI, independent of+the caller's module or language mode.++**A full static scan of `prismTests` found 60 such tests across 9 files:**++| File | Sync tests reaching WebKit |+|------|---------------------------|+| `WebDocumentControllerTests` | 27 |+| `WebContentTerminationWiringTests` | 11 (T-2096) |+| `WebSearchScrollOwnershipTests` | 5 |+| `MermaidRendererTests` | 4 |+| `WebSelectionNoteTests` | 4 |+| `WebScrollabilityReportingTests` | 3 (T-2096) |+| `WebFootnotePopoverTests` | 2 |+| `WebPerfProbeTests` | 2 |+| `WebScrollIntegrationContractTests` | 2 |++`MermaidCSPSpikeTests` is not on that list, and the two largest contributors have+never been named on any ticket. That is the shape of the bug: **the suite the+cascade names is not the suite that caused it.** Swift Testing runs suites+concurrently within the one host, so when a synchronous body somewhere aborts the+process, the blame lands on whichever long-running live-WebKit test was in+flight.++**Why it occurred:** T-1541 diagnosed the mechanism precisely and then fixed one+file. Its own "Prevention" section wrote the rule down - *never touch WebKit from+a synchronous test body in this target* - but nothing enforced it, and the+remaining 60 violations (including 27 in a suite nobody has ever filed a ticket+against) stayed exactly where they were.++## Resolution for the Issue++**Changes made:**++1. **All 60 offending tests are now `async`** (`prismTests/MermaidRendererTests.swift`,+ `prismTests/WebRendering/{WebContentTerminationWiring,WebDocumentController,+ WebFootnotePopover,WebPerfProbe,WebScrollIntegrationContract,+ WebScrollabilityReporting,WebSearchScrollOwnership,WebSelectionNote}Tests.swift`).+ Every one of those suites is already `@MainActor`, so `async` alone buys the+ ABI-level hop; no assertion, helper or body text changed.++2. **`Tools/check-webkit-test-isolation.py`** - a static guard that fails when any+ `@Test` in `prismTests` can reach a WebKit constructor without being both+ `async` and main-actor isolated, following same-file helpers transitively+ (`makeController()` counts). Both halves of the exemption are enforced: an+ `async` test in a suite with no `@MainActor` hops nowhere and runs on the+ cooperative pool exactly like a synchronous one, so exempting on the keyword+ alone would readmit the whole defect through any unannotated suite. The+ keyword itself is read out of the signature's effects clause rather than by+ substring, since `func f() { // make this async later` and a parameter typed+ `(Int) async -> Void` both contain the word and would otherwise exempt a+ synchronous body. Because a+ guard is only a durability argument for the shapes it can *see*, reachability is+ modelled over every type-level declaration form, not just `func`: computed+ properties, property observers, `lazy var`s and stored closures are read-+ triggered; stored-property initialisers, `init`, `deinit` and `subscript` are+ ambient and taint every synchronous test in their suite; nested types are+ triggered by construction, and one written entirely on its declaration line - a+ shape this target already uses five times - is taken whole and read-triggered,+ because a line-based walker has no interior lines to descend into and stepping+ over it would lose a helper silently. That last case is the one parser loss the+ `@Test` cross-check below cannot see, since what goes missing is a helper rather+ than a test. Default arguments, `@autoclosure` arguments and local+ declarations need no rule of their own - the expression is already inside a+ declaration the guard reads - and because the parser consumes a member's whole+ extent in one step, a local `let page = WebPage()` taints only the test that+ writes it.++ It also verifies both of its seed lists. `PRODUCTION_WEBKIT_TYPES` is re-checked+ against the production sources, so a renamed or relocated WebKit constructor+ fails the guard instead of silently blinding it; `TEST_HARNESS_FACTORIES` is+ checked in both directions - a listed harness that no longer builds a page is+ dead, and a static test harness that *does* build one and is called from another+ file but is not listed fails the run. The per-file helper fixpoint cannot see+ cross-file harnesses, so that second direction is the only thing standing+ between the guard and new test infrastructure nobody thought to register. It+ earned its place on first run: `WebNavigationPrecedenceHarness` (used by+ `WebFragmentNavigationPrecedenceTests` and `WebReloadNavigationClaimTests`)+ mounts the real production assembly and had been missed by the hand-written+ list.++ Finally, `scan` cross-checks the number of `@Test` attributes it parsed against+ the number written in each file and fails when any went missing, because a+ parser that quietly loses declarations looks exactly like a clean repository.+ That check found two blind spots in the guard's own first version: a multi-line+ `@Test(...)` attribute made every parameterised test read as an ordinary+ function, and a raw string closing mid-expression (`""")`) desynchronised+ bracket depth for the remainder of the file.++3. **`make verify-test-isolation`**, a prerequisite of `test-quick`, `test` and+ `test-locales`, plus a step in the Linux `checks.yml` job. Half a second of+ static checking runs before any invocation whose result the defect would+ destroy.++4. **`Tools/Tests/test_webkit_test_isolation.py`** - 43 unit tests over the guard's+ detection rules: direct construction, via a helper, via a *nested* helper, the+ async case, each non-`func` declaration form above, both seed-list staleness+ checks, the parser-robustness cases, and five false-positive cases. A guard that+ stopped detecting anything would look exactly like a clean repository, so every+ rule is mutation-tested - removing the rule must turn its test red.++5. **`Tools/Tests/test-make-guards.sh`** - the sweep-control-flow replay stubs+ `python3` in its sandbox, because `test-locales` now has a prerequisite whose+ recipe the replay executes against a tree that has no `Tools/` in it.++**Approach rationale:** T-2219 asks for a remedy "structural enough to cover an+unidentified third case rather than named at the two known ones". Fixing the+named suites would have left 46 of the 60 violations standing, including the+27-test suite that is the most likely true culprit. Fixing all 60 removes today's+instances; the guard is what covers tomorrow's, and it is the only part of this+that can *find* a suite nobody has named - it found `MermaidRendererTests` and+`WebDocumentControllerTests` on its first run.++Nothing is skipped, excluded, or moved to a separate invocation. Executed-test+counts are unchanged.++**Alternatives considered:**+- *Isolate live-WebKit suites into their own xcodebuild invocation* (T-2219's+ fallback) - rejected: it treats the symptom, needs a hand-maintained list of+ "live WebKit" suites that would have omitted `MermaidRendererTests`, doubles+ the sweep's host launches on a machine where host launch is already the+ fragile step, and leaves the off-main touch in place to bite the isolated+ invocation instead.+- *Skip the crashing suites* - explicitly refused, by the ticket and by+ `localisation-tests.yml`'s own comment. It recreates T-1983 in a softer form.+- *Set `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor` on `prismTests`* - the+ mechanically strongest fix, and the one T-1541 named as "plausibly the real+ fix". Rejected for the same reason it was then: it changes the default+ isolation of ~3,900 tests at once, and it would push every test body onto the+ main actor, which is a scheduling change of unknown blast radius for the live+ WebKit suites that currently poll across `await` points. Worth revisiting+ alongside a Swift 6 language-mode migration; not the change to make while the+ suite is the thing being repaired.+- *`#expect(Thread.isMainThread)` before each construction* - rejected on T-1541+ and still rejected: it converts a crash into a flaky failure rather than+ removing the mis-hop.++## Regression Test++The abort is a load-dependent scheduling race: a suite capable of killing the+host passes in isolation, every time (T-1541 recorded 20/20 and a full run that+did not crash on a tree that demonstrably could). No runnable test can pin it.+What *is* decidable is whether any test body is capable of the off-main touch,+and that is what the regression test asserts.++- **Guard:** `Tools/check-webkit-test-isolation.py`, run by `make verify-test-isolation`.+- **Tests of the guard:** `Tools/Tests/test_webkit_test_isolation.py` (43 tests),+ including `RepositoryTests.test_this_repository_is_clean`, which fails if any+ violation is reintroduced anywhere in the target.++**Mutation evidence.** With the fix stashed (`git stash`) and nothing else+changed, `Tools/check-webkit-test-isolation.py` reports:++```+FAIL [webkit-test-isolation]: 60 test(s) can construct WebKit from a synchronous body.+ prismTests/MermaidRendererTests.swift:253: rendererAcceptsCustomTimeout() is synchronous and constructs WebKit directly.+ ...+ prismTests/WebRendering/WebScrollabilityReportingTests.swift:170: scrollabilityChangedDecodes() is synchronous and constructs WebKit via makeController().+```++and exits 1. Restored, it exits 0.++Reverting a **single** test is caught just as precisely - `async` removed from+`scrollabilityChangedDecodes` alone gives:++```+FAIL [webkit-test-isolation]: 1 test(s) can construct WebKit from a synchronous body.+ prismTests/WebRendering/WebScrollabilityReportingTests.swift:170: scrollabilityChangedDecodes() is synchronous and constructs WebKit via makeWebController().+```++So the guard is red exactly when the fix is absent, at both whole-fix and+per-test granularity. It pins all 60, not merely their sum.++**A false-positive pass was also run, and changed the design.** An earlier version+tainted helper names across the whole target rather than per file. That reported 26+extra violations in `KeyboardScrollControllerTests`, whose `makeController()`+builds a `KeyboardScrollController` and no web view at all - it merely shares a+name with `WebDocumentControllerTests`'. The fixpoint is per-file for that reason,+and the shared cross-file harnesses (`SpikeWebPageHarness`, `WebDocumentLiveHarness`)+are seeded by *qualified* name instead, which cannot collide. Both directions are+pinned by `test_flags_a_shared_harness_used_from_another_file` and+`test_a_same_named_helper_in_another_file_is_not_contagious`.++**Run command:** `make verify-test-isolation`++## Affected Files++| File | Change |+|------|--------|+| `prismTests/MermaidRendererTests.swift` | 4 tests made `async` |+| `prismTests/WebRendering/WebContentTerminationWiringTests.swift` | 11 tests made `async` (T-2096) |+| `prismTests/WebRendering/WebDocumentControllerTests.swift` | 27 tests made `async` |+| `prismTests/WebRendering/WebFootnotePopoverTests.swift` | 2 tests made `async` |+| `prismTests/WebRendering/WebPerfProbeTests.swift` | 2 tests made `async` |+| `prismTests/WebRendering/WebScrollIntegrationContractTests.swift` | 2 tests made `async` |+| `prismTests/WebRendering/WebScrollabilityReportingTests.swift` | 3 tests made `async` (T-2096) |+| `prismTests/WebRendering/WebSearchScrollOwnershipTests.swift` | 5 tests made `async` |+| `prismTests/WebRendering/WebSelectionNoteTests.swift` | 4 tests made `async` |+| `prismTests/MainActorHopContractTests.swift` | New: pins the ABI-level main-actor hop the fix depends on |+| `Tools/check-webkit-test-isolation.py` | New: the static guard |+| `Tools/Tests/test_webkit_test_isolation.py` | New: 43 unit tests for the guard's detection rules, each mutation-tested |+| `Tools/Tests/test-make-guards.sh` | Sandbox stubs `python3` for the new prerequisite |+| `Makefile` | New `verify-test-isolation` target; prerequisite of `test-quick`, `test`, `test-locales` |+| `.github/workflows/checks.yml` | Runs `make verify-test-isolation` on every push/PR |+| `docs/agent-notes/development-tooling.md` | The rule, the mechanism, and why the named suite is usually innocent |++## Verification++**Automated:**+- [x] `make verify-test-isolation` passes (guard clean; 43/43 guard unit tests)+- [x] `make verify-make-guards` passes (all 8 checks, including the sweep replay+ that now stubs `python3` for the new prerequisite)+- [x] `make lint` — 0 violations in 546 files+- [x] `xcodebuild build-for-testing` clean for the macOS destination+- [x] `xcodebuild build-for-testing` clean for the iOS Simulator destination+- [x] Mutation: with the fix stashed, the guard reports all 60 tests by name and+ exits 1; restored, it exits 0. The extended guard reports the same 60 in the+ same 9 files against the pre-fix tree, so widening reachability did not move+ the historical count - it only added shapes nobody had written yet+- [x] Mutation: each of the 18 detection rules removed one at a time; every one+ turns its own unit test(s) red and nothing else++**Live runs (iOS Simulator, `-parallel-testing-worker-count 1`):**++The **macOS** destination could not execute anything on this machine: `testmanagerd`+is wedged (up 4d 17h), so every run dies with `The test runner hung before+establishing connection` after ~705 s having executed zero tests. That is T-2146+and it is a different failure from this one — no host abort, no crash report,+nothing executes at all. Six macOS attempts, including two holding the machine's+serialising lock, all identical. The iOS Simulator uses a different test daemon+and runs cleanly, so all live evidence below comes from there.++| Run | Tree | Suites | Result |+|-----|------|--------|--------|+| control | pre-fix | 5 live-WebKit suites | total=93 passed=83 failed=1 |+| post-fix | fixed | same 5 | total=93 passed=82 failed=2 |+| post-fix | fixed | same 5 + probe | total=95 passed=81 failed=5 |+| control solo x2 | pre-fix | `WebContentTerminationWiringTests` | failed=2, failed=2 |+| post-fix solo x3 | fixed | same | failed=6, failed=6, failed=1 |+| post-fix | fixed | the other 5 converted suites + hop contract | total=71 passed=68 failed=3 |+| control | pre-fix | those same suites | total=70 passed=58 failed=**12** |++- **None of the 60 converted tests failed in any run.** Every suite whose tests+ this change touched — `WebDocumentControllerTests`, `MermaidRendererTests`,+ `WebScrollabilityReportingTests`, `WebFootnotePopoverTests`,+ `WebSelectionNoteTests`, `WebSearchScrollOwnershipTests`,+ `WebPerfProbeRoutingTests`, `WebScrollIntegrationContractTests` — was green+ throughout, as was `MermaidCSPSpikeTests`.+- **The failures are pre-existing flakes in `WebContentTerminationWiringTests`,+ and the fix does not touch them.** All six are already-`async` live-`WebPage`+ tests that poll a real navigation:+ `controllerObservesItsOwnPageNavigationStream`,+ `recoveryReasonReachesTheHandlerPerPath`, `loadAfterAbandonmentRestoresRecovery`,+ `readyAfterAbandonmentClearsTheBanner`, `stalledRecoveriesExhaustTheBudget`,+ `cancellingObservationEndsTheRealDrain`. The failing SET is identical in both+ trees — a **pre-fix** control run produced all six, and a post-fix run produced+ only two — so the count varies run to run on either side and the population does+ not. The suite is never green on this destination, before or after. Worth its+ own ticket; it is not this change.+- **The pre-fix tree is not the cleaner one.** On the second suite group the+ control failed 12 (every one a live `WebSelectionNoteTests` test) against the+ fixed tree's 3. Taken with the first group, where the control failed fewer, the+ honest reading is that `prismTests`' live-`WebPage` tests are broadly flaky on+ the iOS Simulator destination in **both** trees and the count is noise. What is+ not noise: across eight runs, **zero failures ever landed on a converted test**.+ Filed as **T-2236**, with those measurements. It also bounds how much this+ evidence can carry: on a destination where the live-`WebPage` population is+ flaky in an unchanged tree, "no converted test failed" is a real signal about+ *which* tests fail, and a weak one about failure counts.+- `MainActorHopContractTests` — the new pin on the ABI-level hop — passed all 64+ cases.+- **Note for whoever reads a bundle here:** xcodebuild's console "Failing tests:"+ list and the result bundle's `failed` count disagree on these runs (six names+ listed, `failed=2` in the bundle) because failures are retried and the retry+ passes. That is the T-2224 blind spot, observed live.++**Not obtained:** a reproduction of the abort itself. It needs a host that+launches, executes, and then dies — and the destination it was historically+observed on cannot launch a host at all right now. It did not reproduce in eight+iOS Simulator runs, but those were 25-95 test runs against a fix that removes the+mechanism; the historical cascades came from ~3,900-test macOS runs. Read that as+"not tested", not as "does not happen on iOS".++That gap matters less than it would for most fixes, for the reason T-1541 already+recorded: a green full run is **necessary but not sufficient** evidence here,+because the crash is probabilistic — T-1541's own baseline run on a tree that+demonstrably could crash did not crash. The proof of this fix is structural. After+it, no test body in the target is capable of constructing WebKit off the main+actor, and the guard makes that a checked property rather than a claim.++## Prevention++- The rule is now enforced rather than written down. That is the whole difference+ between this fix and T-1541's: the mechanism was correctly diagnosed weeks+ earlier and the repository still carried 60 violations, because a report's+ "Prevention" section is not a build step.+- The enforced rule is `async` **and** main-actor isolated. `async` buys a hop to+ the ENCLOSING actor, so an `async` test in a suite with no `@MainActor` hops+ nowhere and is exactly as dangerous as a synchronous one. The guard rejects+ that shape rather than exempting on the keyword.+- When a cascade names a suite, do not start with that suite. Run+ `make verify-test-isolation` first; the guilty suite is whichever one can touch+ WebKit synchronously, not whichever one was on screen.+- If a future test genuinely needs synchronous WebKit construction, the answer is+ `async` plus `await MainActor.run { }` around the construction (T-1541's+ pattern), never an exclusion.++## Related++- T-1541 - the same defect, diagnosed correctly, fixed for one suite+- T-1983 / T-2224 - the result-bundle guard that makes a cascade legible+- T-2146 - the *other* macOS test failure (host hangs before connecting, zero+ tests executed, no crash report); different signature, do not conflate+- T-2236 - filed from this branch's control runs: the live-`WebPage` tests are+ broadly flaky on the iOS Simulator destination in an unchanged tree. Not this+ defect, but it is why the live evidence here is read for *which* tests failed+ rather than for how many+- PR #330 / #331 - earlier attempts at this crash class
diff --git a/docs/agent-notes/development-tooling.md b/docs/agent-notes/development-tooling.mdindex 7c00a08a..8099ac2c 100644--- a/docs/agent-notes/development-tooling.md+++ b/docs/agent-notes/development-tooling.md@@ -5,9 +5,94 @@ - `make lint` runs `swiftlint lint --strict` and may report zero violations but still exit non-zero if SwiftLint cannot write to its cache. This is tracked as T-807. - To distinguish cache-permission failures from lint failures during investigation, run `swiftlint lint --strict --no-cache` after the Makefile target. The Makefile target should still be run first because project tooling is Makefile-based. +## The one rule that keeps full runs legible: no WebKit from a synchronous test++`make verify-test-isolation` (`Tools/check-webkit-test-isolation.py`) fails the+build when a `@Test` function in `prismTests` can reach a WebKit constructor from+a **synchronous** body. `test-quick`, `test` and `test-locales` all depend on it.++Why it is a static check and not a test: the failure it prevents is a+load-dependent scheduling race. A suite capable of aborting the host passes in+isolation, every time — that is precisely how T-1541, T-2096 and T-2219 each got+filed against a *different* suite for the *same* defect. What a run cannot decide,+the source can: whether any test body is capable of the off-main touch at all.++The mechanism (established on T-1541, unchanged): a synchronous `@MainActor`+function has no hop-on-entry. Its isolation is realised only if the CALLER hops,+and the callers here are Swift Testing's macro-generated thunks, compiled in+`prismTests` — Swift 5 language mode, `SWIFT_APPROACHABLE_CONCURRENCY = YES`, no+default actor isolation. Those thunks are `nonisolated(nonsending)`, so they run+on the runner's cooperative pool, and Swift 5 mode has no runtime enforcement to+catch the mis-hop. WebKit's initialisers assert they are on the main thread and+abort the process when they are not. **`async` members of a `@MainActor` type are+immune**: the hop is emitted in the callee and is part of the ABI.++Two things follow that are easy to get wrong:++- **The suite the cascade names is usually not the guilty one.** Swift Testing+ runs suites concurrently *inside one host* (`-parallel-testing-worker-count 1`+ bounds test HOST processes, not in-process concurrency, and `.serialized` only+ orders tests within one suite). When the host aborts, the blame lands on+ whatever long-running live-WebKit test happened to be in flight. T-2219's+ `MermaidCSPSpikeTests` — a single `async` test that renders 26 diagrams and owns+ no synchronous WebKit construction at all — is the ideal candidate for being+ blamed, which is why the same ticket also recorded a cascade with both "known+ crashers" verifiably absent from the bundle. Do not chase the named suite.+- **`@MainActor` on the suite is not a fix by itself, and neither is+ `.serialized`.** `@MainActor` is *necessary* — it supplies the actor there is to+ hop to — but on a synchronous body nothing performs the hop, which is what PR+ #330 discovered when the annotation only moved the trap. `async` supplies the+ other half. Neither half works alone.++When the guard fires, the fix is to mark the test `async` and make sure its suite+carries `@MainActor` — not to skip the suite and not to add an exclusion.+Skipping recreates T-1983 in a softer form.++Four things about the guard itself, all learned by it failing on this repo:++- **The exemption is `async` AND main-actor isolated, not `async` alone.** An+ `async` test in a suite with no global actor hops nowhere and runs on the+ cooperative pool exactly like a synchronous one, so exempting on the keyword+ alone would readmit the whole defect through any suite nobody annotated. For+ the same reason the guard reads `async` out of the signature's effects clause+ rather than by substring: `func f() { // make this async later` and a parameter+ typed `(Int) async -> Void` both contain the word, and both would have exempted+ a synchronous body.++- **Reachability is modelled over every declaration form, not `func`.** A+ computed property, a `lazy var`, a property observer or a stored closure runs+ Swift on a bare *mention* of its name; a stored-property initialiser, `init`,+ `deinit` or `subscript` runs without being named at all and therefore taints+ every synchronous test in its type (Swift Testing builds a fresh suite instance+ per test). A `func`-only parser reports an empty violation list for all of them+ while the test body constructs WebKit exactly as before. Default arguments,+ `@autoclosure` arguments and local declarations need no rule: the expression is+ already inside a declaration the guard reads.+- **`TEST_HARNESS_FACTORIES` is checked in both directions.** The helper fixpoint+ is per-file on purpose (bare-name tainting across the target made+ `KeyboardScrollControllerTests.makeController()` inherit+ `WebDocumentControllerTests.makeController()`'s guilt, for 26 bogus+ violations), so a shared harness is covered only because it is named in that+ list — and a hand-maintained list is exactly what goes stale. The guard now+ discovers static test members that reach WebKit and are called from another+ file, and fails when one is unlisted. It found `WebNavigationPrecedenceHarness`+ on its first run.+- **The parser cross-checks itself against the `@Test` count per file.** A parser+ that silently loses declarations is indistinguishable from a clean repository —+ the same shape as T-1983's zero-tests-executed sweep. That check caught two+ real blind spots: a multi-line `@Test(...)` attribute made every parameterised+ test read as an ordinary function, and a raw string closing mid-expression+ (`""")`) desynchronised bracket depth for the rest of the file. If you touch+ the parser and the count check fires, fix the parser, not the Swift. Know its+ limit, though: it counts lost *tests*, and a lost *helper* is just as silent+ with no counter behind it. That is why the walker records a one-line type+ declaration (`enum Fixture { static func page() -> WebPage { … } }`) whole and+ read-triggered instead of stepping over it — it has no interior lines to descend+ into, and skipping it made every test using that type read as clean.+ ## Test-suite gotchas found while clearing the T-1541/T-1983 backlog -- **`SVGWebViewTests` no longer needs skipping** (fixed on the T-1541 bugfix branch; `specs/bugfixes/svgwebviewtests-offmain-crash/report.md`). Root cause of the full-run host crash: the suite was the only one constructing `WKWebView` in *synchronous* test bodies. A sync `@MainActor` function has no hop-on-entry — its isolation depends on the caller hopping, and swift-testing's `nonisolated(nonsending)` thunks (Swift 5 mode + Approachable Concurrency, no default isolation in the test target) run on the runner's cooperative pool under full-suite load. WebKit's init `RELEASE_ASSERT(main thread)` then killed the host. The `@MainActor` added in 81c1636 could only move the trap. The rule for this target: **touch WebKit only from `async` tests** (actor-isolated async functions hop on entry as ABI); the suite additionally wraps construction in `MainActor.run`, which is runtime-enforced.+- **`SVGWebViewTests` no longer needs skipping** (fixed on the T-1541 bugfix branch; `specs/bugfixes/svgwebviewtests-offmain-crash/report.md`). Root cause of the full-run host crash: the suite was the only one constructing `WKWebView` in *synchronous* test bodies. A sync `@MainActor` function has no hop-on-entry — its isolation depends on the caller hopping, and swift-testing's `nonisolated(nonsending)` thunks (Swift 5 mode + Approachable Concurrency, no default isolation in the test target) run on the runner's cooperative pool under full-suite load. WebKit's init `RELEASE_ASSERT(main thread)` then killed the host. The `@MainActor` added in 81c1636 could only move the trap. The rule for this target: **touch WebKit only from an `async` test in a `@MainActor` suite** (an ACTOR-ISOLATED async function hops on entry as ABI — `async` on its own hops nowhere); the suite additionally wraps construction in `MainActor.run`, which is runtime-enforced. - **The app and test targets have different actor-isolation defaults.** `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor` is set on the app target only; `prismTests` and `prismUITests` have no default. Production code is therefore implicitly main-actor while the same type called from a test is not, which silently turns "safe because everything is on the main actor" into a real race in tests (see `RecentFileEntry.relativeDateFormatter`). Do not assume a production type's isolation holds inside a test. - **`XCTExpectFailure` breaks `check-test-results.sh`'s arithmetic check.** An expected failure is counted in neither `passedTests` nor `failedTests`, so the script prints `WARN: passed+failed+skipped != total`. That warning is benign when the difference equals the number of `XCTExpectFailure`s (currently 2, T-1985 and T-1986); it does not gate the build. - **One `MockURLProtocol` used to be shared by five suites** across `URLDocumentLoaderTests`, `ImageLoaderTests`, and `SVGSourceLoaderTests`. `.serialized` only orders tests *within* a suite, so the suites overwrote each other's static handler and served each other's payloads. Handlers are now registered per scope (`MockURLScope`), carried as a request header set from the session configuration. If you add a networked suite, give it its own scope rather than a global handler.
diff --git a/prismTests/MainActorHopContractTests.swift b/prismTests/MainActorHopContractTests.swiftnew file mode 100644index 00000000..29c295af--- /dev/null+++ b/prismTests/MainActorHopContractTests.swift@@ -0,0 +1,59 @@+//+// MainActorHopContractTests.swift+// prismTests+//+// Pins the language behaviour the whole T-2219/T-2096/T-1541 fix rests on.+//+// The fix for those tickets is that every test able to construct WebKit is+// `async`. That is only a fix because an `async` member of a `@MainActor` type+// hops to the main actor's executor ON ENTRY, as part of the ABI — independently+// of which executor the caller was on. A SYNCHRONOUS `@MainActor` member gets no+// such hop: its isolation is realised only if the caller hops, and in this target+// (Swift 5 language mode, `SWIFT_APPROACHABLE_CONCURRENCY = YES`, no+// `SWIFT_DEFAULT_ACTOR_ISOLATION`) the callers are Swift Testing's+// `nonisolated(nonsending)` thunks running on the cooperative pool. WebKit's+// initialisers abort the shared test host when they run there, and every test+// still queued is then reported failed without having run.+//+// So the hop is load-bearing, and it is behaviour of a toolchain rather than a+// contract this repository controls. If a future Swift release stopped emitting+// it, `Tools/check-webkit-test-isolation.py` would still pass — it checks for+// `async`, which would still be there — and the host aborts would come back with+// every other test in the target green. This is the test that goes red instead.+//+// Parameterised rather than single, because the mis-hop is load-dependent: Swift+// Testing runs the cases concurrently, which is the condition under which a+// missing hop actually lands off-main rather than coincidentally on it.+//+// The synchronous counterpart is deliberately NOT asserted here. It is the+// broken case, it fails only sometimes, and a test that is red on a schedule is+// worse than no test — the static guard covers that direction instead, by+// ensuring no synchronous test body can reach WebKit in the first place.+//++import Foundation+import Testing++@MainActor+@Suite("Main-actor hop contract (T-2219)")+struct MainActorHopContractTests {++ @Test(+ "An async member of a @MainActor suite runs on the main thread",+ arguments: 0..<64+ )+ func asyncBodyRunsOnTheMainThread(_ index: Int) async {+ #expect(+ Thread.isMainThread,+ """+ Case \(index) of an `async` @MainActor test body executed off the main \+ thread, so actor-isolated async functions no longer hop on entry in this \+ toolchain. Marking a test `async` is then no longer enough to keep WebKit \+ construction on the main thread, and the live-WebKit suites can abort the \+ shared test host again (T-1541/T-2096/T-2219). The remedy at that point is \+ an explicit `await MainActor.run { }` around every WebKit construction, \+ plus a rule change in Tools/check-webkit-test-isolation.py.+ """+ )+ }+}
diff --git a/Makefile b/Makefileindex 6fa22ce8..a136ca4f 100644--- a/Makefile+++ b/Makefile@@ -85,6 +85,7 @@ help: @echo " test-locales - Run the test suite under en, en-AU, en-GB, en-US" @echo " test-locales-adhoc - test-locales, ad-hoc signed (CI, no certificate)" @echo " verify-make-guards - Check the test targets cannot report a false pass"+ @echo " verify-test-isolation - Check no test can construct WebKit off-main" @echo " install - Build and install Debug on device" @echo " run - Build, install, and launch Debug on device" @echo ""@@ -183,7 +184,13 @@ build: build-ios build-macos # global state (e.g. MockURLProtocol.handler, NSWindow notifications) that # surface when xcodebuild runs the locale matrix in parallel workers. Run # `make test-locales` to exercise the en-AU/en-GB/en-US configurations.-test-quick:+#+# verify-test-isolation runs FIRST because the defect it catches destroys the+# run it appears in: one off-main WebKit touch aborts the shared host and every+# still-queued test is reported failed without having run (T-1541/T-2096/T-2219).+# Half a second of static checking beats reading a four-figure fictional failure+# count out of a result bundle.+test-quick: verify-test-isolation @rm -rf $(RESULT_BUNDLE) -xcodebuild test \ -project $(PROJECT) \@@ -206,7 +213,7 @@ test-quick: # runs and has the final word. The guard reads the result bundle, which is the # only signal on this project that has never been observed lying. .PHONY: test-test:+test: verify-test-isolation @rm -rf $(RESULT_BUNDLE_IOS) -xcodebuild test \ -project $(PROJECT) \@@ -268,7 +275,7 @@ test-ui: # one CI runs, was the one place the project's own conclusion was not applied. # Tools/Tests/test-make-guards.sh check [8] keeps it that way. .PHONY: test-locales-test-locales:+test-locales: verify-test-isolation $(STRICT) xcodebuild build-for-testing \ -project $(PROJECT) \ -scheme $(SCHEME) \@@ -447,6 +454,25 @@ upload-all: upload upload-macos verify-make-guards: Tools/Tests/test-make-guards.sh +# The other way a test run can report something fictional. A synchronous+# @MainActor test body has no hop-on-entry in this target's build configuration+# (Swift 5 mode, Approachable Concurrency, no default actor isolation), so under+# load it executes on the cooperative pool; WebKit's initialisers assert they are+# on the main thread and abort the shared test host when they are not. Every test+# still queued is then reported failed with no recorded duration — 189 of them in+# one T-2219 run, 233 in another.+#+# The abort is a scheduling race, so no runnable test can pin it: a suite capable+# of crashing the host passes in isolation, every time. What is decidable from the+# source is whether any test body CAN make the off-main touch, which is what this+# checks — and it is the only check that covers a suite nobody has identified yet.+# Its own detection rules are unit-tested, because a guard that stopped detecting+# anything would look exactly like a clean repository.+.PHONY: verify-test-isolation+verify-test-isolation:+ $(STRICT) python3 Tools/check-webkit-test-isolation.py+ $(STRICT) python3 -m unittest Tools.Tests.test_webkit_test_isolation+ # Cleaning .PHONY: clean clean:
diff --git a/prismTests/WebRendering/WebDocumentControllerTests.swift b/prismTests/WebRendering/WebDocumentControllerTests.swiftindex 0597ae09..e6fec5ee 100644--- a/prismTests/WebRendering/WebDocumentControllerTests.swift+++ b/prismTests/WebRendering/WebDocumentControllerTests.swift@@ -49,7 +49,7 @@ struct WebDocumentControllerTests { // MARK: - Generation tagging (both directions) @Test("Outbound commands carry the controller's live generation tag")- func outboundCommandsCarryGeneration() {+ func outboundCommandsCarryGeneration() async { let controller = makeController(sessionID: "abc", parseRevision: 7) let generation = controller.currentGeneration #expect(generation.sessionID == "abc")@@ -66,7 +66,7 @@ struct WebDocumentControllerTests { } @Test("Inbound message matching the live generation is accepted")- func inboundCurrentGenerationAccepted() {+ func inboundCurrentGenerationAccepted() async { let controller = makeController() let result = controller.receive( messageBody: body(type: "ready", generation: controller.currentGeneration)@@ -76,7 +76,7 @@ struct WebDocumentControllerTests { } @Test("Generation that bridges as [AnyHashable: Any] is accepted (real-device WKScriptMessage shape)")- func inboundGenerationAnyHashableAccepted() {+ func inboundGenerationAnyHashableAccepted() async { let controller = makeController() let gen = controller.currentGeneration // A WKScriptMessage body's nested object can bridge as [AnyHashable: Any] on a@@ -92,7 +92,7 @@ struct WebDocumentControllerTests { } @Test("Generation delivered as a JSON string is accepted (defensive bridging)")- func inboundGenerationJSONStringAccepted() {+ func inboundGenerationJSONStringAccepted() async { let controller = makeController() let gen = controller.currentGeneration let json = "{\"sessionID\":\"\(gen.sessionID)\",\"parseRevision\":\(gen.parseRevision),"@@ -102,7 +102,7 @@ struct WebDocumentControllerTests { } @Test("Inbound message from a stale parse revision is dropped")- func inboundStaleRevisionDropped() {+ func inboundStaleRevisionDropped() async { let controller = makeController(sessionID: "s", parseRevision: 5) let stale = BridgeGeneration(sessionID: "s", parseRevision: 4, processGeneration: 0) let result = controller.receive(messageBody: body(type: "ready", generation: stale))@@ -111,7 +111,7 @@ struct WebDocumentControllerTests { } @Test("Inbound message from a stale process generation is dropped (Req 9.6 replay guard)")- func inboundStaleProcessGenerationDropped() {+ func inboundStaleProcessGenerationDropped() async { let controller = makeController() // Simulate a recovery that bumped the process generation. controller.handleProcessTermination(documentURL: URL(string: "prism-doc://document/x")!)@@ -121,7 +121,7 @@ struct WebDocumentControllerTests { } @Test("Inbound message from a different session is dropped (forged document)")- func inboundForeignSessionDropped() {+ func inboundForeignSessionDropped() async { let controller = makeController(sessionID: "mine") let forged = BridgeGeneration(sessionID: "theirs", parseRevision: 1, processGeneration: 0) let result = controller.receive(@@ -133,7 +133,7 @@ struct WebDocumentControllerTests { // MARK: - Message allowlist + forged-message rejection (Req 8.2) @Test("Unknown message types are dropped, never dispatched")- func unknownTypeDropped() {+ func unknownTypeDropped() async { let controller = makeController() let result = controller.receive( messageBody: body(type: "evalArbitraryCode", generation: controller.currentGeneration)@@ -142,14 +142,14 @@ struct WebDocumentControllerTests { } @Test("A non-dictionary body is dropped as a malformed envelope")- func nonDictionaryBodyDropped() {+ func nonDictionaryBodyDropped() async { let controller = makeController() let result = controller.receive(messageBody: "window.location = 'evil'") #expect(result == .dropped(.malformedEnvelope)) } @Test("A known type with a missing payload field is dropped, not crashed")- func malformedPayloadDropped() {+ func malformedPayloadDropped() async { let controller = makeController() // visibleBlock requires domID + fraction; omit fraction. let result = controller.receive(@@ -160,14 +160,14 @@ struct WebDocumentControllerTests { } @Test("A message with no generation tag is dropped")- func missingGenerationDropped() {+ func missingGenerationDropped() async { let controller = makeController() let result = controller.receive(messageBody: ["type": "ready"]) #expect(result == .dropped(.missingGeneration)) } @Test("Every inbound allowlist type decodes to a routed message")- func allowlistTypesDecode() {+ func allowlistTypesDecode() async { let controller = makeController() let generation = controller.currentGeneration var routed: [InboundBridgeMessage] = []@@ -207,7 +207,7 @@ struct WebDocumentControllerTests { // MARK: - selectionCandidate per-state decoding (Req 12.2) @Test("selectionCandidate 'available' decodes with blockID + range + rect")- func selectionCandidateAvailableDecodes() {+ func selectionCandidateAvailableDecodes() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -222,7 +222,7 @@ struct WebDocumentControllerTests { } @Test("selectionCandidate 'declined' decodes with rect only (cross-block, Req 12.4)")- func selectionCandidateDeclinedDecodes() {+ func selectionCandidateDeclinedDecodes() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -235,7 +235,7 @@ struct WebDocumentControllerTests { } @Test("selectionCandidate 'cleared' decodes with no fields")- func selectionCandidateClearedDecodes() {+ func selectionCandidateClearedDecodes() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -247,7 +247,7 @@ struct WebDocumentControllerTests { } @Test("selectionCandidate 'available' missing its range is dropped (forged/incoherent)")- func selectionCandidateAvailableMissingRangeDropped() {+ func selectionCandidateAvailableMissingRangeDropped() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -257,7 +257,7 @@ struct WebDocumentControllerTests { } @Test("selectionCandidate 'declined' missing its rect is dropped")- func selectionCandidateDeclinedMissingRectDropped() {+ func selectionCandidateDeclinedMissingRectDropped() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -267,7 +267,7 @@ struct WebDocumentControllerTests { } @Test("selectionCandidate with an unknown state is dropped")- func selectionCandidateUnknownStateDropped() {+ func selectionCandidateUnknownStateDropped() async { let controller = makeController() let result = controller.receive(messageBody: body( type: "selectionCandidate", generation: controller.currentGeneration,@@ -286,7 +286,7 @@ struct WebDocumentControllerTests { // and would otherwise re-arm the overlay after the clear. @Test("A selectionCandidate arriving before ready is not routed")- func selectionCandidateBeforeReadyDropped() {+ func selectionCandidateBeforeReadyDropped() async { let controller = makeController() var routed: [InboundBridgeMessage] = [] controller.onMessage = { routed.append($0) }@@ -307,7 +307,7 @@ struct WebDocumentControllerTests { } @Test("A selectionCandidate arriving after ready is routed")- func selectionCandidateAfterReadyRouted() {+ func selectionCandidateAfterReadyRouted() async { let controller = makeController() var routed: [InboundBridgeMessage] = [] controller.onMessage = { routed.append($0) }@@ -326,7 +326,7 @@ struct WebDocumentControllerTests { } @Test("A reload re-closes the gate: the outgoing page cannot re-arm mid-load")- func selectionCandidateDroppedAgainAfterReload() {+ func selectionCandidateDroppedAgainAfterReload() async { let controller = makeController(sessionID: "t1852-gate", parseRevision: 4) var routed: [InboundBridgeMessage] = [] controller.test_markReady()@@ -351,7 +351,7 @@ struct WebDocumentControllerTests { // MARK: - Queue-until-ready @Test("Commands sent before ready are queued, not dispatched")- func commandsQueueUntilReady() {+ func commandsQueueUntilReady() async { let controller = makeController() controller.setCommentVisibility(true) controller.applyTheme(theme: "prism-dark", contrast: .standard, variables: [:])@@ -360,7 +360,7 @@ struct WebDocumentControllerTests { } @Test("ready flushes queued non-scroll commands")- func readyFlushesQueue() {+ func readyFlushesQueue() async { let controller = makeController() controller.setCommentVisibility(true) controller.applyTheme(theme: "prism-dark", contrast: .standard, variables: [:])@@ -373,7 +373,7 @@ struct WebDocumentControllerTests { // MARK: - layoutSettled gates scroll restore @Test("scrollTo is held until layoutSettled even after ready")- func scrollHeldUntilLayoutSettled() {+ func scrollHeldUntilLayoutSettled() async { let controller = makeController() controller.scrollTo(blockID: "b-1-0") controller.test_markReady()@@ -387,7 +387,7 @@ struct WebDocumentControllerTests { } @Test("layoutSettled before ready does not dispatch — both milestones required for restore")- func layoutSettledAloneInsufficient() {+ func layoutSettledAloneInsufficient() async { let controller = makeController() controller.scrollTo(blockID: "b-1-0") controller.test_markLayoutSettled()@@ -430,7 +430,7 @@ struct WebDocumentControllerTests { // MARK: - WebContent termination recovery (Req 9.6) @Test("Process termination bumps the generation and re-queues the snapshot for replay")- func terminationReplaysSnapshot() {+ func terminationReplaysSnapshot() async { let controller = makeController() // Establish some native truth, then become ready and drain it. controller.applyTheme(theme: "prism-dark", contrast: .standard, variables: ["--a": "1"])@@ -454,7 +454,7 @@ struct WebDocumentControllerTests { } @Test("Reload for a new revision updates the generation and discards the live queue")- func reloadUpdatesGenerationDiscardsQueue() {+ func reloadUpdatesGenerationDiscardsQueue() async { let controller = makeController(parseRevision: 1) controller.setCommentVisibility(true) // queued at rev 1 controller.load(documentURL: URL(string: "prism-doc://document/x?rev=2")!, parseRevision: 2)
diff --git a/CLAUDE.md b/CLAUDE.mdindex 17acc8b7..049aa928 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -310,6 +310,22 @@ Two related traps live in the Makefile: `make verify-make-guards` asserts both and runs on every push. +The third way a run reports something fictional is the host abort, and it has its+own guard. `make verify-test-isolation` (`Tools/check-webkit-test-isolation.py`,+a prerequisite of `test-quick`, `test` and `test-locales`) fails when a `@Test` in+`prismTests` can reach a WebKit constructor from a **synchronous** body. A sync+`@MainActor` function gets no hop-on-entry in this target's build configuration,+so under load it runs on the cooperative pool, WebKit's main-thread assertion+aborts the shared host, and every still-queued test is reported failed without+having run — 189 fictional failures in one T-2219 run, 233 in another. The rule+is: **touch WebKit only from an `async` test in a `@MainActor` suite**+(T-1541, T-2096, T-2219). Both halves matter — `async` buys a hop to the+ENCLOSING actor, so without `@MainActor` there is no actor to hop to and the+body runs on the cooperative pool exactly as a synchronous one would. The check+is static because the abort is a scheduling race — the guilty suite passes in+isolation every time, and the suite the cascade *names* is usually not the guilty+one. See `docs/agent-notes/development-tooling.md`.+ ### Test Coverage - Unit tests (`prismTests`): parsers, cache, file observer, models
diff --git a/Tools/Tests/test-make-guards.sh b/Tools/Tests/test-make-guards.shindex cadfac08..1c3d9126 100755--- a/Tools/Tests/test-make-guards.sh+++ b/Tools/Tests/test-make-guards.sh@@ -267,6 +267,17 @@ cat > "$SANDBOX/bin/xcbeautify" <<'EOF' #!/bin/bash cat >/dev/null EOF+# python3: test-locales depends on verify-test-isolation, and that prerequisite's+# recipe lines appear in `make -n` output and are replayed here. The real check+# reads the repository's prismTests/ tree, which this sandbox deliberately does+# not have. This test is about the sweep's CONTROL FLOW — that one failing+# configuration does not abort the rest — so the isolation check is stubbed to+# succeed rather than handed a fake tree; its own detection rules are covered by+# Tools/Tests/test_webkit_test_isolation.py.+cat > "$SANDBOX/bin/python3" <<'EOF'+#!/bin/bash+exit 0+EOF # The guard: log which configuration it was asked about, and fail the first one. cat > "$SANDBOX/Tools/check-test-results.sh" <<'EOF' #!/bin/bash@@ -274,7 +285,8 @@ printf '%s\n' "${2:-}" >> guard.log [ "${2:-}" = "test-locales: en (base)" ] && exit 1 exit 0 EOF-chmod +x "$SANDBOX/bin/xcodebuild" "$SANDBOX/bin/xcbeautify" "$SANDBOX/Tools/check-test-results.sh"+chmod +x "$SANDBOX/bin/xcodebuild" "$SANDBOX/bin/xcbeautify" "$SANDBOX/bin/python3" \+ "$SANDBOX/Tools/check-test-results.sh" # Reassemble `make -n` output into logical recipe lines: a line ending in `\` # continues into the next, exactly as make hands the block to one shell.
diff --git a/prismTests/WebRendering/WebContentTerminationWiringTests.swift b/prismTests/WebRendering/WebContentTerminationWiringTests.swiftindex af4e0d6e..9750c60d 100644--- a/prismTests/WebRendering/WebContentTerminationWiringTests.swift+++ b/prismTests/WebRendering/WebContentTerminationWiringTests.swift@@ -244,7 +244,7 @@ struct WebContentTerminationWiringTests { // MARK: - 3. What the controller does with the signal @Test("An observed termination runs the recovery the handler implements")- func observedTerminationTriggersRecovery() {+ func observedTerminationTriggersRecovery() async { let controller = makeController() controller.applyTheme(theme: "prism-dark", contrast: .standard, variables: ["--a": "1"]) controller.load(@@ -271,7 +271,7 @@ struct WebContentTerminationWiringTests { } @Test("An ordinary navigation failure neither recovers nor stops observation")- func navigationFailureKeepsObservingWithoutRecovering() {+ func navigationFailureKeepsObservingWithoutRecovering() async { // The re-subscription half of the fix. `page.navigations` ends when it throws, // and it throws for benign failures, so returning `false` here would let one // bad link permanently disarm crash recovery for the rest of the session.@@ -289,7 +289,7 @@ struct WebContentTerminationWiringTests { } @Test("A termination before any load does not reload, but keeps observing")- func terminationBeforeFirstLoadIsSurvivable() {+ func terminationBeforeFirstLoadIsSurvivable() async { let controller = makeController() #expect(controller.applyNavigationOutcome(.webContentTerminated) == true) #expect(@@ -299,7 +299,7 @@ struct WebContentTerminationWiringTests { } @Test("Recovery stops after repeated attempts that never reach ready")- func unproductiveRecoveriesAreCapped() {+ func unproductiveRecoveriesAreCapped() async { // A WebContent process that cannot be relaunched would otherwise have the // observer reload in a hot loop forever. let controller = makeController()@@ -317,7 +317,7 @@ struct WebContentTerminationWiringTests { } @Test("A recovery that reaches layoutSettled restores the retry budget")- func productiveRecoveryResetsTheBudget() {+ func productiveRecoveryResetsTheBudget() async { let controller = makeController() controller.load( documentURL: URL(string: "prism-doc://document/x?rev=1")!, parseRevision: 1@@ -343,7 +343,7 @@ struct WebContentTerminationWiringTests { } @Test("Crashes that keep landing after ready but before layoutSettled stay in the same chain")- func crashesAfterReadyButBeforeLayoutSettledStayInTheSameChain() {+ func crashesAfterReadyButBeforeLayoutSettledStayInTheSameChain() async { // T-2107 regression: `markReady` clears `recoveryInFlight` well before the page // is actually stable. A WebContent process that reliably crashes after `ready` // but before `layoutSettled` must still accumulate toward the cap and eventually@@ -375,7 +375,7 @@ struct WebContentTerminationWiringTests { // MARK: - 4. A recovery reload that fails as an ordinary navigation @Test("A recovery reload that fails as an ordinary navigation is retried")- func failedRecoveryReloadIsRetried() {+ func failedRecoveryReloadIsRetried() async { // The sharp edge: `handleProcessTermination` reloads on the relaunched // process, and that reload can fail PROVISIONALLY — which the stream reports // as an ordinary `failedProvisionalNavigation`, not as a termination. Treating@@ -402,7 +402,7 @@ struct WebContentTerminationWiringTests { } @Test("Recovery reloads that keep failing as navigation failures exhaust the budget")- func failedRecoveryReloadsChargeTheBudget() {+ func failedRecoveryReloadsChargeTheBudget() async { // The retry above must be BOUNDED by the same budget a repeated crash is, or // the new path just moves the hot loop rather than removing it. let controller = makeController()@@ -419,7 +419,7 @@ struct WebContentTerminationWiringTests { } @Test("An ordinary navigation failure outside a recovery does not charge the budget")- func benignNavigationFailureDoesNotChargeTheBudget() {+ func benignNavigationFailureDoesNotChargeTheBudget() async { // A bad link on a healthy, ready page must not consume crash-recovery budget. let controller = makeController() controller.load(@@ -443,7 +443,7 @@ struct WebContentTerminationWiringTests { } @Test("A recovery that reaches ready makes a later navigation failure benign again")- func readyAfterRecoveryRestoresBenignFailures() {+ func readyAfterRecoveryRestoresBenignFailures() async { // The exact discrimination the whole design rests on, in the direction the other // tests do not travel: `recoveryInFlight` must come back DOWN on the recovered // page's `ready`, or the next bad link in the document would be misread as a@@ -470,7 +470,7 @@ struct WebContentTerminationWiringTests { // MARK: - 5. Abandonment is recoverable, and visible @Test("Abandoning recovery raises a user-visible surface")- func abandonmentIsSurfacedToTheUser() {+ func abandonmentIsSurfacedToTheUser() async { let controller = makeController() controller.load( documentURL: URL(string: "prism-doc://document/x?rev=1")!, parseRevision: 1
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.ymlindex a9d864c0..a3f3b411 100644--- a/.github/workflows/checks.yml+++ b/.github/workflows/checks.yml@@ -36,6 +36,15 @@ jobs: - name: Verify test targets cannot report a false pass run: make verify-make-guards + # The other way a run reports something fictional. A test that constructs+ # WebKit from a synchronous body aborts the shared test host under load,+ # and every still-queued test is then reported failed without having run+ # (189 of them in one T-2219 run, 233 in another). The check is static+ # because the abort is a scheduling race — the guilty suite passes in+ # isolation, every time — so it runs here, on Linux, in under a second.+ - name: Verify no test can construct WebKit off the main thread+ run: make verify-test-isolation+ - name: Check for large files run: | max_size=1048576 # 1MB
diff --git a/prismTests/WebRendering/WebSearchScrollOwnershipTests.swift b/prismTests/WebRendering/WebSearchScrollOwnershipTests.swiftindex df1bc39e..5f3de6a3 100644--- a/prismTests/WebRendering/WebSearchScrollOwnershipTests.swift+++ b/prismTests/WebRendering/WebSearchScrollOwnershipTests.swift@@ -359,7 +359,7 @@ struct WebSearchScrollOwnershipTests { } @Test("An undelivered reveal outranks the stored-position restore for its load")- func undeliveredRevealBlocksRestore() {+ func undeliveredRevealBlocksRestore() async { let controller = makeUnreadyController() // A user navigation while the page is still loading: the reveal queues. controller.setSearchState(json: "{\"query\":\"n\",\"blocks\":{}}", reveal: true)@@ -381,7 +381,7 @@ struct WebSearchScrollOwnershipTests { } @Test("A reveal that dispatches straight through raises no claim")- func deliveredRevealDoesNotBlockRestore() {+ func deliveredRevealDoesNotBlockRestore() async { let controller = makeUnreadyController() controller.test_markReady() controller.test_markLayoutSettled()@@ -391,7 +391,7 @@ struct WebSearchScrollOwnershipTests { } @Test("A non-reveal push never blocks the restore")- func nonRevealPushDoesNotBlockRestore() {+ func nonRevealPushDoesNotBlockRestore() async { let controller = makeUnreadyController() controller.setSearchState(json: "{\"query\":\"n\",\"blocks\":{}}", reveal: false) controller.restoreScroll(blockID: "b-saved-0")@@ -399,7 +399,7 @@ struct WebSearchScrollOwnershipTests { } @Test("The snapshot replays search state without the reveal once delivered")- func snapshotStripsDeliveredReveal() {+ func snapshotStripsDeliveredReveal() async { let controller = makeUnreadyController() controller.test_markReady() controller.test_markLayoutSettled()@@ -416,7 +416,7 @@ struct WebSearchScrollOwnershipTests { } @Test("An undelivered reveal survives the load, re-queued after the restore target")- func undeliveredRevealSurvivesLoadAfterRestoreTarget() {+ func undeliveredRevealSurvivesLoadAfterRestoreTarget() async { let controller = makeUnreadyController() // The restore target is already part of native truth for this load… controller.restoreScroll(blockID: "b-saved-0")
diff --git a/prismTests/WebRendering/WebSelectionNoteTests.swift b/prismTests/WebRendering/WebSelectionNoteTests.swiftindex e7774dae..0245aebd 100644--- a/prismTests/WebRendering/WebSelectionNoteTests.swift+++ b/prismTests/WebRendering/WebSelectionNoteTests.swift@@ -497,7 +497,7 @@ struct WebSelectionNoteTests { } @Test("A reload for a new parse revision clears the selection affordance")- func reloadClearsSelectionAffordance() {+ func reloadClearsSelectionAffordance() async { let affordance = armedAffordance() #expect(affordance.canAddNote == true) @@ -512,7 +512,7 @@ struct WebSelectionNoteTests { } @Test("A same-revision reload (image-access retry) clears the selection affordance")- func sameRevisionReloadClearsSelectionAffordance() {+ func sameRevisionReloadClearsSelectionAffordance() async { // The iOS grant-folder-access path reloads at the SAME revision. The page is // still replaced, so the affordance is just as stale as after a re-parse. let affordance = armedAffordance()@@ -523,7 +523,7 @@ struct WebSelectionNoteTests { } @Test("WebContent process recovery clears the selection affordance")- func processRecoveryClearsSelectionAffordance() {+ func processRecoveryClearsSelectionAffordance() async { let affordance = armedAffordance() let controller = makeController() controller.selectionAffordance = affordance@@ -605,7 +605,7 @@ struct WebSelectionNoteTests { /// the controller, and leaves every assembly test above green. Source-structural, /// matching the repo's other wiring pins (FootnotePresentationHostTests). @Test("DocumentScrollContent passes the affordance into makeAssembly")- func scrollContentPassesAffordanceToAssembly() throws {+ func scrollContentPassesAffordanceToAssembly() async throws { let source = try String( contentsOf: URL(fileURLWithPath: #filePath) .deletingLastPathComponent() // prismTests/WebRendering
diff --git a/prismTests/MermaidRendererTests.swift b/prismTests/MermaidRendererTests.swiftindex 25e4f7eb..19b431b5 100644--- a/prismTests/MermaidRendererTests.swift+++ b/prismTests/MermaidRendererTests.swift@@ -250,7 +250,7 @@ struct MermaidRendererTests { /// Test that MermaidRenderer initializes with configurable timeout. @Test("MermaidRenderer accepts custom timeout")- func rendererAcceptsCustomTimeout() {+ func rendererAcceptsCustomTimeout() async { let renderer = MermaidRenderer(timeoutSeconds: 5) // If we get here without errors, the renderer initialized correctly _ = renderer@@ -258,7 +258,7 @@ struct MermaidRendererTests { /// Test that MermaidRenderer initializes with default timeout. @Test("MermaidRenderer uses default 10 second timeout")- func rendererUsesDefaultTimeout() {+ func rendererUsesDefaultTimeout() async { let renderer = MermaidRenderer() // Default timeout is 10 seconds - initialization should succeed _ = renderer@@ -268,7 +268,7 @@ struct MermaidRendererTests { /// Test that MermaidRenderer initializes on iOS. /// Requirement 3.4: Attach hidden WKWebView to UIWindow on iOS. @Test("iOS: MermaidRenderer initializes and attempts window attachment")- func iOSRendererInitializesWithWindowAttachment() {+ func iOSRendererInitializesWithWindowAttachment() async { let renderer = MermaidRenderer() // On iOS, attachToWindowLegacy is called during init // If we get here, initialization succeeded@@ -288,7 +288,7 @@ struct MermaidRendererTests { /// Actual (before fix): The retain cycle keeps the renderer alive forever, /// so the weak reference still points to a live object. @Test("MermaidRenderer deallocates when released (no retain cycle)")- func rendererDeallocatesWhenReleased() {+ func rendererDeallocatesWhenReleased() async { weak var weakRenderer: MermaidRenderer? autoreleasepool { let renderer = MermaidRenderer()
diff --git a/prismTests/WebRendering/WebScrollabilityReportingTests.swift b/prismTests/WebRendering/WebScrollabilityReportingTests.swiftindex 106768d9..4b342972 100644--- a/prismTests/WebRendering/WebScrollabilityReportingTests.swift+++ b/prismTests/WebRendering/WebScrollabilityReportingTests.swift@@ -167,7 +167,7 @@ struct WebScrollabilityReportingTests { } @Test("scrollabilityChanged decodes through the audited bridge allowlist")- func scrollabilityChangedDecodes() {+ func scrollabilityChangedDecodes() async { let controller = makeWebController() let body: [String: Any] = [ "type": "scrollabilityChanged",@@ -178,7 +178,7 @@ struct WebScrollabilityReportingTests { } @Test("A scrollabilityChanged without a boolean flag is dropped, not defaulted")- func scrollabilityChangedMalformedDropped() {+ func scrollabilityChangedMalformedDropped() async { let controller = makeWebController() let body: [String: Any] = [ "type": "scrollabilityChanged",@@ -189,7 +189,7 @@ struct WebScrollabilityReportingTests { } @Test("A stale-generation scrollabilityChanged is dropped")- func scrollabilityChangedStaleGenerationDropped() {+ func scrollabilityChangedStaleGenerationDropped() async { let controller = makeWebController() let stale = BridgeGeneration(sessionID: "t1932", parseRevision: 0, processGeneration: 0) let body: [String: Any] = [
diff --git a/prismTests/WebRendering/WebScrollIntegrationContractTests.swift b/prismTests/WebRendering/WebScrollIntegrationContractTests.swiftindex 6f8c4739..b05693f1 100644--- a/prismTests/WebRendering/WebScrollIntegrationContractTests.swift+++ b/prismTests/WebRendering/WebScrollIntegrationContractTests.swift@@ -39,7 +39,7 @@ struct WebScrollIntegrationContractTests { } @Test("scrollDirectionChanged decodes through the audited bridge allowlist")- func scrollDirectionChangedDecodes() {+ func scrollDirectionChangedDecodes() async { let controller = makeController() let body: [String: Any] = [ "type": "scrollDirectionChanged",@@ -52,7 +52,7 @@ struct WebScrollIntegrationContractTests { } @Test("A malformed scrollDirectionChanged payload is dropped, not defaulted")- func scrollDirectionChangedMalformedDropped() {+ func scrollDirectionChangedMalformedDropped() async { let controller = makeController() let body: [String: Any] = [ "type": "scrollDirectionChanged",
diff --git a/prismTests/WebRendering/WebPerfProbeTests.swift b/prismTests/WebRendering/WebPerfProbeTests.swiftindex d98040e0..a3b843ea 100644--- a/prismTests/WebRendering/WebPerfProbeTests.swift+++ b/prismTests/WebRendering/WebPerfProbeTests.swift@@ -34,7 +34,7 @@ struct WebPerfProbeRoutingTests { } @Test("perfSample is accepted and routed to onMessage with its numeric payload")- func perfSampleRouted() {+ func perfSampleRouted() async { let controller = makeController() var received: InboundBridgeMessage? controller.onMessage = { received = $0 }@@ -52,7 +52,7 @@ struct WebPerfProbeRoutingTests { } @Test("diagFailure is accepted and routed with its category (Req 11.5)")- func diagFailureRouted() {+ func diagFailureRouted() async { let controller = makeController() var received: InboundBridgeMessage? controller.onMessage = { received = $0 }
diff --git a/prismTests/WebRendering/WebFootnotePopoverTests.swift b/prismTests/WebRendering/WebFootnotePopoverTests.swiftindex 245078ca..ae77d8b1 100644--- a/prismTests/WebRendering/WebFootnotePopoverTests.swift+++ b/prismTests/WebRendering/WebFootnotePopoverTests.swift@@ -94,7 +94,7 @@ struct WebFootnotePopoverTests { } @Test("present then reset leaves no footnote id and empty content")- func presentThenReset() {+ func presentThenReset() async { let page = FootnotePopoverWebPage() page.present(footnoteId: "1", data: footnoteData(), settings: RenderSettings()) #expect(page.footnoteId == "1")@@ -244,7 +244,7 @@ struct WebFootnotePopoverTests { } @Test("An unchanged re-present does not reload; a changed input does")- func rePresentIsDedupedByRenderKey() {+ func rePresentIsDedupedByRenderKey() async { let page = FootnotePopoverWebPage() let data = footnoteData()
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8c650b61..8fde0c44 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Live-WebKit test suites no longer abort the shared test host, which used to turn a whole run into a fictional four-figure failure count (T-2219, T-2096). One process hosts the entire unit-test target, so an abort reports every still-queued test as a failure it never ran — 189 of them in one observed run, 233 in another. The cause was the one T-1541 diagnosed and fixed for a single suite: a synchronous `@MainActor` test body gets no hop-on-entry in this target's build configuration, so under load it runs on the cooperative pool and WebKit's main-thread assertion kills the process. Sixty tests across nine suites could still do that, including the two the tickets name and two suites no ticket had ever mentioned — which is why the abort kept being attributed to a different suite each time, usually whichever long-running live-WebKit test happened to be in flight. All sixty are now `async`, which hops as part of the ABI, and `make verify-test-isolation` fails the build if a synchronous test can reach WebKit again. Nothing is skipped or excluded; the number of tests executed is unchanged. - CI now runs the test suite instead of only appearing to (T-1983). The per-locale sweep is the only job that can execute tests, and it reported success while executing none: its build failed for want of a signing certificate on the runner, that failure was swallowed, and nothing checked that any test had run. The sweep now signs ad-hoc so the build succeeds without a certificate, every test target hands its result bundle to the zero-test guard — per locale configuration, not once at the end — and any recipe whose failure must be believed carries `$(STRICT)`, because `.SHELLFLAGS` is silently ignored by the GNU Make 3.81 that macOS ships. `make verify-make-guards` asserts all of this on every push. - Copy notes is now the primary notes action (T-1577). On iPhone the document screen's toolbar shows Copy notes instead of Share with Notes, which moved into the notes pane alongside copy; on iPad and Mac the top toolbar shows copy leading the export button. Every copy button appears exactly when the copy output would contain at least one note under the current export settings, each action carries an accessibility label and help text, and an export blocked by the paywall from inside the pane now retries fully — including the author-name prompt and its confirmation toast — after a purchase completes. - The test suite covering notes-action placement was retargeted to the new contracts (T-1577): the T-138 share-button parity tests became the Share-with-Notes placement contract, and visibility/payload tests now assert the shared copy-availability predicate and the single export payload call site. Two device-only checks are recorded in the spec for manual verification.
Both PRs edit Makefile in the same region: #379 rewrites the comment block above verify-make-guards and appends Tools/Tests/test-check-test-results.sh to its recipe, while this PR inserts the new verify-test-isolation target immediately after that recipe. Both also insert at the top of the CHANGELOG's ### Changed list. Git will report a textual conflict in both files, but both sides are pure insertions into the same region — resolve by keeping both. There is no semantic interaction: the two targets are independent, and verify-test-isolation does not go through check-test-results.sh.
The macOS destination cannot launch a test host at all (T-2146, testmanagerd wedged; six attempts, ~705 s each, zero tests executed), and the historical cascades came from ~3,900-test macOS runs. The author records this as "not tested", not "does not happen" — the correct posture. Given the crash is probabilistic (T-1541's own baseline run on a tree that demonstrably could crash did not crash), a green run would have been necessary but not sufficient anyway. The weight is carried by the structural argument, which is checkable and now checked — including, after this review, against three shapes that previously slipped through it.
Measured, not asserted: the historical 60 break down as 7 direct constructions and 53 through three same-file func helpers. The entire non-func member walker — computed properties, observers, lazy vars, stored closures, ambient init/deinit/subscript, nested types — has zero current and zero historical hits. The report is honest about this ("it only added shapes nobody had written yet") so it is not an overclaim, and for a guard that is the whole durability argument the prophylaxis is defensible. It cuts both ways, though: that speculative machinery is where two of the three real bugs were hiding. Reviewing it hard was worth it; growing it further without a concrete shape to catch would not be.
Three are worth pre-reading. A parse desync reports "the guard parsed N of M @Test declarations… Fix the parser, not the file" — correct advice, but someone who just added a regex literal will not connect the two. A one-line-type violation reads "constructs WebKit via Fixture" rather than naming a member, because the members were never split out. And the new isolation failure reads "is `async` but neither it nor its suite is @MainActor" — accurate, but it will be the first time most readers learn the rule has two halves. All acceptable; none self-explanatory.
A review scratch directory was briefly committed and then filtered out of the three review commits with git filter-branch. The branch has not been pushed since, so no force-push is needed beyond the ordinary first push — but if this branch was already pushed before the review, the push will need --force-with-lease. The final tree is 19 files against origin/main, exactly the set the PR started with.