Pre-push review: T-2198 test-only changes bypass the sole test workflow

diff --git a/.github/workflows/localisation-tests.yml b/.github/workflows/localisation-tests.ymlindex 446be6f1..cac227aa 100644--- a/.github/workflows/localisation-tests.yml+++ b/.github/workflows/localisation-tests.yml@@ -5,6 +5,13 @@ on:     branches: [main]     paths:       - 'prism/**'+      - 'prismTests/**'+      - 'prismUITests/**'+      - 'prism.xcodeproj/**'+      - 'prism.xctestplan'+      # Read from disk at RUN TIME by the parity/samples suites; nothing+      # compiles it, which is how it was missed the first time (T-2198).+      - 'samples/**'       - 'Tools/**'       - 'specs/localisation/**'       - 'Makefile'@@ -13,6 +20,13 @@ on:     branches: [main]     paths:       - 'prism/**'+      - 'prismTests/**'+      - 'prismUITests/**'+      - 'prism.xcodeproj/**'+      - 'prism.xctestplan'+      # Read from disk at RUN TIME by the parity/samples suites; nothing+      # compiles it, which is how it was missed the first time (T-2198).+      - 'samples/**'       - 'Tools/**'       - 'specs/localisation/**'       - 'Makefile'

diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.ymlindex a3f3b411..5a908ca5 100644--- a/.github/workflows/checks.yml+++ b/.github/workflows/checks.yml@@ -45,6 +45,19 @@ jobs:       - name: Verify no test can construct WebKit off the main thread         run: make verify-test-isolation +      # localisation-tests.yml is the only workflow that actually runs tests,+      # and it is paths-filtered, which means it fails silently (by skipping)+      # rather than loudly when its filter misses an input. A test-only+      # change — a test file, test target membership, the test plan, Xcode+      # project settings, or a document under samples/ that the parity suites+      # open at run time — used to match none of its paths and merge with no+      # test run at all (T-2198). This workflow has no paths filter, so it+      # cannot be bypassed the same way; the guard asserts that too, along+      # with the presence of this step, since a filter added here or a step+      # deleted here would silence it exactly as quietly.+      - name: Verify the sole test workflow's paths filter covers every build/test input+        run: make verify-workflow-triggers+       - name: Check for large files         run: |           max_size=1048576  # 1MB

diff --git a/Tools/check-workflow-triggers.py b/Tools/check-workflow-triggers.pynew file mode 100644index 00000000..8f5122cf--- /dev/null+++ b/Tools/check-workflow-triggers.py@@ -0,0 +1,358 @@+#!/usr/bin/env python3+"""Fail when the only test-executing workflow can skip test-only changes.++Why this guard exists+----------------------+`.github/workflows/localisation-tests.yml` is the repository's sole+test-executing workflow — `checks.yml` runs on Linux and never invokes+xcodebuild. Both its `on.push.paths` and `on.pull_request.paths` filters used+to omit `prismTests/**`, `prismUITests/**`, `prism.xctestplan`, and+`prism.xcodeproj/**`, so a push or PR that only changed a test file, test+target membership, the test plan, or Xcode project settings ran no build and+no tests at all (T-2198). A compile-broken test, or a coverage-disabling plan+edit, could merge with a fully green run that had tested none of it.++What counts as an input+-----------------------+Not "what Xcode compiles" — "what the suite READS while it runs". That+distinction is the one the first fix got wrong: it covered the compiled+targets and the build tooling and still missed `samples/**`, which no target+compiles but three suites open from disk at run time+(`ParityFixtureSupport.samplesDirectory()` resolves the repo-root directory+from `#filePath`; `SamplesComplianceTests` enumerates `samples/*.md`;+`OffMainEmitTests` times `samples/large-html-heavy.md`). A samples-only+commit therefore ran no tests either. When adding a required path, ask what+the tests open, not what the project file lists.++This does a light textual parse of the workflow's two `paths:` lists (the+file's structure is a simple, fixed, flat list under each trigger) rather+than depending on a YAML library that may not be installed on every runner,+and asserts every path pattern that can change what `make test-locales-adhoc`+builds or runs is present in BOTH lists, so the two filters cannot silently+drift apart again.++It fails closed. Every line under a trigger that this parser does not+recognise — a quoted key, a `paths-ignore:`, an inline `paths: [...]`, a `!`+negation, a flow-style continuation — is collected and REFUSED rather than+skipped, because "the parser saw nothing here" and "there is no filter here"+are the same thing to a reader of the result and opposite things to CI.++It also guards its own runner: `checks.yml` is what executes this check on+every push, so a `paths` filter appearing THERE — or the loss of the step that+invokes it — would silence the guard by the same mechanism the guard exists to+catch.+"""++import re+import sys+from pathlib import Path++REPO_ROOT = Path(__file__).resolve().parent.parent+WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "localisation-tests.yml"+# The workflow that RUNS this guard. It must stay unfiltered; see+# check_guard_workflow().+GUARD_WORKFLOW_PATH = REPO_ROOT / ".github" / "workflows" / "checks.yml"+# The command `checks.yml` must still invoke. A guard nothing runs is the+# same silence as a filter that skips. Matched as a `run:` step (or a line of+# a `run: |` block), never as a substring, so a passing mention of the target+# in a COMMENT — this file's own explanation of the rule, for one — cannot+# stand in for the step.+GUARD_INVOCATION = "make verify-workflow-triggers"+GUARD_INVOCATION_RE = re.compile(+    rf"^[^\S\n]*(?:run:[^\S\n]*)?{re.escape(GUARD_INVOCATION)}[^\S\n]*$",+    re.MULTILINE,+)++# Every input that can change what the per-locale sweep builds or runs — and+# that includes what the suite READS AT RUN TIME, not only what the project+# compiles. `samples/**` is the entry that makes the difference explicit: no+# target compiles it, three suites open it from disk.+REQUIRED_PATHS = [+    "prism/**",+    "prismTests/**",+    "prismUITests/**",+    "prism.xcodeproj/**",+    "prism.xctestplan",+    "samples/**",+    "Tools/**",+    "specs/localisation/**",+    "Makefile",+    ".github/workflows/localisation-tests.yml",+]++# The `on:` block itself. The parser only recognises triggers INSIDE it, so a+# job named `push:` under `jobs:` cannot register as a trigger and report a+# workflow as unconditionally triggered. An inline value (`on: [push]`) does+# not open a block, so it leaves both triggers unlocated — a fail-closed+# outcome, since this parser cannot read the flow form.+ON_KEY_RE = re.compile(r"^(?:on|'on'|\"on\"):\s*(?:#.*)?$")+# The two triggers `check()` reads.+TRIGGER_RE = re.compile(r"^  (push|pull_request):\s*(?:#.*)?$")+# Any other two-space-indented key under `on:` (`workflow_dispatch:`,+# `schedule:`, `workflow_call:` ...) is a SIBLING trigger: it ends the+# current trigger's scope so its own nested keys (`inputs:` ...) are not+# misread as filters on the previous trigger. It never STARTS a scope, so a+# quoted trigger name (`'push':`) is a sibling rather than a located trigger+# — the guard then reports push as not found, which is the fail-closed side.+SIBLING_KEY_RE = re.compile(r"^  [^\s#-][^:]*:")+# Any unindented key (`jobs:`, `env:`, ...) ends the `on:` block.+TOP_LEVEL_KEY_RE = re.compile(r"^[^\s#-][^:]*:")+# A filter key directly under a trigger (`paths:`, `paths-ignore:`,+# `branches:` ...), with whatever follows the colon captured so an inline+# value (`paths: ['prism/**']`) can be recognised and refused. A trailing+# comment (`paths:  # see T-2198`) is not a value: the key still opens a+# block. The key may be quoted: `'paths-ignore':` is the same filter as+# `paths-ignore:`, and a key group of `[A-Za-z_-]+` did not match it, so the+# whole line fell through unmatched and the filter vanished — `paths-ignore`+# read as "no filter at all" and passed with zero failures.+FILTER_KEY_RE = re.compile(+    r"^    (?:'([^']*)'|\"([^\"]*)\"|([^\s'\"#-][^:]*?))\s*:\s*(.*?)\s*(?:#.*)?$"+)+# Filter keys that cannot change WHICH FILES trigger the workflow, so the+# parser may ignore them (and the nested content they open). Anything else is+# refused rather than skipped — `paths-ignore:` in particular is a paths+# filter this parser does not understand, and treating it as "no filter"+# would read an exclusion list as an unconditional trigger and pass silently.+IGNORABLE_FILTER_KEYS = frozenset(+    {"branches", "branches-ignore", "tags", "tags-ignore", "types"}+)+COMMENT_RE = re.compile(r"^\s*#")+# One list entry: single-quoted, double-quoted, or bare, with optional+# trailing whitespace or a trailing comment. A stricter pattern (6-space,+# single-quoted, nothing after) silently DROPPED any entry that did not+# match, and every entry after it, which read as "missing" only by luck. The+# indent floor is four rather than six because YAML permits a sequence at the+# same indentation as its key.+PATH_ENTRY_RE = re.compile(+    r"^\s{4,}-\s*(?:'([^']*)'|\"([^\"]*)\"|([^'\"#\s][^#]*?))\s*(?:#.*)?$"+)+++def _first_group(groups):+    return next(g for g in groups if g is not None)+++def extract_trigger_filters(text: str) -> tuple:+    """Parse the `on:` block.++    Returns `(triggers, unsupported)`: `triggers` maps each trigger name+    (push, pull_request) the parser LOCATED to the entries of its `paths:`+    filter, in file order, or to None when the trigger has no `paths:`+    filter at all — it then runs unconditionally, which trivially satisfies+    "every test-only change gets a test signal". `unsupported` lists+    `(trigger, token)` pairs for everything under a trigger the parser+    cannot interpret: an unreadable filter key (`paths-ignore`, an inline+    `paths: [...]`) named by its key, and any other unrecognised line named+    by its stripped text.++    Nothing under a trigger is skipped except the nested content of a key+    the parser has already dealt with: an ignorable one (`types:` and its+    list of event names, which cannot express a path filter) or one it has+    just REFUSED (the entries under a `paths-ignore:` add nothing to the+    failure its key already produced). Everything else is collected.++    "Located but unfiltered" (None) and "never located" (absent) are kept+    distinct on purpose: a trigger this parser did not find — the file+    re-indented to four spaces, a quoted `'push':`, or an `on:` block with+    neither trigger — tells the caller nothing about whether the workflow+    runs, and the callers must fail on it rather than read absence as+    "unconditional".+    """+    paths: dict = {}+    unsupported: list = []+    in_on = False+    current = None+    in_paths = False+    in_opaque_block = False+    for line in text.splitlines():+        if COMMENT_RE.match(line) or not line.strip():+            continue+        if TOP_LEVEL_KEY_RE.match(line):+            in_on = bool(ON_KEY_RE.match(line))+            current = None+            in_paths = False+            in_opaque_block = False+            continue+        if not in_on:+            continue+        trigger_match = TRIGGER_RE.match(line)+        if trigger_match:+            current = trigger_match.group(1)+            in_paths = False+            in_opaque_block = False+            paths.setdefault(current, None)+            continue+        if SIBLING_KEY_RE.match(line):+            current = None+            in_paths = False+            in_opaque_block = False+            continue+        if current is None:+            continue+        key_match = FILTER_KEY_RE.match(line)+        if key_match:+            *key_forms, inline_value = key_match.groups()+            key = _first_group(key_forms)+            in_paths = False+            in_opaque_block = False+            if key == "paths" and not inline_value:+                in_paths = True+                paths[current] = []+            else:+                if key not in IGNORABLE_FILTER_KEYS:+                    unsupported.append((current, key))+                # Either way the key is settled, so whatever it nests is+                # not a filter this parser still has to read.+                in_opaque_block = True+            continue+        if in_paths:+            entry_match = PATH_ENTRY_RE.match(line)+            if entry_match:+                paths[current].append(_first_group(entry_match.groups()))+                continue+            # A line inside a `paths:` block that is not a readable entry.+            # Dropping it would under-report the filter's contents, so it is+            # refused like any other unreadable filter.+            unsupported.append((current, line.strip()))+            in_paths = False+            continue+        if in_opaque_block:+            continue+        unsupported.append((current, line.strip()))+    return paths, unsupported+++def extract_trigger_paths(text: str) -> dict:+    """Map each located trigger under `on:` to its `paths:` entries (None+    when it has no `paths:` filter); see extract_trigger_filters."""+    return extract_trigger_filters(text)[0]+++def stale_required_paths(repo_root: Path) -> list:+    """Return the REQUIRED_PATHS entries whose target no longer exists in+    the repository. A required pattern that points at nothing keeps the+    check green while guarding a path that cannot change any more — the+    list must move with the repository layout."""+    stale = []+    for pattern in REQUIRED_PATHS:+        target = pattern[: -len("/**")] if pattern.endswith("/**") else pattern+        if not (repo_root / target).exists():+            stale.append(pattern)+    return stale+++def _unreadable_filter_failure(where: str, trigger: str, token: str) -> str:+    return (+        f"{where}on.{trigger} has a filter this guard cannot read: {token!r} — "+        "express every trigger filter as a plain `paths:` list (no "+        "`paths-ignore:`, no inline `paths: [...]`, no flow style), or extend "+        "the guard, so a skip-by-filter cannot pass unchecked (T-2198)"+    )+++def check(text: str) -> list:+    """Return human-readable failures; empty when the file is fine."""+    failures = []+    triggers, unsupported = extract_trigger_filters(text)+    for trigger, token in unsupported:+        failures.append(_unreadable_filter_failure("", trigger, token))+    for trigger in ("push", "pull_request"):+        if trigger not in triggers:+            # Never located. Either the workflow no longer has this trigger+            # or the file is shaped in a way this parser cannot read (e.g.+            # four-space indentation, a quoted `'push':`); both mean the+            # guard has verified nothing, so fail closed — the same class as+            # `paths-ignore`.+            failures.append(+                f"on.{trigger} was not found — the guard cannot verify a "+                "trigger it cannot locate; keep the two-space `on:` layout "+                "and both triggers, or extend the guard (T-2198)"+            )+            continue+        entries = triggers[trigger]+        if entries is None:+            # Located, with no `paths:` filter at all — see the docstring+            # on extract_trigger_filters for why that is not a failure.+            continue+        for entry in entries:+            if entry.startswith("!"):+                # GitHub's `!` excludes matching files from the filter. This+                # guard checks membership, and membership cannot see an+                # exclusion: `['prismTests/**', '!prismTests/**']` listed+                # every required path and triggered on none of them. Rather+                # than model glob negation, refuse it.+                failures.append(+                    f"on.{trigger}.paths contains the negation '{entry}' — a "+                    "`!` pattern can cancel a required path while still "+                    "listing it, and this guard cannot evaluate what it "+                    "excludes; remove it or extend the guard (T-2198)"+                )+        for required in REQUIRED_PATHS:+            if required not in entries:+                failures.append(+                    f"on.{trigger}.paths is missing '{required}' — a change "+                    "only there would run no test-executing workflow (T-2198)"+                )+    return failures+++def check_guard_workflow(text: str) -> list:+    """Return failures for `checks.yml`, the workflow that RUNS this guard.++    The guard only helps while it executes on every change. If `checks.yml`+    ever gained a `paths` filter, a change outside that filter would skip the+    guard AND the test workflow together — the T-2198 hole reopened one level+    up, and invisible for exactly the same reason: a skipped workflow looks+    like nothing needed checking. The same is true of the step disappearing,+    so the invocation is asserted as well as the trigger shape.+    """+    failures = []+    if not GUARD_INVOCATION_RE.search(text):+        failures.append(+            f"checks.yml no longer runs `{GUARD_INVOCATION}` — this guard only "+            "protects the test workflow while something invokes it on every "+            "push (T-2198)"+        )+    triggers, unsupported = extract_trigger_filters(text)+    for trigger, token in unsupported:+        failures.append(_unreadable_filter_failure("checks.yml ", trigger, token))+    for trigger in ("push", "pull_request"):+        if trigger not in triggers:+            failures.append(+                f"checks.yml on.{trigger} was not found — the workflow that "+                "runs this guard must trigger on both push and pull_request "+                "(T-2198)"+            )+        elif triggers[trigger] is not None:+            failures.append(+                f"checks.yml on.{trigger} has a `paths:` filter — the workflow "+                "that runs this guard must stay unfiltered, or a change it "+                "filters out skips the guard as well as the tests (T-2198)"+            )+    return failures+++def main() -> int:+    failures = []+    for path in (WORKFLOW_PATH, GUARD_WORKFLOW_PATH):+        if not path.exists():+            print(f"FAIL [workflow-triggers]: {path} not found", file=sys.stderr)+            return 1+    failures += [+        f"REQUIRED_PATHS entry '{pattern}' points at nothing in the repository — "+        "update the guard's list to match the current layout"+        for pattern in stale_required_paths(REPO_ROOT)+    ]+    failures += check(WORKFLOW_PATH.read_text(encoding="utf-8"))+    failures += check_guard_workflow(GUARD_WORKFLOW_PATH.read_text(encoding="utf-8"))+    if failures:+        for failure in failures:+            print(f"FAIL [workflow-triggers]: {failure}", file=sys.stderr)+        print(f"{len(failures)} check(s) failed.", file=sys.stderr)+        return 1+    print("All workflow-trigger checks passed.")+    return 0+++if __name__ == "__main__":+    sys.exit(main())

diff --git a/Tools/Tests/test_workflow_triggers.py b/Tools/Tests/test_workflow_triggers.pynew file mode 100644index 00000000..5b6f5331--- /dev/null+++ b/Tools/Tests/test_workflow_triggers.py@@ -0,0 +1,572 @@+"""Unit tests for Tools/check-workflow-triggers.py (T-2198).++The guard is a light textual parser rather than a full YAML implementation,+so its own extraction logic is pinned against fixtures here — the same+"a guard that quietly stopped checking anything looks exactly like a clean+repository" reasoning as test_webkit_test_isolation.py.++Fixtures that must contain a COMPLETE required list are generated from+`guard.REQUIRED_PATHS` rather than spelled out, so adding a required input+does not silently leave half a dozen hand-written fixtures behind. The+constant itself is pinned by `RequiredPathsTests`, and the real workflow file+by `RepositoryTests`, so generating the fixtures cannot hide a shrinking list.++The script has a hyphen in its filename, so it is loaded via importlib.+"""++import importlib.util+import tempfile+import unittest+from pathlib import Path++TOOLS_DIR = Path(__file__).resolve().parent.parent+SCRIPT_PATH = TOOLS_DIR / "check-workflow-triggers.py"+++def _load_script():+    spec = importlib.util.spec_from_file_location("check_workflow_triggers", SCRIPT_PATH)+    module = importlib.util.module_from_spec(spec)+    spec.loader.exec_module(module)+    return module+++guard = _load_script()+++def paths_block(indent: str = "      ") -> str:+    """The complete required list as single-quoted YAML sequence entries."""+    return "".join(f"{indent}- '{pattern}'\n" for pattern in guard.REQUIRED_PATHS)+++def loose_paths_block() -> str:+    """The same list spelled every way YAML allows: double quotes, single+    quotes, bare scalars, trailing whitespace, trailing and interleaved+    comments, and a sequence at the same indentation as its key. The old+    PATH_ENTRY_RE accepted only `      - '...'` with nothing after, so the+    first entry it did not match ended the list and every later entry was+    dropped — reported as "missing" only by coincidence."""+    styles = [+        lambda p: f'      - "{p}"\n',+        lambda p: f"      - '{p}'   \n",+        lambda p: f"      # a comment between entries\n      - {p}\n",+        lambda p: f"      - '{p}' # trailing comment\n",+        lambda p: f"    - '{p}'\n",+    ]+    return "".join(+        styles[i % len(styles)](pattern)+        for i, pattern in enumerate(guard.REQUIRED_PATHS)+    )+++GOOD_WORKFLOW = f"""\+name: Localisation Tests++on:+  push:+    branches: [main]+    paths:+{paths_block()}\+  pull_request:+    branches: [main]+    paths:+{paths_block()}\++jobs:+  test-locales:+    runs-on: macos-latest+"""++# The pre-fix shape (T-2198): both filters omit the test-target and+# project-settings inputs.+REGRESSED_WORKFLOW = """\+name: Localisation Tests++on:+  push:+    branches: [main]+    paths:+      - 'prism/**'+      - 'Tools/**'+      - 'specs/localisation/**'+      - 'Makefile'+      - '.github/workflows/localisation-tests.yml'+  pull_request:+    branches: [main]+    paths:+      - 'prism/**'+      - 'Tools/**'+      - 'specs/localisation/**'+      - 'Makefile'+      - '.github/workflows/localisation-tests.yml'++jobs:+  test-locales:+    runs-on: macos-latest+"""++# The shape this escalated pass fixes: everything the first fix added, but+# still missing the run-time fixture directory.+SAMPLES_MISSING_WORKFLOW = GOOD_WORKFLOW.replace("      - 'samples/**'\n", "")++UNFILTERED_WORKFLOW = """\+name: Localisation Tests++on:+  push:+    branches: [main]+  pull_request:+    branches: [main]++jobs:+  test-locales:+    runs-on: macos-latest+"""++# A job id spelled with underscores only, and a `paths:`-shaped key under it.+# Before TRIGGER_RE was anchored, `test_locales:` matched as a trigger and+# reset the parse state, so anything after it was attributed to the wrong key.+UNDERSCORE_JOB_WORKFLOW = GOOD_WORKFLOW.replace("  test-locales:", "  test_locales:") + """\+    paths:+      - 'not-a-trigger/**'+"""++# A job literally named `push`. `jobs:` is a top-level key, so the parser+# leaves the `on:` block there; without that scoping, `  push:` under `jobs:`+# registered as a located, unfiltered trigger and passed.+PUSH_NAMED_JOB_WORKFLOW = """\+on:+  pull_request:+    branches: [main]+    paths:+""" + paths_block() + """\++jobs:+  push:+    runs-on: macos-latest+"""++# `paths-ignore:` is a paths filter the parser does not understand. Before it+# was refused, the trigger was classed "unfiltered" and passed silently.+PATHS_IGNORE_WORKFLOW = """\+name: Localisation Tests++on:+  push:+    branches: [main]+    paths-ignore:+      - 'docs/**'+  pull_request:+    branches: [main]+    paths:+""" + paths_block() + """\++jobs:+  test-locales:+    runs-on: macos-latest+"""++# The same filter with a QUOTED key. `FILTER_KEY_RE`'s key group was+# `[A-Za-z_-]+`, which does not match `'paths-ignore'`, so the line matched+# nothing at all and was skipped: triggers={'push': None}, unsupported=[],+# zero failures. YAML treats the two spellings identically; the guard now+# does too.+QUOTED_PATHS_IGNORE_WORKFLOW = PATHS_IGNORE_WORKFLOW.replace(+    "    paths-ignore:", "    'paths-ignore':"+)++# A quoted `paths:` key is still a paths filter and must be read as one.+QUOTED_PATHS_KEY_WORKFLOW = GOOD_WORKFLOW.replace("    paths:", '    "paths":')++# A negation. Membership cannot see an exclusion, so before this was refused+# the list below satisfied every required path while triggering on none of+# the test files it named.+NEGATED_REQUIRED_WORKFLOW = """\+on:+  push:+    paths:+""" + paths_block() + """\+      - '!prismTests/**'+  pull_request:+    paths:+""" + paths_block() + """\++jobs:+  test-locales:+    runs-on: macos-latest+"""++# A third trigger between and after the two the guard reads, and an ignorable+# key that opens a nested block. Before any two-space-indented sibling key+# ended the current scope, `workflow_dispatch:` was invisible and its nested+# `inputs:` was reported as an unsupported filter on whichever of+# push/pull_request came last — a false failure the moment someone added a+# manual-dispatch trigger.+SIBLING_TRIGGER_WORKFLOW = """\+on:+  push:+    paths:+""" + paths_block() + """\+  workflow_dispatch:+    inputs:+      locale:+        description: 'Locale to test'+  pull_request:+    types:+      - opened+      - synchronize+    paths:+""" + paths_block() + """\+  schedule:+    - cron: '0 3 * * 1'+  workflow_call:+    inputs:+      paths:+        type: string++jobs:+  test-locales:+    runs-on: macos-latest+"""++INLINE_PATHS_WORKFLOW = """\+on:+  push:+    paths: ['prism/**']+"""++# The flow form of `on:` itself. It opens no block this parser can read, so+# neither trigger is located and the guard fails rather than assuming.+INLINE_ON_WORKFLOW = """\+on: [push, pull_request]++jobs:+  test-locales:+    runs-on: macos-latest+"""++# A trailing comment after the key is not an inline value: the block still+# opens and its entries are read.+COMMENTED_PATHS_KEY_WORKFLOW = GOOD_WORKFLOW.replace(+    "  push:\n    branches: [main]\n    paths:\n",+    "  push:\n    branches: [main]\n    paths:  # every build/test input (T-2198)\n",+)++# The good workflow re-indented to four spaces per level. The parser's+# regexes are anchored to two-space indentation, so it locates neither+# trigger; before "not found" was distinguished from "found, unfiltered",+# that read as two unconditional triggers and passed silently.+FOUR_SPACE_WORKFLOW = "\n".join(+    " " * (2 * (len(line) - len(line.lstrip(" ")))) + line.lstrip(" ")+    for line in GOOD_WORKFLOW.splitlines()+) + "\n"++# An `on:` block with neither push nor pull_request: nothing runs the tests+# on a change, which is exactly the T-2198 gap in a different shape.+MISSING_TRIGGER_WORKFLOW = """\+on:+  workflow_dispatch:+    inputs:+      locale:+        description: 'Locale to test'++jobs:+  test-locales:+    runs-on: macos-latest+"""++LOOSELY_FORMATTED_WORKFLOW = """\+on:+  push:+    paths:+""" + loose_paths_block() + """\++jobs:+  test-locales:+    runs-on: macos-latest+"""++# checks.yml's shape: both triggers, no paths filter anywhere, and the step+# that actually invokes the guard.+GUARD_WORKFLOW_OK = """\+name: Checks++on:+  push:+    branches: [main]+  pull_request:+    branches: [main]++jobs:+  file-checks:+    runs-on: ubuntu-latest+    steps:+      - name: Verify the sole test workflow's paths filter+        run: make verify-workflow-triggers+"""++GUARD_WORKFLOW_FILTERED = GUARD_WORKFLOW_OK.replace(+    "  push:\n    branches: [main]\n",+    "  push:\n    branches: [main]\n    paths:\n      - 'Tools/**'\n",+)+++class RequiredPathsTests(unittest.TestCase):+    """The fixtures are generated from REQUIRED_PATHS, so the constant is+    pinned here instead — otherwise deleting an entry would delete the+    fixtures' expectation along with it."""++    def test_required_paths_are_the_expected_set(self):+        self.assertEqual(+            set(guard.REQUIRED_PATHS),+            {+                "prism/**",+                "prismTests/**",+                "prismUITests/**",+                "prism.xcodeproj/**",+                "prism.xctestplan",+                # Not compiled by any target; opened from disk at run time by+                # ParityFixtureSupport / SamplesComplianceTests / OffMainEmitTests.+                "samples/**",+                "Tools/**",+                "specs/localisation/**",+                "Makefile",+                ".github/workflows/localisation-tests.yml",+            },+        )+++class ExtractTriggerPathsTests(unittest.TestCase):+    def test_loosely_formatted_entries_are_all_read(self):+        triggers = guard.extract_trigger_paths(LOOSELY_FORMATTED_WORKFLOW)+        self.assertEqual(triggers["push"], guard.REQUIRED_PATHS)+        # The fixture deliberately has no pull_request trigger; the only+        # failure must be that, never a "missing" path entry.+        failures = guard.check(LOOSELY_FORMATTED_WORKFLOW)+        self.assertEqual(len(failures), 1)+        self.assertIn("on.pull_request was not found", failures[0])++    def test_paths_ignore_is_reported_as_unsupported(self):+        paths, unsupported = guard.extract_trigger_filters(PATHS_IGNORE_WORKFLOW)+        # Located, but with no readable `paths:` filter.+        self.assertIsNone(paths["push"])+        self.assertEqual(unsupported, [("push", "paths-ignore")])++    def test_quoted_filter_key_is_not_skipped(self):+        paths, unsupported = guard.extract_trigger_filters(QUOTED_PATHS_IGNORE_WORKFLOW)+        self.assertIsNone(paths["push"])+        self.assertEqual(unsupported, [("push", "paths-ignore")])++    def test_quoted_paths_key_still_opens_the_block(self):+        paths, unsupported = guard.extract_trigger_filters(QUOTED_PATHS_KEY_WORKFLOW)+        self.assertEqual(unsupported, [])+        self.assertEqual(paths["push"], guard.REQUIRED_PATHS)+        self.assertEqual(paths["pull_request"], guard.REQUIRED_PATHS)++    def test_inline_paths_list_is_reported_as_unsupported(self):+        _, unsupported = guard.extract_trigger_filters(INLINE_PATHS_WORKFLOW)+        self.assertEqual(unsupported, [("push", "paths")])++    def test_trailing_comment_on_paths_key_opens_a_block(self):+        paths, unsupported = guard.extract_trigger_filters(COMMENTED_PATHS_KEY_WORKFLOW)+        self.assertEqual(unsupported, [])+        self.assertEqual(paths["push"], guard.REQUIRED_PATHS)+        self.assertEqual(guard.check(COMMENTED_PATHS_KEY_WORKFLOW), [])++    def test_four_space_indentation_locates_no_trigger(self):+        paths, unsupported = guard.extract_trigger_filters(FOUR_SPACE_WORKFLOW)+        self.assertEqual(paths, {})+        self.assertEqual(unsupported, [])++    def test_sibling_triggers_end_the_current_scope(self):+        paths, unsupported = guard.extract_trigger_filters(SIBLING_TRIGGER_WORKFLOW)+        self.assertEqual(unsupported, [])+        self.assertEqual(paths["push"], guard.REQUIRED_PATHS)+        self.assertEqual(paths["pull_request"], guard.REQUIRED_PATHS)+        self.assertEqual(set(paths), {"push", "pull_request"})+        self.assertEqual(guard.check(SIBLING_TRIGGER_WORKFLOW), [])++    def test_job_ids_are_not_mistaken_for_triggers(self):+        triggers = guard.extract_trigger_paths(UNDERSCORE_JOB_WORKFLOW)+        self.assertEqual(set(triggers), {"push", "pull_request"})+        self.assertNotIn("not-a-trigger/**", triggers["pull_request"])+        self.assertEqual(guard.check(UNDERSCORE_JOB_WORKFLOW), [])++    def test_a_job_named_push_is_not_a_trigger(self):+        triggers = guard.extract_trigger_paths(PUSH_NAMED_JOB_WORKFLOW)+        self.assertEqual(set(triggers), {"pull_request"})+        failures = guard.check(PUSH_NAMED_JOB_WORKFLOW)+        self.assertEqual(len(failures), 1)+        self.assertIn("on.push was not found", failures[0])++    def test_extracts_both_trigger_lists_in_order(self):+        triggers = guard.extract_trigger_paths(GOOD_WORKFLOW)+        self.assertEqual(triggers["push"], guard.REQUIRED_PATHS)+        self.assertEqual(triggers["pull_request"], guard.REQUIRED_PATHS)++    def test_trigger_with_no_paths_filter_is_present_but_none(self):+        # Located-but-unfiltered must stay distinguishable from never-located.+        triggers = guard.extract_trigger_paths(UNFILTERED_WORKFLOW)+        self.assertEqual(triggers, {"push": None, "pull_request": None})++    def test_unreadable_line_under_a_trigger_is_refused(self):+        # A key at filter depth that opens no readable block, with no filter+        # key of its own to blame it on. Skipping it would leave the trigger+        # looking unfiltered.+        workflow = """\+on:+  push:+    if: github.actor != 'dependabot[bot]'+"""+        _, unsupported = guard.extract_trigger_filters(workflow)+        self.assertEqual(unsupported, [("push", "if")])++    def test_folded_scalar_paths_value_is_refused_once(self):+        # `paths: >` is an inline value, so the key is refused; its folded+        # continuation adds nothing the failure does not already say.+        workflow = """\+on:+  push:+    paths: >+      prism/**+"""+        paths, unsupported = guard.extract_trigger_filters(workflow)+        self.assertIsNone(paths["push"])+        self.assertEqual(unsupported, [("push", "paths")])+++class CheckTests(unittest.TestCase):+    def test_good_workflow_passes(self):+        self.assertEqual(guard.check(GOOD_WORKFLOW), [])++    def test_regressed_workflow_is_caught_in_both_triggers(self):+        failures = guard.check(REGRESSED_WORKFLOW)+        self.assertTrue(any("prismTests/**" in f and "push" in f for f in failures))+        self.assertTrue(any("prismTests/**" in f and "pull_request" in f for f in failures))+        self.assertTrue(any("prism.xctestplan" in f for f in failures))+        self.assertTrue(any("prism.xcodeproj/**" in f for f in failures))+        self.assertTrue(any("samples/**" in f for f in failures))++    def test_missing_samples_is_caught_in_both_triggers(self):+        failures = guard.check(SAMPLES_MISSING_WORKFLOW)+        self.assertEqual(len(failures), 2)+        self.assertTrue(any("samples/**" in f and "on.push" in f for f in failures))+        self.assertTrue(+            any("samples/**" in f and "on.pull_request" in f for f in failures)+        )++    def test_paths_ignore_fails_instead_of_passing_silently(self):+        failures = guard.check(PATHS_IGNORE_WORKFLOW)+        self.assertEqual(len(failures), 1)+        self.assertIn("paths-ignore", failures[0])+        self.assertIn("push", failures[0])++    def test_quoted_paths_ignore_fails_instead_of_passing_silently(self):+        failures = guard.check(QUOTED_PATHS_IGNORE_WORKFLOW)+        self.assertEqual(len(failures), 1)+        self.assertIn("paths-ignore", failures[0])+        self.assertIn("push", failures[0])++    def test_negated_required_path_fails(self):+        failures = guard.check(NEGATED_REQUIRED_WORKFLOW)+        self.assertEqual(len(failures), 1)+        self.assertIn("!prismTests/**", failures[0])+        self.assertIn("on.push.paths", failures[0])++    def test_inline_on_value_fails_instead_of_passing_silently(self):+        failures = guard.check(INLINE_ON_WORKFLOW)+        self.assertEqual(len(failures), 2)+        self.assertTrue(any("on.push was not found" in f for f in failures))+        self.assertTrue(any("on.pull_request was not found" in f for f in failures))++    def test_unfiltered_trigger_is_not_flagged(self):+        # No paths filter at all means the trigger runs unconditionally,+        # which trivially covers every test-only change.+        self.assertEqual(guard.check(UNFILTERED_WORKFLOW), [])++    def test_four_space_indented_file_fails_instead_of_passing_silently(self):+        failures = guard.check(FOUR_SPACE_WORKFLOW)+        self.assertEqual(len(failures), 2)+        self.assertTrue(any("on.push was not found" in f for f in failures))+        self.assertTrue(any("on.pull_request was not found" in f for f in failures))++    def test_missing_trigger_fails_instead_of_passing_silently(self):+        failures = guard.check(MISSING_TRIGGER_WORKFLOW)+        self.assertEqual(len(failures), 2)+        self.assertTrue(any("on.push was not found" in f for f in failures))+        self.assertTrue(any("on.pull_request was not found" in f for f in failures))+++class GuardWorkflowTests(unittest.TestCase):+    """checks.yml runs this guard on every push; a filter there would let a+    change skip the guard by the same mechanism the guard exists to catch."""++    def test_unfiltered_guard_workflow_passes(self):+        self.assertEqual(guard.check_guard_workflow(GUARD_WORKFLOW_OK), [])++    def test_filtered_guard_workflow_fails(self):+        failures = guard.check_guard_workflow(GUARD_WORKFLOW_FILTERED)+        self.assertEqual(len(failures), 1)+        self.assertIn("checks.yml on.push", failures[0])+        self.assertIn("`paths:` filter", failures[0])++    def test_paths_ignore_on_the_guard_workflow_fails(self):+        text = GUARD_WORKFLOW_OK.replace(+            "  push:\n    branches: [main]\n",+            "  push:\n    branches: [main]\n    paths-ignore:\n      - 'docs/**'\n",+        )+        failures = guard.check_guard_workflow(text)+        self.assertTrue(any("paths-ignore" in f for f in failures))++    def test_guard_workflow_that_stopped_running_the_guard_fails(self):+        text = GUARD_WORKFLOW_OK.replace("make verify-workflow-triggers", "make lint")+        failures = guard.check_guard_workflow(text)+        self.assertEqual(len(failures), 1)+        self.assertIn("no longer runs", failures[0])++    def test_commented_out_invocation_does_not_count_as_running_it(self):+        # The target's name appears in prose all over this repository; only a+        # `run:` step (or a line of a `run: |` block) counts as invoking it.+        text = GUARD_WORKFLOW_OK.replace(+            "        run: make verify-workflow-triggers",+            "        # run: make verify-workflow-triggers\n        run: true",+        )+        failures = guard.check_guard_workflow(text)+        self.assertEqual(len(failures), 1)+        self.assertIn("no longer runs", failures[0])++    def test_block_scalar_invocation_counts(self):+        text = GUARD_WORKFLOW_OK.replace(+            "        run: make verify-workflow-triggers",+            "        run: |\n          make verify-workflow-triggers",+        )+        self.assertEqual(guard.check_guard_workflow(text), [])++    def test_real_guard_workflow_is_unfiltered(self):+        text = guard.GUARD_WORKFLOW_PATH.read_text(encoding="utf-8")+        self.assertEqual(guard.check_guard_workflow(text), [])+        self.assertEqual(+            guard.extract_trigger_paths(text), {"push": None, "pull_request": None}+        )+++class RepositoryTests(unittest.TestCase):+    """Pin the guard against the real repository, not just fixtures."""++    def test_real_workflow_passes(self):+        text = guard.WORKFLOW_PATH.read_text(encoding="utf-8")+        self.assertEqual(guard.check(text), [])+        self.assertEqual(guard.stale_required_paths(guard.REPO_ROOT), [])++    def test_stale_required_path_is_reported(self):+        with tempfile.TemporaryDirectory() as tmp:+            root = Path(tmp)+            for pattern in guard.REQUIRED_PATHS:+                target = pattern[:-3] if pattern.endswith("/**") else pattern+                if target != "prism.xctestplan":+                    (root / target).parent.mkdir(parents=True, exist_ok=True)+                    (root / target).mkdir() if pattern.endswith("/**") else (root / target).touch()+            self.assertEqual(guard.stale_required_paths(root), ["prism.xctestplan"])+++if __name__ == "__main__":+    unittest.main()

diff --git a/Makefile b/Makefileindex 46383de9..b88fd0e2 100644--- a/Makefile+++ b/Makefile@@ -86,6 +86,7 @@ help: 	@echo "    test-locales-adhoc - test-locales, ad-hoc signed (CI, no certificate)" 	@echo "    verify-make-guards - Check the test targets cannot report a false pass" 	@echo "    verify-test-isolation - Check no test can construct WebKit off-main"+	@echo "    verify-workflow-triggers - Check the sole test workflow's paths filter covers every build/test input" 	@echo "    install     - Build and install Debug on device" 	@echo "    run         - Build, install, and launch Debug on device" 	@echo ""@@ -477,6 +478,28 @@ verify-test-isolation: 	$(STRICT) python3 Tools/check-webkit-test-isolation.py 	$(STRICT) python3 -m unittest Tools.Tests.test_webkit_test_isolation +# localisation-tests.yml is the repository's only test-executing workflow+# (checks.yml, which runs this target, stays on Linux and never invokes+# xcodebuild). Its paths filters used to omit prismTests/**, prismUITests/**,+# prism.xctestplan, prism.xcodeproj/** and samples/**, so a push or PR that+# only touched a test file, test target membership, the test plan, Xcode+# project settings, or a sample document skipped the workflow entirely and+# merged with no build and no test run at all (T-2198). This asserts every+# input that can change what test-locales-adhoc builds or runs is present in+# both the push and pull_request paths lists, so the two filters cannot drift+# apart again.+#+# "Input" here means what the suite READS at run time, not what Xcode+# compiles. samples/** is the entry that makes the difference concrete: no+# target compiles it, and three suites open it from disk — which is exactly+# how the first pass at this guard missed it. The guard also checks its own+# runner, since a paths filter on checks.yml, or the loss of the step that+# invokes this target, would silence it by the mechanism it exists to catch.+.PHONY: verify-workflow-triggers+verify-workflow-triggers:+	$(STRICT) python3 Tools/check-workflow-triggers.py+	$(STRICT) python3 -m unittest Tools.Tests.test_workflow_triggers+ # Cleaning .PHONY: clean clean:

diff --git a/CLAUDE.md b/CLAUDE.mdindex d549b483..5927c832 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -332,6 +332,24 @@ is static because the abort is a scheduling race — the guilty suite passes in isolation every time, and the suite the cascade *names* is usually not the guilty one. See `docs/agent-notes/development-tooling.md`. +The fourth way is for the tests never to be scheduled at all. `localisation-tests.yml`+is the only workflow that runs `xcodebuild`, and its `paths` filters omitted+`prismTests/**`, `prismUITests/**`, `prism.xctestplan`, `prism.xcodeproj/**` and+`samples/**`, so a test-only change skipped it and merged green having tested+nothing (T-2198). `make verify-workflow-triggers` (`Tools/check-workflow-triggers.py`,+run by `checks.yml` on every push) asserts both filters list every input that can+change what the sweep builds or runs, refuses filter keys it cannot read+(`paths-ignore:`, quoted or not, would otherwise pass as "unfiltered"), refuses a+`!` negation (membership cannot see an exclusion, so `['prismTests/**',+'!prismTests/**']` listed every required path and triggered on none of them), and+fails when a required path no longer exists in the repository. The required list is+derived from what the suite READS AT RUN TIME, not from what Xcode compiles:+`samples/**` is compiled by no target and opened from disk by three suites+(`ParityFixtureSupport.samplesDirectory()`, `SamplesComplianceTests`,+`OffMainEmitTests`), which is how the guard's first pass shipped still missing it.+The guard also checks its own runner — a `paths` filter on `checks.yml`, or the+loss of the step that invokes it, would silence it by the same mechanism.+ ### Test Coverage  - Unit tests (`prismTests`): parsers, cache, file observer, models

diff --git a/docs/agent-notes/development-tooling.md b/docs/agent-notes/development-tooling.mdindex daf09a20..db03c9da 100644--- a/docs/agent-notes/development-tooling.md+++ b/docs/agent-notes/development-tooling.md@@ -5,6 +5,38 @@ - `make lint` runs `swiftlint lint --strict` and may report zero violations but still exit non-zero if SwiftLint cannot write to its cache. This is tracked as T-807. - To distinguish cache-permission failures from lint failures during investigation, run `swiftlint lint --strict --no-cache` after the Makefile target. The Makefile target should still be run first because project tooling is Makefile-based. +## The sole test workflow must not skip test-only changes++`make verify-workflow-triggers` (`Tools/check-workflow-triggers.py`, unit tests in+`Tools/Tests/test_workflow_triggers.py`) is the sibling of `verify-make-guards` and+`verify-test-isolation` for the failure mode where nothing runs because GitHub never+scheduled it: `localisation-tests.yml` is the only workflow that invokes `xcodebuild`,+and a `paths` filter that omits an input skips silently rather than failing (T-2198).+The guard is a textual parse, not YAML, so it fails closed on anything it cannot read+— a `paths-ignore:` filter (quoted or not), an inline `paths: [...]`, a `!` negation,+or any other unrecognised line under a trigger is a FAIL, never "unfiltered" — and+`REQUIRED_PATHS` is checked against the repository so a renamed directory cannot+leave the list guarding a path that no longer exists. It also checks `checks.yml`,+the workflow that runs it: a `paths` filter there, or the deletion of the step, would+silence the guard by the very mechanism it exists to catch.++**What belongs in `REQUIRED_PATHS` is what the suite READS AT RUN TIME, not what+Xcode compiles.** That is the distinction the guard's first pass got wrong. It+covered the compiled targets (`prism/**`, `prismTests/**`, `prismUITests/**`), the+project and plan (`prism.xcodeproj/**`, `prism.xctestplan`) and the build tooling+(`Tools/**`, `Makefile`, `specs/localisation/**` — read by the `validate-localisation.py`+build phase) and still missed `samples/**`, which no target compiles and three suites+open from disk: `ParityFixtureSupport.samplesDirectory()` resolves the repo-root+directory from `#filePath`, `SamplesComplianceTests` enumerates `samples/*.md` at run+time, and `OffMainEmitTests` times `samples/large-html-heavy.md`. A samples-only+commit ran no tests. When adding a directory, ask what the tests open.++Audited and deliberately NOT required, because nothing the sweep runs reads them:+`prism-notes-js/` (unreferenced by the Xcode project), `package.json` /+`.stylelintrc.cjs` (CSS lint tooling, not run by `test-locales-adhoc`),+`.swiftlint.yml` (no SwiftLint build phase), `ExportOptions.plist` (archiving),+`docs/`, and the rest of `specs/`.+ ## The one rule that keeps full runs legible: no WebKit from a synchronous test  `make verify-test-isolation` (`Tools/check-webkit-test-isolation.py`) fails the

diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bf353052..8a7eea32 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed  - Live-WebKit test suites no longer abort the shared test host, which used to turn a whole run into a fictional four-figure failure count (T-2219, T-2096). One process hosts the entire unit-test target, so an abort reports every still-queued test as a failure it never ran — 189 of them in one observed run, 233 in another. The cause was the one T-1541 diagnosed and fixed for a single suite: a synchronous `@MainActor` test body gets no hop-on-entry in this target's build configuration, so under load it runs on the cooperative pool and WebKit's main-thread assertion kills the process. Sixty tests across nine suites could still do that, including the two the tickets name and two suites no ticket had ever mentioned — which is why the abort kept being attributed to a different suite each time, usually whichever long-running live-WebKit test happened to be in flight. All sixty are now `async`, which hops as part of the ABI, and `make verify-test-isolation` fails the build if a synchronous test can reach WebKit again. Nothing is skipped or excluded; the number of tests executed is unchanged.+- A change that touches only test files, test target membership, the test plan, Xcode project settings, or a document under `samples/` now runs the test suite in CI instead of merging with a green run that tested none of it (T-2198). The per-locale sweep is the sole workflow that executes tests, and it is paths-filtered; both its `push` and `pull_request` filters omitted `prismTests/**`, `prismUITests/**`, `prism.xctestplan`, `prism.xcodeproj/**` and `samples/**`, so a compile-broken test, a coverage-disabling plan edit, or a sample document the parity suites open at run time produced no build and no test run at all — a gap that is silent by construction, because a skipped workflow looks exactly like nothing needed checking. `samples/**` is worth naming separately because it is the shape the first fix missed: no target compiles it, so a list derived from "what Xcode builds" omits it, while `ParityFixtureSupport`, `SamplesComplianceTests` and `OffMainEmitTests` all read it from disk while the suite runs. The required list is now derived from what the tests OPEN, not what the project compiles. Both filters cover it, and `make verify-workflow-triggers` — run from the unfiltered `checks.yml`, so it cannot itself be bypassed the same way — fails when the two lists drift apart again, when a filter key is one it cannot read (`paths-ignore:`, quoted or not; an inline `paths: [...]`; any unrecognised line under a trigger), when a `!` negation appears (membership cannot see an exclusion, so `['prismTests/**', '!prismTests/**']` listed every required path while triggering on none of them), when a required path no longer exists in the repository, and when `checks.yml` itself grows a paths filter or drops the step that invokes the guard. - CI now runs the test suite instead of only appearing to (T-1983). The per-locale sweep is the only job that can execute tests, and it reported success while executing none: its build failed for want of a signing certificate on the runner, that failure was swallowed, and nothing checked that any test had run. The sweep now signs ad-hoc so the build succeeds without a certificate, every test target hands its result bundle to the zero-test guard — per locale configuration, not once at the end — and any recipe whose failure must be believed carries `$(STRICT)`, because `.SHELLFLAGS` is silently ignored by the GNU Make 3.81 that macOS ships. `make verify-make-guards` asserts all of this on every push. - `Tools/check-test-results.sh` no longer reports OK on two more shapes of run that were not a clean pass (T-2224, T-1993). A test that fails on its first attempt and passes on a retry only ever left its final result in the bundle's counts, so the failure was invisible — measured directly on PR #377, where a macOS run exited 65 with `** TEST FAILED **` while the bundle reported 60/60 passed because two `WebContentTerminationWiringTests` cases needed a retry. The checker now walks the bundle's per-test attempt history and fails the run when any test needed one, naming which. Separately, an interrupted, cancelled, or infrastructure-failed run can leave a readable partial bundle with passing-looking counts and zero recorded failures while its own top-level `result` field sits at "unknown" — never resolving to a verdict at all. The checker now refuses to report success on such a run, and the count-arithmetic sanity check that used to only warn on a mismatch now fails closed, the same as everything else this script guards. That guard distinguishes "did not resolve" from "did not pass": `result` is typed as the same five-value enum as an individual test's result (`Passed`, `Failed`, `Skipped`, `Expected Failure`, `unknown`), so a run resolving to Skipped or Expected Failure is a legitimate outcome and is allowed through — demanding Passed or Failed would have turned an ordinary `-only-testing:` selection that lands entirely on disabled tests into a hard failure blamed on an interrupted test host that never existed. A run that executed nothing because every selected test was skipped still fails, since it verifies exactly as much as running no tests at all, but it now fails as a zero-executed run and says so rather than borrowing the infrastructure-failure diagnosis. The retry scan searches a test case's whole subtree rather than its direct children, which is what makes it work at all on `make test` and `make test-ui`: neither passes `-only-test-configuration`, so every attempt on those runs sits under a `Test Plan Configuration` node instead of directly under the test case, and a direct-children scan finds nothing there — silently, on half the pre-push matrix. It also no longer depends on attempt ordering or on there being more than one attempt: any recorded failed attempt under a test that did not end up failed is the laundering shape, however the attempts are listed. Three further shapes now fail instead of reporting OK, all of them cases where the checker previously drew a clean conclusion from data it had not actually understood: a per-test tree containing a `nodeType` or result value outside the published enums (a one-character drift is enough to make the scan match nothing), a tree with no test-case node in it at all while the summary counts tests, and a bundle that contradicts itself by recording a test case as failed while its `failedTests` count is zero. `make verify-make-guards` now runs a dedicated regression suite (`Tools/Tests/test-check-test-results.sh`) covering these shapes alongside the existing zero-test and cascade-failure cases. One limitation is worth stating outright rather than leaving implied: the per-attempt node shape the retry scan reads (`Repetition` / `Test Case Run`) is derived from `xcresulttool`'s published schema and is **not** confirmed against a captured retried bundle — repeated attempts to produce one on a loaded machine died with `** BUILD INTERRUPTED **`, and a sweep of the readable historical bundles on this machine found none containing such a node. The scan accepts either shape at any depth for that reason, and the fixtures pin its parsing, its descent through the configuration layer, and its control flow — not the attempt-node shape itself. The nesting the fixtures *do* model faithfully (a `Test Plan` root, and test-case nodes whose children are `Test Plan Configuration` nodes) was measured against real bundles from this project. - Copy notes is now the primary notes action (T-1577). On iPhone the document screen's toolbar shows Copy notes instead of Share with Notes, which moved into the notes pane alongside copy; on iPad and Mac the top toolbar shows copy leading the export button. Every copy button appears exactly when the copy output would contain at least one note under the current export settings, each action carries an accessibility label and help text, and an export blocked by the paywall from inside the pane now retries fully — including the author-name prompt and its confirmation toast — after a purchase completes.

diff --git a/specs/bugfixes/test-only-changes-bypass-workflow/report.md b/specs/bugfixes/test-only-changes-bypass-workflow/report.mdnew file mode 100644index 00000000..3984d563--- /dev/null+++ b/specs/bugfixes/test-only-changes-bypass-workflow/report.md@@ -0,0 +1,294 @@+# Bugfix Report: Test-Only Changes Bypass the Sole Test Workflow++**Date:** 2026-08-29+**Status:** Fixed++## Description of the Issue++`.github/workflows/localisation-tests.yml` is the repository's only+test-executing workflow (`checks.yml` runs Make guard structure checks on+Ubuntu and never invokes `xcodebuild`). Both its `on.push.paths` and+`on.pull_request.paths` filters omitted `prismTests/**`, `prismUITests/**`,+`prism.xctestplan`, `prism.xcodeproj/**`, and `samples/**`. A push or PR that+only touched one of those — a test file, test target membership, test-plan+configuration, Xcode project settings, or a sample document three suites read+from disk while they run — matched none of the listed path globs, so GitHub+Actions skipped the workflow entirely and the change merged with no build and+no test run.++**Reproduction steps:**+1. On a branch, edit only a file under `prismTests/**` (e.g. add a test that+   fails to compile) or only `prism.xctestplan` (e.g. remove a test target+   from the plan).+2. Open a PR against `main`.+3. Observe that `localisation-tests.yml` does not trigger — its `paths`+   filter has no entry matching the changed file — while `checks.yml` (which+   never runs `xcodebuild`) reports green. The PR shows a fully passing check+   suite despite the change never having been built or tested.++**Impact:** High. A compile-broken test, a test silently removed from+`prism.xctestplan`, or an Xcode project setting change that disables+coverage could merge into `main` with an all-green check suite, since no+workflow in the repository would have run a single test against it.++## Investigation Summary++- **Symptoms examined:** Ticket description and the two workflow files+  directly; no runtime reproduction was needed since the defect is a static+  configuration gap (a `paths` glob list), not runtime behaviour.+- **Code inspected:** `.github/workflows/localisation-tests.yml` (lines 3-19),+  `.github/workflows/checks.yml` (confirmed it is Ubuntu-only and never calls+  `make test-locales`/`xcodebuild`), the repository root layout (confirmed+  `prismTests/`, `prismUITests/`, `prism.xcodeproj/`, and `prism.xctestplan`+  all live at the repo root, none nested under `prism/`), and the existing+  `Tools/Tests/test-make-guards.sh` / `Tools/check-webkit-test-isolation.py`+  pattern for how this project already writes static CI-configuration+  guards.+- **Hypotheses tested:** None needed — the gap between "inputs that affect+  `make test-locales-adhoc`" and "inputs listed in the `paths` filter" is+  directly visible by comparing the filter against the repo's file layout.++## Discovered Root Cause++**Defect type:** Missing validation / incomplete configuration (CI trigger+path filter that does not cover every build/test input).++**Why it occurred:** The `paths` filters were written to cover the main+application source (`prism/**`), build tooling (`Tools/**`, `Makefile`), and+the localisation overrides the workflow was originally built to sweep+(`specs/localisation/**`), but were never updated to also cover the test+targets themselves or the Xcode project/test-plan configuration that governs+what `xcodebuild` builds and runs. Since this workflow is a `push`/`pull_request`+path-filtered trigger, GitHub Actions silently skips it — rather than failing+— when no changed file matches any glob, so the gap produced no visible+symptom until specifically audited.++**Contributing factors:** `checks.yml`, the only other workflow that runs on+every push, does Make-guard structural checks on Ubuntu and never invokes+`xcodebuild`, so there was no secondary safety net to catch a test-only+change that `localisation-tests.yml` skipped.++## Resolution for the Issue++**Changes made:**+- `.github/workflows/localisation-tests.yml` — added `prismTests/**`,+  `prismUITests/**`, `prism.xcodeproj/**`, `prism.xctestplan` and `samples/**`+  to both the `on.push.paths` and `on.pull_request.paths` filters, so every+  input that can change what `make test-locales-adhoc` builds or runs now+  triggers the workflow. `samples/**` was added in the final review round;+  see "Review fixes".+- `Tools/check-workflow-triggers.py` (new) — a static guard, parsing the+  workflow's two `paths:` lists and asserting every required path pattern is+  present in both, so the two filters cannot silently drift apart again.+- `Tools/Tests/test_workflow_triggers.py` (new) — unit tests pinning the+  guard's own parsing/checking logic against fixtures (a fixed good+  workflow, the pre-fix regressed shape, an unfiltered-trigger case, an+  underscore-only job id, a `paths-ignore` filter, an inline `paths: [...]`+  value, and a loosely formatted list), plus a test that runs the guard+  against the real workflow file.+- `CHANGELOG.md` — `[Unreleased]` entry for the fix.+- `Makefile` — added a `verify-workflow-triggers` target running the new+  checker and its unit tests, following the existing+  `verify-make-guards`/`verify-test-isolation` pattern, plus a help-text+  line.+- `.github/workflows/checks.yml` — added a step running+  `make verify-workflow-triggers` so the guard runs on every push/PR (this+  workflow has no `paths` filter, so it cannot itself be bypassed the same+  way).++**Approach rationale:** The ticket's suggested fix offered two options:+remove the path filters entirely, or include every source/test/build input.+Expanding the filter (rather than removing it) preserves the intent of the+`paths` filter — skipping the expensive (tens-of-minutes) per-locale sweep+for changes that cannot affect its outcome, e.g. documentation-only edits —+while closing the actual gap. A static guard was added because the ticket+explicitly suggested "a workflow-trigger regression check if practical",+and a textual/regex parse (rather than a YAML library dependency) matches+this project's existing style for CI-configuration guards+(`Tools/check-webkit-test-isolation.py`) and avoids depending on PyYAML being+present on every runner.++**Review fixes:**+- `TRIGGER_RE` originally matched any two-space-indented lowercase key, so a+  job id without a hyphen (`test_locales:`) would have been parsed as a+  trigger and reset the parse state mid-file. It is now anchored to+  `push|pull_request`, any top-level key ends the `on:` block, and a+  fixture pins it (commit 541847b5).+- A filter key the parser cannot read (`paths-ignore:`, an inline+  `paths: [...]`) used to class the trigger as "unfiltered" and pass+  silently; it is now a FAIL. `PATH_ENTRY_RE` tolerates double quotes, bare+  scalars, trailing whitespace and comments so a later entry is never+  dropped, `REQUIRED_PATHS` is checked against the repository layout so the+  list cannot go stale, and failures use the sibling guards'+  `FAIL [workflow-triggers]: ...` format.+- The trigger scope only ended on a top-level key or another+  `push`/`pull_request`, so a sibling trigger such as `workflow_dispatch:`+  was invisible and its nested `inputs:` was reported as an unsupported+  filter on the previous trigger — a false failure. Any two-space-indented+  key under `on:` now ends the current scope (without starting one), and+  `SIBLING_TRIGGER_WORKFLOW` pins it.+- A trigger the parser never LOCATED (the file re-indented to four spaces,+  or an `on:` block holding only `workflow_dispatch:`) was indistinguishable+  from a located trigger with no `paths:` filter, so it passed silently —+  the same fail-open class as `paths-ignore`. The parser now records every+  `push`/`pull_request` it finds (`None` when unfiltered) and `check()`+  FAILs on one that is absent; `FOUR_SPACE_WORKFLOW` and+  `MISSING_TRIGGER_WORKFLOW` pin it. `paths:  # comment` is also read as a+  block header rather than an inline value.+- **`samples/**` was missing from the filters and from `REQUIRED_PATHS`, so+  a samples-only commit still ran no tests.** The list had been derived from+  what Xcode COMPILES; `samples/` is compiled by no target and read from+  disk by three suites while they run — `ParityFixtureSupport.samplesDirectory()`+  resolves the repo-root directory from `#filePath`, `SamplesComplianceTests`+  enumerates `samples/*.md` at run time, and `OffMainEmitTests` times+  `samples/large-html-heavy.md`. The derivation rule is now stated wherever+  the list is documented: an input is what the suite READS AT RUN TIME, not+  what the project file lists. A full audit of the test tree (`#filePath`,+  `Bundle.*`, `Process`, and every repo-relative literal under `prismTests/`+  and `prismUITests/`, plus the three `shellScript` build phases in+  `project.pbxproj`) found `samples/**` to be the only uncovered input; the+  result is tabulated under "Runtime inputs audited" below.+- `FILTER_KEY_RE`'s key group was `[A-Za-z_-]+`, which does not match a+  QUOTED key, so `'paths-ignore':` under `on.push` matched nothing at all and+  was silently skipped: `triggers={'push': None}`, `unsupported=[]`, zero+  failures — the exact fail-open the previous round had just closed for the+  unquoted spelling. The key group now accepts single-quoted, double-quoted+  and bare keys, and, more importantly, the parser no longer skips ANY line+  under a trigger: a line that is neither a readable filter key nor a+  readable `paths:` entry is collected as unsupported and fails. The only+  lines passed over are the nested contents of a key already settled — an+  ignorable one (`types:` and its event list) or one just refused+  (`paths-ignore:`'s own entries add nothing to the failure its key already+  produced). `QUOTED_PATHS_IGNORE_WORKFLOW`, `QUOTED_PATHS_KEY_WORKFLOW`,+  `test_unreadable_line_under_a_trigger_is_refused` and+  `test_folded_scalar_paths_value_is_refused_once` pin it.+- A `!` negation entry satisfied a required path by membership:+  `['prismTests/**', '!prismTests/**']` listed every required pattern and+  triggered on none of the files it named. Rather than model glob negation,+  any `!` entry in a `paths:` list is now a failure+  (`NEGATED_REQUIRED_WORKFLOW`).+- The parser recognised `push:`/`pull_request:` anywhere at two-space+  indentation, so a JOB named `push` under `jobs:` registered as a located,+  unfiltered trigger and passed. Triggers are now recognised only inside the+  `on:` block; an inline `on: [push, pull_request]` opens no readable block,+  so it locates neither trigger and fails closed. `PUSH_NAMED_JOB_WORKFLOW`+  and `INLINE_ON_WORKFLOW` pin both.+- The guard depended on `checks.yml` staying unfiltered and on the step that+  runs it, and checked neither. `check_guard_workflow()` now asserts+  `checks.yml` has both triggers, no `paths`/`paths-ignore` filter on either,+  and still invokes `make verify-workflow-triggers` — a filter added there,+  or the step deleted, would silence the guard by the very mechanism it+  exists to catch. `GuardWorkflowTests` covers all three, including against+  the real file.+- The fixtures that must hold a COMPLETE required list are now generated from+  `guard.REQUIRED_PATHS` instead of being spelled out six times, because+  adding `samples/**` otherwise meant editing six literal lists and any one+  left behind would have silently weakened its own test. The constant is+  pinned by `RequiredPathsTests` and the real workflow by `RepositoryTests`,+  so generating the fixtures cannot hide a shrinking list.++**Runtime inputs audited (the basis for `REQUIRED_PATHS`):**++| Input | Read by | Covered as |+|-------|---------|------------|+| `prism/` sources and `Resources/` (`document.css`, `WebRenderer/*.js`, `mermaid.min.js`, `onboarding.md`, …) | compiled; `Bundle.main` lookups in ~15 suites; `ProductionSourceScan`, `TOCDetailsHeadingsTests`, `FootnotePresentationHostTests`, `KeyboardScrollControllerTests`, `RemoteRefreshFlowTests`, `PaywallPresenterTests`, `WebSelectionNoteTests`, `WebFootnotePopoverTests` and `DocumentLoadingIndicatorWiringTests` scan it from `#filePath` | `prism/**` |+| `prismTests/`, incl. `WebRendering/Fixtures/**` read via `#filePath` | compiled + read at run time | `prismTests/**` |+| `prismUITests/` | compiled | `prismUITests/**` |+| Project settings, build phases, `Package.resolved` | `xcodebuild` | `prism.xcodeproj/**` |+| Test plan (targets, locale configurations, coverage) | `xcodebuild -testPlan prism` | `prism.xctestplan` |+| `samples/*.md` | `ParityFixtureSupport.samplesDirectory()`, `SamplesComplianceTests`, `OffMainEmitTests` | `samples/**` *(added this round)* |+| `Tools/validate-localisation.py`, `Tools/stamp-commit-hash.sh` | `project.pbxproj` build phases | `Tools/**` |+| `Tools/check-test-results.sh`, `Tools/check-webkit-test-isolation.py` | Makefile targets the sweep depends on | `Tools/**` |+| `specs/localisation/en-AU-overrides.json` | the `validate-localisation.py` build phase | `specs/localisation/**` |+| The sweep's own recipe | `make test-locales-adhoc` | `Makefile` |+| The workflow itself | GitHub Actions | `.github/workflows/localisation-tests.yml` |++Audited and deliberately NOT required, because nothing `make+test-locales-adhoc` reads them: `prism-notes-js/` (zero references in+`project.pbxproj`), `package.json` / `package-lock.json` / `.stylelintrc.cjs`+(CSS lint tooling, not invoked by the sweep), `.swiftlint.yml` (the project+has no SwiftLint build phase), `ExportOptions.plist` (archiving only),+`docs/`, the non-localisation parts of `specs/`, and+`.github/workflows/checks.yml` (which carries no paths filter, so it always+runs — and the guard now asserts that).++**Alternatives considered:**+- **Remove the `paths` filters entirely:** simplest, and guarantees no future+  input can be missed — but it would make every push/PR run the full+  tens-of-minutes macOS sweep, including for documentation-only changes,+  which the filter exists specifically to avoid.+- **A full YAML-parsing guard (PyYAML):** more robust to arbitrary+  restructuring of the workflow file, but adds a dependency that may not be+  installed on the Ubuntu runner `checks.yml` uses, and the existing+  `check-webkit-test-isolation.py` precedent already favours dependency-free+  textual parsing for workflow/Makefile checks in this repo.++## Regression Test++**Test file:** `Tools/check-workflow-triggers.py` (guard) and+`Tools/Tests/test_workflow_triggers.py` (unit tests for the guard's parsing+logic)++**Test name:** `Tools/check-workflow-triggers.py` run directly asserts the+real workflow file; `test_regressed_workflow_is_caught_in_both_triggers`+pins the parsing logic against the pre-fix shape.++**What it verifies:** That `.github/workflows/localisation-tests.yml`'s+`on.push.paths` and `on.pull_request.paths` filters both include every path+pattern that can change what `make test-locales-adhoc` builds or runs+(`prism/**`, `prismTests/**`, `prismUITests/**`, `prism.xcodeproj/**`,+`prism.xctestplan`, `samples/**`, `Tools/**`, `specs/localisation/**`,+`Makefile`, and the workflow file itself), that no entry is a `!` negation,+and that `checks.yml` — the workflow that runs the guard — stays unfiltered+and still invokes it.++**Run command:**+```+python3 Tools/check-workflow-triggers.py+python3 -m unittest Tools.Tests.test_workflow_triggers -v+# or, via the wired Makefile target:+make verify-workflow-triggers+```++Confirmed red before the fix: running the checker against the unmodified+`localisation-tests.yml` reported 8 failures (4 missing paths x 2 triggers).+Confirmed green after the fix: 0 failures.++## Affected Files++| File | Change |+|------|--------|+| `.github/workflows/localisation-tests.yml` | Added `prismTests/**`, `prismUITests/**`, `prism.xcodeproj/**`, `prism.xctestplan`, `samples/**` to both trigger `paths` filters |+| `.github/workflows/checks.yml` | Added a step running `make verify-workflow-triggers` |+| `Tools/check-workflow-triggers.py` | New static guard for the workflow's trigger paths |+| `Tools/Tests/test_workflow_triggers.py` | New unit tests for the guard |+| `Makefile` | New `verify-workflow-triggers` target + help text |+| `CHANGELOG.md` | `[Unreleased]` entry |+| `CLAUDE.md`, `docs/agent-notes/development-tooling.md` | Document the guard alongside `verify-make-guards` / `verify-test-isolation` |++## Verification++**Automated:**+- [x] Regression test passes (`make verify-workflow-triggers`)+- [x] `make verify-make-guards` and `make verify-test-isolation` unaffected/passing+- [x] `make lint` (SwiftLint — no Swift files touched, but run for completeness)++**Manual verification:**+- Diffed `.github/workflows/localisation-tests.yml`'s `paths` lists against+  the actual repo-root layout (`ls -la`) to confirm the added entries match+  real paths (`prism.xcodeproj/`, `prism.xctestplan`, `prismTests/`,+  `prismUITests/`).++## Prevention++**Recommendations to avoid similar bugs:**+- Keep `Tools/check-workflow-triggers.py`'s `REQUIRED_PATHS` list in sync+  whenever a new top-level source/test/build directory is added — it will+  fail closed (report a missing path) if it drifts.+- Prefer no `paths` filter at all on a repository's only test-executing+  workflow unless the filter is actively guarded against omission, since a+  path-filtered trigger fails silently (skips) rather than loudly.++## Related++- Transit ticket: T-2198