Pre-push review #2: T-2224 — check-test-results.sh hardening

diff --git a/Tools/check-test-results.sh b/Tools/check-test-results.shindex 6c8e55de..811be9f3 100755--- a/Tools/check-test-results.sh+++ b/Tools/check-test-results.sh@@ -18,9 +18,30 @@ #   * CI's per-locale sweep reported success for months while launching no tests at #     all — the test bundle could not be signed on the runner, and nothing checked #     that anything had run (T-1983).+#   * A run can retry a failing test and pass on the second attempt. The bundle's+#     top-level counts record only the FINAL outcome, so the first-attempt failure+#     disappears entirely from passedTests/failedTests. Measured on PR #377: a+#     macOS run exited 65 with "** TEST FAILED **" while the bundle reported+#     60/60 passed, because two WebContentTerminationWiringTests cases failed on+#     their first attempt and passed on retry (T-2224).+#   * An interrupted/cancelled/infrastructure-failed run can leave a readable+#     PARTIAL bundle: one or more passing tests, zero recorded failures, and a+#     top-level result of "unknown" rather than "Passed". Counts alone look like+#     a clean pass (T-1993). #-# The zero-test guard below is the specific fix for that last one: a run that-# executes nothing is a failure, whatever else it claims.+# The zero-test guard below is the fix for the "launched nothing" shape, together+# with its skipped-run variant (a run whose every selected test was skipped, which+# verifies just as little while keeping a non-zero total). The top-level RESULT+# guard (T-1993) and the retry scan (T-2224) are the fixes for the other two: a run+# that never RESOLVED at all, and a run whose green counts were only reached by+# silently discarding a first-attempt failure.+#+# Note the difference between "did not resolve" and "did not pass". Only `unknown`+# means the run never reached a verdict; `Skipped` and `Expected Failure` are+# resolved, legitimate outcomes per the schema's TestResult enum. Failing them as+# though they were corruption would be a false FAIL carrying a misleading+# explanation, which is its own harm — see the abnormal-result guard for the full+# reasoning. # # Counts come from the bundle's TOP-LEVEL summary via a JSON parse. Do not go back # to grepping: the summary also carries per-device figures under@@ -28,6 +49,21 @@ # first-match grep silently mixes the two scopes and produces nonsense like # "total=3932 passed=4051" (passed exceeding total). That bug shipped in the first # version of this script and was caught only because the numbers failed to add up.+#+# This script is deliberately self-sufficient: every check below is derived from+# the result bundle alone. It does not need xcodebuild's exit status passed in+# (the Makefile discards it before this script even runs, via the `-` prefix on+# the xcodebuild line — see the Makefile's testing section), because the bundle+# already carries a stronger signal than that exit code ever did: a top-level+# `result` field that is not simply "did the shell command exit 0", and, on any+# test that was retried, one node per attempt with its own individual result+# somewhere beneath the Test Case.+#+# One rule governs every check below, and it is worth stating once rather than+# re-deriving per guard: a check that finds nothing in a tree it does not+# understand has not passed, it has abstained. So the per-test walk reports what it+# could not read, and the shell treats that report as a failure — the same way a+# missing bundle, an unparseable summary, or an absent python3 are failures.  set -uo pipefail @@ -64,7 +100,18 @@ if ! command -v python3 >/dev/null 2>&1; then     exit 1 fi -# Top-level counts only. Emits: total passed failed skipped+# Top-level counts plus the two fields T-1993 and T-2224 need:+#   result           — the bundle's own verdict on the run (Passed / Failed /+#                       Skipped / "Expected Failure" / unknown). An interrupted or+#                       cancelled run can leave passing-looking counts behind while+#                       this stays "unknown" — that mismatch IS the signal (T-1993).+#   expectedFailures — a required top-level count alongside the other four; folding+#                       it into the arithmetic sanity check below closes the gap+#                       T-1993 found in the old total==passed+failed+skipped check.+# Emits: total passed failed skipped expectedFailures result+# `result` is emitted LAST and read into bash's last variable on purpose: its enum+# includes "Expected Failure", which contains a space, and `read` only keeps a+# multi-word value intact when it is the final field. COUNTS=$(printf '%s' "$SUMMARY" | python3 -c ' import json, sys try:@@ -72,51 +119,72 @@ try: except Exception:     raise SystemExit(1) print(d.get("totalTestCount", 0), d.get("passedTests", 0),-      d.get("failedTests", 0), d.get("skippedTests", 0))+      d.get("failedTests", 0), d.get("skippedTests", 0),+      d.get("expectedFailures", 0), d.get("result", "")) ') || {     echo "FAIL [$LABEL]: could not parse the test summary as JSON" >&2     exit 1 } -read -r TOTAL PASSED FAILED SKIPPED <<< "$COUNTS"+read -r TOTAL PASSED FAILED SKIPPED EXPECTED_FAILURES RESULT <<< "$COUNTS"  # Every count must be a number before anything below compares them. Without this an # empty or malformed value makes the comparisons error out and fall through to the # success path — the fail-open bug described above.-for pair in "TOTAL:$TOTAL" "PASSED:$PASSED" "FAILED:$FAILED" "SKIPPED:$SKIPPED"; do-    name="${pair%%:*}"; value="${pair#*:}"+#+# Three ways a value can be unusable, all of them fatal:+#+#   * not digits at all — `[ -eq ]` ERRORS on it under `set -uo pipefail` (no `-e`),+#     which SKIPS the guard and falls through to OK;+#   * implausibly large — a plausibility bound, not an overflow guard. Bash uses+#     64-bit arithmetic here, so `[ -eq ]` only errors ("integer expression+#     expected") at 20 digits; between 11 and 19 it compares fine. Ten digits+#     (up to 9999999999) is far past any real test count, and a value beyond it+#     means the field was not a count in the first place;+#   * a leading zero — bash arithmetic reads `08` as octal and fails with "value too+#     great for base", leaving `$(( ))` targets unassigned; the next `set -u`+#     expansion then kills the script with a bare "unbound variable" and no hint of+#     what actually went wrong. It still fails closed, but a guard whose whole job is+#     explaining itself should not exit on an unexplained bash error.+#+# None of the three is reachable from a real xcresult bundle; that is precisely why+# none of them may be the one place this script quietly passes or dies mute.+validate_count() {+    local name="$1" value="$2"+    shift 2     case "$value" in         ''|*[!0-9]*)             echo "FAIL [$LABEL]: $name came back as '${value}', which is not a count." >&2-            echo "  Refusing to report success on an unverified run." >&2-            exit 1+            ;;+        0?*)+            echo "FAIL [$LABEL]: $name came back as '${value}', which has a leading zero." >&2+            echo "  Bash arithmetic would read that as octal and error out mid-comparison." >&2+            ;;+        *)+            if [ "${#value}" -le 10 ]; then+                return 0+            fi+            echo "FAIL [$LABEL]: $name came back as '${value}', which is implausibly large." >&2             ;;     esac-    # Digits alone are not enough. A value too large for a machine integer passes the-    # glob above and then makes `[ -eq ]` / `[ -ne ]` ERROR under `set -uo pipefail`-    # (no `-e`), which skips the guard and prints OK — the exact fail-open shape this-    # script exists to prevent, reached through magnitude rather than a malformed-    # string. Unreachable for a real xcresult bundle, but the whole value of this tool-    # is that it cannot quietly pass, so bound it. Ten digits allows up to 9999999999.-    if [ "${#value}" -gt 10 ]; then-        echo "FAIL [$LABEL]: $name came back as '${value}', which is implausibly large." >&2-        echo "  Refusing to report success on an unverified run." >&2-        exit 1+    if [ "$#" -gt 0 ]; then+        printf '%s\n' "$@" >&2     fi-done+    echo "  Refusing to report success on an unverified run." >&2+    exit 1+} -echo "[$LABEL] total=$TOTAL passed=$PASSED failed=$FAILED skipped=$SKIPPED"+validate_count TOTAL "$TOTAL"+validate_count PASSED "$PASSED"+validate_count FAILED "$FAILED"+validate_count SKIPPED "$SKIPPED"+validate_count EXPECTED_FAILURES "$EXPECTED_FAILURES" -# Sanity-check the arithmetic. If the parts do not reconstruct the whole, the-# figures are being read from mismatched scopes and nothing below can be trusted.-EXPECTED=$((PASSED + FAILED + SKIPPED))-if [ "$TOTAL" -ne "$EXPECTED" ]; then-    echo "WARN [$LABEL]: passed+failed+skipped ($EXPECTED) != total ($TOTAL) — counts may be unreliable" >&2-fi+echo "[$LABEL] total=$TOTAL passed=$PASSED failed=$FAILED skipped=$SKIPPED expectedFailures=$EXPECTED_FAILURES result=$RESULT"  # --- The zero-test guard (T-1983) --------------------------------------------# The whole point of the script. A green run that executed nothing is the most-# dangerous result there is, because it is indistinguishable from success.+# The whole point of the original script. A green run that executed nothing is+# the most dangerous result there is, because it is indistinguishable from success. if [ "$TOTAL" -eq 0 ]; then     echo "FAIL [$LABEL]: the run executed ZERO tests." >&2     echo "  This is treated as a failure, not a pass. Common causes:" >&2@@ -129,6 +197,431 @@ if [ "$TOTAL" -eq 0 ]; then     exit 1 fi +# --- The abnormal-result guard (T-1993) ---------------------------------------+# totalTestCount/passedTests/failedTests are the counts a human reads first, but+# they are not the bundle's own verdict on the run — `result` is. An interrupted,+# cancelled, or infrastructure-failed run (a wedged test host, a killed xcodebuild)+# can leave a readable PARTIAL bundle behind: some tests recorded as passed, zero+# recorded as failed, and `result` sitting at "unknown" because the run never+# reached a real conclusion. Gating on counts alone reports that as OK; gating on+# `result` does not.+#+# What counts as abnormal is the schema's business, not a guess. `result` is typed+# as the TestResult enum, and the SAME enum is used for the top-level run verdict+# as for an individual test (confirmed against+# `xcrun xcresulttool get test-results summary --schema`):+#+#     ["Passed", "Failed", "Skipped", "Expected Failure", "unknown"]+#+# So "Skipped" and "Expected Failure" are LEGITIMATE run outcomes, not corruption —+# a `-only-testing:` selection resolving entirely to skipped tests is a normal thing+# to ask for, and this repo has both narrow `-only-testing:` workflows and dozens of+# `.disabled(...)` tests. An earlier version of this guard required Passed/Failed and+# would have hard-failed those runs with the "interrupted/cancelled" diagnosis below,+# which is not merely a wrong verdict but a MISLEADING one — it sends the reader+# hunting an infrastructure fault that never happened. `unknown` is the genuine+# abnormal shape, and anything outside the enum is a schema change this script has+# not been taught, so both fail closed here.+#+# "Failed" is allowed through: that shape already has a dedicated, more informative+# branch below (the cascade-vs-genuine split). A run that resolved to Skipped or+# Expected Failure without executing anything is NOT waved through either — it is+# caught by the nothing-executed guard immediately below, which gives it the+# accurate diagnosis instead of this one.+case "$RESULT" in+    Passed|Failed|Skipped|"Expected Failure") ;;+    *)+        echo "FAIL [$LABEL]: the bundle's top-level result is '$RESULT', which is not a" >&2+        echo "  resolved outcome. The pass/fail counts above may look clean — that is" >&2+        echo "  exactly the trap: an interrupted, cancelled, or infrastructure-failed run" >&2+        echo "  can leave a PARTIAL bundle with passing-looking counts and zero recorded" >&2+        echo "  failures, because the run itself never reached a real verdict (T-1993)." >&2+        echo "  Refusing to report success on a run that never resolved." >&2+        exit 1+        ;;+esac++# Sanity-check the arithmetic, now including expectedFailures (T-1993 found the+# original version of this check incomplete, not just non-fatal — see below for+# why it now fails closed instead of warning). If the parts do not reconstruct the+# whole, the figures are being read from mismatched scopes and nothing below can+# be trusted.+EXPECTED_TOTAL=$((PASSED + FAILED + SKIPPED + EXPECTED_FAILURES))+if [ "$TOTAL" -ne "$EXPECTED_TOTAL" ]; then+    echo "FAIL [$LABEL]: passed+failed+skipped+expectedFailures ($EXPECTED_TOTAL) != total ($TOTAL)." >&2+    echo "  The counts do not reconstruct the whole, which means they were read from" >&2+    echo "  mismatched scopes or a corrupt/partial bundle. Nothing below this line can" >&2+    echo "  be trusted, so this is a failure rather than a warning (T-1993)." >&2+    exit 1+fi++# --- The nothing-executed guard (T-1983, skipped-run variant) -----------------+# The zero-test guard above catches "the run selected nothing". This catches the+# other way to verify nothing while looking healthy: every selected test was+# SKIPPED. totalTestCount counts skipped tests, so that run sails past `TOTAL -eq 0`+# with internally consistent counts and a perfectly legitimate top-level result of+# "Skipped" — and no test body ever ran.+#+# This is the same hazard as T-1983 (a green-looking run that verified nothing), so+# it gets the same verdict, but it needs its OWN diagnosis. Reporting it as an+# interrupted/infrastructure-failed run — which is what the previous guard did — is+# a correct FAIL with a wrong explanation, and that is its own kind of damage: the+# next person chases a wedged host that was never there.+#+# Deliberately placed AFTER the arithmetic check, not before it. Only once the parts+# reconstruct the whole is "executed nothing" guaranteed to mean "everything was+# skipped" — otherwise a corrupt bundle reporting total=5 with all four parts at+# zero would land here and be explained as five skipped tests, when the honest+# answer is the one the check above gives: these counts do not add up and nothing+# can be read from them. Same discipline as the rest of this guard: never hand out+# a confident diagnosis the data does not support.+EXECUTED=$((PASSED + FAILED + EXPECTED_FAILURES))+if [ "$EXECUTED" -eq 0 ]; then+    echo "FAIL [$LABEL]: the run executed ZERO tests — all $SKIPPED of them were skipped." >&2+    # The counts adding up does not by itself make the bundle coherent. A result of+    # 'Failed' alongside zero executed tests is a contradiction — something failed+    # that never ran — so do not tell the reader nothing is corrupt when the bundle+    # is, on its face, saying two incompatible things. Same discipline as above:+    # never hand out a confident diagnosis the data does not support.+    case "$RESULT" in+        Passed|Skipped|"Expected Failure")+            echo "  The counts are internally consistent and the bundle's own result is" >&2+            echo "  '$RESULT', so nothing here is corrupt; the run simply never ran a test" >&2+            echo "  body, which verifies exactly as much as running no tests at all (T-1983)." >&2+            echo "  Common causes:" >&2+            echo "    - '-only-testing:' selected only tests that are disabled or skipped" >&2+            echo "      (a '.disabled(...)' trait, or an XCTSkip / availability guard);" >&2+            echo "    - the test plan's configuration skipped every test it selected." >&2+            ;;+        *)+            echo "  The counts are internally consistent, but the bundle's own result is" >&2+            echo "  '$RESULT' — which cannot be reconciled with executing nothing, since" >&2+            echo "  a test that never ran cannot have failed. Treat this bundle as" >&2+            echo "  untrustworthy rather than as a skipped run: the disagreement is the" >&2+            echo "  signal (T-2224). Check whether the run died before or during test" >&2+            echo "  execution — a wedged or aborted host reports this shape." >&2+            ;;+    esac+    exit 1+fi++# --- Retry scan + cascade classification (T-2224 / T-1541) -------------------+# One `tests` tree walk answers three questions:+#+#   1. Did any test need a retry to reach its final result? The top-level counts+#      above only ever record the LAST attempt, so a test that failed once and+#      passed on retry is invisible to them — it looks identical to a test that+#      passed cleanly. That is precisely how PR #377 got through: two+#      WebContentTerminationWiringTests cases failed first-attempt and passed on+#      retry, and the bundle's 60/60 hid it completely (T-2224). Each attempt is+#      recorded as its own node (`Repetition` or `Test Case Run`) somewhere BELOW+#      the Test Case, with its own `result` — the per-attempt history the+#      top-level counts throw away.+#   2. Does the tree contradict the summary — a Test Case recorded Failed while+#      failedTests is zero — or is the tree simply not something this script+#      understands? Either way its silence proves nothing.+#   3. Of any FAILED tests, which are cascade artefacts rather than genuine+#      failures? (Unchanged from the T-1541 fix below.)+#+# All three come from the same `xcrun xcresulttool get test-results tests` call, so+# it is made once and reused, rather than shelling out three times.+#+# LIMITATION, STATED PLAINLY: the per-attempt node shape is SCHEMA-DERIVED, NOT+# CONFIRMED AGAINST A CAPTURED RETRY BUNDLE. What is verified is only that+# `xcrun xcresulttool get test-results tests --schema` lists "Repetition" and+# "Test Case Run" as TestNodeType values distinct from "Test Case", and marks+# `result` optional on every TestNode. What is NOT verified is which of the two+# xcresulttool actually emits for a retried test, or where beneath the Test Case it+# puts it — hence a scan that accepts either type at any depth and tolerates a+# missing result on an attempt whose child carries one. Attempts to capture a real+# retried bundle on this machine failed: every+# `-retry-tests-on-failure -test-iterations 2` run died with+# `** BUILD INTERRUPTED **` under load and left an unreadable bundle, and a sweep of+# the readable historical bundles on this machine (834 bundles / 55,986 Test Case+# nodes across the wider sweep) found ZERO containing a Repetition or Test Case Run+# node — no past run on this project used retries. The fixtures pin the parsing,+# the descent through the configuration layer, and the control flow against+# hand-written JSON; they cannot pin the shape. Treat the attempt-node types as the+# least-verified part of this script, and if a real retried bundle is ever captured,+# diff it against Tools/Tests/Fixtures/check-test-results/retried-then-passed/.+# What is NOT a guess is the layer the scan had to learn to descend through: the+# Test Case -> Test Plan Configuration nesting is measured on real bundles from this+# project, and is what `make test` and `make test-ui` produce on every run.+TESTS_JSON=$(xcrun xcresulttool get test-results tests --path "$BUNDLE" 2>/dev/null)+if [ -z "$TESTS_JSON" ]; then+    echo "FAIL [$LABEL]: could not read the per-test tree from $BUNDLE." >&2+    echo "  The top-level counts said $TOTAL test(s) ran, but the detail this script" >&2+    echo "  needs to rule out a retry-laundered failure is unreadable. Refusing to" >&2+    echo "  report success on a run that cannot be fully verified." >&2+    exit 1+fi++ANALYSIS=$(printf '%s' "$TESTS_JSON" | python3 -c '+import json, sys++# The two enums this tree is typed with, copied verbatim from+# "xcrun xcresulttool get test-results tests --schema". Anything outside them is a+# schema this script has NOT been taught, and is reported rather than skipped past+# — see the UNREADABLE handling in the shell below for why that has to be fatal.+RESULTS = {"Passed", "Failed", "Skipped", "Expected Failure", "unknown"}+NODE_TYPES = {+    "Test Plan", "Unit test bundle", "UI test bundle", "Test Suite", "Test Case",+    "Device", "Test Plan Configuration", "Arguments", "Repetition", "Test Case Run",+    "Failure Message", "Source Code Reference", "Attachment", "Expression",+    "Test Value", "Runtime Warning",+}+ATTEMPT_TYPES = {"Repetition", "Test Case Run"}++problems = []+retried = []+counts = {"cases": 0, "failed_cases": 0, "cascade": 0}++def clean(text):+    # One marker per line, so a name containing a newline (or any other control+    # character) must not be able to forge a second marker line. Runs of ordinary+    # spaces are preserved: the shell reads these back with sed, not awk, precisely+    # so that a test name spelled with double spaces comes out as the author wrote it.+    return "".join(" " if (ord(c) < 0x20 or ord(c) == 0x7f) else c for c in str(text))++def kids(node):+    children = node.get("children")+    return children if isinstance(children, list) else []++def collect_attempts(node, acc, label):+    # Attempts are gathered from the WHOLE subtree of a Test Case, not from its+    # direct children. That is the T-2224 review fix, and it is not a special case+    # for one node type: what the schema guarantees is that Test Case is the+    # leaf-most node carrying test IDENTITY, and that everything below it is detail+    # ABOUT that one test — grouping layers (Test Plan Configuration, Arguments,+    # Device) and leaf annotations (Failure Message, ...). So "every attempt at this+    # test" is "every attempt node anywhere beneath this Test Case", at whatever+    # depth the grouping layers happen to put it.+    #+    # Reading direct children only was structurally inert on the two pre-push targets+    # that matter most. "make test" and "make test-ui" pass no+    # -only-test-configuration, so every Test Case gets one Test Plan Configuration+    # child PER CONFIGURATION and no attempt node is ever a direct child. Measured on+    # this machine: of the readable historical bundles carrying more than one+    # configuration, EVERY Test Case had Test Plan Configuration children and nothing+    # else structural (e.g. 93 cases / 372 configuration nodes / 4 configurations).+    # A guard that silently collects nothing on half the pre-push matrix is the same+    # silent non-firing this script exists to eliminate.+    if not isinstance(node, dict):+        problems.append("a node under %s is not a JSON object" % label)+        return+    before = len(acc)+    for child in kids(node):+        collect_attempts(child, acc, label)+    if node.get("nodeType") in ATTEMPT_TYPES:+        # Post-order on purpose: children are collected first, so an attempt node+        # that carries no result of its own (the schema marks "result" optional) is+        # satisfied by a nested attempt that does. Only when NOTHING in the subtree+        # produced a result is the attempt unreadable — and an unreadable attempt is+        # reported, never treated as "no failure here".+        result = node.get("result")+        if result in RESULTS:+            acc.append(result)+        elif result is not None:+            problems.append("an attempt node under %s has result %r, which is not a TestResult"+                            % (label, clean(result)))+        elif len(acc) == before:+            problems.append("an attempt node under %s records no result, and neither does "+                            "anything nested inside it" % label)++def walk(node, path):+    if not isinstance(node, dict):+        problems.append("the tests tree contains a node that is not a JSON object")+        return+    node_type = node.get("nodeType")+    name = node.get("name")+    if node_type not in NODE_TYPES:+        problems.append("unrecognised nodeType %r in the tests tree" % clean(node_type))+    here = path + [name] if (node_type and name) else path+    label = "/".join(p for p in here if p) or "(unnamed)"++    if node_type == "Test Case":+        counts["cases"] += 1+        result = node.get("result")+        if result is not None and result not in RESULTS:+            problems.append("test case %s has result %r, which is not a TestResult"+                            % (label, clean(result)))+        if result == "Failed":+            counts["failed_cases"] += 1+            # A cascade artefact is a failed Test Case with NO recorded duration:+            # the test never ran because the host was already dead when it was+            # queued. A genuine failure always carries a duration, even a very+            # small one. See the FAILED>0 branch below for the full rationale.+            if node.get("duration") is None and node.get("durationInSeconds") is None:+                counts["cascade"] += 1++        # A Failed attempt anywhere, while the Test Case itself did not end Failed,+        # means the run only reported success because a retry papered over a real+        # failure. Deliberately NOT keyed on ordering or on attempt count:+        #+        #   * ordering — the old rule looked only at attempts[:-1], so a bundle that+        #     lists attempts newest-first (which nothing here can rule out, since no+        #     real retried bundle exists to check) hid the failure in the slot the+        #     rule skipped;+        #   * count — the old rule required more than one attempt node, so a single+        #     recorded attempt of Failed under a Test Case that ended Passed sailed+        #     through. That combination is a contradiction on its face, and this+        #     script treats bundle-versus-bundle contradictions as signal, not noise.+        #+        # What it is still NOT keyed on is the mere EXISTENCE of several attempt+        # nodes. XCTest measure(metrics:) repeats its block many times inside ONE+        # test case (prismUITests testLaunchPerformance does this on every+        # "make test-ui"), and whether xcresulttool represents those iterations as+        # Repetition children is not something any bundle available here could+        # settle. It does not need settling: a passing performance test records no+        # Failed iteration, so this branch cannot fire for it however the iterations+        # are represented. The only way such a test trips the guard is if an+        # iteration genuinely recorded Failed while the case still ended green —+        # which IS the laundering shape.+        attempts = []+        for child in kids(node):+            collect_attempts(child, attempts, label)+        if result != "Failed" and "Failed" in attempts:+            retried.append(label)++    for child in kids(node):+        walk(child, here)++try:+    document = json.load(sys.stdin)+except Exception:+    raise SystemExit(1)++if not isinstance(document, dict):+    problems.append("the tests output is not a JSON object")+    roots = []+else:+    roots = document.get("testNodes")+    if not isinstance(roots, list):+        problems.append("the tests output has no testNodes array")+        roots = []++for root in roots:+    walk(root, [])++print("CASES %d" % counts["cases"])+print("FAILEDCASES %d" % counts["failed_cases"])+print("CASCADE %d" % counts["cascade"])+print("RETRIED %d" % len(retried))+seen = []+for problem in problems:+    if problem not in seen:+        seen.append(problem)+for problem in seen[:5]:+    print("UNREADABLE %s" % clean(problem))+if len(seen) > 5:+    # NOT "of the same kind": seen is deduplicated across ALL problem kinds and+    # truncated by position, so the hidden remainder can be a different kind+    # entirely — a nodeType drift pushed off the list by five unrelated per-case+    # problems. Say what is true (how many distinct problems are unshown) rather+    # than characterising problems that are not being printed.+    print("UNREADABLE ... and %d more distinct problem(s), not shown" % (len(seen) - 5))+for name in retried:+    print("RETRIED_NAME %s" % clean(name))+') || {+    echo "FAIL [$LABEL]: could not parse the per-test tree as JSON." >&2+    echo "  Refusing to report success on a run that cannot be fully verified." >&2+    exit 1+}++CASES=$(printf '%s\n' "$ANALYSIS" | awk '$1=="CASES"{print $2; exit}')+FAILED_CASES=$(printf '%s\n' "$ANALYSIS" | awk '$1=="FAILEDCASES"{print $2; exit}')+CASCADE=$(printf '%s\n' "$ANALYSIS" | awk '$1=="CASCADE"{print $2; exit}')+RETRIED=$(printf '%s\n' "$ANALYSIS" | awk '$1=="RETRIED"{print $2; exit}')++# Validate these the same way the summary counts are validated above, rather than+# defaulting them to 0 and wrapping the comparisons in `2>/dev/null`. That older+# form silently swallowed a non-numeric value: `[ "$RETRIED" -gt 0 ]` ERRORS on a+# non-number, the redirect hid the error, and the retry guard then quietly did not+# fire — a fail-OPEN path inside the one check whose entire purpose is to stop a+# retry-laundered failure reporting success. These values are script-controlled+# (printed by the embedded Python above), so this should be unreachable; that is+# precisely why it must not be the one place that fails open if it ever is reached.+UNPARSED_TREE_HINT="  The per-test tree parsed, but its summary line did not. Refusing to+  report success on a run whose retry history could not be checked."+validate_count CASES "$CASES" "$UNPARSED_TREE_HINT"+validate_count FAILED_CASES "$FAILED_CASES" "$UNPARSED_TREE_HINT"+validate_count CASCADE "$CASCADE" "$UNPARSED_TREE_HINT"+validate_count RETRIED "$RETRIED" "$UNPARSED_TREE_HINT"++# --- The tree-comprehension guard (T-2224) ------------------------------------+# Everything below reads meaning out of the per-test tree. If the walk understood+# NOTHING in that tree, every one of those readings is vacuously clean: no attempt+# nodes found means no retries found, which is indistinguishable from a run that+# genuinely had none. That is the same false green as T-1983, one level down.+#+# Two independent ways to notice, both fatal:+#+#   1. The walk met a nodeType or a result value outside the published enums. A+#      one-character drift ("Test Case" -> "TestCase") is enough to make the retry+#      scan collect nothing, and it would otherwise be silent. This is the same+#      stance the abnormal-result guard takes on the top-level `result`: a schema+#      this script has not been taught is a reason to stop, not to guess. It does+#      mean a future Xcode that adds a TestNodeType will fail this check until the+#      list above is updated — an accepted, loud, one-line cost, chosen over the+#      alternative of a guard that goes quiet exactly when the format moves.+#   2. The counts say tests ran, but the tree contains no Test Case node at all —+#      an empty object, an empty array, or a tree whose case nodes are spelled+#      something this script does not recognise.+# Read with a here-string, NOT `printf ... | grep -q`. Under `set -o pipefail` that+# pipeline returns 141 (SIGPIPE) instead of 0 whenever grep matches and exits while+# printf still has output to write — which is exactly when the UNREADABLE line is+# followed by RETRIED_NAME lines. Measured on this machine at ~0.3-0.6% of runs+# under parallel load: the `if` then reads FALSE on a tree that IS unreadable and+# the guard silently does not fire. A guard against silent success must not have a+# silent-skip path of its own; a here-string has no pipeline and no SIGPIPE.+if grep -q '^UNREADABLE ' <<< "$ANALYSIS"; then+    echo "FAIL [$LABEL]: the per-test tree is readable JSON but this script does not" >&2+    echo "  understand its shape:" >&2+    printf '%s\n' "$ANALYSIS" | sed -n 's/^UNREADABLE /    - /p' >&2+    echo "  An unrecognised tree cannot be checked for retry-laundered failures, and a" >&2+    echo "  check that finds nothing in a tree it cannot read is not a pass (T-2224)." >&2+    exit 1+fi++if [ "$CASES" -eq 0 ]; then+    echo "FAIL [$LABEL]: the summary counts $TOTAL test(s), but the per-test tree" >&2+    echo "  contains no Test Case node at all. The two disagree, so the tree cannot be" >&2+    echo "  used to rule out a retry-laundered failure — and a retry scan over a tree" >&2+    echo "  with no test cases in it finds nothing for the same reason an empty file" >&2+    echo "  would (T-2224). Refusing to report success on a run that cannot be verified." >&2+    exit 1+fi++if [ "$RETRIED" -gt 0 ]; then+    echo "  $RETRIED test(s) only reached their final result after a retry — an" >&2+    echo "  attempt FAILED and is invisible in the counts above:" >&2+    printf '%s\n' "$ANALYSIS" | sed -n 's/^RETRIED_NAME /    - /p' >&2++    if [ "$FAILED" -eq 0 ]; then+        echo "FAIL [$LABEL]: refusing to report success — $RETRIED test(s) passed only on retry." >&2+        echo "  A first-attempt failure that a retry silently erases is exactly the shape" >&2+        echo "  that let PR #377's failing run report 60/60 (T-2224)." >&2+        exit 1+    fi+fi++# --- The failed-case contradiction guard (T-2224) ------------------------------+# The summary's failedTests and the tree's own Test Case results are two recordings+# of the same fact. When they disagree — a case recorded Failed while the summary+# counts zero failures — the run is not a pass, whichever recording is wrong. This+# is the T-2224 thesis (the bundle and reality disagreeing) applied to two figures+# this script already holds in memory, and the old version simply discarded one of+# them. Measured on the readable historical bundles on this machine, the two agree+# EXACTLY in every case (e.g. 5331 Test Case nodes, 60 recorded Failed, and every+# bundle's failedTests equal to its own failed-case count), so this fires only on a+# genuine contradiction.+if [ "$FAILED" -eq 0 ] && [ "$FAILED_CASES" -gt 0 ]; then+    echo "FAIL [$LABEL]: the summary reports zero failed tests, but the per-test tree" >&2+    echo "  records $FAILED_CASES test case(s) as Failed. The bundle contradicts itself, so its" >&2+    echo "  green counts cannot be trusted — the disagreement is the signal (T-2224)." >&2+    exit 1+fi+ if [ "$FAILED" -gt 0 ]; then     # Separate real failures from cascade artefacts. When a test traps it kills the     # test host, and every test still queued is reported as failed without ever having@@ -137,9 +630,6 @@ if [ "$FAILED" -gt 0 ]; then     # tells you whether you are looking at N broken tests or one crasher plus wreckage.     # (The console prints those queued tests as "0.000 seconds"; the result bundle,     # which is what this script reads, records no duration for them at all.)-    # A cascade artefact is a failed Test Case with NO recorded duration: the test-    # never ran, because the host was already dead, so nothing timed it. A genuine-    # failure always carries a duration, even a very small one.     #     # Do not switch this to a numeric comparison against zero. Measured on a real     # bundle: the 779 artefacts have `duration: None`, while genuine failures report@@ -156,29 +646,14 @@ if [ "$FAILED" -gt 0 ]; then     # a test that traps during setup or teardown, before timing starts, could also     # end up with no duration and be reported as wreckage. The split is a triage aid,     # not an authority — the FAILED count above it is the number that gates the build.-    CASCADE=$(xcrun xcresulttool get test-results tests --path "$BUNDLE" 2>/dev/null | python3 -c '-import json, sys--def walk(node, acc):-    if isinstance(node, dict):-        if node.get("result") == "Failed" and node.get("nodeType") in ("Test Case", "TestCase"):-            if node.get("duration") is None and node.get("durationInSeconds") is None:-                acc[0] += 1-        for v in node.values():-            walk(v, acc)-    elif isinstance(node, list):-        for v in node:-            walk(v, acc)--acc = [0]-try:-    walk(json.load(sys.stdin), acc)-except Exception:-    raise SystemExit(1)-print(acc[0])-') || CASCADE=""--    if [ -n "$CASCADE" ] && [ "$CASCADE" -gt 0 ] 2>/dev/null; then+    #+    # The split is only printed when it subtracts coherently. CASCADE counts Failed+    # Test Case nodes in the tree; FAILED comes from the summary. Every real bundle+    # on this machine agrees on those two figures, but if one ever exceeded the other+    # the subtraction below would report "~-1 look like genuine failures", which is+    # not a triage aid, it is noise. Say nothing rather than something false; the+    # FAILED count above still gates the build either way.+    if [ "$CASCADE" -gt 0 ] && [ "$CASCADE" -le "$FAILED" ]; then         REAL=$((FAILED - CASCADE))         echo "  of which ~$CASCADE never ran (no recorded duration — the host crashed" >&2         echo "  mid-run and these were still queued; they are not results)" >&2@@ -190,4 +665,26 @@ print(acc[0])     exit 1 fi +# Belt-and-braces: FAILED is 0, and the abnormal-result guard above already+# narrowed RESULT to the four resolved TestResult values. Of those, "Failed" with+# zero recorded test failures is a whole-run failure (e.g. a build error inside the+# test action) with nothing to point at — still not a pass. "Skipped" and+# "Expected Failure" ARE passes: they are resolved outcomes, and the+# nothing-executed guard above has already established that at least one test+# actually ran to a verdict.+#+# Written as a positive allowlist rather than `!= "Passed"` so it stays fail-closed+# on its own terms: if the enum ever grows a value the guard above learns to accept,+# this line does not silently start passing it too.+case "$RESULT" in+    Passed|Skipped|"Expected Failure") ;;+    *)+        echo "FAIL [$LABEL]: no individual test is recorded as failed, but the bundle's" >&2+        echo "  top-level result is '$RESULT', which is not a successful outcome. That is" >&2+        echo "  a whole-run failure with nothing to point at — a build error inside the" >&2+        echo "  test action, for example. Refusing to report success." >&2+        exit 1+        ;;+esac+ echo "[$LABEL] OK"

diff --git a/Tools/Tests/test-check-test-results.sh b/Tools/Tests/test-check-test-results.shnew file mode 100755index 00000000..1fd0a772--- /dev/null+++ b/Tools/Tests/test-check-test-results.sh@@ -0,0 +1,372 @@+#!/bin/bash+#+# test-check-test-results.sh — regression tests for Tools/check-test-results.sh+# (T-2224 / T-1993).+#+# Two shapes let a bad run report OK before this suite existed:+#+#   * T-2224: a test can FAIL on its first attempt and PASS on a retry. The+#     bundle's top-level counts record only the final attempt, so the failure+#     is invisible to total/passed/failed — measured directly on PR #377 (a+#     macOS run exited 65 with "** TEST FAILED **" while the bundle reported+#     60/60 passed).+#   * T-1993: an interrupted/cancelled/infrastructure-failed run can leave a+#     readable PARTIAL bundle with passing-looking counts (zero recorded+#     failures) while the bundle's own top-level `result` field is "unknown"+#     rather than "Passed" — the run never reached a real verdict.+#+# Both are invisible to a black-box "does it exit 0" smoke test on a real+# xcodebuild run: reproducing them for real requires flaky live tests or a+# wedged host, which is exactly the kind of thing this suite must not depend+# on to be repeatable. Instead each fixture under+# Tools/Tests/Fixtures/check-test-results/<scenario>/ IS a directory standing+# in for an .xcresult bundle, containing summary.json and tests.json — the+# JSON `xcrun xcresulttool get test-results {summary,tests}` would print for+# that bundle.+#+# WHAT THE FIXTURES ARE, EXACTLY: every one of them is HAND-WRITTEN. None was+# captured from a real bundle. What they are checked against is the published+# schema (`xcrun xcresulttool get test-results summary --schema` /+# `... tests --schema`) and, for node NESTING, the shape measured by reading+# real .xcresult bundles on this machine: a `Test Plan` root, and — in every+# multi-configuration bundle — `Test Case` nodes whose children are+# `Test Plan Configuration` nodes, one per configuration, with no attempt node+# ever a direct child of the Test Case.+#+# WHAT THEY STILL DO NOT MODEL, said plainly because a green suite otherwise+# implies coverage it has not earned: no real RETRIED bundle exists to copy.+# A sweep of the readable historical bundles on this machine found zero+# `Repetition` and zero `Test Case Run` nodes, and every attempt to produce one+# with `-retry-tests-on-failure` died before writing a readable bundle. So the+# retry fixtures pin the script's PARSING, its descent through the+# configuration layer, and its control flow — they cannot pin the attempt-node+# shape itself. That limitation is the whole reason the script accepts either+# attempt type at any depth, and the reason it now fails closed on an attempt+# node it cannot read rather than counting it as "no failure here".+#+# The stub `xcrun` below never touches the network or a real bundle: it reads+# <bundle-path>/summary.json or <bundle-path>/tests.json depending on which+# xcresulttool subcommand was requested, where <bundle-path> is whatever this+# suite passed as the bundle argument — i.e. the fixture directory itself.+#+# Usage: Tools/Tests/test-check-test-results.sh++set -uo pipefail++REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)+cd "$REPO_ROOT" || exit 2++SCRIPT="$REPO_ROOT/Tools/check-test-results.sh"+FIXTURES="$REPO_ROOT/Tools/Tests/Fixtures/check-test-results"++FAILURES=0++pass() { echo "  ok   — $1"; }+fail() { echo "  FAIL — $1" >&2; FAILURES=$((FAILURES + 1)); }++SANDBOX=$(mktemp -d)+trap 'rm -rf "$SANDBOX"' EXIT+mkdir -p "$SANDBOX/bin"++# Stub xcrun: only understands `xcresulttool get test-results {summary,tests}+# --path <dir>`, and serves that directory's summary.json / tests.json. Any+# other invocation exits 1, so a real xcrun elsewhere on PATH is never reached+# by accident.+cat > "$SANDBOX/bin/xcrun" <<'EOF'+#!/bin/bash+subcmd="${4:-}"+path=""+prev=""+for arg in "$@"; do+    if [ "$prev" = "--path" ]; then path="$arg"; fi+    prev="$arg"+done+case "${1:-}/${2:-}/${3:-}/$subcmd" in+    xcresulttool/get/test-results/summary) exec cat "$path/summary.json" ;;+    xcresulttool/get/test-results/tests)   exec cat "$path/tests.json" ;;+    *) exit 1 ;;+esac+EOF+chmod +x "$SANDBOX/bin/xcrun"++# run <fixture-name-or-real-path> <label> — invoke the real script under the+# stub, capture exit code + combined output.+OUT=""+STATUS=0+run() {+    OUT=$(PATH="$SANDBOX/bin:$PATH" "$SCRIPT" "$1" "$2" 2>&1)+    STATUS=$?+}++# Every assertion below searches "$OUT" through these two helpers, which use a+# here-string rather than `printf '%s' "$OUT" | grep …`. Under `set -o pipefail`+# that pipeline returns 141 (SIGPIPE) rather than 0 when grep matches early and+# exits while printf is still writing, so an assertion that SHOULD pass reports a+# failure instead. Measured here at roughly 1 run in 200 under parallel load — and+# a merge-gate self-test that cries wolf is a self-test people learn to re-run+# until it is green, which is the same disease as a gate that stays silent.+saw()  { grep -qF -- "$1" <<< "$OUT"; }+sawi() { grep -qiF -- "$1" <<< "$OUT"; }++expect_pass() {+    local fixture="$1" desc="$2"+    run "$FIXTURES/$fixture" "$fixture"+    if [ "$STATUS" -eq 0 ] && grep -q ' OK$' <<< "$OUT"; then+        pass "$desc"+    else+        fail "$desc (exit=$STATUS)"$'\n'"$OUT"+    fi+}++expect_fail() {+    local fixture="$1" desc="$2" needle="$3"+    run "$FIXTURES/$fixture" "$fixture"+    if [ "$STATUS" -eq 0 ]; then+        fail "$desc: reported success (exit 0) — this is the false-green shape"$'\n'"$OUT"+        return+    fi+    if [ -n "$needle" ] && ! saw "$needle"; then+        fail "$desc: failed (good) but did not explain why — expected to see: $needle"$'\n'"$OUT"+        return+    fi+    pass "$desc"+}++echo "--- Regressions: shapes the script already caught ---------------------"++# The zero-test guard (T-1983) is the reason this script exists at all, so it+# needs a needle only IT can produce. Asserting "executed ZERO tests" alone did+# not pin it: deleting the guard outright lets the run fall through to the+# nothing-executed guard further down, which prints the same phrase — the suite+# stayed green with the original guard removed. The signing/launch diagnosis is+# unique to this branch.+expect_fail "zero-tests" "zero executed tests still fails" "could not be signed or launched"+expect_fail "genuine-failure-cascade" "genuine failures still fail, with the cascade split reported" \+    "of which ~2 never ran"+expect_fail "corrupt-total" "a non-numeric total still fails" "which is not a count"+expect_fail "leading-zero-count" \+    "a count spelled with a leading zero fails with a diagnosis, not a bare 'unbound variable'" \+    "which has a leading zero"++echo+echo "--- T-1993: abnormal / partial runs must not report OK ----------------"++# Both needles must be text ONLY the abnormal-result guard can print. The obvious+# phrasing — "top-level result is 'unknown', which is not a" — is a prefix of the+# FINAL result allowlist's message too ("…which is not a successful outcome"), so a+# fixture needling it stays green with the abnormal-result guard deleted outright:+# the run simply falls through and is caught 400 lines later by a different guard+# printing a superset of the needle. That is the same unpinned-guard trap the+# zero-test check above documents, and it was live here. "never resolved" is unique+# to this branch.+expect_fail "unknown-result-partial" \+    "a partial bundle with passing-looking counts but result=unknown fails" \+    "Refusing to report success on a run that never resolved."+expect_fail "wedged-host" \+    "a wedged host (total=1 failed=1 result=unknown) fails with the abnormal-result diagnosis" \+    "Refusing to report success on a run that never resolved."+expect_fail "mismatched-counts" \+    "passed+failed+skipped+expectedFailures != total fails closed (no longer just a warning)" \+    "do not reconstruct the whole"+expect_fail "mismatched-counts-silent" \+    "a count mismatch with zero recorded failures fails closed (this is the shape the old WARN let through)" \+    "do not reconstruct the whole"++echo+echo "--- The abnormal-result guard must not over-reach --------------------"+# The top-level `result` field is typed as the SAME TestResult enum as a+# per-test result — ["Passed","Failed","Skipped","Expected Failure","unknown"]+# — confirmed against `xcrun xcresulttool get test-results summary --schema`.+# An earlier version of the guard accepted only Passed/Failed and rejected+# everything else as the T-1993 "interrupted run" shape, which turns two+# LEGITIMATE outcomes into a false FAIL carrying a misleading diagnosis. These+# pin the distinction: `unknown` is abnormal (above), Skipped/Expected Failure+# are not, and a run that resolved fine but executed nothing fails for the+# accurate reason instead.++expect_pass "skipped-with-executed" \+    "a run whose top-level result is Skipped but which executed tests reports OK"+expect_pass "expected-failure-result" \+    "a run whose top-level result is Expected Failure reports OK"++expect_fail "all-skipped-selection" \+    "a selection that resolved entirely to skipped tests fails as ZERO executed" \+    "executed ZERO tests"+run "$FIXTURES/all-skipped-selection" "all-skipped-selection"+if saw 'all 2 of them were skipped' && ! sawi 'interrupted'; then+    pass "the all-skipped diagnosis names the real cause, not an interrupted run"+else+    fail "all-skipped run got the wrong explanation (a correct FAIL with a misleading reason)"$'\n'"$OUT"+fi++# The counts adding up does not make the bundle coherent. A result of 'Failed'+# with zero tests executed is a contradiction — something failed that never ran —+# so the nothing-executed diagnosis must NOT reassure the reader that nothing is+# corrupt. Same rule the rest of this guard follows: never state a confidence the+# data does not support. Both wordings still exit 1, so only the explanation is at+# stake, which is precisely what this script exists to get right.+expect_fail "failed-result-zero-executed" \+    "a 'Failed' bundle that executed nothing fails as ZERO executed" \+    "executed ZERO tests"+run "$FIXTURES/failed-result-zero-executed" "failed-result-zero-executed"+if saw 'cannot be reconciled with executing nothing' && ! saw 'nothing here is corrupt'; then+    pass "a 'Failed' zero-executed bundle is called untrustworthy, not merely skipped"+else+    fail "'Failed' with zero executed was explained as a clean skipped run"$'\n'"$OUT"+fi++# The nothing-executed guard sits AFTER the arithmetic check on purpose. A bundle+# claiming total=5 with every part at zero also "executed nothing", but the honest+# diagnosis there is that the counts do not add up — not that five tests were+# skipped, which the data does not say. Pins the ordering, since swapping the two+# guards still fails the run and would look fine without this.+expect_fail "corrupt-zero-parts" \+    "a total>0 bundle with all parts zero is diagnosed as bad arithmetic, not as skipped tests" \+    "do not reconstruct the whole"+run "$FIXTURES/corrupt-zero-parts" "corrupt-zero-parts"+if saw 'were skipped'; then+    fail "corrupt bundle was explained as skipped tests — a diagnosis the counts do not support"$'\n'"$OUT"+else+    pass "corrupt bundle is not misreported as an all-skipped run"+fi++echo+echo "--- T-2224: a retry must not launder a first-attempt failure -----------"++expect_fail "retried-then-passed" \+    "a test that failed first-attempt and passed on retry fails the run" \+    "passed only on retry"+run "$FIXTURES/retried-then-passed" "retried-then-passed"+if saw 'controllerObservesItsOwnPageNavigationStream' \+        && saw 'loadAfterAbandonmentRestoresRecovery'; then+    pass "both retried tests are named individually, not just counted"+else+    fail "retried tests are not surfaced by name"$'\n'"$OUT"+fi++expect_fail "retried-then-passed-nested" \+    "the fallback also catches a Repetition node whose result lives on a nested Test Case Run" \+    "passed only on retry"++# The guard keys on a FAILED earlier attempt, never on "this case has several+# attempt nodes". That distinction is what makes XCTest's measure(metrics:)+# — which repeats its block many times inside one test case, as+# prismUITests/testLaunchPerformance does on every `make test-ui` — safe here+# regardless of how xcresulttool represents those iterations.+expect_pass "multi-attempt-all-passed" \+    "a test case with several passing attempt nodes (the measure() shape) is not flagged as a retry"++echo+echo "--- T-2224 review: the retry scan must reach through the config layer --"+# THE shape the original fixtures did not model, and the reason a green suite+# coexisted with a guard that could not fire. Reading a Test Case's DIRECT+# children only finds nothing on any bundle with more than one test plan+# configuration, because every attempt then sits under a Test Plan Configuration+# node instead. `make test` and `make test-ui` pass no -only-test-configuration,+# so they ALWAYS produce that shape: on two of the four pre-push targets the+# guard was structurally inert. Measured on real bundles from this project:+# 4 configurations, 93 Test Case nodes, 372 Test Plan Configuration children,+# and no attempt node ever a direct child of a Test Case.++expect_fail "multi-config-retried" \+    "a retry hidden under a Test Plan Configuration node is still caught (the make test / make test-ui shape)" \+    "passed only on retry"+run "$FIXTURES/multi-config-retried" "multi-config-retried"+if saw 'controllerObservesItsOwnPageNavigationStream'; then+    pass "the retried test under a configuration node is named, not just counted"+else+    fail "the multi-config retried test was not surfaced by name"$'\n'"$OUT"+fi++expect_pass "multi-config-clean" \+    "a clean multi-configuration run (Test Plan root, four config nodes per case) still reports OK"++echo+echo "--- T-2224 review: the retry scan must fail closed --------------------"+# Three fail-OPEN paths the first version of the scan had. Each one made a+# retry-laundered failure report OK, which is the exact thing this script was+# written to stop.++expect_fail "retried-newest-first" \+    "a Failed attempt listed LAST (newest-first ordering) is still caught" \+    "passed only on retry"+expect_fail "retried-single-attempt" \+    "a single recorded attempt of Failed under a non-Failed Test Case is caught" \+    "passed only on retry"+expect_fail "attempt-unknown-result" \+    "an attempt result outside the TestResult enum fails closed rather than counting as no failure" \+    "which is not a TestResult"+expect_fail "attempt-no-result" \+    "an attempt node with no result anywhere in its subtree fails closed" \+    "records no result"++echo+echo "--- T-2224 review: a tree the walk understands nothing in -------------"+# A scan that finds no retries in a tree it cannot read has not passed, it has+# abstained. All three of these used to report OK.++expect_fail "nodetype-drift" \+    "a one-character nodeType drift ('TestCase') fails closed instead of silently finding nothing" \+    "unrecognised nodeType"+expect_fail "empty-tree-object" \+    "an empty JSON object where the tests tree should be fails closed" \+    "no testNodes array"+expect_fail "empty-tree-array" \+    "an empty JSON array where the tests tree should be fails closed" \+    "not a JSON object"+# Distinct from the three above, and the only fixture that pins the count guard+# on its own: every node here is a recognised type with a recognised result, so+# nothing is reported as unreadable — the tree is simply empty of test cases+# while the summary insists three tests ran. A retry scan over it finds nothing+# for the same reason an empty file would.+expect_fail "tree-without-test-cases" \+    "a well-formed tree containing no Test Case node at all fails when the summary counts tests" \+    "contains no Test Case node at all"++echo+echo "--- T-2224 review: the bundle must not contradict itself --------------"+# The T-2224 thesis applied to two figures the script already holds: a Test Case+# recorded Failed while the summary counts zero failures. Both readings describe+# the same run, so a disagreement means the green counts cannot be trusted —+# whichever of the two is wrong.++expect_fail "failed-case-zero-failed-count" \+    "a Test Case recorded Failed while failedTests is zero fails the run" \+    "records 1 test case(s) as Failed"++echo+echo "--- The final result allowlist ---------------------------------------"+# Deleting the last allowlist in the script produces a false green, and nothing+# used to notice: a bundle whose top-level result is 'Failed' with zero recorded+# test failures passes every earlier guard (Failed IS a resolved outcome, the+# counts add up, tests executed, no retries) and is caught only there.++expect_fail "failed-result-no-failed-tests" \+    "a top-level result of 'Failed' with zero failed tests is still a failure" \+    "whole-run failure with nothing to point at"++echo+echo "--- Reporting fidelity ------------------------------------------------"+# The retried names are read back with sed, not awk. Rebuilding \$0 in awk after+# blanking \$1 collapses every run of whitespace, so a test name spelled with+# doubled spaces was reported as a name that does not exist.++run "$FIXTURES/retried-name-spacing" "retried-name-spacing"+if saw 'a  doubled   space  name()'; then+    pass "a retried test name keeps its internal whitespace verbatim"+else+    fail "the retried test name was reformatted before being reported"$'\n'"$OUT"+fi++echo+echo "--- Clean shapes must still pass, unchanged ----------------------------"++expect_pass "clean-pass" "a clean run (including an expected-failure test) reports OK"++echo++if [ "$FAILURES" -gt 0 ]; then+    echo "$FAILURES check(s) failed." >&2+    exit 1+fi+echo "All check-test-results.sh checks passed."

diff --git a/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/summary.json b/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/summary.jsonnew file mode 100644index 00000000..46c4c501--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000002,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Skipped",+  "totalTestCount": 2,+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 2,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/tests.json b/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/tests.jsonnew file mode 100644index 00000000..8fdc4a11--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/all-skipped-selection/tests.json@@ -0,0 +1,39 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Skipped",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Skipped",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebViewPoolTests",+              "result": "Skipped",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testDisabledOne()",+                  "result": "Skipped",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testDisabledTwo()",+                  "result": "Skipped",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/attempt-no-result/summary.json b/Tools/Tests/Fixtures/check-test-results/attempt-no-result/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/attempt-no-result/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/attempt-no-result/tests.json b/Tools/Tests/Fixtures/check-test-results/attempt-no-result/tests.jsonnew file mode 100644index 00000000..c02c4e7e--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/attempt-no-result/tests.json@@ -0,0 +1,64 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "duration": "0.9s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/summary.json b/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/tests.json b/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/tests.jsonnew file mode 100644index 00000000..b6a2f97d--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/attempt-unknown-result/tests.json@@ -0,0 +1,65 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "result": "Borked",+                      "duration": "0.9s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/clean-pass/summary.json b/Tools/Tests/Fixtures/check-test-results/clean-pass/summary.jsonnew file mode 100644index 00000000..5a0be932--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/clean-pass/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 6,+  "passedTests": 5,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 1,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/clean-pass/tests.json b/Tools/Tests/Fixtures/check-test-results/clean-pass/tests.jsonnew file mode 100644index 00000000..27ef0166--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/clean-pass/tests.json@@ -0,0 +1,48 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testOne()",+                  "result": "Passed",+                  "duration": "0.01s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testTwo()",+                  "result": "Passed",+                  "duration": "0.02s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testExpectedFailure()",+                  "result": "Expected Failure",+                  "duration": "0.01s",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/corrupt-total/summary.json b/Tools/Tests/Fixtures/check-test-results/corrupt-total/summary.jsonnew file mode 100644index 00000000..7f1352f9--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/corrupt-total/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": "not-a-number",+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/summary.json b/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/summary.jsonnew file mode 100644index 00000000..4fe02a80--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000005,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 5,+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/tests.json b/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/tests.jsonnew file mode 100644index 00000000..48de21a7--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/corrupt-zero-parts/tests.json@@ -0,0 +1,5 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/empty-tree-array/summary.json b/Tools/Tests/Fixtures/check-test-results/empty-tree-array/summary.jsonnew file mode 100644index 00000000..c92fc422--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/empty-tree-array/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 3,+  "passedTests": 3,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/empty-tree-array/tests.json b/Tools/Tests/Fixtures/check-test-results/empty-tree-array/tests.jsonnew file mode 100644index 00000000..fe51488c--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/empty-tree-array/tests.json@@ -0,0 +1 @@+[]diff --git a/Tools/Tests/Fixtures/check-test-results/empty-tree-object/summary.json b/Tools/Tests/Fixtures/check-test-results/empty-tree-object/summary.jsonnew file mode 100644index 00000000..c92fc422--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/empty-tree-object/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 3,+  "passedTests": 3,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/empty-tree-object/tests.json b/Tools/Tests/Fixtures/check-test-results/empty-tree-object/tests.jsonnew file mode 100644index 00000000..0967ef42--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/empty-tree-object/tests.json@@ -0,0 +1 @@+{}diff --git a/Tools/Tests/Fixtures/check-test-results/expected-failure-result/summary.json b/Tools/Tests/Fixtures/check-test-results/expected-failure-result/summary.jsonnew file mode 100644index 00000000..de99fa0a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/expected-failure-result/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000004,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Expected Failure",+  "totalTestCount": 2,+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 2,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/expected-failure-result/tests.json b/Tools/Tests/Fixtures/check-test-results/expected-failure-result/tests.jsonnew file mode 100644index 00000000..4e7d220c--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/expected-failure-result/tests.json@@ -0,0 +1,41 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Expected Failure",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Expected Failure",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "KnownIssueTests",+              "result": "Expected Failure",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testKnownIssueOne()",+                  "result": "Expected Failure",+                  "duration": "0.01s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testKnownIssueTwo()",+                  "result": "Expected Failure",+                  "duration": "0.01s",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/summary.json b/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/summary.jsonnew file mode 100644index 00000000..284ad0cc--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 2,+  "passedTests": 2,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/tests.json b/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/tests.jsonnew file mode 100644index 00000000..2390042f--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-case-zero-failed-count/tests.json@@ -0,0 +1,58 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "name": "someTest()",+                  "nodeIdentifier": "SampleTests/someTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/someTest()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                },+                {+                  "name": "otherTest()",+                  "nodeIdentifier": "SampleTests/otherTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/otherTest()",+                  "nodeType": "Test Case",+                  "result": "Failed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/summary.json b/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/summary.jsonnew file mode 100644index 00000000..e48aa0f0--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Failed",+  "totalTestCount": 2,+  "passedTests": 2,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/tests.json b/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/tests.jsonnew file mode 100644index 00000000..312d4732--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-result-no-failed-tests/tests.json@@ -0,0 +1,58 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "name": "someTest()",+                  "nodeIdentifier": "SampleTests/someTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/someTest()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                },+                {+                  "name": "otherTest()",+                  "nodeIdentifier": "SampleTests/otherTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/otherTest()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/summary.json b/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/summary.jsonnew file mode 100644index 00000000..b7e77acd--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Failed",+  "totalTestCount": 3,+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 3,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/tests.json b/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/tests.jsonnew file mode 100644index 00000000..0eb0f89b--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/failed-result-zero-executed/tests.json@@ -0,0 +1,45 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Failed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Failed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebViewPoolTests",+              "result": "Skipped",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testOne()",+                  "result": "Skipped",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testTwo()",+                  "result": "Skipped",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testThree()",+                  "result": "Skipped",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/summary.json b/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/summary.jsonnew file mode 100644index 00000000..7497dd63--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Failed",+  "totalTestCount": 3,+  "passedTests": 0,+  "failedTests": 3,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/tests.json b/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/tests.jsonnew file mode 100644index 00000000..c3181501--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/genuine-failure-cascade/tests.json@@ -0,0 +1,48 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Failed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Failed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Failed",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testCrasher()",+                  "result": "Failed",+                  "duration": "0.068s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testQueuedAfterCrash1()",+                  "result": "Failed",+                  "duration": null,+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testQueuedAfterCrash2()",+                  "result": "Failed",+                  "duration": null,+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/leading-zero-count/summary.json b/Tools/Tests/Fixtures/check-test-results/leading-zero-count/summary.jsonnew file mode 100644index 00000000..3e5c3e2b--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/leading-zero-count/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 8,+  "passedTests": "08",+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/leading-zero-count/tests.json b/Tools/Tests/Fixtures/check-test-results/leading-zero-count/tests.jsonnew file mode 100644index 00000000..bc9a196b--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/leading-zero-count/tests.json@@ -0,0 +1,49 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "name": "someTest()",+                  "nodeIdentifier": "SampleTests/someTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/someTest()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/summary.json b/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/summary.jsonnew file mode 100644index 00000000..70bcbe93--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000005,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 5,+  "passedTests": 3,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/tests.json b/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/tests.jsonnew file mode 100644index 00000000..48de21a7--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/mismatched-counts-silent/tests.json@@ -0,0 +1,5 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/mismatched-counts/summary.json b/Tools/Tests/Fixtures/check-test-results/mismatched-counts/summary.jsonnew file mode 100644index 00000000..895b6586--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/mismatched-counts/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 10,+  "passedTests": 7,+  "failedTests": 1,+  "skippedTests": 1,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/mismatched-counts/tests.json b/Tools/Tests/Fixtures/check-test-results/mismatched-counts/tests.jsonnew file mode 100644index 00000000..e3e2fae4--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/mismatched-counts/tests.json@@ -0,0 +1,19 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": []+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/summary.json b/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/summary.jsonnew file mode 100644index 00000000..7ba44d18--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000020,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/tests.json b/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/tests.jsonnew file mode 100644index 00000000..dfcab78a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-attempt-all-passed/tests.json@@ -0,0 +1,56 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "UI test bundle",+          "name": "prismUITests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "prismUITests",+              "result": "Passed",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testLaunchPerformance()",+                  "result": "Passed",+                  "duration": "12.4s",+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Iteration 1",+                      "result": "Passed",+                      "duration": "2.4s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Iteration 2",+                      "result": "Passed",+                      "duration": "2.4s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Iteration 3",+                      "result": "Passed",+                      "duration": "2.4s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-config-clean/summary.json b/Tools/Tests/Fixtures/check-test-results/multi-config-clean/summary.jsonnew file mode 100644index 00000000..284ad0cc--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-config-clean/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 2,+  "passedTests": 2,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-config-clean/tests.json b/Tools/Tests/Fixtures/check-test-results/multi-config-clean/tests.jsonnew file mode 100644index 00000000..b2caaabe--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-config-clean/tests.json@@ -0,0 +1,138 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    },+    {+      "configurationId": "2",+      "configurationName": "en-AU"+    },+    {+      "configurationId": "3",+      "configurationName": "en-US"+    },+    {+      "configurationId": "4",+      "configurationName": "en-GB"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebDocumentControllerTests",+              "result": "Passed",+              "children": [+                {+                  "name": "selectionCandidateAfterReadyRouted()",+                  "nodeIdentifier": "WebDocumentControllerTests/selectionCandidateAfterReadyRouted()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebDocumentControllerTests/selectionCandidateAfterReadyRouted()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en (base)",+                      "nodeIdentifier": "1",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-AU",+                      "nodeIdentifier": "2",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-US",+                      "nodeIdentifier": "3",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-GB",+                      "nodeIdentifier": "4",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    }+                  ]+                },+                {+                  "name": "loadAfterAbandonmentRestoresRecovery()",+                  "nodeIdentifier": "WebDocumentControllerTests/loadAfterAbandonmentRestoresRecovery()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebDocumentControllerTests/loadAfterAbandonmentRestoresRecovery()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en (base)",+                      "nodeIdentifier": "1",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-AU",+                      "nodeIdentifier": "2",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-US",+                      "nodeIdentifier": "3",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-GB",+                      "nodeIdentifier": "4",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-config-retried/summary.json b/Tools/Tests/Fixtures/check-test-results/multi-config-retried/summary.jsonnew file mode 100644index 00000000..284ad0cc--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-config-retried/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 2,+  "passedTests": 2,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/multi-config-retried/tests.json b/Tools/Tests/Fixtures/check-test-results/multi-config-retried/tests.jsonnew file mode 100644index 00000000..267539b1--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/multi-config-retried/tests.json@@ -0,0 +1,156 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    },+    {+      "configurationId": "2",+      "configurationName": "en-AU"+    },+    {+      "configurationId": "3",+      "configurationName": "en-US"+    },+    {+      "configurationId": "4",+      "configurationName": "en-GB"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en (base)",+                      "nodeIdentifier": "1",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-AU",+                      "nodeIdentifier": "2",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed",+                      "children": [+                        {+                          "nodeType": "Repetition",+                          "name": "Repetition 1",+                          "result": "Failed",+                          "duration": "0.9s",+                          "durationInSeconds": 0.9,+                          "children": []+                        },+                        {+                          "nodeType": "Repetition",+                          "name": "Repetition 2",+                          "result": "Passed",+                          "duration": "0.3s",+                          "durationInSeconds": 0.3,+                          "children": []+                        }+                      ]+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-US",+                      "nodeIdentifier": "3",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-GB",+                      "nodeIdentifier": "4",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    }+                  ]+                },+                {+                  "name": "cleanPassthroughCase()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/cleanPassthroughCase()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/cleanPassthroughCase()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en (base)",+                      "nodeIdentifier": "1",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-AU",+                      "nodeIdentifier": "2",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-US",+                      "nodeIdentifier": "3",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    },+                    {+                      "duration": "0.21s",+                      "durationInSeconds": 0.208,+                      "name": "en-GB",+                      "nodeIdentifier": "4",+                      "nodeType": "Test Plan Configuration",+                      "result": "Passed"+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/nodetype-drift/summary.json b/Tools/Tests/Fixtures/check-test-results/nodetype-drift/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/nodetype-drift/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/nodetype-drift/tests.json b/Tools/Tests/Fixtures/check-test-results/nodetype-drift/tests.jsonnew file mode 100644index 00000000..0982bb68--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/nodetype-drift/tests.json@@ -0,0 +1,49 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "name": "someTest()",+                  "nodeIdentifier": "SampleTests/someTest()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/someTest()",+                  "nodeType": "TestCase",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/summary.json b/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/tests.json b/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/tests.jsonnew file mode 100644index 00000000..3c4d32a0--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-name-spacing/tests.json@@ -0,0 +1,65 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": [+                {+                  "name": "a  doubled   space  name()",+                  "nodeIdentifier": "SampleTests/a  doubled   space  name()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/SampleTests/a  doubled   space  name()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "result": "Failed",+                      "duration": "0.9s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-newest-first/summary.json b/Tools/Tests/Fixtures/check-test-results/retried-newest-first/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-newest-first/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-newest-first/tests.json b/Tools/Tests/Fixtures/check-test-results/retried-newest-first/tests.jsonnew file mode 100644index 00000000..80ef9c85--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-newest-first/tests.json@@ -0,0 +1,65 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "result": "Failed",+                      "duration": "0.9s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/summary.json b/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/summary.jsonnew file mode 100644index 00000000..4641209a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/tests.json b/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/tests.jsonnew file mode 100644index 00000000..995e2f5a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-single-attempt/tests.json@@ -0,0 +1,58 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [+    {+      "deviceId": "D1",+      "deviceName": "iPhone 17 Pro",+      "architecture": "arm64",+      "modelName": "iPhone 17 Pro",+      "osVersion": "26.0"+    }+  ],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifier": "WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeIdentifierURL": "test://com.apple.xcode/prism/prismTests/WebContentTerminationWiringTests/controllerObservesItsOwnPageNavigationStream()",+                  "nodeType": "Test Case",+                  "result": "Passed",+                  "duration": "0.1s",+                  "durationInSeconds": 0.104,+                  "children": [+                    {+                      "nodeType": "Test Case Run",+                      "name": "Run 1",+                      "result": "Failed",+                      "duration": "0.9s",+                      "children": []+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/summary.json b/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/summary.jsonnew file mode 100644index 00000000..43deb0f0--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 1,+  "passedTests": 1,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/tests.json b/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/tests.jsonnew file mode 100644index 00000000..77728e58--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-then-passed-nested/tests.json@@ -0,0 +1,59 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SomeFlakySuite",+              "result": "Passed",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "nestedRepetitionResult()",+                  "result": "Passed",+                  "duration": "1.0s",+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "children": [+                        {+                          "nodeType": "Test Case Run",+                          "name": "Run 1",+                          "result": "Failed",+                          "children": []+                        }+                      ]+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "children": [+                        {+                          "nodeType": "Test Case Run",+                          "name": "Run 1",+                          "result": "Passed",+                          "children": []+                        }+                      ]+                    }+                  ]+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-then-passed/summary.json b/Tools/Tests/Fixtures/check-test-results/retried-then-passed/summary.jsonnew file mode 100644index 00000000..8b366127--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-then-passed/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000060,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 60,+  "passedTests": 60,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/retried-then-passed/tests.json b/Tools/Tests/Fixtures/check-test-results/retried-then-passed/tests.jsonnew file mode 100644index 00000000..d10f9ab3--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/retried-then-passed/tests.json@@ -0,0 +1,78 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebContentTerminationWiringTests",+              "result": "Passed",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "controllerObservesItsOwnPageNavigationStream()",+                  "result": "Passed",+                  "duration": "1.2s",+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "result": "Failed",+                      "duration": "0.9s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    }+                  ]+                },+                {+                  "nodeType": "Test Case",+                  "name": "loadAfterAbandonmentRestoresRecovery()",+                  "result": "Passed",+                  "duration": "0.8s",+                  "children": [+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 1",+                      "result": "Failed",+                      "duration": "0.5s",+                      "children": []+                    },+                    {+                      "nodeType": "Repetition",+                      "name": "Repetition 2",+                      "result": "Passed",+                      "duration": "0.3s",+                      "children": []+                    }+                  ]+                },+                {+                  "nodeType": "Test Case",+                  "name": "cleanPassthroughCase()",+                  "result": "Passed",+                  "duration": "0.05s",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/summary.json b/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/summary.jsonnew file mode 100644index 00000000..477725fe--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000010,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Skipped",+  "totalTestCount": 3,+  "passedTests": 2,+  "failedTests": 0,+  "skippedTests": 1,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/tests.json b/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/tests.jsonnew file mode 100644index 00000000..0dc7e853--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/skipped-with-executed/tests.json@@ -0,0 +1,47 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Skipped",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Skipped",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "WebViewPoolTests",+              "result": "Skipped",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testRuns()",+                  "result": "Passed",+                  "duration": "0.01s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testAlsoRuns()",+                  "result": "Passed",+                  "duration": "0.02s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testDisabled()",+                  "result": "Skipped",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/summary.json b/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/summary.jsonnew file mode 100644index 00000000..c92fc422--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism \u00b7 Debug \u00b7 iOS Simulator",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 3,+  "passedTests": 3,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/tests.json b/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/tests.jsonnew file mode 100644index 00000000..9cad1d27--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/tree-without-test-cases/tests.json@@ -0,0 +1,31 @@+{+  "testPlanConfigurations": [+    {+      "configurationId": "1",+      "configurationName": "en (base)"+    }+  ],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Passed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Passed",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "Passed",+              "children": []+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/summary.json b/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/summary.jsonnew file mode 100644index 00000000..4b4bf6e8--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000003,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "unknown",+  "totalTestCount": 3,+  "passedTests": 3,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/tests.json b/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/tests.jsonnew file mode 100644index 00000000..4898114a--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/unknown-result-partial/tests.json@@ -0,0 +1,48 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "unknown",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "unknown",+          "children": [+            {+              "nodeType": "Test Suite",+              "name": "SampleTests",+              "result": "unknown",+              "children": [+                {+                  "nodeType": "Test Case",+                  "name": "testOne()",+                  "result": "Passed",+                  "duration": "0.01s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testTwo()",+                  "result": "Passed",+                  "duration": "0.02s",+                  "children": []+                },+                {+                  "nodeType": "Test Case",+                  "name": "testThree()",+                  "result": "Passed",+                  "duration": "0.01s",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/wedged-host/summary.json b/Tools/Tests/Fixtures/check-test-results/wedged-host/summary.jsonnew file mode 100644index 00000000..aa50915d--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/wedged-host/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "unknown",+  "totalTestCount": 1,+  "passedTests": 0,+  "failedTests": 1,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/wedged-host/tests.json b/Tools/Tests/Fixtures/check-test-results/wedged-host/tests.jsonnew file mode 100644index 00000000..c3362392--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/wedged-host/tests.json@@ -0,0 +1,33 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": [+    {+      "nodeType": "Test Plan",+      "name": "prism",+      "result": "Failed",+      "children": [+        {+          "nodeType": "Unit test bundle",+          "name": "prismTests",+          "result": "Failed",+          "children": [+            {+              "nodeType": "Test Case",+              "name": "prismTests",+              "result": "Failed",+              "duration": null,+              "children": [+                {+                  "nodeType": "Failure Message",+                  "name": "test runner hung before establishing connection",+                  "children": []+                }+              ]+            }+          ]+        }+      ]+    }+  ]+}diff --git a/Tools/Tests/Fixtures/check-test-results/zero-tests/summary.json b/Tools/Tests/Fixtures/check-test-results/zero-tests/summary.jsonnew file mode 100644index 00000000..3084585e--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/zero-tests/summary.json@@ -0,0 +1,16 @@+{+  "title": "Test - prism",+  "startTime": 1000000000,+  "finishTime": 1000000001,+  "environmentDescription": "prism · Debug · macOS",+  "topInsights": [],+  "result": "Passed",+  "totalTestCount": 0,+  "passedTests": 0,+  "failedTests": 0,+  "skippedTests": 0,+  "expectedFailures": 0,+  "statistics": [],+  "devicesAndConfigurations": [],+  "testFailures": []+}diff --git a/Tools/Tests/Fixtures/check-test-results/zero-tests/tests.json b/Tools/Tests/Fixtures/check-test-results/zero-tests/tests.jsonnew file mode 100644index 00000000..48de21a7--- /dev/null+++ b/Tools/Tests/Fixtures/check-test-results/zero-tests/tests.json@@ -0,0 +1,5 @@+{+  "testPlanConfigurations": [],+  "devices": [],+  "testNodes": []+}

diff --git a/Makefile b/Makefileindex 6fa22ce8..487055b5 100644--- a/Makefile+++ b/Makefile@@ -440,12 +440,16 @@ upload-macos: archive-macos upload-all: upload upload-macos 	@echo "Uploaded both platforms to App Store Connect" -# Verification of the Makefile's own safety nets. Both defects it checks for were-# invisible in a passing run — their only symptom was a green light on nothing —-# so they are asserted rather than trusted. See T-1983.+# Verification of the Makefile's own safety nets, and of the result-bundle guard+# they all delegate to. Every defect these two scripts check for was invisible in+# a passing run — the only symptom was a green light on a run that either+# executed nothing (T-1983), never reached a real verdict (T-1993), or silently+# had a first-attempt failure erased by a retry (T-2224) — so they are asserted+# rather than trusted. .PHONY: verify-make-guards verify-make-guards: 	Tools/Tests/test-make-guards.sh+	Tools/Tests/test-check-test-results.sh  # Cleaning .PHONY: clean

diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8c650b61..fc997335 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed  - CI now runs the test suite instead of only appearing to (T-1983). The per-locale sweep is the only job that can execute tests, and it reported success while executing none: its build failed for want of a signing certificate on the runner, that failure was swallowed, and nothing checked that any test had run. The sweep now signs ad-hoc so the build succeeds without a certificate, every test target hands its result bundle to the zero-test guard — per locale configuration, not once at the end — and any recipe whose failure must be believed carries `$(STRICT)`, because `.SHELLFLAGS` is silently ignored by the GNU Make 3.81 that macOS ships. `make verify-make-guards` asserts all of this on every push.+- `Tools/check-test-results.sh` no longer reports OK on two more shapes of run that were not a clean pass (T-2224, T-1993). A test that fails on its first attempt and passes on a retry only ever left its final result in the bundle's counts, so the failure was invisible — measured directly on PR #377, where a macOS run exited 65 with `** TEST FAILED **` while the bundle reported 60/60 passed because two `WebContentTerminationWiringTests` cases needed a retry. The checker now walks the bundle's per-test attempt history and fails the run when any test needed one, naming which. Separately, an interrupted, cancelled, or infrastructure-failed run can leave a readable partial bundle with passing-looking counts and zero recorded failures while its own top-level `result` field sits at "unknown" — never resolving to a verdict at all. The checker now refuses to report success on such a run, and the count-arithmetic sanity check that used to only warn on a mismatch now fails closed, the same as everything else this script guards. That guard distinguishes "did not resolve" from "did not pass": `result` is typed as the same five-value enum as an individual test's result (`Passed`, `Failed`, `Skipped`, `Expected Failure`, `unknown`), so a run resolving to Skipped or Expected Failure is a legitimate outcome and is allowed through — demanding Passed or Failed would have turned an ordinary `-only-testing:` selection that lands entirely on disabled tests into a hard failure blamed on an interrupted test host that never existed. A run that executed nothing because every selected test was skipped still fails, since it verifies exactly as much as running no tests at all, but it now fails as a zero-executed run and says so rather than borrowing the infrastructure-failure diagnosis. The retry scan searches a test case's whole subtree rather than its direct children, which is what makes it work at all on `make test` and `make test-ui`: neither passes `-only-test-configuration`, so every attempt on those runs sits under a `Test Plan Configuration` node instead of directly under the test case, and a direct-children scan finds nothing there — silently, on half the pre-push matrix. It also no longer depends on attempt ordering or on there being more than one attempt: any recorded failed attempt under a test that did not end up failed is the laundering shape, however the attempts are listed. Three further shapes now fail instead of reporting OK, all of them cases where the checker previously drew a clean conclusion from data it had not actually understood: a per-test tree containing a `nodeType` or result value outside the published enums (a one-character drift is enough to make the scan match nothing), a tree with no test-case node in it at all while the summary counts tests, and a bundle that contradicts itself by recording a test case as failed while its `failedTests` count is zero. `make verify-make-guards` now runs a dedicated regression suite (`Tools/Tests/test-check-test-results.sh`) covering these shapes alongside the existing zero-test and cascade-failure cases. One limitation is worth stating outright rather than leaving implied: the per-attempt node shape the retry scan reads (`Repetition` / `Test Case Run`) is derived from `xcresulttool`'s published schema and is **not** confirmed against a captured retried bundle — repeated attempts to produce one on a loaded machine died with `** BUILD INTERRUPTED **`, and a sweep of the readable historical bundles on this machine found none containing such a node. The scan accepts either shape at any depth for that reason, and the fixtures pin its parsing, its descent through the configuration layer, and its control flow — not the attempt-node shape itself. The nesting the fixtures *do* model faithfully (a `Test Plan` root, and test-case nodes whose children are `Test Plan Configuration` nodes) was measured against real bundles from this project. - Copy notes is now the primary notes action (T-1577). On iPhone the document screen's toolbar shows Copy notes instead of Share with Notes, which moved into the notes pane alongside copy; on iPad and Mac the top toolbar shows copy leading the export button. Every copy button appears exactly when the copy output would contain at least one note under the current export settings, each action carries an accessibility label and help text, and an export blocked by the paywall from inside the pane now retries fully — including the author-name prompt and its confirmation toast — after a purchase completes. - The test suite covering notes-action placement was retargeted to the new contracts (T-1577): the T-138 share-button parity tests became the Share-with-Notes placement contract, and visibility/payload tests now assert the shared copy-availability predicate and the single export payload call site. Two device-only checks are recorded in the spec for manual verification. - A `prism://open?url=…` link naming anything other than a web address is now rejected as soon as it arrives rather than a moment later during the download (T-2140). The address it names must start with `http` or `https`, matching what Prism already requires of an address you type in yourself. The message you see is unchanged — "Only http and https URLs are supported." — it just appears without a download being attempted first. A `prism://open` link that names no address at all now says so too, with "The URL is not valid."; previously nothing happened when you followed one, which was indistinguishable from the link never having arrived.

diff --git a/docs/agent-notes/development-tooling.md b/docs/agent-notes/development-tooling.mdindex 7c00a08a..dc4a8d2f 100644--- a/docs/agent-notes/development-tooling.md+++ b/docs/agent-notes/development-tooling.md@@ -9,7 +9,8 @@  - **`SVGWebViewTests` no longer needs skipping** (fixed on the T-1541 bugfix branch; `specs/bugfixes/svgwebviewtests-offmain-crash/report.md`). Root cause of the full-run host crash: the suite was the only one constructing `WKWebView` in *synchronous* test bodies. A sync `@MainActor` function has no hop-on-entry — its isolation depends on the caller hopping, and swift-testing's `nonisolated(nonsending)` thunks (Swift 5 mode + Approachable Concurrency, no default isolation in the test target) run on the runner's cooperative pool under full-suite load. WebKit's init `RELEASE_ASSERT(main thread)` then killed the host. The `@MainActor` added in 81c1636 could only move the trap. The rule for this target: **touch WebKit only from `async` tests** (actor-isolated async functions hop on entry as ABI); the suite additionally wraps construction in `MainActor.run`, which is runtime-enforced. - **The app and test targets have different actor-isolation defaults.** `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor` is set on the app target only; `prismTests` and `prismUITests` have no default. Production code is therefore implicitly main-actor while the same type called from a test is not, which silently turns "safe because everything is on the main actor" into a real race in tests (see `RecentFileEntry.relativeDateFormatter`). Do not assume a production type's isolation holds inside a test.-- **`XCTExpectFailure` breaks `check-test-results.sh`'s arithmetic check.** An expected failure is counted in neither `passedTests` nor `failedTests`, so the script prints `WARN: passed+failed+skipped != total`. That warning is benign when the difference equals the number of `XCTExpectFailure`s (currently 2, T-1985 and T-1986); it does not gate the build.+- **`check-test-results.sh`'s arithmetic check now folds in `expectedFailures`, and it gates the build.** An expected failure is counted in neither `passedTests` nor `failedTests`, so the original `passed+failed+skipped != total` check tripped on every run containing an `XCTExpectFailure` (currently 2, T-1985 and T-1986) and printed a `WARN` everyone learned to ignore. Fixed in both directions (T-1993): `expectedFailures` is part of the sum, so those runs now reconcile exactly, and a mismatch is a hard FAIL rather than a warning — counts that do not reconstruct the whole mean nothing below them can be trusted. Do not expect a benign warning here any more; if this fires, the bundle is wrong.+- **`check-test-results.sh` fails closed on anything it cannot read, including the per-test tree.** Beyond the counts it reads `xcrun xcresulttool get test-results tests` and refuses to report OK when the tree contains an unrecognised `nodeType` or result value, contains no `Test Case` node while the summary counts tests, records a `Test Case` as `Failed` while `failedTests` is zero, or shows any test whose final result was reached only after a failed attempt (T-2224). Two things worth knowing before debugging a surprise FAIL: a future Xcode that adds a `TestNodeType` will trip the unrecognised-node check until the enum copy inside the script is updated (deliberate — a guard that goes quiet exactly when the format moves is worse), and the retry scan walks a Test Case's WHOLE subtree rather than its direct children, because on any run without `-only-test-configuration` — i.e. `make test` and `make test-ui` — a Test Case's direct children are `Test Plan Configuration` nodes and nothing else structural (measured: 5,627 of 5,627 cases across every multi-configuration bundle on this machine, four configurations each). Where a per-attempt node would land inside that subtree is inference, not measurement — no bundle on this machine has ever contained a `Repetition` or `Test Case Run` node, which is exactly why the scan accepts either type at any depth. Regression suite: `Tools/Tests/test-check-test-results.sh`, run by `make verify-make-guards`. - **One `MockURLProtocol` used to be shared by five suites** across `URLDocumentLoaderTests`, `ImageLoaderTests`, and `SVGSourceLoaderTests`. `.serialized` only orders tests *within* a suite, so the suites overwrote each other's static handler and served each other's payloads. Handlers are now registered per scope (`MockURLScope`), carried as a request header set from the session configuration. If you add a networked suite, give it its own scope rather than a global handler.  ## "The test runner hung before establishing connection" on macOS (T-2146)

diff --git a/Tools/Tests/test-check-test-results.sh b/Tools/Tests/test-check-test-results.shindex 075f07c7..1fd0a772 100755--- a/Tools/Tests/test-check-test-results.sh+++ b/Tools/Tests/test-check-test-results.sh@@ -98,10 +98,20 @@ run() {     STATUS=$? } +# Every assertion below searches "$OUT" through these two helpers, which use a+# here-string rather than `printf '%s' "$OUT" | grep …`. Under `set -o pipefail`+# that pipeline returns 141 (SIGPIPE) rather than 0 when grep matches early and+# exits while printf is still writing, so an assertion that SHOULD pass reports a+# failure instead. Measured here at roughly 1 run in 200 under parallel load — and+# a merge-gate self-test that cries wolf is a self-test people learn to re-run+# until it is green, which is the same disease as a gate that stays silent.+saw()  { grep -qF -- "$1" <<< "$OUT"; }+sawi() { grep -qiF -- "$1" <<< "$OUT"; }+ expect_pass() {     local fixture="$1" desc="$2"     run "$FIXTURES/$fixture" "$fixture"-    if [ "$STATUS" -eq 0 ] && printf '%s' "$OUT" | grep -q ' OK$'; then+    if [ "$STATUS" -eq 0 ] && grep -q ' OK$' <<< "$OUT"; then         pass "$desc"     else         fail "$desc (exit=$STATUS)"$'\n'"$OUT"@@ -115,7 +125,7 @@ expect_fail() {         fail "$desc: reported success (exit 0) — this is the false-green shape"$'\n'"$OUT"         return     fi-    if [ -n "$needle" ] && ! printf '%s' "$OUT" | grep -qF "$needle"; then+    if [ -n "$needle" ] && ! saw "$needle"; then         fail "$desc: failed (good) but did not explain why — expected to see: $needle"$'\n'"$OUT"         return     fi@@ -141,12 +151,20 @@ expect_fail "leading-zero-count" \ echo echo "--- T-1993: abnormal / partial runs must not report OK ----------------" +# Both needles must be text ONLY the abnormal-result guard can print. The obvious+# phrasing — "top-level result is 'unknown', which is not a" — is a prefix of the+# FINAL result allowlist's message too ("…which is not a successful outcome"), so a+# fixture needling it stays green with the abnormal-result guard deleted outright:+# the run simply falls through and is caught 400 lines later by a different guard+# printing a superset of the needle. That is the same unpinned-guard trap the+# zero-test check above documents, and it was live here. "never resolved" is unique+# to this branch. expect_fail "unknown-result-partial" \     "a partial bundle with passing-looking counts but result=unknown fails" \-    "top-level result is 'unknown', which is not a"+    "Refusing to report success on a run that never resolved." expect_fail "wedged-host" \     "a wedged host (total=1 failed=1 result=unknown) fails with the abnormal-result diagnosis" \-    "top-level result is 'unknown', which is not a"+    "Refusing to report success on a run that never resolved." expect_fail "mismatched-counts" \     "passed+failed+skipped+expectedFailures != total fails closed (no longer just a warning)" \     "do not reconstruct the whole"@@ -175,8 +193,7 @@ expect_fail "all-skipped-selection" \     "a selection that resolved entirely to skipped tests fails as ZERO executed" \     "executed ZERO tests" run "$FIXTURES/all-skipped-selection" "all-skipped-selection"-if printf '%s' "$OUT" | grep -qF 'all 2 of them were skipped' \-        && ! printf '%s' "$OUT" | grep -qiF 'interrupted'; then+if saw 'all 2 of them were skipped' && ! sawi 'interrupted'; then     pass "the all-skipped diagnosis names the real cause, not an interrupted run" else     fail "all-skipped run got the wrong explanation (a correct FAIL with a misleading reason)"$'\n'"$OUT"@@ -192,8 +209,7 @@ expect_fail "failed-result-zero-executed" \     "a 'Failed' bundle that executed nothing fails as ZERO executed" \     "executed ZERO tests" run "$FIXTURES/failed-result-zero-executed" "failed-result-zero-executed"-if printf '%s' "$OUT" | grep -qF 'cannot be reconciled with executing nothing' \-        && ! printf '%s' "$OUT" | grep -qF 'nothing here is corrupt'; then+if saw 'cannot be reconciled with executing nothing' && ! saw 'nothing here is corrupt'; then     pass "a 'Failed' zero-executed bundle is called untrustworthy, not merely skipped" else     fail "'Failed' with zero executed was explained as a clean skipped run"$'\n'"$OUT"@@ -208,7 +224,7 @@ expect_fail "corrupt-zero-parts" \     "a total>0 bundle with all parts zero is diagnosed as bad arithmetic, not as skipped tests" \     "do not reconstruct the whole" run "$FIXTURES/corrupt-zero-parts" "corrupt-zero-parts"-if printf '%s' "$OUT" | grep -qF 'were skipped'; then+if saw 'were skipped'; then     fail "corrupt bundle was explained as skipped tests — a diagnosis the counts do not support"$'\n'"$OUT" else     pass "corrupt bundle is not misreported as an all-skipped run"@@ -221,8 +237,8 @@ expect_fail "retried-then-passed" \     "a test that failed first-attempt and passed on retry fails the run" \     "passed only on retry" run "$FIXTURES/retried-then-passed" "retried-then-passed"-if printf '%s' "$OUT" | grep -q 'controllerObservesItsOwnPageNavigationStream' \-        && printf '%s' "$OUT" | grep -q 'loadAfterAbandonmentRestoresRecovery'; then+if saw 'controllerObservesItsOwnPageNavigationStream' \+        && saw 'loadAfterAbandonmentRestoresRecovery'; then     pass "both retried tests are named individually, not just counted" else     fail "retried tests are not surfaced by name"$'\n'"$OUT"@@ -256,7 +272,7 @@ expect_fail "multi-config-retried" \     "a retry hidden under a Test Plan Configuration node is still caught (the make test / make test-ui shape)" \     "passed only on retry" run "$FIXTURES/multi-config-retried" "multi-config-retried"-if printf '%s' "$OUT" | grep -q 'controllerObservesItsOwnPageNavigationStream'; then+if saw 'controllerObservesItsOwnPageNavigationStream'; then     pass "the retried test under a configuration node is named, not just counted" else     fail "the multi-config retried test was not surfaced by name"$'\n'"$OUT"@@ -336,7 +352,7 @@ echo "--- Reporting fidelity ------------------------------------------------" # doubled spaces was reported as a name that does not exist.  run "$FIXTURES/retried-name-spacing" "retried-name-spacing"-if printf '%s' "$OUT" | grep -qF 'a  doubled   space  name()'; then+if saw 'a  doubled   space  name()'; then     pass "a retried test name keeps its internal whitespace verbatim" else     fail "the retried test name was reformatted before being reported"$'\n'"$OUT"diff --git a/Tools/check-test-results.sh b/Tools/check-test-results.shindex 7cae12a3..811be9f3 100755--- a/Tools/check-test-results.sh+++ b/Tools/check-test-results.sh@@ -136,9 +136,11 @@ read -r TOTAL PASSED FAILED SKIPPED EXPECTED_FAILURES RESULT <<< "$COUNTS" # #   * not digits at all — `[ -eq ]` ERRORS on it under `set -uo pipefail` (no `-e`), #     which SKIPS the guard and falls through to OK;-#   * too large for a machine integer — passes a digits-only glob and then errors in-#     exactly the same way, i.e. the same fail-open shape reached through magnitude.-#     Ten digits allows up to 9999999999;+#   * implausibly large — a plausibility bound, not an overflow guard. Bash uses+#     64-bit arithmetic here, so `[ -eq ]` only errors ("integer expression+#     expected") at 20 digits; between 11 and 19 it compares fine. Ten digits+#     (up to 9999999999) is far past any real test count, and a value beyond it+#     means the field was not a count in the first place; #   * a leading zero — bash arithmetic reads `08` as octal and fails with "value too #     great for base", leaving `$(( ))` targets unassigned; the next `set -u` #     expansion then kills the script with a bare "unbound variable" and no hint of@@ -512,7 +514,12 @@ for problem in problems: for problem in seen[:5]:     print("UNREADABLE %s" % clean(problem)) if len(seen) > 5:-    print("UNREADABLE ... and %d more of the same kind" % (len(seen) - 5))+    # NOT "of the same kind": seen is deduplicated across ALL problem kinds and+    # truncated by position, so the hidden remainder can be a different kind+    # entirely — a nodeType drift pushed off the list by five unrelated per-case+    # problems. Say what is true (how many distinct problems are unshown) rather+    # than characterising problems that are not being printed.+    print("UNREADABLE ... and %d more distinct problem(s), not shown" % (len(seen) - 5)) for name in retried:     print("RETRIED_NAME %s" % clean(name)) ') || {@@ -560,7 +567,14 @@ validate_count RETRIED "$RETRIED" "$UNPARSED_TREE_HINT" #   2. The counts say tests ran, but the tree contains no Test Case node at all — #      an empty object, an empty array, or a tree whose case nodes are spelled #      something this script does not recognise.-if printf '%s\n' "$ANALYSIS" | grep -q '^UNREADABLE '; then+# Read with a here-string, NOT `printf ... | grep -q`. Under `set -o pipefail` that+# pipeline returns 141 (SIGPIPE) instead of 0 whenever grep matches and exits while+# printf still has output to write — which is exactly when the UNREADABLE line is+# followed by RETRIED_NAME lines. Measured on this machine at ~0.3-0.6% of runs+# under parallel load: the `if` then reads FALSE on a tree that IS unreadable and+# the guard silently does not fire. A guard against silent success must not have a+# silent-skip path of its own; a here-string has no pipeline and no SIGPIPE.+if grep -q '^UNREADABLE ' <<< "$ANALYSIS"; then     echo "FAIL [$LABEL]: the per-test tree is readable JSON but this script does not" >&2     echo "  understand its shape:" >&2     printf '%s\n' "$ANALYSIS" | sed -n 's/^UNREADABLE /    - /p' >&2@@ -632,7 +646,14 @@ if [ "$FAILED" -gt 0 ]; then     # a test that traps during setup or teardown, before timing starts, could also     # end up with no duration and be reported as wreckage. The split is a triage aid,     # not an authority — the FAILED count above it is the number that gates the build.-    if [ "$CASCADE" -gt 0 ]; then+    #+    # The split is only printed when it subtracts coherently. CASCADE counts Failed+    # Test Case nodes in the tree; FAILED comes from the summary. Every real bundle+    # on this machine agrees on those two figures, but if one ever exceeded the other+    # the subtraction below would report "~-1 look like genuine failures", which is+    # not a triage aid, it is noise. Say nothing rather than something false; the+    # FAILED count above still gates the build either way.+    if [ "$CASCADE" -gt 0 ] && [ "$CASCADE" -le "$FAILED" ]; then         REAL=$((FAILED - CASCADE))         echo "  of which ~$CASCADE never ran (no recorded duration — the host crashed" >&2         echo "  mid-run and these were still queued; they are not results)" >&2diff --git a/docs/agent-notes/development-tooling.md b/docs/agent-notes/development-tooling.mdindex aa847a65..dc4a8d2f 100644--- a/docs/agent-notes/development-tooling.md+++ b/docs/agent-notes/development-tooling.md@@ -10,7 +10,7 @@ - **`SVGWebViewTests` no longer needs skipping** (fixed on the T-1541 bugfix branch; `specs/bugfixes/svgwebviewtests-offmain-crash/report.md`). Root cause of the full-run host crash: the suite was the only one constructing `WKWebView` in *synchronous* test bodies. A sync `@MainActor` function has no hop-on-entry — its isolation depends on the caller hopping, and swift-testing's `nonisolated(nonsending)` thunks (Swift 5 mode + Approachable Concurrency, no default isolation in the test target) run on the runner's cooperative pool under full-suite load. WebKit's init `RELEASE_ASSERT(main thread)` then killed the host. The `@MainActor` added in 81c1636 could only move the trap. The rule for this target: **touch WebKit only from `async` tests** (actor-isolated async functions hop on entry as ABI); the suite additionally wraps construction in `MainActor.run`, which is runtime-enforced. - **The app and test targets have different actor-isolation defaults.** `SWIFT_DEFAULT_ACTOR_ISOLATION = MainActor` is set on the app target only; `prismTests` and `prismUITests` have no default. Production code is therefore implicitly main-actor while the same type called from a test is not, which silently turns "safe because everything is on the main actor" into a real race in tests (see `RecentFileEntry.relativeDateFormatter`). Do not assume a production type's isolation holds inside a test. - **`check-test-results.sh`'s arithmetic check now folds in `expectedFailures`, and it gates the build.** An expected failure is counted in neither `passedTests` nor `failedTests`, so the original `passed+failed+skipped != total` check tripped on every run containing an `XCTExpectFailure` (currently 2, T-1985 and T-1986) and printed a `WARN` everyone learned to ignore. Fixed in both directions (T-1993): `expectedFailures` is part of the sum, so those runs now reconcile exactly, and a mismatch is a hard FAIL rather than a warning — counts that do not reconstruct the whole mean nothing below them can be trusted. Do not expect a benign warning here any more; if this fires, the bundle is wrong.-- **`check-test-results.sh` fails closed on anything it cannot read, including the per-test tree.** Beyond the counts it reads `xcrun xcresulttool get test-results tests` and refuses to report OK when the tree contains an unrecognised `nodeType` or result value, contains no `Test Case` node while the summary counts tests, records a `Test Case` as `Failed` while `failedTests` is zero, or shows any test whose final result was reached only after a failed attempt (T-2224). Two things worth knowing before debugging a surprise FAIL: a future Xcode that adds a `TestNodeType` will trip the unrecognised-node check until the enum copy inside the script is updated (deliberate — a guard that goes quiet exactly when the format moves is worse), and the retry scan walks a Test Case's WHOLE subtree because on any run without `-only-test-configuration` — i.e. `make test` and `make test-ui` — every per-attempt node sits under a `Test Plan Configuration` node rather than directly under the case. Regression suite: `Tools/Tests/test-check-test-results.sh`, run by `make verify-make-guards`.+- **`check-test-results.sh` fails closed on anything it cannot read, including the per-test tree.** Beyond the counts it reads `xcrun xcresulttool get test-results tests` and refuses to report OK when the tree contains an unrecognised `nodeType` or result value, contains no `Test Case` node while the summary counts tests, records a `Test Case` as `Failed` while `failedTests` is zero, or shows any test whose final result was reached only after a failed attempt (T-2224). Two things worth knowing before debugging a surprise FAIL: a future Xcode that adds a `TestNodeType` will trip the unrecognised-node check until the enum copy inside the script is updated (deliberate — a guard that goes quiet exactly when the format moves is worse), and the retry scan walks a Test Case's WHOLE subtree rather than its direct children, because on any run without `-only-test-configuration` — i.e. `make test` and `make test-ui` — a Test Case's direct children are `Test Plan Configuration` nodes and nothing else structural (measured: 5,627 of 5,627 cases across every multi-configuration bundle on this machine, four configurations each). Where a per-attempt node would land inside that subtree is inference, not measurement — no bundle on this machine has ever contained a `Repetition` or `Test Case Run` node, which is exactly why the scan accepts either type at any depth. Regression suite: `Tools/Tests/test-check-test-results.sh`, run by `make verify-make-guards`. - **One `MockURLProtocol` used to be shared by five suites** across `URLDocumentLoaderTests`, `ImageLoaderTests`, and `SVGSourceLoaderTests`. `.serialized` only orders tests *within* a suite, so the suites overwrote each other's static handler and served each other's payloads. Handlers are now registered per scope (`MockURLScope`), carried as a request header set from the session configuration. If you add a networked suite, give it its own scope rather than a global handler.  ## "The test runner hung before establishing connection" on macOS (T-2146)