Pre-push review: T-1983 CI locale-sweep zero-tests fix

diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.ymlindex 452d310..a9d864c 100644--- a/.github/workflows/checks.yml+++ b/.github/workflows/checks.yml@@ -29,6 +29,13 @@ jobs:           fi           echo "No merge conflict markers found" +      # Asserts the Makefile cannot report a passing test run that executed no+      # tests. Cheap, and it runs on every push because the failure mode it+      # guards against is silent by construction — the per-locale sweep reported+      # success on zero tests for months (T-1983).+      - name: Verify test targets cannot report a false pass+        run: make verify-make-guards+       - name: Check for large files         run: |           max_size=1048576  # 1MB

diff --git a/.github/workflows/localisation-tests.yml b/.github/workflows/localisation-tests.ymlindex 87ea24c..d3a016a 100644--- a/.github/workflows/localisation-tests.yml+++ b/.github/workflows/localisation-tests.yml@@ -18,10 +18,24 @@ on:       - 'Makefile'       - '.github/workflows/localisation-tests.yml' +# The sweep used to take 70 seconds because it ran nothing. Now that it runs the+# suite four times over it costs tens of minutes of macOS runner time, so a rapid+# series of pushes must not leave several of them racing each other. On main the+# in-progress run is allowed to finish (newer pushes queue instead): this is the+# only test-executing job, and cancelling it would leave commits with no test+# signal at all — a hole when bisecting a regression later.+concurrency:+  group: ${{ github.workflow }}-${{ github.ref }}+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}+ jobs:   test-locales:     name: Per-locale test sweep     runs-on: macos-latest+    # This job used to finish in about 70 seconds because it ran nothing at all+    # (T-1983). A real sweep is four full runs of the suite plus the UI tests, so+    # it takes tens of minutes; the timeout is here to bound a hang, not the work.+    timeout-minutes: 90     steps:       - uses: actions/checkout@v4 @@ -32,6 +46,26 @@ jobs:         run: |           xcodebuild -version           xcrun --show-sdk-version+          make --version | head -1 +      # test-locales-adhoc is what makes this job capable of running anything. The+      # runner has no "Mac Development" certificate and no provisioning profile,+      # so the default signing made the build fail — and, because the failure was+      # swallowed (see the Makefile's $(STRICT)), every later step then tried to+      # launch a prism.app that had never been built. Each xcodebuild invocation+      # inside the target now writes a result bundle that+      # Tools/check-test-results.sh reads, so a run that executes zero tests fails+      # this job instead of quietly passing it.       - name: Run per-locale test sweep-        run: make test-locales+        run: make test-locales-adhoc++      # The bundles are the only record of what actually ran. Keep them when the+      # sweep fails, so a CI failure can be diagnosed without re-running it.+      - name: Upload test result bundles+        if: failure()+        uses: actions/upload-artifact@v4+        with:+          name: test-locales-results+          path: DerivedData/TestResults-*.xcresult+          retention-days: 7+          if-no-files-found: ignore

diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5224d5a..ce13838 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0  ### Changed +- CI now runs the test suite instead of only appearing to (T-1983). The per-locale sweep is the only job that can execute tests, and it reported success while executing none: its build failed for want of a signing certificate on the runner, that failure was swallowed, and nothing checked that any test had run. The sweep now signs ad-hoc so the build succeeds without a certificate, every test target hands its result bundle to the zero-test guard — per locale configuration, not once at the end — and any recipe whose failure must be believed carries `$(STRICT)`, because `.SHELLFLAGS` is silently ignored by the GNU Make 3.81 that macOS ships. `make verify-make-guards` asserts all of this on every push. - Copy notes is now the primary notes action (T-1577). On iPhone the document screen's toolbar shows Copy notes instead of Share with Notes, which moved into the notes pane alongside copy; on iPad and Mac the top toolbar shows copy leading the export button. Every copy button appears exactly when the copy output would contain at least one note under the current export settings, each action carries an accessibility label and help text, and an export blocked by the paywall from inside the pane now retries fully — including the author-name prompt and its confirmation toast — after a purchase completes. - The test suite covering notes-action placement was retargeted to the new contracts (T-1577): the T-138 share-button parity tests became the Share-with-Notes placement contract, and visibility/payload tests now assert the shared copy-availability predicate and the single export payload call site. Two device-only checks are recorded in the spec for manual verification. 

diff --git a/CLAUDE.md b/CLAUDE.mdindex 2f08372..7674934 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -297,6 +297,17 @@ Adding an en-AU divergence: - **Before pushing (pre-push-review)**: Use `make build-ios`, `make build-macos`, `make test`, and `make test-ui` — both builds must pass with zero warnings and zero errors - **For commits**: No tests required — lint only (`make lint`) +### Why the Make Targets Look Paranoid++Every target that runs tests writes its own `-resultBundlePath` and hands it to `Tools/check-test-results.sh`, which fails the run when the bundle reports zero executed tests. `test-locales` does this per locale configuration. This is not belt-and-braces: the CI sweep reported success while running zero tests for months (T-1983), and a `-only-testing:` filter that matches nothing does the same locally.++Two related traps live in the Makefile:++- `/usr/bin/make` on macOS is **GNU Make 3.81**, which silently ignores `.SHELLFLAGS`. Recipes therefore get no `-e` and no `pipefail`, so `xcodebuild | xcbeautify` reports xcbeautify's exit 0 over a failed build. Any recipe whose failure must be believed is prefixed with `$(STRICT)`.+- CI has no signing certificate, so the workflow runs `make test-locales-adhoc`, which signs ad-hoc against `prism/prism-ci.entitlements` (the app's entitlements minus the iCloud keys, which would demand a provisioning profile).++`make verify-make-guards` asserts both and runs on every push.+ ### Test Coverage  - Unit tests (`prismTests`): parsers, cache, file observer, models

diff --git a/Makefile b/Makefileindex 7d747fd..d0cb91e 100644--- a/Makefile+++ b/Makefile@@ -1,8 +1,52 @@ # Prism Makefile  SHELL = /bin/bash++# .SHELLFLAGS is honoured by GNU Make 3.82 and later. /usr/bin/make on macOS —+# what a developer runs, and what the CI runner runs — is 3.81, which IGNORES+# this line without a word. Recipes here therefore run as plain `/bin/bash -c`:+# no `-e`, and crucially no `pipefail`. Keep the line for anyone on gmake 4.x,+# but never rely on it. .SHELLFLAGS = -eo pipefail -c +# $(STRICT) is what actually makes a recipe honest, on every make version.+#+# Without pipefail, `xcodebuild ... | xcbeautify` reports XCBEAUTIFY's exit+# status, which is 0 whatever xcodebuild did. That is not hypothetical: the+# per-locale CI sweep printed "** TEST BUILD FAILED **", built nothing, launched+# nothing, and reported success for months (T-1983). The Makefile looked+# protected the whole time, because .SHELLFLAGS was sitting right there.+#+# Prefix any recipe whose failure must be believed. Inside a multi-command+# recipe the `-e` half also matters: it stops the recipe from marching on past+# an UNEXPECTED failure. Commands whose failure is anticipated and handled —+# the sweep's per-configuration guard calls, say — must be defused explicitly+# (`|| ...`), or `-e` turns "collect every result" into "abort on the first".+STRICT = set -eo pipefail;++# Code signing. A developer Mac signs with the team's Apple Development identity;+# a CI runner has no certificate and no provisioning profile, which is exactly+# why the sweep's build failed there and every subsequent step ran against a+# `prism.app` that did not exist. The `-adhoc` targets at the bottom of the+# testing section sign ad-hoc instead — enough for macOS to launch the binaries+# (arm64 will not run an unsigned one) — and swap in prism/prism-ci.entitlements,+# which drops the iCloud keys that would otherwise force a provisioning profile.+# See that file for what is kept and why.+#+# This is deliberately NOT a variable you set on make's command line. A+# command-line variable is exported into every recipe's environment, and+# xcodebuild reads its environment as build settings: the first attempt called it+# CODESIGN, which is a real Xcode setting naming the codesign TOOL, and CI died+# with `error: unable to spawn process 'adhoc'`. `unexport` does not reliably+# prevent that either — GNU Make 4.x always exports command-line variables, 3.81+# does not — so the flags are applied as a TARGET-SPECIFIC variable, which is+# never exported on any version.+ADHOC_SIGNING = CODE_SIGN_IDENTITY=- CODE_SIGN_STYLE=Manual DEVELOPMENT_TEAM= \+	PROVISIONING_PROFILE_SPECIFIER= CODE_SIGN_ENTITLEMENTS=prism/prism-ci.entitlements++# Empty for normal work; overridden per target by the `-adhoc` wrappers.+SIGNING_FLAGS =+ SCHEME = prism PROJECT = prism.xcodeproj BUNDLE_ID = me.nore.ig.prism@@ -39,6 +83,8 @@ help: 	@echo "    test        - Run full test suite on iOS Simulator" 	@echo "    test-ui     - Run UI tests only" 	@echo "    test-locales - Run the test suite under en, en-AU, en-GB, en-US"+	@echo "    test-locales-adhoc - test-locales, ad-hoc signed (CI, no certificate)"+	@echo "    verify-make-guards - Check the test targets cannot report a false pass" 	@echo "    install     - Build and install Debug on device" 	@echo "    run         - Build, install, and launch Debug on device" 	@echo ""@@ -60,6 +106,9 @@ help: 	@echo "" 	@echo "Device targets use DEVICE_MODEL (default: iPhone 17 Pro)" 	@echo "Override with: make install DEVICE_MODEL='iPhone 16'"+	@echo ""+	@echo "On a machine with no signing certificate (CI), use the -adhoc"+	@echo "variants, e.g. make test-locales-adhoc."  # Linting # Use --no-cache so lint runs don't depend on a writable user/global SwiftLint@@ -88,32 +137,41 @@ lint-css-fix: node_modules # Building DERIVED_DATA = ./DerivedData -# Result bundle for the test targets. The bundle is the only trustworthy record of+# Result bundles for the test targets. The bundle is the only trustworthy record of # what a run did: xcodebuild's exit code and console output have both been observed # claiming success for runs that executed nothing (T-1983). Tools/check-test-results.sh # reads it and fails on zero tests.+#+# Every test invocation gets its OWN path, and removes it before running. A shared+# or leftover bundle would let the guard pass on evidence from a different run,+# which is the same fail-open it exists to prevent. RESULT_BUNDLE = $(DERIVED_DATA)/TestResults.xcresult+RESULT_BUNDLE_IOS = $(DERIVED_DATA)/TestResults-ios.xcresult+RESULT_BUNDLE_UI = $(DERIVED_DATA)/TestResults-ui.xcresult+RESULT_BUNDLE_LOCALES_UI = $(DERIVED_DATA)/TestResults-locales-ui.xcresult  .PHONY: build-ios build-ios:-	xcodebuild build \+	$(STRICT) xcodebuild build \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		$(DEST_TIMEOUT) \ 		-destination 'platform=iOS Simulator,name=iPhone 17 Pro' \ 		-configuration $(CONFIG) \ 		-derivedDataPath $(DERIVED_DATA) \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY)  .PHONY: build-macos build-macos: clean-	xcodebuild build \+	$(STRICT) xcodebuild build \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		$(DEST_TIMEOUT) \ 		-destination 'platform=macOS' \ 		-configuration $(CONFIG) \ 		-derivedDataPath $(DERIVED_DATA) \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY)  .PHONY: build@@ -139,39 +197,65 @@ test-quick: 		-only-test-configuration "en (base)" \ 		-parallel-testing-worker-count 1 \ 		-only-testing:prismTests \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY) 	@Tools/check-test-results.sh $(RESULT_BUNDLE) "test-quick" +# The `-` prefix on each xcodebuild line below is deliberate and must stay: make+# ignores that command's exit status so the guard on the following line always+# runs and has the final word. The guard reads the result bundle, which is the+# only signal on this project that has never been observed lying. .PHONY: test test:-	xcodebuild test \+	@rm -rf $(RESULT_BUNDLE_IOS)+	-xcodebuild test \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		$(DEST_TIMEOUT) \ 		-destination 'platform=iOS Simulator,name=iPhone 17 Pro' \ 		-configuration Debug \ 		-derivedDataPath $(DERIVED_DATA) \+		-resultBundlePath $(RESULT_BUNDLE_IOS) \ 		-parallel-testing-worker-count 1 \ 		-maximum-concurrent-test-simulator-destinations 1 \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY)+	@Tools/check-test-results.sh $(RESULT_BUNDLE_IOS) "test"  .PHONY: test-ui test-ui:-	xcodebuild test \+	@rm -rf $(RESULT_BUNDLE_UI)+	-xcodebuild test \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		$(DEST_TIMEOUT) \ 		-destination 'platform=iOS Simulator,name=iPhone 17 Pro' \ 		-configuration Debug \ 		-derivedDataPath $(DERIVED_DATA) \+		-resultBundlePath $(RESULT_BUNDLE_UI) \ 		-only-testing:prismUITests \ 		-parallel-testing-worker-count 1 \ 		-maximum-concurrent-test-simulator-destinations 1 \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY)-+	@Tools/check-test-results.sh $(RESULT_BUNDLE_UI) "test-ui"++# This is the target CI runs. Each of the five test invocations below gets its own+# result bundle and its own guard call, because a sweep is only as honest as its+# least-checked configuration: with one guard at the end, three of the four locales+# could launch nothing and still report green.+#+# The guard calls are COLLECTED, never allowed to abort. Everything after the+# build runs in one $(STRICT) shell, and under its `set -e` an undefused guard+# failure would kill that shell on the first failing configuration — the+# remaining locales and the prismUITests run would never be attempted. An+# aborted sweep hides its later configurations just as effectively as a false+# green does (run 31366718394: "en (base)" failed and en-AU/en-GB/en-US were+# never tried). So each guard failure is recorded in $failed and the target+# fails once, at the end, after every configuration has run and reported. .PHONY: test-locales test-locales:-	xcodebuild build-for-testing \+	$(STRICT) xcodebuild build-for-testing \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		$(DEST_TIMEOUT) \@@ -179,9 +263,14 @@ test-locales: 		-configuration Debug \ 		-derivedDataPath $(DERIVED_DATA) \ 		-testPlan prism \+		$(SIGNING_FLAGS) \ 		$(PIPE_PRETTY)+	$(STRICT) failed=""; \ 	for cfg in "en (base)" "en-AU" "en-GB" "en-US"; do \+		slug=$$(printf '%s' "$$cfg" | tr -c 'A-Za-z0-9' '-'); \+		bundle="$(DERIVED_DATA)/TestResults-locale-$$slug.xcresult"; \ 		echo "==> test-locales: $$cfg"; \+		rm -rf "$$bundle"; \ 		xcodebuild test-without-building \ 			-project $(PROJECT) \ 			-scheme $(SCHEME) \@@ -189,12 +278,16 @@ test-locales: 			-destination 'platform=macOS' \ 			-configuration Debug \ 			-derivedDataPath $(DERIVED_DATA) \+			-resultBundlePath "$$bundle" \ 			-testPlan prism \ 			-only-test-configuration "$$cfg" \ 			-skip-testing:prismUITests \-			$(PIPE_PRETTY) || exit 1; \-	done-	@echo "==> test-locales: prismUITests (locale-forcing tests run once)"+			$(PIPE_PRETTY) || true; \+		Tools/check-test-results.sh "$$bundle" "test-locales: $$cfg" \+			|| failed="$$failed [$$cfg]"; \+	done; \+	echo "==> test-locales: prismUITests (locale-forcing tests run once)"; \+	rm -rf $(RESULT_BUNDLE_LOCALES_UI); \ 	xcodebuild test-without-building \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \@@ -202,10 +295,25 @@ test-locales: 		-destination 'platform=macOS' \ 		-configuration Debug \ 		-derivedDataPath $(DERIVED_DATA) \+		-resultBundlePath $(RESULT_BUNDLE_LOCALES_UI) \ 		-testPlan prism \ 		-only-test-configuration "en (base)" \ 		-only-testing:prismUITests \-		$(PIPE_PRETTY)+		$(PIPE_PRETTY) || true; \+	Tools/check-test-results.sh $(RESULT_BUNDLE_LOCALES_UI) "test-locales: prismUITests" \+		|| failed="$$failed [prismUITests]"; \+	if [ -n "$$failed" ]; then \+		echo "FAIL [test-locales]: failing configuration(s):$$failed" >&2; \+		exit 1; \+	fi++# What CI runs. A separate target rather than a command-line variable, for the+# reason given beside ADHOC_SIGNING: a command-line variable would reach+# xcodebuild as a build setting. Target-specific variables apply to the target and+# everything it depends on, so test-locales picks these flags up unchanged.+.PHONY: test-locales-adhoc+test-locales-adhoc: SIGNING_FLAGS = $(ADHOC_SIGNING)+test-locales-adhoc: test-locales  # Device deployment DEVICE_MODEL ?= iPhone 17 Pro@@ -221,7 +329,7 @@ install: 		exit 1; \ 	fi 	@echo "Building $(CONFIG) for device $(DEVICE_ID)..."-	xcodebuild build \+	$(STRICT) xcodebuild build \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		-destination 'id=$(DEVICE_ID)' \@@ -269,7 +377,7 @@ EXPORT_PATH_MACOS = ./build/export-macos  .PHONY: archive archive:-	xcodebuild archive \+	$(STRICT) xcodebuild archive \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		-destination 'generic/platform=iOS' \@@ -281,7 +389,7 @@ archive:  .PHONY: archive-macos archive-macos:-	xcodebuild archive \+	$(STRICT) xcodebuild archive \ 		-project $(PROJECT) \ 		-scheme $(SCHEME) \ 		-destination 'generic/platform=macOS' \@@ -316,6 +424,13 @@ upload-macos: archive-macos upload-all: upload upload-macos 	@echo "Uploaded both platforms to App Store Connect" +# Verification of the Makefile's own safety nets. Both defects it checks for were+# invisible in a passing run — their only symptom was a green light on nothing —+# so they are asserted rather than trusted. See T-1983.+.PHONY: verify-make-guards+verify-make-guards:+	Tools/Tests/test-make-guards.sh+ # Cleaning .PHONY: clean clean:

diff --git a/Tools/Tests/test-make-guards.sh b/Tools/Tests/test-make-guards.shnew file mode 100755index 0000000..3dd049e--- /dev/null+++ b/Tools/Tests/test-make-guards.sh@@ -0,0 +1,345 @@+#!/bin/bash+#+# test-make-guards.sh — regression tests for T-1983.+#+# The per-locale CI sweep reported success for months while executing ZERO+# tests. Two independent Makefile defects let that happen; two more appeared+# while fixing them. None is visible in a passing local run, which is precisely+# why the first two survived: the only symptom is a green light.+#+#   1. EXIT CODES. `xcodebuild ... | xcbeautify` reported xcbeautify's exit 0,+#      so "** TEST BUILD FAILED **" made a target SUCCEED. The Makefile's+#      `.SHELLFLAGS = -eo pipefail -c` does not prevent this: .SHELLFLAGS+#      arrived in GNU Make 3.82, and /usr/bin/make on macOS — the make a+#      developer and the CI runner both invoke — is 3.81, which ignores the+#      variable silently. Recipes must therefore carry the strictness+#      themselves, via the Makefile's $(STRICT) prefix.+#+#   2. ZERO-TEST GUARD. Only test-quick handed its result bundle to+#      Tools/check-test-results.sh. `test`, `test-ui` and `test-locales` — the+#      last being the one CI runs — asserted nothing about whether any test had+#      run, so "no tests executed" was indistinguishable from "all tests+#      passed".+#+#   3. EXPORTED SWITCHES. Fixing (1) and (2) introduced a third: a variable set+#      on make's command line is exported into every recipe's environment, and+#      xcodebuild reads its environment as build settings. The signing switch was+#      first called CODESIGN, which is a real Xcode setting naming the codesign+#      TOOL, and the CI build died with "unable to spawn process 'adhoc'". Ad-hoc+#      signing is therefore applied through a target-specific variable, which no+#      make version exports.+#+#   4. SHORT-CIRCUITED SWEEP. Fixing (2) introduced a fourth: the per-locale+#      guard call ran undefused under $(STRICT)'s `set -e`, so the FIRST failing+#      locale killed the loop's shell and the remaining locales and the+#      prismUITests guard never ran at all (PR #354 run 31366718394). An aborted+#      sweep hides its later configurations as effectively as a false green.+#+# Tests 1-5 are black-box: they read `make -n` output, not the Makefile text, so+# a target that reaches the guard by any route passes. Test 6 is partly textual,+# because the defect it pins is the NAME of a variable. Test 7 goes further and+# EXECUTES the expanded test-locales recipe against stubbed tools, because a+# mid-loop abort is control flow — invisible to any reading of the recipe text.+#+# Usage: Tools/Tests/test-make-guards.sh++set -uo pipefail++REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)+cd "$REPO_ROOT" || exit 2++FAILURES=0++pass() { echo "  ok   — $1"; }+fail() { echo "  FAIL — $1" >&2; FAILURES=$((FAILURES + 1)); }++# Targets that run tests. Every one of them must be guarded.+TEST_TARGETS=(test-quick test test-ui test-locales)++echo "make: $(make --version | head -1)"+echo++# --- 1. The $(STRICT) prefix really does propagate a piped failure -----------+# Uses the project's own definition rather than a copy, so the test fails if+# STRICT is deleted, renamed, or weakened.+#+# The fixture deliberately carries NO .SHELLFLAGS line. On GNU Make >= 3.82 —+# the ubuntu runner that executes verify-make-guards is 4.x — .SHELLFLAGS+# alone would fail the probe, and a weakened STRICT (say, `set -e;` without+# pipefail) would pass unnoticed on exactly the machine that enforces this+# unattended. STRICT must prove itself in isolation, on every make version.+echo "[1] \$(STRICT) propagates a failing pipeline"++STRICT_DEF=$(grep -E '^STRICT[[:space:]]*=' Makefile | head -1)+if [ -z "$STRICT_DEF" ]; then+    fail "Makefile defines no STRICT variable — piped recipes cannot report failure reliably"+else+    TMPDIR_T=$(mktemp -d)+    trap 'rm -rf "$TMPDIR_T"' EXIT+    {+        echo 'SHELL = /bin/bash'+        echo "$STRICT_DEF"+        echo '.PHONY: probe'+        echo 'probe:'+        # shellcheck disable=SC2016  # $(STRICT) must reach the fixture Makefile unexpanded+        printf '\t$(STRICT) (exit 65) | cat\n'+    } > "$TMPDIR_T/Makefile"++    if make -f "$TMPDIR_T/Makefile" probe >/dev/null 2>&1; then+        fail "a recipe using \$(STRICT) still succeeded when its xcodebuild-equivalent exited 65"+    else+        pass "$STRICT_DEF"+    fi++    # Informational control: show whether this make honours .SHELLFLAGS at all.+    {+        echo 'SHELL = /bin/bash'+        echo '.SHELLFLAGS = -eo pipefail -c'+        echo '.PHONY: probe'+        echo 'probe:'+        printf '\t(exit 65) | cat\n'+    } > "$TMPDIR_T/Makefile.control"+    if make -f "$TMPDIR_T/Makefile.control" probe >/dev/null 2>&1; then+        echo "  note — this make IGNORES .SHELLFLAGS (GNU Make < 3.82); \$(STRICT) is the only protection"+    else+        echo "  note — this make honours .SHELLFLAGS, but macOS's /usr/bin/make (3.81) does not; \$(STRICT) is still required"+    fi+fi+echo++# --- 2. Every test target reaches the zero-test guard ------------------------+echo "[2] every test target runs Tools/check-test-results.sh"+for target in "${TEST_TARGETS[@]}"; do+    recipe=$(make -n "$target" 2>/dev/null)+    if [ -z "$recipe" ]; then+        fail "$target: could not expand the recipe with 'make -n'"+        continue+    fi+    if printf '%s' "$recipe" | grep -q 'check-test-results.sh'; then+        pass "$target"+    else+        fail "$target: nothing asserts that any test ran — a zero-test run would report success"+    fi+done+echo++# --- 3. Every test target writes its own result bundle, freshly --------------+# The guard reads a bundle. If a target does not write one, or reuses another+# target's, or leaves a previous run's bundle in place, the guard can pass on+# evidence from a different run — a fail-open in the thing that exists to fail+# closed.+echo "[3] every test target writes a fresh, private result bundle"+declare -a SEEN_BUNDLES=()+for target in "${TEST_TARGETS[@]}"; do+    recipe=$(make -n "$target" 2>/dev/null)+    bundles=$(printf '%s' "$recipe" | grep -oE '\-resultBundlePath +"?[^ "\\]+' | sed -E 's/^-resultBundlePath +"?//')+    if [ -z "$bundles" ]; then+        fail "$target: passes no -resultBundlePath, so the guard has nothing to read"+        continue+    fi+    if ! printf '%s' "$recipe" | grep -q 'rm -rf'; then+        fail "$target: never removes the previous result bundle — a stale one could satisfy the guard"+        continue+    fi+    dup=""+    while read -r b; do+        [ -z "$b" ] && continue+        for prev in ${SEEN_BUNDLES[@]+"${SEEN_BUNDLES[@]}"}; do+            [ "$b" = "$prev" ] && dup="$b"+        done+        SEEN_BUNDLES+=("$b")+    done <<< "$bundles"+    if [ -n "$dup" ]; then+        fail "$target: shares result bundle $dup with another target"+    else+        pass "$target ($(printf '%s' "$bundles" | tr '\n' ' '))"+    fi+done+echo++# --- 4. test-locales checks every configuration, not just the last -----------+# The sweep is a loop. One bundle reused across four locale configurations would+# leave three of them unverified, and a per-configuration guard is the only way+# to know each one launched anything.+echo "[4] test-locales guards each locale configuration separately"+locales_recipe=$(make -n test-locales 2>/dev/null)+guard_hits=$(printf '%s' "$locales_recipe" | grep -c 'check-test-results.sh')+if [ "${guard_hits:-0}" -lt 2 ]; then+    fail "test-locales calls the guard $guard_hits time(s); the loop body and the UI-test run each need one"+else+    pass "guard invoked $guard_hits times"+fi+if printf '%s' "$locales_recipe" | grep -qE '\-resultBundlePath +"?[^ "]*\$'; then+    pass "the loop's result bundle varies per configuration"+else+    fail "test-locales uses a fixed result bundle path — later configurations would overwrite earlier ones"+fi+echo++# --- 5. Recipes that pipe through xcbeautify are strict ----------------------+# xcbeautify exits 0 whatever xcodebuild did. Any recipe that pipes into it and+# is not prefixed with $(STRICT) is reporting xcbeautify's opinion of the build.+# (Targets whose xcodebuild line is deliberately prefixed with make's `-` are+# exempt: there the guard, not the exit code, is the authority.)+#+# XCBEAUTIFY=x forces the pipe into the expansion. The Makefile only emits+# `| xcbeautify` when the tool is installed, and on the ubuntu runner that+# executes verify-make-guards it never is — an earlier version skipped this+# whole check there, so CI asserted nothing about the macOS pipelines it+# exists to protect. A command-line variable is safe HERE, unlike defect 3's,+# because under `make -n` nothing executes, so nothing reads its environment.+echo "[5] recipes piping through xcbeautify are prefixed with \$(STRICT)"+for target in build-ios build-macos install archive archive-macos test-locales; do+    recipe=$(make -n XCBEAUTIFY=x "$target" 2>/dev/null)+    if ! printf '%s' "$recipe" | grep -q 'xcbeautify'; then+        echo "  skip — $target does not pipe through xcbeautify"+        continue+    fi+    if printf '%s' "$recipe" | grep -q 'set -eo pipefail'; then+        pass "$target"+    else+        fail "$target: pipes into xcbeautify without \$(STRICT) — a failed build would report success"+    fi+done+echo++# --- 6. Ad-hoc signing is applied without a command-line variable ------------+# Fixing 1 and 2 introduced this one, and CI found both attempts at it. A variable+# set on make's command line is exported into every recipe's environment, and+# xcodebuild reads its environment as build settings: the switch was first called+# CODESIGN, which is a real Xcode setting naming the codesign TOOL, and the build+# died with "unable to spawn process 'adhoc'". `unexport` is not a fix — GNU Make+# 4.x always exports command-line variables, 3.81 does not, so the guard passed+# locally and failed on CI. The flags are therefore target-specific, which is+# never exported on any version. Partly a text check, because the defect is a NAME.+echo "[6] ad-hoc signing is applied without a command-line variable"++if grep -qE '^[[:space:]]*CODESIGN[[:space:]]*:{0,2}[?+]?=' Makefile; then+    fail "a variable named CODESIGN is defined — that name IS an Xcode build setting (the codesign tool path)"+else+    pass "no variable named CODESIGN"+fi++adhoc_recipe=$(make -n test-locales-adhoc 2>/dev/null)+if printf '%s' "$adhoc_recipe" | grep -q 'CODE_SIGN_IDENTITY=-'; then+    pass "test-locales-adhoc passes ad-hoc signing settings to xcodebuild"+else+    fail "test-locales-adhoc does not pass CODE_SIGN_IDENTITY=- — CI cannot build without a certificate"+fi+if printf '%s' "$adhoc_recipe" | grep -q 'CODE_SIGN_ENTITLEMENTS=prism/prism-ci.entitlements'; then+    pass "and signs against the CI entitlements"+else+    fail "test-locales-adhoc does not use prism/prism-ci.entitlements — the iCloud keys demand a provisioning profile"+fi+if printf '%s' "$(make -n test-locales 2>/dev/null)" | grep -q 'CODE_SIGN_IDENTITY'; then+    fail "plain test-locales carries ad-hoc signing settings — local runs would stop signing normally"+else+    pass "plain test-locales is unaffected"+fi+echo++# --- 7. A failing configuration does not abort the sweep ----------------------+# The defect this pins is control flow, so no reading of the recipe text can+# catch it: the guard call sat undefused under $(STRICT)'s `set -e`, and the+# first failing locale killed the shell mid-loop — the remaining locales and the+# prismUITests guard were never attempted (PR #354 run 31366718394). This test+# EXECUTES the real expanded recipe: it takes `make -n test-locales`, replays+# each recipe line the way make would (its own shell, stop at the first line+# that fails), with `xcodebuild` stubbed to fail its test runs and the guard+# stubbed to fail the FIRST configuration. The sweep is honest only if every+# configuration still reports and the target still fails at the end.+echo "[7] a failing configuration still lets every later configuration run"++SANDBOX=$(mktemp -d)+mkdir -p "$SANDBOX/bin" "$SANDBOX/Tools"++# xcodebuild: builds succeed, test runs fail — the sweep must survive that.+cat > "$SANDBOX/bin/xcodebuild" <<'EOF'+#!/bin/bash+case "${1:-}" in+    build-for-testing) exit 0 ;;+    *) exit 65 ;;+esac+EOF+# xcbeautify: PIPE_PRETTY may or may not be in the expanded recipe; stub it so+# the pipeline works either way.+cat > "$SANDBOX/bin/xcbeautify" <<'EOF'+#!/bin/bash+cat >/dev/null+EOF+# The guard: log which configuration it was asked about, and fail the first one.+cat > "$SANDBOX/Tools/check-test-results.sh" <<'EOF'+#!/bin/bash+printf '%s\n' "${2:-}" >> guard.log+[ "${2:-}" = "test-locales: en (base)" ] && exit 1+exit 0+EOF+chmod +x "$SANDBOX/bin/xcodebuild" "$SANDBOX/bin/xcbeautify" "$SANDBOX/Tools/check-test-results.sh"++# Reassemble `make -n` output into logical recipe lines: a line ending in `\`+# continues into the next, exactly as make hands the block to one shell.+#+# --no-print-directory is load-bearing. This script normally runs as a sub-make+# (`make verify-make-guards`), and GNU Make 4.x puts `-w` into sub-makes'+# MAKEFLAGS, so the inner `make -n` here prints `make[1]: Entering directory+# ...` lines into stdout — which the replay below would then execute as+# commands and die on before reaching any guard. That is exactly how this+# test's first version failed on CI while passing when run directly. The+# `make*:` skip is a second belt for any chatter a flag cannot suppress.+sweep_cmds=()+acc=""+while IFS= read -r line; do+    case "$line" in+        make:*|make\[*\]:*) continue ;;+    esac+    acc+="$line"$'\n'+    case "$line" in+        *\\) ;;+        *)+            [ -n "${acc//[[:space:]]/}" ] && sweep_cmds+=("$acc")+            acc=""+            ;;+    esac+done < <(make --no-print-directory -n test-locales 2>/dev/null)++if [ "${#sweep_cmds[@]}" -eq 0 ]; then+    fail "could not expand the test-locales recipe with 'make -n'"+else+    # Replay with make's semantics: each logical line in its own shell, and the+    # first non-zero line ends the target.+    sweep_status=0+    for cmd in "${sweep_cmds[@]}"; do+        if ! (cd "$SANDBOX" && PATH="$SANDBOX/bin:$PATH" bash -c "$cmd") \+                >> "$SANDBOX/sweep.log" 2>&1; then+            sweep_status=1+            break+        fi+    done++    if [ "$sweep_status" -eq 0 ]; then+        fail "test-locales reported success although a configuration's guard failed"+    else+        pass "a failing configuration still fails the target"+    fi++    missing=""+    for expected in "test-locales: en (base)" "test-locales: en-AU" \+            "test-locales: en-GB" "test-locales: en-US" "test-locales: prismUITests"; do+        grep -qxF "$expected" "$SANDBOX/guard.log" 2>/dev/null || missing="$missing [$expected]"+    done+    if [ -n "$missing" ]; then+        fail "configurations never reached their guard after the first one failed:$missing"+        echo "  --- replay output (last 20 lines) ---" >&2+        tail -20 "$SANDBOX/sweep.log" 2>/dev/null | sed 's/^/  | /' >&2+    else+        pass "all five guard calls ran despite the first configuration failing"+    fi+fi+rm -rf "$SANDBOX"+echo++if [ "$FAILURES" -gt 0 ]; then+    echo "$FAILURES check(s) failed." >&2+    exit 1+fi+echo "All make-guard checks passed."

diff --git a/prism/prism-ci.entitlements b/prism/prism-ci.entitlementsnew file mode 100644index 0000000..f2dbe77--- /dev/null+++ b/prism/prism-ci.entitlements@@ -0,0 +1,24 @@+<?xml version="1.0" encoding="UTF-8"?>+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">+<!--+  Entitlements for ad-hoc signed CI builds (make test-locales-adhoc), used only+  to run the test suite on a machine with no signing certificate. See T-1983.++  This is prism.entitlements minus the iCloud keys. Those are "restricted"+  entitlements: Xcode refuses to build a target carrying them without a matching+  provisioning profile ("prism requires a provisioning profile"), which a CI+  runner cannot have. Nothing under test needs iCloud — ExportCounter's store is+  injected through KeyValueStoreProtocol in tests.++  com.apple.security.network.client is kept deliberately. WKWebView on macOS+  spawns WebContent and GPU processes that talk over network IPC, and without it+  they fail to launch silently, which would take the whole WebKit rendering path+  down with no visible error. It is not a restricted entitlement, so it costs+  nothing here.+-->+<plist version="1.0">+<dict>+	<key>com.apple.security.network.client</key>+	<true/>+</dict>+</plist>

diff --git a/specs/bugfixes/ci-locale-sweep-zero-tests/report.md b/specs/bugfixes/ci-locale-sweep-zero-tests/report.mdnew file mode 100644index 0000000..5f5dfc1--- /dev/null+++ b/specs/bugfixes/ci-locale-sweep-zero-tests/report.md@@ -0,0 +1,301 @@+# Bugfix Report: CI Per-Locale Test Sweep Reports Success While Running Zero Tests++**Date:** 2026-08-10+**Status:** Fixed+**Ticket:** T-1983++## Description of the Issue++`.github/workflows/localisation-tests.yml` ("Per-locale test sweep") is the only CI+job that runs on macOS and therefore the only one that can execute the test suite.+It reported success on every run while executing zero tests. Every green tick on+this repository meant "lint passed" — no test has ever been validated by CI.++**Reproduction steps:**++1. `gh run view 30820938599 --log` (the last main-branch run before this fix).+2. Observe `** TEST BUILD FAILED **` in the `make test-locales` step, followed by+   `No signing certificate "Mac Development" found ... (in target 'prismTests')`.+3. Observe, for each of the four locale configurations and the UI-test step,+   `Could not launch "prismTests" ... The file "prism.app" couldn't be opened+   because there is no such file`.+4. Observe zero executed tests, and the job reporting **success** in 1m54s. A real+   sweep is four full runs of the suite plus the UI tests.++**Impact:** High. Two failure modes compounded: CI could not run the tests, and+nothing noticed. Every "confirmed pre-existing on main" conclusion recorded on+other tickets rests on local runs alone. PR #330 had already closed half of this+(the guard script + `make test-quick`); the CI half — the half that was actually+lying — was untouched.++## Investigation Summary++- **Symptoms examined:** the raw CI log rather than the job summary. Two distinct+  facts: the *build* failed, and `make` then carried on to run four test+  configurations against a build product that did not exist, finishing with exit 0.+- **Code inspected:** `Makefile` (`test-locales`, `test`, `test-ui`, `build-*`),+  `Tools/check-test-results.sh`, `.github/workflows/localisation-tests.yml`,+  `prism.xcodeproj` signing settings, `prism/prism.entitlements`.+- **Hypotheses tested:**+  - *"xcbeautify masks the exit code"* — the ticket explicitly warned against+    assuming this, since `.SHELLFLAGS = -eo pipefail -c` appears to prevent it and+    that same reasoning error has been made on this repo before. Tested directly+    rather than assumed and found **true**, for a reason that is not the obvious one.+  - *"the sweep's `|| exit 1` catches it"* — no: `|| exit 1` tests the same+    masked pipeline status.+  - *"ad-hoc signing alone fixes the runner"* — tested locally and **false**: the+    build still fails with `"prism" requires a provisioning profile`.++## Discovered Root Cause++Two independent defects, either of which alone still leaves a silent hole.++### 1. `.SHELLFLAGS` does nothing on the make that actually runs++`.SHELLFLAGS` was introduced in **GNU Make 3.82**. `/usr/bin/make` on macOS —+what the developer runs and what the CI runner runs — is **GNU Make 3.81**, which+ignores the variable **silently**. Measured directly:++```make+SHELL = /bin/bash+.SHELLFLAGS = -eo pipefail -c+probe:+	(exit 65) | cat+	@echo CONTINUED+```++```+$ make --version | head -1+GNU Make 3.81+$ make probe+(exit 65) | cat+CONTINUED+make exit: 0+```++So every recipe ran as plain `/bin/bash -c`, without `-e` and without `pipefail`.+`xcodebuild ... | xcbeautify` reported xcbeautify's exit status, which is 0+whatever xcodebuild did. `** TEST BUILD FAILED **` was therefore a *successful*+recipe, and the sweep marched on into a directory with no `prism.app` in it.++**Defect type:** false assumption about the toolchain, invisible because the+Makefile *looked* protected — the mitigation was present, just inert.++### 2. Nothing asserted that any test ran++`Tools/check-test-results.sh` (PR #330) fails a run whose result bundle reports+`totalTestCount == 0`, but only `make test-quick` called it. `test`, `test-ui` and+`test-locales` — the last being the target CI runs — passed no+`-resultBundlePath` at all, so there was nothing to check and no check to make.++### 3. (The reason the build failed) No signing identity on the runner++The runner has no "Mac Development" certificate and no provisioning profile, and+the project signs with a fixed `DEVELOPMENT_TEAM`. Ad-hoc signing alone is not+enough: `prism.entitlements` carries iCloud keys, which are *restricted*+entitlements, and Xcode refuses to build a target carrying them without a matching+profile.++**Why it occurred:** each layer's failure was swallowed by the next. A missing+certificate produced a failed build; a masked exit code turned that into a passing+step; an unchecked test count turned "launched nothing" into "all tests passed".++## Resolution for the Issue++**Changes made:**++- `Makefile` — added `STRICT = set -eo pipefail;` and prefixed every recipe whose+  failure must be believed (`build-ios`, `build-macos`, `install`, `archive`,+  `archive-macos`, `test-locales`). `.SHELLFLAGS` is kept for anyone on gmake 4.x,+  with a comment stating that it does nothing here.+- `Makefile` — `test`, `test-ui` and `test-locales` now write a result bundle and+  hand it to `Tools/check-test-results.sh`. `test-locales` does so **per+  configuration**: five invocations, five bundles, five guard calls. Each bundle is+  removed immediately before its run, so a leftover bundle cannot satisfy the guard.+- `Makefile` — new `test-locales-adhoc` target, which applies+  `CODE_SIGN_IDENTITY=- CODE_SIGN_STYLE=Manual DEVELOPMENT_TEAM=+  PROVISIONING_PROFILE_SPECIFIER= CODE_SIGN_ENTITLEMENTS=prism/prism-ci.entitlements`+  as a **target-specific** variable and depends on `test-locales`. Deliberately not+  a command-line variable — see "What CI caught" below.+- `prism/prism-ci.entitlements` — new. `prism.entitlements` minus the iCloud keys,+  keeping `com.apple.security.network.client` (WKWebView's WebContent and GPU+  processes fail to launch without it, silently).+- `.github/workflows/localisation-tests.yml` — runs `make test-locales-adhoc`,+  bounds the job at 90 minutes, and uploads the result bundles on failure.+- `.github/workflows/checks.yml` — runs `make verify-make-guards` on every push.+- `Tools/Tests/test-make-guards.sh` — new; the regression test, below.++**Approach rationale:** the guard is the part that has to hold. Signing is one+cause of "no tests ran" and it is now fixed, but the guard converts *any* future+cause — a renamed test plan, a filtered configuration, a `-only-testing:` typo,+another expired certificate — into a red job rather than a green one. The two+halves are deliberately independent: if the ad-hoc signing regresses, CI fails+loudly instead of returning to silence.++**Alternatives considered:**++- **Provision a real signing certificate on CI** — needs a secret and its+  rotation, and buys nothing over ad-hoc for a test run that never ships.+- **`CODE_SIGNING_ALLOWED=NO`** — arm64 macOS will not execute an unsigned binary,+  so the tests would fail to launch for a new reason.+- **Strip all entitlements on CI** — would drop+  `com.apple.security.network.client` and take the whole WebKit rendering path+  down with no visible error. Hence a CI-specific entitlements file rather than+  none.+- **Require gmake 4.x** — would fix `.SHELLFLAGS` but adds a toolchain prerequisite+  for every contributor and every runner, to buy something a two-word prefix buys+  outright.+- **Drop the sweep from CI and run it locally only** (floated on the ticket) —+  rejected for now: with the guard in place the job can no longer lie, so it is+  worth keeping while we see what it reports. Revisit if the runner proves unable+  to host the suite.++## Regression Test++**Test file:** `Tools/Tests/test-make-guards.sh`+**Run command:** `make verify-make-guards`++Seven checks. Tests 1-5 are black-box — they read `make -n` output, not Makefile+text, so a target that reaches the guard by any route passes. Test 6 is partly+textual, because the defect it pins is the NAME of a variable. Test 7 executes+the expanded `test-locales` recipe against stubbed tools, because a mid-loop+abort is control flow, invisible to any reading of the recipe text.++1. `$(STRICT)`, taken from the real Makefile, makes a failing pipeline fail its+   target — in a fixture that deliberately carries no `.SHELLFLAGS`, so a+   weakened definition cannot hide behind a make that honours it (the ubuntu+   runner's 4.x does). Also reports whether the local make honours+   `.SHELLFLAGS` at all.+2. Every test target (`test-quick`, `test`, `test-ui`, `test-locales`) reaches+   `Tools/check-test-results.sh`.+3. Every test target passes its own `-resultBundlePath` and removes it first.+4. `test-locales` calls the guard more than once and its loop bundle varies per+   configuration — one bundle for four locales would leave three unverified.+5. Every recipe piping into xcbeautify (`build-ios`, `build-macos`, `install`,+   `archive`, `archive-macos`, `test-locales`) carries `$(STRICT)`. The pipe is+   forced into the expansion with `make -n XCBEAUTIFY=x`, so the check also+   runs on machines where xcbeautify is not installed — the ubuntu runner that+   executes `verify-make-guards` included.+6. Ad-hoc signing is applied without a command-line variable: no `CODESIGN`+   variable is defined, `test-locales-adhoc` passes the ad-hoc settings and the+   CI entitlements, and plain `test-locales` is unaffected.+7. A failing configuration does not abort the sweep: the expanded recipe is+   replayed against stubbed `xcodebuild`/guard binaries with the first locale's+   guard failing, and all five guard calls must still run and the target must+   still fail at the end.++Against the unfixed Makefile it reports **19 failures**; against the fix, all pass.++## Affected Files++| File | Change |+|------|--------|+| `Makefile` | `STRICT` prefix, ad-hoc signing wrapper, per-invocation result bundles + guard calls, `verify-make-guards` target |+| `Tools/Tests/test-make-guards.sh` | New — regression test for all four defect classes |+| `prism/prism-ci.entitlements` | New — profile-free entitlements for ad-hoc CI builds |+| `.github/workflows/localisation-tests.yml` | `test-locales-adhoc`, timeout, artifact upload on failure |+| `.github/workflows/checks.yml` | Runs `make verify-make-guards` |++## Verification++**Automated:**++- [x] `make verify-make-guards` — 19 failures before the fix, all checks pass after+- [x] `shellcheck Tools/Tests/test-make-guards.sh` — clean+- [x] `xcodebuild build-for-testing` with the ad-hoc flags — `** TEST BUILD+      SUCCEEDED **`, where the same command with default signing fails on the+      runner and the same command with ad-hoc signing but the *original*+      entitlements fails locally with `"prism" requires a provisioning profile`+- [x] `codesign -d --entitlements -` on the ad-hoc product confirms the signature+      carries `com.apple.security.network.client`, `get-task-allow`, the app+      sandbox and the test-host temporary exceptions++**Manual verification:**++- Read the CI log of run 30820938599 to establish the failure mode from evidence+  rather than inference, and confirmed the build failed *before* the sweep began.+- Reproduced the `.SHELLFLAGS` no-op directly against `/usr/bin/make` 3.81.++### What CI caught++Three defects in the fix itself, none visible locally, all found by the PR's own+runs. Recording them because they are the same failure class as the bug.++1. The signing switch was first a command-line variable, `CODESIGN=adhoc`. Make+   **exports** command-line variables into every recipe's environment, and+   xcodebuild reads its environment as build settings — and `CODESIGN` is a real+   Xcode setting: the path to the codesign *tool*. The build died with+   `error: unable to spawn process 'adhoc'`.+2. The first repair renamed it and added `unexport`, verified against+   `/usr/bin/make` 3.81. `unexport` does **not** apply to command-line variables on+   GNU Make 4.x, so the guard test passed on macOS and failed on the ubuntu job —+   a version-dependent assumption dressed as protection, exactly like+   `.SHELLFLAGS`. The mechanism was removed rather than patched: ad-hoc signing is+   now a target-specific variable, which no make version exports.+3. The per-locale guard call first ran undefused under `$(STRICT)`'s `set -e`, so+   the FIRST failing locale killed the sweep's shell: en-AU, en-GB, en-US and the+   prismUITests guard were never attempted (run 31366718394). An aborted sweep+   hides its later configurations as effectively as a false green does. Guard+   failures are now collected into an accumulator and the target fails once, at+   the end, after every configuration has run — pinned by the script's test 7,+   which replays the expanded recipe against stubs, because a mid-loop abort is+   control flow that no reading of the recipe text can catch.++### Confirmed on CI++Run 31366718394, the PR's own sweep:++```+[test-locales: en (base)] total=4240 passed=4051 failed=151 skipped=37+FAIL [test-locales: en (base)]: 151 test(s) failed.+```++The first real CI test result this repository has ever produced. Both halves hold:+the build succeeds on a runner with no certificate and the test host launches+(4,240 tests executed, against zero on every previous run), and the guard failed+the job on genuine failures rather than reporting green. Time to+first-configuration failure was about 9 minutes, against 57-70 seconds for the+run-nothing version.++That same run also exposed defect 3 above: it aborted after "en (base)" instead+of sweeping on, so the later configurations were never attempted. The+collect-then-fail repair landed after that run (commit 7de2c74) and is pinned by+the script's test 7; it has not yet had a red-locale CI run to demonstrate+itself on.++What the honest sweep then reveals is separate work, filed as **T-2146**: 151+failures, almost all live WebKit tests (143 of 189 failure messages are+`.loadTimedOut` from `SpikeWebPageHarness`, which allows 30 seconds per load — so+the WebContent process is not loading on the runner at all), plus three SIGSEGVs+and a few wall-clock performance budgets that are hardware-sensitive on shared+runners. Whether that is the runner environment or the ad-hoc signing introduced+here is not established; T-2146 carries the evidence and a way to separate the two.++Those tests were deliberately NOT skipped to make the job green. Skipping them+would recreate a softer version of this bug: a sweep that passes by not looking.++**Not verifiable locally:** whether an ad-hoc-signed test host launches under+XCTest. Attempts on this machine hit `The test runner hung before establishing+connection`, with several stale `prism` test hosts from other worktrees running+concurrently and sharing the bundle identifier — a contended-machine artefact, not+a signing result.++## Prevention++- **Never trust a mitigation you have not seen fire.** `.SHELLFLAGS` sat in this+  Makefile looking like protection for as long as the bug existed. Both defects+  here were invisible in a passing run; the only symptom was a green light.+- **Assert on evidence, not exit codes.** The result bundle is the only signal on+  this project that has never been observed lying. Any new target that runs tests+  gets a bundle and a guard call — `make verify-make-guards` now enforces that.+- **A CI job that finishes suspiciously fast is a defect report.** 70 seconds for+  a four-locale sweep was visible on every run for months.++## Related++- T-1983 (this ticket); PR #330 / commit 81c16361 — the guard script and+  `make test-quick` wiring, the other half of this fix+- T-1541 — crash cascade making local runs unreadable, which the guard's+  cascade-split report addresses+- T-1967 — two WebRendering tests whose "pre-existing" status was only ever+  established locally, which is a direct consequence of this bug