Runtime diagnostics

Why some flaws show only on a running screen, what the library gives to catch them, how an app turns diagnostics on only in its own debug build, and how this is tested. The problem is not Pantrix's own: it is seen the same way in every Compose Multiplatform project that uses automation tags, colours chosen at run time or texts that come from outside.

Everything said here about Compose was read in the sources of Compose Multiplatform 1.12.2, or seen in a test that runs in this repository. What is said about Kotlin and the Gradle plugins was read in the sources of Kotlin 2.4.21 and of the Compose Gradle plugin 1.12.2. Which was seen where is written in sections 7 and 8; what could not be verified is counted apart in section 8.

1. The problem

Symptom

• A test looks for a button with onNodeWithTag("Button_save") and says "Expected exactly '1' node but found '2' nodes". The same key was given in two places, but the two clash only while both are on the screen at the same time: a list's row and the detail it opens, two examples on a catalogue page, the two pages of a transition.

• A brand's colour comes from a server, a theme's colour from the user's choice. The screenshot test draws with its own colours; no test sees whether the text on the run-time colour can be read.

• A checkbox silently shrinks when the row that holds it gives it less room. In the screenshot the box is the same; the area a finger can press has narrowed.

• An icon button's contentDescription comes empty from a resource. When the screen reader reaches the button, it says nothing.

• The app opens in German, but the library has no German words: the components' own words ("Close", "Clear") stay in English.

• A screen's root and a field inside it listen to the same shortcut. When the key is pressed the outer one runs; the inner one never runs, and no one notices.

Cause

• Lint looks at the source. It cannot know a colour's value at run time, a name that comes from a resource, or how many nodes carry the same tag at the same time.

• A screenshot sees only what is drawn. It does not see the semantic name, the automation tag or the area that can be pressed.

• Compose does not look for repeats. Modifier.testTag only adds a property to the semantics (TestTagElement, SemanticsModifierNode); two nodes can carry the same tag. The test finders, for their part, count every node that is attached to the tree and not deactivated, even one not placed, in all roots (SemanticsOwner.getAllSemanticsNodes). The modifiers of a deactivated node are detached from the tree (onDetach), and attached again when it is reused (onAttach): the rows a scrolled list recycles are not counted.

• Some repeats are passing. A page transition shows two pages together for a few frames; a row that opens squeezes what is inside it for a moment. A check that looks at every frame counts these as flaws too.

2. The solution: four pieces

Without the first piece the second has no meaning: a check that looks from outside the component does not know a button's colour or name. Without the third, the first two are of use only in a test; without the fourth, a transition produces a finding every time.

3. With Pantrix

The library gives all four pieces (com.pantrixui.foundation.diagnostics; the inspector in com.pantrixui.components.overlay.inspector):

3.1. The six checks

The theme's 34 pairs: for each of the six roles (primary, success, info, warning, danger, neutral), onColor on color, onContainer on container, and color on background and on surface; the page's texts onSurface, textPrimary and textSecondary, and the marks borderStrong and focusRing, on background and on surface.

A ground with no fill. A ground that shows what is under it, like the ground of a button with no fill, is measured on the theme's background and surface if the component stands on one of the theme's own grounds, and the lower ratio is counted. If the place the component stands on is a fill of another colour that gives its own content colour (ProvidePantrixContentColor, as a dark card does), such a pair is not measured: what is under it is not known.

3.2. Only in the debug build

Pantrix does not know what kind of build it runs in; the app tells it. The plainest way is for each platform's entry point to give common code a debug:

// commonMain
@Composable
fun App(debug: Boolean) {
    val findings = remember { PantrixFindings(then = PantrixDiagnostics.Log) }
    PantrixTheme(diagnostics = if (debug) findings else PantrixDiagnostics.Off) {
        if (debug) PantrixDiagnosticsInspector(findings) { Screens() } else Screens()
    }
}

debug does not change during a build's life, so the if is safe here. For a setting that is turned on and off while the app runs, see the first pitfall in section 7.

Each platform knows the debug build its own way:

3.3. In a test

@Test
fun the_checkout_screen_has_no_findings() = runComposeUiTest {
    val findings = PantrixFindings()
    setContent { PantrixTheme(diagnostics = findings) { CheckoutScreen() } }
    // What layout finds is reported once it has lasted half a second, a repeated tag a second and a half.
    mainClock.advanceTimeBy(2_000)
    assertEquals(emptyList(), findings.all)
}

A finding's message is enough in the test's failure message: it says in English what is wrong and how it is put right. values carries what was measured (in LowContrast, ratio, needed, foreground, background and, if it was measured, ground; in SmallTouchTarget, width, height and needed).

3.4. Your own component

The app's own component takes part in the checks too. The component itself knows its name and its colours:

@Composable
fun Swatch(name: String, fill: Color, label: Color) {
    Column(
        modifier = Modifier
            .checkName("Swatch", name)
            .checkContrast("Swatch") { listOf(ContrastPair("label on fill", label, fill, ContrastNeed.Text)) }
            .background(fill)
            .padding(12.dp)
    ) {
        PantrixText(text = name, color = label)
    }
}

The pairs of checkContrast are asked for only while diagnostics are on, and again when the diagnostics or the theme's colours change. A disabled component gives no pairs, since no one is expected to read it.

3.5. A part with a receiver of its own

A theme inside can take its own receiver: PantrixTheme(diagnostics = own) { … }. That part's findings go only to own, not to the outer one. The demo in this repository shows the examples of the "Diagnostics" page this way: each example's flaw is gathered in its own receiver, and does not fall into the app's own inspector.

4. Without Pantrix

The checks made from inside the component (colour, name, touch target) need a library or the app's own component layer: code that looks from outside does not know which colour a button draws on which. The two pieces below, though, can be copied into any project as they are.

4.1. The contrast ratio (commonMain)

import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.compositeOver
import androidx.compose.ui.graphics.isSpecified
import androidx.compose.ui.graphics.luminance

/** WCAG 2.2: text reads at 4.5 and above, large text and a control's mark at 3 and above. */
fun contrastRatio(foreground: Color, background: Color): Float {
    require(foreground.isSpecified && background.isSpecified) { "An unspecified colour has no contrast." }
    val drawn = if (foreground.alpha < 1f) foreground.compositeOver(background) else foreground
    val one = drawn.luminance() + 0.05f
    val other = background.luminance() + 0.05f
    return maxOf(one, other) / minOf(one, other)
}

This is the body of the library's contrastRatio. A foreground colour that shows what is behind it is measured as it is drawn, put over the ground; the ground is measured as it is given.

4.2. Repeated tags in a test (commonTest)

import androidx.compose.ui.semantics.SemanticsProperties
import androidx.compose.ui.test.SemanticsMatcher
import androidx.compose.ui.test.SemanticsNodeInteractionsProvider

/** The tags two or more nodes carry, and how many nodes carry each. */
fun SemanticsNodeInteractionsProvider.repeatedTestTags(): Map<String, Int> =
    onAllNodes(SemanticsMatcher.keyIsDefined(SemanticsProperties.TestTag), useUnmergedTree = true)
        .fetchSemanticsNodes()
        .groupingBy { it.config[SemanticsProperties.TestTag] }
        .eachCount()
        .filterValues { it > 1 }

At the end of a screen test, assertEquals(emptyMap(), repeatedTestTags()). It counts what the test finders count: unplaced nodes too, recycled rows not. The moment the test looks is a single moment; if it looks in the middle of a transition, it finds a passing repeat too.

5. What the checks do not do

• Compose's own Modifier.testTag is not counted. Only the tags pantrixTestTag puts go into the register.

• A pressable area that does not use minTouchTarget is not measured. The app's own clickable finds no SmallTouchTarget.

• A chart's own series colours are not measured. What is measured in charts are the text and mark pairs of the style and of the colours class.

• iOS's switch thumb is not measured. In the iOS style (thumbAlwaysLight) the thumb is always light and stands apart from its track by its shadow; a colour ratio says nothing there.

• Pictures, gradients, and translucent colours on a filled ground are not measured. What is under them is not known.

• There is no check for an overlay that falls on the fold. Pantrix's overlays stay on one side of the fold (ModalRoom); this check would never fire on Pantrix's own overlays.

• There are no levels. Every finding has a kind; the app decides which one stops a release.

• The messages are in English only. A developer reads them, not the app's user. The inspector's own words (diagnostics, findingCount, notOnScreen) are in PantrixStrings in Turkish, English and Arabic.

6. Testing

• The receiver: every test gives its own PantrixFindings to the theme and looks at all. PantrixDiagnostics.Log is needless in a test.

• The clock: layout's findings are reported once they have lasted half a second, a repeated tag once it has lasted a second and a half. In a test whose clock advances by itself, waitForIdle() is enough; in a test with mainClock.autoAdvance = false, the clock is advanced by hand: by at least 500 ms for layout's findings, by at least 1500 ms for a repeated tag. That a short repeat is not reported is tested this way too: the repeat is removed before its time is up.

• All the screens: the demo in this repository draws every page in DemoPagesTest in two languages and under the app's own shortcut, moves to an example's code and to the API tab, and gathers the findings for each page; no page has findings. DiagnosticsSamplesTest looks at whether each example of the two diagnostics pages shows only its own findings: the examples have their own receivers, and their findings do not fall into the app's.

• The place: boundsOf(finding) is the place of the finding's node in the window; the inspector's frame is drawn from it. A finding with no node (the theme's colours, the language) gives null.

7. Pitfalls

8. Verification

Tests

Behaviours broken by hand

Each behaviour was broken once by hand, and which test turned red was looked at; then it was put back.

• The demo: the inspector wrapped in a condition (DemoShellTest's test red); findings not cleared on turning off (SettingsPageTest's test red); the old form of the two examples of the "Diagnostics" page, ⌘K and the 32dp box (DiagnosticsSamplesTest red: zero squeezes, the shadowed shortcut twice); the two examples of the "Diagnostics Inspector" page sharing the same keys (DiagnosticsSamplesTest red: "Findings: 4").

• The timing: the half second brought down to zero. Six tests turned red: the short repeat, the squeeze and the suspicion, the report that comes after half a second, the node that changes its tag, and the desk asking for frames.

• Free while off: Off counted as on too. Four tests turned red: the theme, the tag register, the squeezed target and the contrast.

• iOS's log: going back to NSLog("%@", line). PantrixDiagnosticsOnIosTest's process crashed with signal 11.

• A ground with no fill: a ground that shows what is under it measured on the theme's grounds on every fill. In the first form no test turned red: the test used a colour that reads on the page too. The test was written again with a colour that does not read on the page; when the break was tried again, it turned red.

Not verified

• The platforms' debug ways. The code of the table in 3.2 was not compiled or run in this repository. Platform.isDebugBinary was read in Kotlin/Native 2.4.21's Platform.kt, the reading of Xcode's configuration in the Kotlin Gradle plugin 2.4.21's XcodeEnvironment.kt, and that the run task is a JavaExec in the Compose Gradle plugin 1.12.2's configureJvmApplication.kt. Android's FLAG_DEBUGGABLE and BuildConfig's default in Android Gradle plugin 8 are from Android's documentation.

• The real logs. In the iOS simulator the line itself was seen in the unified log (xcrun simctl spawn <udid> log stream), and in the browser's console as a warning. Logcat and the desktop's standard error were not looked at; the tests there look at the text of the log line.

• A screen reader. How the inspector and its list are read aloud was not listened to.

9. Where it is in this repository

10. What is left open

• A chart's own series colours are not measured (see section 5).

• There is no check for an overlay that falls on the fold; it would not fire on Pantrix's own overlays.

• On real devices and in the browser only the demo itself was looked at (see section 8).