
AI Evals: 72 Runs Expose Citation and Grounding Failures

VMware Fusion: Windows ARM Audit and Evidence Limits

Tree Testing Before a Navigation Redesign: The Decision We Actually Had
An administrator knows exactly what needs to change: require a second authentication factor, permit a webhook to reach an internal host, or stop an automated account-deactivation email. The harder question is where the administrator would look for that control. In an enterprise product, a setting can be technically available, correctly documented, and still difficult to locate. Yet moving it to a shorter menu path is not automatically an improvement. A new label can create a competing interpretation, and a newly grouped setting can cross a security or ownership boundary.
For this study, I wanted to separate what an information architecture model can establish from what tree testing with people must still demonstrate. The evidence package compares two text-only versions of a GitLab Self-Managed administration navigation area. It uses 36 common destinations and 12 realistic administration tasks. A reproducible structural audit checks which locations move, where hierarchy depth changes, and which labels overlap. It does not invent participants, first clicks, navigation times, or a winning design.
The distinction matters more than it might initially appear. A team can produce two polished menu proposals, calculate path lengths, and call the shorter option more usable. That conclusion skips the essential question: what do people believe a category contains when they are trying to complete an actual task? Tree testing is designed to observe that behavior, but the structural groundwork and the human tree testing experiment are different stages. Here, the former was executed; the latter is prepared but not conducted.
The central finding is deliberately mixed. The alternative tree leaves ten of twelve modeled task depths unchanged, shortens one path, and lengthens one. It also increases the number of first-level options from ten to eleven. Several proposed names make a category’s purpose more explicit, while other changes introduce genuine competing interpretations. There is no measured participant success rate and no defensible overall winner. That is not a failed research exercise. It is an honest boundary around a useful design decision.
Decision Brief: A Structural Audit Is Not a Human Tree Test
Decision question: Should a self-managed administration menu be reorganized around clearer task-oriented categories, or should its established structure remain the safer starting point for a participant test?
Observed material: A version-specific set of GitLab Community Edition administration menu registrations, page templates, and official documentation. Measured material: Properties of two researcher-defined navigation trees, validated against 12 fixed task destinations. Interpretation: Hypotheses about discoverability, category ownership, and ambiguity. Missing evidence: Actual administrator choices and paths through either tree.
My decision would be to keep the documented structure as the comparator, selectively test the most meaningful changes, and treat security-sensitive regressions as guardrails. I would not ship the entire alternative based on a path-depth audit alone. The research package is classified LIMITED — STRUCTURAL IA EXPERIMENT, not as an observed human A/B tree-testing result.
The method distinction follows Nielsen Norman Group’s explanation of tree testing: participants receive realistic tasks and choose locations in a text hierarchy. Researchers can then examine destinations, first choices, directness, and detours. None of those behavioral measures can be calculated from a researcher’s predetermined correct path. This article makes that boundary explicit because a believable-looking table can otherwise disguise the absence of participants.
Enterprise Navigation Testing Setup: Product, Version, and Evidence
A bounded GitLab administration model, not a live production redesign
The reference environment was GitLab Self-Managed Community Edition, Free, version 18.11.0, specifically the English Admin > Settings area. The baseline was reconstructed from the version-tagged Settings menu source, linked view templates, and the corresponding administration documentation. These sources were treated as a point-in-time record, not as a claim about every currently deployed GitLab interface.
A full instance administrator was the assumed role. If Admin Mode is configured, access must first be enabled. Group Owners, project Maintainers, ordinary users, and GitLab.com users were outside the tested scope; the official Admin area documentation distinguishes these boundaries. The audit starts inside Settings, which means the broader application navigation, search, shortcuts, direct links, and completed configuration actions were excluded.
There is an additional modeling caveat: the source-backed first-level Settings menu is represented, but the 36 lower-level targets are a deliberate sample, not a complete inventory of every possible administration control. The model combines sidebar pages, in-page sections, and integration rows as hierarchy nodes. One modeled edge therefore should not be read as one physical click in a live interface. No authenticated instance or pixel-level visual state was inspected.
Why this product, rather than a private portfolio interface?
The candidate review considered publicly visible material for RE Workspace, GitLab Self-Managed, Strapi, and Directus. The available RE Workspace material did not establish a sufficiently detailed public, verifiable control hierarchy for this exercise. GitLab offered release-tagged menu code and task documentation that could support a frozen reference tree. That made the comparison auditable without inspecting a private product or inventing functionality.
This is an independent research exercise about a GitLab administration model, not a GitLab-sponsored study or a test of RE Workspace. I am interested in the transferable decision method: define a specific administrative job, preserve the exact destination across design variants, then ask what each proposed category helps or obscures. The same method could later be applied to a private workspace navigation with appropriate access and genuine administrator participants.
What was frozen before the audit?
The research locked three inputs on November 9, 2026, at 11:33:04 UTC: the baseline tree, the proposed tree, and the 12 task definitions. Exact SHA-256 hashes were stored in a freeze manifest. A Python standard-library audit then checked parent relationships, cycles, stable destination IDs, task mappings, display paths, duplicate labels, token overlaps, and task-to-category wording. Separate validation code recomputed all 72 condition-to-destination paths and regenerated the result files byte for byte.
Alongside those deterministic checks, an unblinded, model-assisted analyst walkthrough examined plausible conflicts between labels and task intentions. Those judgments are not independent user observations. They inform hypotheses to test, and their possible biases must remain attached to the interpretation. A local, keyboard-oriented tree-testing instrument and a participant protocol were prepared, but no human sessions were run and no participant dataset entered this analysis.
| Component | Actually executed | What it supports |
|---|---|---|
| Tagged source review | Yes | Reference navigation and documented control scope |
| Two-tree validation | Yes | Stable targets, valid links, preserved comparison inputs |
| Twelve-task structural audit | Yes | Path depth, branch choices, moves, and label diagnostics |
| Model-assisted walkthrough | Yes | Explicitly provisional ambiguity and ownership hypotheses |
| Local research instrument QA | Logic checks only | Planned collection flow, not field usability |
| Participant tree testing | No: zero sessions | No first-choice, success, time, or backtracking findings |
Two Navigation Trees, the Same 36 Destinations
Tree A: The documented administration structure
Tree A represents ten top-level choices inside Settings: General, Search, Integrations, Repository, CI/CD, Reporting, Metrics and profiling, Network, Appearance, and Preferences. It has 36 destination IDs across 46 nodes including the root. For example, project visibility defaults sit in General, while search-specific visibility controls sit in Search. Both sections are labeled “Visibility and access controls,” even though they do different jobs. That duplicate was not invented for the test; it comes from the sampled source structure.
Tree A also preserves an important policy boundary: instance-wide outbound request rules are grouped under Network. A developer looking for webhook connectivity might intuitively start with Integrations, but moving a security-sensitive host allowlist there can change how the control’s ownership is understood. A familiar label is not automatically ideal, yet the rationale for relocating it must be more substantial than making a webhook-related task appear to have a more obvious parent.
Tree B: A targeted researcher proposal, not a verified improvement
Tree B keeps exactly the same 36 destinations and documented restrictions. It renames General to Identity & Access, Repository to Projects & Repositories, Metrics and profiling to Monitoring & Telemetry, and Network to Traffic & Request Limits. It moves five destinations, including import/export settings, diff limits, one Jira app control, outbound requests, and the email section. A new Webhook connectivity subgroup appears beneath Integrations, while the email section becomes a direct first-level Email notifications choice.
These changes are meaningful enough to investigate, but the direction of benefit is not obvious. The new identity label sounds closer to authentication tasks; the enlarged projects category may attract someone searching for default project visibility even though that destination remains under access. Putting two Jira-related controls together creates topical consistency while also making two distinct configurations adjacent. Promoting Email changes both depth and how broadly that destination is described. A redesign can improve one local signpost while weakening another.
| Structural property | Tree A | Tree B |
|---|---|---|
| Common destination IDs | 36 | 36 |
| First-level choices | 10 | 11 |
| Total nodes, including Settings root | 46 | 47 |
| Destination depths: 1 / 2 / 3 edges | 1 / 35 / 0 | 2 / 33 / 1 |
| Mean depth across 36 destinations | 1.972 | 1.972 |
| Mean depth across 12 tasks | 2.000 | 2.000 |
| Largest sampled first-level destination load | 8 (General) | 6 (Projects & Repositories) |
| Exact duplicate-label groups | 1 | 1 |
Neither average depth nor the count of repeated labels resolves navigation quality. They measure properties of the structures, not how people interpret them. The real research value comes from inspecting the tasks where relocation changes context, especially when a label suggests one kind of ownership and the destination performs another. This is where an information architecture audit becomes a good input to tree testing instead of a substitute for it.
Task-Based Navigation Audit: Twelve Jobs an Administrator Might Need to Do
The test scenarios were written in natural American English for a full instance administrator. Each scenario specifies one bounded outcome, such as enforcing a second factor for every user or allowing one internal hostname for webhook egress. Correct destination IDs were fixed across both trees. The task wording does not merely repeat the full breadcrumb trail, although some necessary product terms can still cue one branch more strongly than another. The audit reports those literal lexical effects separately from inferred user comprehension.
| Task | Tree A canonical route | Tree B canonical route | Depth |
|---|---|---|---|
| T01 — Require two-factor authentication | General → Sign-in restrictions | Identity & Access → Sign-in restrictions | 2 → 2 |
| T02 — Disable self-registration | General → New user account restrictions | Identity & Access → New user account restrictions | 2 → 2 |
| T03 — Default new projects to private | General → Visibility and access controls | Identity & Access → Visibility and access controls | 2 → 2 |
| T04 — Allow project archive imports | General → Import and export settings | Projects & Repositories → Import and export settings | 2 → 2 |
| T05 — Configure automatic pipeline defaults | CI/CD → Continuous Integration and Deployment | Same route | 2 → 2 |
| T06 — Restrict runner creation | CI/CD → Runners | Same route | 2 → 2 |
| T07 — Change the shared Jira URL | Integrations → Jira issues | Same route | 2 → 2 |
| T08 — Allow one internal webhook host | Network → Outbound requests | Integrations → Webhook connectivity → Outbound requests | 2 → 3 |
| T09 — Limit anonymous API calls | Network → User and IP rate limits | Traffic & Request Limits → User and IP rate limits | 2 → 2 |
| T10 — Disable automatic telemetry sharing | Metrics and profiling → Usage statistics | Monitoring & Telemetry → Usage statistics | 2 → 2 |
| T11 — Configure periodic repository checks | Repository → Repository maintenance | Projects & Repositories → Repository maintenance | 2 → 2 |
| T12 — Stop account-deactivation messages | Preferences → Email | Email notifications | 2 → 1 |

Ten tasks retain the same modeled depth, T12 becomes one level shorter, and T08 becomes one level longer. Three tasks change first-level node ID; nine have different displayed path text because relabeling can change the words in a breadcrumb without moving its underlying destination. There are five direct-parent changes among all 36 destinations. The mean task depth remains exactly two in each model. Every one of these statements is a structural result, not an observed result of tree testing with participants.
Why path length is not time on task
For T12, the proposal places Email notifications directly under Settings instead of under Preferences. The modeled path is shorter, but that does not tell us whether an administrator would recognize the target more quickly. The original Email section includes settings beyond notification messages, including a private commit email hostname. A new parent label can make one task sound more obvious while making other responsibilities less predictable. A shorter path with a narrower label is an experiment hypothesis, not a demonstrated time saving.
T08 exposes the reverse trade-off. Moving Outbound requests beneath Integrations and a new Webhook connectivity subgroup makes the canonical path longer. That subgroup has only one modeled child, so it adds an intermediate choice without distinguishing between competing destination leaves in the sample. A support engineer diagnosing a broken webhook may like the new context; an instance administrator applying an outbound security rule may expect the policy under Network. The extra level is measured. Which context users understand is not.
What the Information Architecture Testing Revealed About Labels and Ownership
Identity is more explicit, but project privacy still crosses concepts
Changing General to Identity & Access gives T01 and T02 a clearer semantic neighborhood. Requiring two-factor authentication and preventing open self-registration fit that label more naturally than they fit a generic category name. I would consider those plausible improvements worth testing, while recognizing that experienced administrators might already know the established General location. A familiar route has practical value even when the label is not descriptive.
T03 is the counterexample that prevents a clean victory. It asks where to set new project visibility to private by default. In Tree B, the destination remains under Identity & Access while Projects & Repositories becomes another prominent root choice. The phrase “projects” may pull attention toward the wrong category. The structural script identifies an exact word overlap with that competing root; it does not establish that real people would choose it. I would not claim that the identity relabel improved all access-related tasks without measuring these two plausible interpretations.
Putting two Jira controls together does not merge their purposes
In Tree A, the GitLab for Jira App configuration sits under General while Jira issues sits under Integrations. Tree B relocates the app under Integrations, alongside Jira issues. The thematic grouping is understandable: both involve Jira. But one relates to the development-panel app’s connection details, while the other configures the shared Jira issues integration used by inheriting projects or groups. Treating the same vendor name as sufficient navigation guidance risks hiding an important distinction in what the settings control.
The T07 route to Jira issues is unchanged in both models, but the neighboring options change. That is why I would measure which Jira label administrators select first and whether they can distinguish the two functions from the task wording. The audit’s token-overlap detector reports the shared word, but no participant confusion was observed. Sometimes consolidating related settings improves recognition; sometimes it simply moves the difficult choice one step deeper. The correct answer depends on task behavior, not the tidiness of the category diagram.
Webhook egress makes a security policy look like an integration feature
The most consequential trade-off in this sample is T08. The task concerns permission for one internal hostname while keeping other internal hosts blocked. In the source-backed baseline, that policy lives under Network > Outbound requests; GitLab’s tagged webhook security documentation provides the relevant host-access context. The proposal moves the destination into Integrations > Webhook connectivity > Outbound requests.
A feature-oriented interpretation says a webhook setting should be near other integrations. A policy-oriented interpretation says a host allowlist affects the instance’s outbound request boundary and belongs under network controls. Both are plausible, but their consequences are not symmetric. If an administrator misunderstands an egress restriction, the cost of a wrong decision could be more serious than taking an extra moment to find an email preference. The research did not measure incidents, error severity, or user intent, so I would not assign invented risk weights. I would make T08 a guardrail in the next tree testing study with participants and avoid treating an aggregate navigation score as permission to weaken the distinction.
A broader repository category can move friction rather than remove it
Moving Import and export settings from General to Projects & Repositories creates a more obvious task-and-destination association for T04 in the model. Yet the broader category gains additional options, increasing immediate choices from four to six. T11, locating periodic repository maintenance, receives no shorter path and now competes with more neighboring controls. An information architecture decision needs to examine both the task receiving a new home and the tasks already living there.
This is one reason I resist single-metric design arguments. A move can improve semantic alignment for a migration task while increasing local choice complexity for a maintenance task. Neither the path-depth count nor a shared-word diagnostic proves that administrators are helped or harmed. The proposed category is worth comparing in tree testing, not automatically adopting.
Navigation Tree Sensitivity Checks: Rename, Regroup, or Add a Level?
The tree testing preparation did not stop at comparing complete A and B packages. It also created two diagnostic hybrid variants: A’s topology with B’s shared category names, and B’s topology with A’s shared category names. These variants do not represent human test arms and do not isolate a causal treatment effect. They help identify which purely structural observations come from moving destinations and which literal task-to-category word matches depend on naming.
When A retains its old placements but takes B’s broader project label, an import task still lives under the renamed identity branch while the word “projects” cues a competitor. With B’s placement and A’s older labels, the import destination stays in Repository and the webhook rule still adds a level. Email promotion changes structural depth regardless of a category name. These examples show that tree testing should separate a question about labeling from a question about grouping wherever the next tree testing experiment can do so without losing realistic task context.
A simple exact-token screen counted one correct task-to-root match in A and three in B. It also matched an incorrect B root for T03 and T07. This metric knows nothing about synonyms, plural forms, acronyms, learned conventions, or user intent. It even misses apparently relevant labels when the task uses different vocabulary. I treat it as a quality-control prompt to review wording, never as an information-scent score or an estimate of first-click success.
Novice expectations and experienced administrator habits may disagree
A new administrator might prefer Identity & Access or Traffic & Request Limits because they describe tasks in plain language. An experienced GitLab administrator might reach General or Network without hesitation because the route is learned. Neither group was observed here. The practical implication is to recruit relevant experience strata rather than assume that labels which sound clearer to an analyst also work better for the people who maintain a system every day.
Task criticality is not another invented ranking weight
Access control, outbound host permissions, migration, monitoring, and email preferences may impose different operational consequences. But this study did not observe usage frequency, incident severity, or actual task completion. I would ask the product owner to define what an unacceptable regression means before tree testing participant enrollment. For example, the next study could flag a meaningful reduction in correct egress-policy destinations as a reason not to ship B, even if some lower-risk tasks improve. That is a design guardrail, not a post-hoc statistical threshold fabricated from this structural dataset.
What Earlier Human Tree Testing Studies Contribute, and What They Do Not
The source review found a published 2021 GitLab.com navigation case study describing separate human studies for group navigation, project navigation, and project-level settings. The reported groups had 30, 52, and 70 participants respectively. The project-settings findings include examples of people considering adjacent feature areas when searching for a setting. This is valuable context for designing competing-category tasks, but it is not a participant dataset for the instance-admin trees in this article. The version, permission scope, navigation model, task wording, and available traces differ. I do not merge those counts with UX-08 or claim they show how current instance administrators would navigate. The source is the published GitLab navigation case study by Katherine Okpara.
A different study by Kuric, Demcak, and Krajcovic compared ways of presenting an online magazine information architecture. The accompanying research repository documents dataset structure and licensing. That study is relevant because a tree’s presentation can influence measured choices even when its hierarchy remains the same. It does not validate either enterprise navigation proposal here. The external records were not redistributed or used as participant rows for this research, and their CC BY-NC repository license was treated as a separate reuse consideration.
This is the part of research synthesis most easily mishandled. A genuine human tree-testing study can support a useful methodological caution without supplying a success rate for a different product and task. More sources do not automatically turn incompatible datasets into one experiment. I would rather retain a narrow evidence claim than decorate an unobserved enterprise navigation outcome with an unrelated participant number.
From Structural Information Architecture Testing to a Real Participant Study
The research package includes a standalone local HTML tree-testing instrument, a JSON collection schema, a consent-aware protocol, and separate frozen correct destinations. The instrument allows a facilitator to assign Tree A or B and one of two six-task panels. It records the participant’s actual menu choices, returns, submitted destination, skipped tasks, and relative elapsed times only after a genuine session. No external scripts, automatic uploads, or background storage are required by the delivered instrument. Its internal logic passed offline tests, but a full tree testing browser, keyboard, and assistive-technology pilot was not completed; that is a pre-enrollment requirement, not a detail to ignore.
Before a human tree testing comparison, I would recruit adults with a legitimate administrative responsibility for a self-managed source-control or DevOps platform and distinguish experienced GitLab instance administrators from relevant newcomers. Each participant should see only one tree condition. Assignment should be balanced within experience groups and the two task panels; tasks can be presented in randomized order within a participant’s assigned panel. Consent, personal-contact records, and exported pseudonymous navigation traces must have separate access and retention controls.
The primary outcome in the proposed protocol is a task-balanced comparison of accepted final destinations, with success assessed separately for all twelve tasks. First choices, direct success, backtracking, and elapsed time are secondary observations. A submitted wrong destination is not rescued by the fact that the person briefly visited the correct area before leaving it. Skipped, incomplete, and missing trials are handled explicitly rather than silently removed from a denominator. The protocol defines a future study; it does not establish any of those outcomes today.
What would count as evidence in an actual tree test?
| Question | What a participant trial records | What this audit knows |
|---|---|---|
| Can the person find the correct destination? | Submitted final destination and task-specific success | The frozen correct destination only |
| Where do they look first? | First selected root category | Category labels and canonical path |
| Did they take a direct route? | Actual sequence, revisions, and backtracking | One canonical route with no observed detours |
| How long did the task take? | Measured duration under defined timing rules | Number of modeled edges, not time |
| Which menu is preferable? | Comparable results with appropriate uncertainty and risk guards | Competing hypotheses; no proven winner |
A future quantitative tree testing comparison would also need an enrollment and stopping rule that is independent of which design happens to be ahead halfway through recruitment. Repeated tasks from the same person are not independent observations. Participant-level resampling or an appropriately specified analysis must preserve that dependency. The research protocol provides an illustrative planning precision calculation, but it is not a completed sample-size justification for every organization or an instruction to recruit a fixed number without regard to feasibility and consequences.
Another design detail is easy to miss: if one tree-testing interface expands labels differently, displays path history differently, or exposes more visual feedback, the comparison is no longer only about labels and grouping. The future instrument uses the same basic presentation for both models, drawing on evidence that presentation differences can affect navigation results. Even so, the proposed A/B treatment bundles label, order, grouping, and promotion changes. It can test the package’s behavior, not identify the causal contribution of each individual change without a subsequent targeted design.
What I Would Recommend to an Enterprise UX Team
My first recommendation is to resist the temptation to label this result a redesign win. Ten identical task depths and one advantage offset by one disadvantage are not a success score. In the structural record, B is neither uniformly simpler nor uniformly more complex. It is a different allocation of choices and terminology. The question worth testing is whether the new ownership model helps administrators choose the correct destination under realistic constraints.
I would prioritize the identity and request-limit labels for focused participant checks, while treating project visibility as a counterexample that could expose a hidden cost. I would test the relocated import/export destination alongside repository maintenance, because one branch gains a task and another inherits additional choices. I would explicitly test both Jira configurations to see whether their adjacency aids recognition or creates a near-duplicate choice. And I would preserve outbound host policy as a safety-sensitive guardrail rather than allowing an average success figure to hide a regression.
I would also refuse to confuse research execution with organizational permission. A structural audit may justify testing a label. A participant study may justify considering a new route. A permission-sensitive setting may still require implementation review, documentation updates, migration paths, administrator training, and rollback planning. Findings inform those decisions; they do not automatically authorize them.
A practical navigation decision record
For every proposed move, I would keep a short record with the existing destination ID and path, the alternative path, the task it is intended to help, the competing task it might harm, the kind of evidence available, and the observation that would change the decision. The final field is essential. If no result could persuade a team to retain the current location, the so-called experiment is only a presentation for an already chosen redesign.
| Proposed change | Possible benefit | Counterrisk | Next evidence |
|---|---|---|---|
| Identity & Access naming | Explicit sign-in and account ownership | Project visibility competes with Projects & Repositories | First-choice and final-destination outcomes for T01–T03 |
| Project import relocation | Closer conceptual fit for migration | More neighboring repository options | Compare T04 and T11 paths |
| Jira controls together | Shared integration neighborhood | Two different Jira functions become adjacent | Confusion between the two eligible destinations for T07 |
| Webhook connectivity subgroup | Feature-oriented discoverability hypothesis | Longer route and weaker network-policy framing | T08 correctness and misrouting guardrail |
| Direct Email notifications | One less modeled level for T12 | Narrow label for a broader existing Email section | T12 plus additional non-notification email tasks |
The emphasis on competing evidence also applies when a measured change looks favorable. If a shorter email path improves observed task success, I would still check the section’s other responsibilities before treating that promotion as complete. If the new webhook grouping generates fewer wrong first choices, I would examine whether administrators can still recognize that they are editing an instance-wide outbound-access policy. Clear navigation must communicate the right scope, not merely guide a click toward a reachable leaf.
Reproducibility, Source Integrity, and Threats to Validity
The audit can be recreated with the frozen Tree A and Tree B JSON files, the task CSV, the freeze manifest, the analyst walkthrough, and the Python standard-library script. The supplied independent verifier uses a separate breadth-first traversal rather than relying solely on the audit’s parent-path algorithm. It checks all 72 condition-to-destination routes, the 36 common IDs, the two depth distributions, and the T08/T12 deltas. It also confirms that malformed trees are rejected and that the result files can be regenerated byte for byte.
The verification distinguishes executable software QA from research evidence about people. The offline Node.js tests exercise the instrument’s consent gating, menu navigation, backtracking, task submission, skipping, stopping, and local export shape. They do not create participant traces. The research log also records one source locator whose requested line endpoint exceeded the tagged file; the corrected source range was documented without changing the frozen input or computed results. This is how a reproducible case study should handle an identified source-reference issue: preserve the original record and attach a narrow correction.
Several limitations materially affect interpretation. The lower-level menu inventory was sampled, so branch loads are sample-specific. The underlying code and documentation do not prove how an actual configured installation looks under every feature flag. The model blends page navigation and sections as equivalent edges. Task prompts and alternatives came from the same research process, and some wording cues the proposed labels. The analyst walkthrough was model-assisted and unblinded. The structural script does not understand synonyms or organizational context. No independent human reviewer established inter-rater agreement for the subjective interpretations.
Most importantly, this research cannot estimate a population’s mental model, task success rate, first-choice distribution, path directness, time to completion, satisfaction, productivity, or preference for one tree. Nor can it establish that the proposed menu would work for paid editions, a different product version, a project Maintainer, or ordinary GitLab.com users. The evidence is still useful: it shows exactly what was proposed, where it changes the structure, why selected moves are contestable, and how to prepare a legitimate next tree testing study.
Frequently Asked Questions About Tree Testing Enterprise Navigation
Is a structural navigation audit the same as tree testing?
No. A structural audit can verify hierarchy validity, target preservation, category depth, duplicate labels, and route changes. Tree testing adds participants who attempt real tasks using the hierarchy and generate observed choices. In UX-08, the structural experiment was completed and a future participant instrument was prepared, but there were no participant sessions. Calling its path counts a user success rate would be a methodological error.
Does the shorter navigation path always perform better?
No. A shorter route can have a less recognizable label, more competing top-level choices, or an inappropriate scope. Conversely, an added level may clarify category ownership if the intermediate choice is meaningful. Here the email route shortened and the webhook route lengthened; neither fact establishes actual time savings or confusion. The next tree testing study should measure destination accuracy and detours instead of treating depth as a proxy for usability.
Can AI agents replace participants in tree testing?
Not for claims about human expectations or navigation behavior. An AI-assisted reviewer can flag duplicate terms, hypothesize ambiguities, or help audit a navigation structure, but its simulated choices are not participant traces. The UX-08 walkthrough is therefore classified as inference. Real tree testing requires appropriate human participation and observation for claims about first choices, success, and backtracking.
What should an enterprise navigation study measure beyond success?
At minimum, distinguish the submitted final destination, first selected category, direct versus indirect paths, returns to earlier categories, and elapsed time under clear rules. For consequential administrative controls, also assess whether a wrong destination could imply a misunderstanding of permission scope or policy ownership. Task-by-task results matter because an aggregate improvement can coexist with a regression in a critical workflow. Each measure needs a valid denominator and documented exclusions.
Can published GitLab navigation research validate this proposed tree?
No. Earlier published studies can inform task design and identify plausible feature-area versus settings confusion. They do not measure the 18.11.0 instance-admin hierarchy or the newly proposed tree unless the people, tasks, conditions, and recorded paths actually match. The contextual GitLab.com and online-magazine studies in this report are cited as separate evidence, not merged into a fictitious A/B result.
What should happen after the structural experiment?
Conduct a consented pilot of the local instrument, verify keyboard and assistive-technology behavior in real browsers, refine ambiguous tasks in a new documented version if needed, and freeze the participant study before enrollment. Assign comparable participants to one tree, preserve actual interaction paths and missingness, then analyze results with task and risk boundaries intact. That would allow a subsequent article to report genuine human tree testing findings without retroactively treating this structural analysis as user evidence.
Conclusion: A Better Menu Is a Hypothesis Until People Navigate It
In this case, the engineering and information architecture checks did what they should: they confirmed that two valid trees share the same 36 destinations, described every consequential move, and exposed trade-offs hidden by a casual glance at the labels. The alternative made some intentions more explicit while widening the root, moving a network policy into an integration context, and preserving a real duplicate label. Its average modeled route was no shorter. None of those results reveals what users would actually choose.
My conclusion is to test selected changes instead of presenting the redesign as proven. Tree testing is most useful when it can contradict a designer’s expectation, including mine. The quality of the decision depends on preserving the distinction between the structure we can calculate, the interpretation we can defend, and the behavior we still need to observe. An honest, reproducible tree testing preparation exercise can end with a sharper research question rather than a victory banner.
Related Reading and Further Work
For the evidence boundaries behind this case study, see my Small UX Research Samples: What They Can and Cannot Tell You and UX Research Triangulation: When Findings Conflict. The adjacent decision problem, including how research should influence authority without claiming more than it shows, is explored in When Stakeholder Certainty Overrules User Research. Browse the UX & UI Design articles collection for the wider research and design series.
My public website and project portfolio document related work across software engineering, product systems, and UX. For this study I examined publicly available GitLab material rather than a private RE Workspace installation; the research should not be confused with a reported usability test of one of my products.
Author’s Note and Follow the Work
I approach information architecture as a product and engineering decision, not a menu-cleanup exercise. Names, hierarchy, permission boundaries, established habits, and test evidence belong in the same discussion, but they cannot be collapsed into one tidy score. I would rather publish the limits of a real structural study than imply an unperformed participant experiment succeeded. That standard is especially important when a navigation decision can affect an administrator’s ability to find security-sensitive controls.
For new research notes, code, and practical work, follow GitHub, LinkedIn, and X. My website also collects selected articles and product work. The next step for this particular case is a real, permissioned tree testing participant study; until that happens, the findings remain a documented navigation-structure comparison.




