Testing¶

Summary: Project Eternal uses UE5's Automation Framework with BDD-style DEFINE_SPEC tests. Unit tests run synchronously with manual mocks, integration tests use LatentIt against a real backend. Shared helpers in TestHelpers.h create pre-configured items and save data. Tests live under Private/Tests/ and are stripped from packaged builds.

Table of Contents¶

Framework
Test Structure
Writing Tests
Shared Helpers
Mocking
Integration Tests
Proving Guard Tests Are Real (Mutation Testing)
Running Tests
Source References
Recent Changes

Framework¶

BDD-Style with DEFINE_SPEC¶

All tests use DEFINE_SPEC (Describe/It/BeforeEach/AfterEach) rather than IMPLEMENT_SIMPLE_AUTOMATION_TEST. This matches the BDD style already used by the ProceduralDungeon plugin in this project.

DEFINE_SPEC(FMyFeatureSpec,
    "ProjectEternal.Unit.MyFeature",
    EAutomationTestFlags_ApplicationContextMask | EAutomationTestFlags::ProductFilter)

void FMyFeatureSpec::Define()
{
    Describe("Feature behavior", [this]()
    {
        It("should do expected thing", [this]()
        {
            TestEqual("Value", Actual, Expected);
        });
    });
}

UE 5.8 Flag¶

Use EAutomationTestFlags_ApplicationContextMask (underscore, not ::) — this changed in UE 5.5.

Test Filter Naming¶

Pattern	Scope
`ProjectEternal.Unit.*`	Unit tests (no external dependencies)
`ProjectEternal.Persistence.*`	Persistence specs (local file I/O + mock remote provider)
`ProjectEternal.Content.*`	Content-shaped suites over real assets (e.g. `Content.Dungeon.GenerationRegression`, ailment/chain tuning)
`ProjectEternal.Editor.*`	Editor-module specs (`ProjectEternalEditor`): validation cores, stamp/export tooling
`ProjectEternal.Integration.*`	Integration tests (requires running server)

CI runs Persistence+Unit+Content+Editor in one pass and Integration separately (pr-validation.yml); a filter that matches zero tests fails the step rather than passing silently.

Test Structure¶

Runtime specs live under Source/ProjectEternal/Private/Tests/, one folder per system (file names are <Feature>.spec.cpp; replication functional tests are *Test.cpp/.h pairs):

Source/ProjectEternal/Private/Tests/
├── TestHelpers.h            # Shared factory functions
├── TestItemGridWidget.h     # UI test widget
├── Mocks/                   # MockPersistenceApiService, MockItemRegistry
├── Abilities/  Audio/  Automap/  Balance/  Combat/  Dungeon/  Equipment/
├── Inventory/  Itemization/  Persistence/  Replication/  Session/  UI/  Vendor/
└── *.spec.cpp               # a few loose cross-system specs at the root
                             #   (CombatEffectLayers, Compendium*, GreedSystem, QuestTagCount,
                             #    RemnantItems, ScreenEffects, TooltipFormat)

Combat/ is by far the largest folder (dozens of specs); Replication/ holds the functional replication tests plus their unit-level *.spec.cpp siblings. Functional-test headers live in Public/Tests/.

Editor-only specs live in the editor module, Source/ProjectEternalEditor/Private/Tests/ (18 .spec.cpp files as of 2026-08: validation cores — ability, kit, domain, itemization, manifest, montage table, narrative — plus stamp/export tooling, audio audit and Build Lab GC). They use the ProjectEternal.Editor.* filter and run in CI alongside the unit filters.

For current counts, list the folders rather than trusting a table here — the per-spec inventory this section used to carry went stale within weeks.

Writing Tests¶

Naming Convention¶

Spec class: F{System}{Feature}Spec Filter: "ProjectEternal.{Unit|Integration}.{System}.{Feature}"

UObject Lifetime¶

Tests create UObjects outside of a normal game world. Use TStrongObjectPtr to prevent garbage collection:

TStrongObjectPtr<UMyObject> Obj(NewObject<UMyObject>());
// Obj stays alive until TStrongObjectPtr goes out of scope

Shared State Across Lambdas¶

Describe/BeforeEach/It blocks are lambdas. To share UObject-owning pointers between them, wrap in TSharedRef:

Describe("SaveAndLoad", [this]()
{
    auto Provider = MakeShared<TStrongObjectPtr<ULocalPersistenceProvider>>();

    BeforeEach([this, Provider]()
    {
        *Provider = TStrongObjectPtr(NewObject<ULocalPersistenceProvider>());
    });

    It("should save data", [this, Provider]()
    {
        (*Provider)->SaveCharacter(Id, Data, Callback);
    });
});

Cleanup¶

Use AfterEach when tests create side effects (files, state):

AfterEach([this, CharacterId]()
{
    // Delete test save file
    IFileManager::Get().Delete(*GetSaveFilePath(*CharacterId));
});

Known Limitations¶

ensure() assertions do not suppress cleanly with AddExpectedError. Avoid writing tests that intentionally trigger ensure() failure paths. If a function guards a null path with ensure(), skip that edge case test and document why.

Shared Helpers¶

Private/Tests/TestHelpers.h provides inline factory functions in the EternalTestHelpers namespace:

Function	Returns	Fragment
`CreateTestBasicItem`	`UItemObject*`	None (minimal item)
`CreateTestEquipmentItem`	`UItemObject*`	`FEquipmentFragment` with CritChance prefix + BaseDamage implicit
`CreateTestConsumableItem`	`UItemObject*`	`FConsumableFragment` with configurable usages
`CreateTestStackableItem`	`UItemObject*`	`FStackableFragment` with configurable stack count
`BuildTestModifiers`	`FEquipmentModifiers`	One entry in each of 4 arrays (implicits, prefixes, suffixes, scaling)
`CreateTestStonePlateItem`	`UItemObject*`	`FStonePlateFragment` with 4x4 plate layout
`CreateTestCharacterSaveData`	`FCharacterSaveData`	Fully populated: items, inventory slots, equipment slots, quests, world map, exploration

All helpers use FGameplayTag::RequestGameplayTag(Name, false) for safe tag lookup during CDO construction.

Footgun: Mutate the Default Fragment, Don't Append a Second¶

FItemManifest's constructor already seeds a default FGridFragment, and GetFragmentOfType<T>() returns the first matching fragment. When a helper needs to set an item's grid footprint (e.g. CreateTestSizedItem), it must mutate the existing default via GetFragmentOfTypeMutable<FGridFragment>() — do not append a second FGridFragment. A footprint set on an appended fragment is silently ignored because lookups return the seeded first one. This exact bug made a "2x4" item secretly resolve as 1x1.

Do	Don't
`Manifest.GetFragmentOfTypeMutable<FGridFragment>()->Dimensions = Size;`	`Manifest.Fragments.Add(FGridFragment{...});` (ignored)

Related: merge match is on GetItemType() (a tag) — set it with SetItemType to a registered tag; empty tags never MatchesTagExact and will never merge.

Replication Functional Tests¶

Why Functional Tests?¶

Unit tests (DEFINE_SPEC) verify logic in isolation. Replication tests verify that state changes on the server propagate to clients correctly. These require a real multiplayer PIE session (server world + client world in one process); they DO run under -NullRHI through Tools/run_functional_tests.ps1 (see Running below).

Which layer tests what¶

Layer	Runs where	Proves	Use for
Unit spec (`*.spec.cpp`)	Headless, `Tools/run_tests.ps1`, CI	Mechanics in isolation: damage/DoT math, tag contracts, charge state machine, container logic. Real ASCs + probe abilities, no mocks.	Any rule expressible without a world or a second machine.
Replication functional test (`*ReplicationTest`)	PIE with a client world, `Tools/run_functional_tests.ps1`	Server→client propagation: OnRep-driven state, subobject/manifest replication, tag replication.	Anything a client must see (charges, equipment, status tags, quest state).
Live PIE cheat harness (`EternalCheatToolset` over MCP, `.claude/skills/pie-test`)	Agent- or owner-driven PIE, exploratory	Emergent behaviour: montages, windows, VFX, kits, feel. Not repeatable by itself.	Verifying a change end-to-end once; promote the check to a functional test when it becomes regression-worthy.

Cheats are the fixture library of the live layer, the same role a test rig's NewObject plays for specs. Contract for a new cheat: log the reason on refusal, forward authority-needing cheats to the server (ForwardCheatToServer), return structured data where an assertion needs it (GetInventory), and expose it as an EternalCheatToolset tool so an agent can call it directly. Don't build UI for cheats.

Architecture¶

┌──────────────────────────────┐
│  AReplicationTestBase        │
│  (extends AFunctionalTest)   │
│  • World/player accessors    │
│  • WaitForReplication()      │
│  • bEnabled toggle           │
│  • Data asset item creation  │
└──────────────┬───────────────┘
               │ inherits
    ┌──────────┴──────────┐
    │ AEquipItemTest      │
    │ AQuestStartTest     │
    │ ACombatHitTest      │
    │ ... (20 test actors)│
    └─────────────────────┘

How They Work¶

Test actors are placed in ReplicationTestMap with AReplicationTestGameMode
PIE runs as Listen Server + 2 players
Each test auto-starts on the server after a staggered delay
Server mutates state, waits N frames for replication, verifies on client
Results logged to Output Log ([ReplicationTest] TestName: PASSED/FAILED)

Test GameMode¶

AReplicationTestGameMode inherits AEternalGameMode but: - Skips persistence (deferred auto-load disabled via SetDisabled()) - Skips world map travel (calls AGameMode::HandleStartingNewPlayer directly) - Skips SpawnManager gating (calls AGameModeBase::SpawnDefaultPawnFor directly) - Sets CharacterClassInfo on PlayerState from parent's field

Data Asset Support¶

Item-based tests have UPROPERTY(EditAnywhere) fields for UItemManifestDataAsset. Assign real production items in the editor for maximum coverage. Synthetic fallback items are used if no data asset is assigned (for CI without editor setup).

Adding New Replication Tests¶

Create header in Public/Tests/ inheriting AReplicationTestBase
Create implementation in Private/Tests/Replication/
Override RunReplicationTest() — mutate on server, WaitForReplication(), verify on client
Place actor in ReplicationTestMap
Assign data assets if the test uses items

Running Replication Tests¶

Headless (agent / CI):

Tools\run_functional_tests.ps1 [-Dedicated] [-NoBuild] [-EnginePath <path>] [-Map <path>] [-TimeoutSec 300]
→ === PASS=n FAIL=n SKIPPED=n QUEUED=n (Listen|Dedicated) ===   exit 0 pass / 1 fail / 2 never finished

It runs Eternal.Test.RunReplicationSuite (editor console command, ProjectEternalEditor/Private/Tests/ReplicationSuiteRunner.cpp) under -nullrhi: PIEs the map with the topology the tests need, lets the BeginPlay queue run, ends PIE, exits with the verdict. -Dedicated = client + in-process dedicated server (the real MP topology; tests that need a listen host FinishTestSkipped there and are counted separately). Run both before claiming MP coverage.

Why not Automation RunTests Project.Functional Tests: the engine's automation map loader hardcodes one player and no separate server, so every replication test lands in a single world with no client. The base class now fails (not skips) in that case so the stock runner cannot go falsely green. Under automation the BeginPlay queue is also disabled (the framework starts each test itself).

Timing: WaitForReplication waits N frames AND at least MinReplicationWaitSeconds (0.35 s) — headless PIE runs at ~100 FPS, where 10 frames is shorter than a 10 Hz actor's net update.

In-editor: set Number of Players to 2, Net Mode to Listen Server (or Client + in-process server), press Play on ReplicationTestMap. Results in Output Log as [ReplicationTest] Name: PASSED/FAILED/SKIPPED plus a SUITE DONE line.

Mocking¶

Approach¶

Manual mocks via UCLASS inheritance — no external mocking framework. Mocks override virtual methods and expose captured state for assertions.

MockPersistenceApiService¶

Replaces real HTTP calls for URemotePersistenceProvider tests.

Configuration (set before test): - bShouldSucceed — control success/failure - MockStatusCode, MockErrorMessage — error details - bReturnMalformedJson — test deserialization error path

Captured state (inspect after test): - SaveCallCount, LoadCallCount — call counts - LastSaveCharacterId, LastSaveData — captured arguments

Realistic deserialization: On load, the mock serializes FCharacterSaveData to real JSON via FJsonObjectConverter, then deserializes into FApiResponse.Content. This exercises the same deserialization path as production code.

MockItemRegistry¶

Replaces asset-based item manifest lookup for CreateItemFromSaveState tests.

Registration helpers: RegisterBasicManifest, RegisterEquipmentManifest, RegisterConsumableManifest, RegisterStackableManifest — populate a TMap<FString, FItemManifest> instead of loading data assets.

Integration Tests¶

ServerPersistence.spec.cpp¶

Requires a running eternal-server at http://localhost:5065/ with PostgreSQL.

Key patterns: - Uses LatentIt with FDoneDelegate for async HTTP operations - Creates unique account + character per test to avoid collision - Wires up UEternalApiSubsystem with all three services against the real server - Verifies full HTTP save/load pipeline including JSON serialization through real network

Filter: ProjectEternal.Integration.Persistence

These tests are separated from unit tests so CI can run unit tests without a backend.

Proving Guard Tests Are Real (Mutation Testing)¶

A test that guards a behavior (a revert-on-fail, a count-0 rejection, a broadcast) is only meaningful if it can actually fail when that behavior breaks. A test that still passes after you sabotage the production code is tautological — it asserts nothing.

The bar: every guard test must be proven real by mutation before it is trusted.

Procedure¶

Break the guard — introduce a plausible mutation into the production code the test covers (e.g. delete the early-return that rejects a count of 0, remove the displaced-fit revert, skip the PostReplicatedChange broadcast).
Run the test — confirm it now FAILs. If it still passes, the assertion is not actually exercising the guard — fix the test.
Restore — revert the mutation and confirm the test passes again (PASS=n FAIL=0).

Guidance¶

Situation	Verdict
Test fails on a plausible mutation, passes when restored	Real — keep it
Test passes even with the guard removed	Tautological — rewrite until a mutation breaks it
Proving a broadcast fired	Bind a probe `UFUNCTION() void OnChanged(){ ++Count; }` via `AddDynamic` to the multicast and assert `Count == 1` — asserting "didn't crash" is not enough

This was applied to the itemization count-0 merge guard, the swap displaced-fit revert, and the PostReplicatedChange broadcast.

Running Tests¶

CI (Automated)¶

Unit and mock-based tests run automatically in PR validation. See CI/CD Pipeline for details.

CLI (Headless)¶

# Run all unit/mock tests (same filter used by CI)
UnrealEditor-Cmd.exe <project.uproject> \
    -ExecCmds="Automation RunTests ProjectEternal.Persistence;Quit" \
    -NullRHI -unattended

# Run specific test group
UnrealEditor-Cmd.exe <project.uproject> \
    -ExecCmds="Automation RunTests ProjectEternal.Persistence.Types;Quit" \
    -NullRHI -unattended

# Run integration tests (requires eternal-server running)
UnrealEditor-Cmd.exe <project.uproject> \
    -ExecCmds="Automation RunTests ProjectEternal.Integration;Quit" \
    -NullRHI -unattended

CLI (Headless Runner Script)¶

The wrapper script builds (closing the editor first), runs the requested filter headless, and prints a PASS=n FAIL=n summary parsed from LogAutomationController results — the preferred way to run tests from the terminal.

powershell -File Tools/run_tests.ps1 -Filter <name> [-NoBuild] [-EnginePath <path>]

Flag	Purpose
`-Filter <name>`	Test filter, by prefix — e.g. `ProjectEternal.Unit.Inventory`
`-NoBuild`	Skip the build step and run the already-compiled binaries
`-EnginePath <path>`	Explicit engine root

Engine path resolution order: -EnginePath argument → UE_ENGINE_PATH environment variable → error if neither is set.

Live Coding blocks UBT — the build step cannot run while the editor holds a Live Coding session. Close the editor before running with a build (force-kill is acceptable), or pass -NoBuild to run existing binaries.

Note: the runner lives at Tools/run_tests.ps1 (moved from Saved/).

A sibling runner, Tools/run_validation.ps1, drives the content-validation commandlet with the same flag/engine-resolution conventions — see Content Validation. Content-shaped test suites also exist: ProjectEternal.Content.Dungeon.GenerationRegression runs every UDomainDungeonConfig × 8 fixed seeds (topology mapping, reachability, same-seed determinism) via the normal test runner.

Editor¶

Session Frontend > Automation tab > filter by ProjectEternal.

Adding New Test Files¶

Create Private/Tests/{System}/{Feature}.spec.cpp
Use DEFINE_SPEC with appropriate filter path
Add shared helpers to TestHelpers.h if reusable
Add mocks to Private/Tests/Mocks/ if needed
No build system changes needed — UE discovers spec files automatically

Source References¶

File	Purpose
`Private/Tests/TestHelpers.h`	Shared item/data creation helpers
`Private/Tests/Mocks/MockPersistenceApiService.h`	Mock API service for remote provider tests
`Private/Tests/Mocks/MockItemRegistry.h`	Mock item registry for reconstruction tests
`Private/Tests/Persistence/*.spec.cpp`	All persistence test specs
`Public/Tests/ReplicationTestBase.h`	Functional test base class for replication tests
`Public/Tests/ReplicationTestGameMode.h`	Test GameMode for multiplayer PIE
`Private/Tests/Replication/*.cpp`	All replication test implementations
`Content/Maps/FunctionalTests/ReplicationTestMap`	Test map with placed test actors

C++ Style Guide — Code conventions
Content Validation — Save/push/PR gates for data assets; validation cores are spec-tested
CI/CD Pipeline — Automated test execution in PR validation
API Layer — Backend communication (tested by integration tests)
Replication Overview — Replication concepts tested by functional tests
Persistence architecture — ImplementationDocs/archive/PersistenceLayer.md

Recent Changes¶

Date	Change	Reason
2026-08-20	Replaced the stale per-file tree + per-spec count tables with a directory-level map; documented the editor test module and the `Persistence`/`Content`/`Editor` filters CI runs	File-level inventories rot; the filter table now matches `pr-validation.yml`
2026-07-08	Point to `run_validation.ps1` + note `Content.Dungeon.GenerationRegression` suite	Validation spine + B-8 regression suite shipped
2026-07-03	Add mutation-testing bar, headless runner (`Tools/run_tests.ps1`) CLI, TestHelpers default-fragment footgun	Promote automation-test patterns from session learnings
2026-03-14	Add replication functional test documentation	20 functional tests covering 11 replicated systems
2026-03	Initial documentation	Document testing patterns established with persistence layer