spine-runtimes/todos/done/2025-01-11-03-02-54-test-suite.md

create a folder test/ and write a comprehensive test suite

Status: In Progress Created: 2025-01-11T03:02:54 Started: 2025-01-11T03:11:22 Agent PID: 15570

CRITICAL:

• NEVER never check a chceckbox and move on to the next checkbox unless the user has confirmed completion of the current checkbox!
• NEVER modify code if there's not a checkbox that instructs you to do so!

Original Todo

• create a folder test/ and write a comprehensive test suite
  • For each core runtime
    • Write a program that takes as input a skeleton (.json or .skel) and atlas file path and animation name
    • Loads a SkeletonData and outputs EVERYTHING in a simple, diffable text format
    • Creates a skeleton from the SkeletonData, an AnimationState, sets the Animation on track 0, updates and applies the state to the skeleton, then outputs EVERYTHING in a diffable text format
    • The best approach is likely to create a Printer interface that can print each relevant type in the diffable format, with a specific indentation level, so the output represents the hierarchy of the data
    • See docs/project-description.md for the core runtimes and their location
    • The test/ folder should have simple language specific build scripts that build/pull in the core runtime for that language, and create an exectuable program we can invoke
      • build.gradle for Java, directly pulling in the spine-libgdx project via settings.gradle
      • CMakeLists.txt for C and C++
      • package.json/tsconfig.json for Typescript
      • Let's ignore the other core runtimes for now
    • The programs must be headless, which means we need to ensure when loading the atlases, texture loading actually doesn't happen.
    • The goal is to be able to construct bash or nodejs test suites that can find errors in the non-reference runtimes quickly by comparing actually loaded and "applied" data between teh runtimes

Description

Create a comprehensive test suite that compares the output of all core Spine runtimes (Java, C++, C, TypeScript) to ensure consistency. The test suite will consist of headless command-line programs for each runtime that load skeleton data and output all internal state in a diffable text format. This will enable automated comparison testing between the reference implementation (spine-libgdx) and all ports. Each test program must compile directly against the runtime's source code, not pull published versions from package repositories.

The test programs will print both SkeletonData (setup pose/static data) and Skeleton (runtime pose after animation) states, including all nested types such as bones, slots, skins, attachments, constraints, and animations.

Updated Implementation Plan (DebugPrinter in each runtime)

• Remove test/ directory approach (deprecated)
• Implement DebugPrinter in spine-libgdx-tests:
  • Create DebugPrinter.java in spine-libgdx/spine-libgdx-tests/src
  • Add command line argument parsing for skeleton, atlas, animation
  • Create Printer class for outputting all types in hierarchical format
  • Ensure it can run headless (no GL requirements)
  • Merged Printer and HeadlessTextureLoader into single DebugPrinter.java file
  • Added gradle task runDebugPrinter for easy execution
• Implement DebugPrinter in spine-cpp:
  • Create spine-cpp/tests directory with DebugPrinter.cpp
  • Update spine-cpp/CMakeLists.txt with conditional target
  • Add Printer class matching Java output format
  • Ensure headless operation
• Implement DebugPrinter in spine-c:
  • Create spine-c/tests directory with debug-printer.c
  • Update spine-c/CMakeLists.txt with conditional target
  • Add printer functions matching output format
  • Ensure headless operation
  • Fix spine-c/codegen/src/ir-generator.ts to use .buffer() for string getters (currently returns address of temporary String object)
    • Run generator via: npx tsx spine-c/codegen/src/index.ts
    • This regenerates spine-c/src/generated/*.cpp files
    • Fixed by handling "const String &" (with space) in addition to "const String&"
    • Verified: String methods now properly return .buffer() values (version, hash, etc. display correctly)
  • Issues found by DebugPrinter comparison:
    • spine-c API missing file-based skeleton loader that sets name from filename
      • C++ has readSkeletonDataFile() which sets name, spine-c only exposes content-based loader
      • Result: SkeletonData name is empty in spine-c output
      • Fixed: Modified spine_skeleton_data_load_json/binary to accept path parameter and extract name
    • Coordinate system inconsistency: Java shows scaleY=1.0, C/C++ show scaleY=-1.0, use Bone::setYDown(false) to match Java
• Implement DebugPrinter in spine-ts/spine-core:
  • Create tests/DebugPrinter.ts
  • Update tsconfig.json to exclude tests/ so tests are not bundled
  • Add Printer class matching output format
  • Ensure it runs with npx tsx without build step
• Create test runner script (compare-with-reference-impl.ts):
  • Run each runtime's DebugPrinter with same inputs
  • Compare outputs and report differences
  • TypeScript script with shebang for direct execution
  • Automatically builds C/C++/Java if needed
  • Saves outputs to tests/output/ directory
  • Shows line-by-line differences when outputs don't match
• Make animation parameter optional in all DebugPrinters:
  • If animation not provided, call skeleton.setToSetupPose() instead
  • Update Java DebugPrinter
  • Update C++ DebugPrinter
  • Update C DebugPrinter
  • Update TypeScript DebugPrinter
  • Update compare-with-reference-impl.ts to handle optional animation
• Fix locale issues - all DebugPrinters should use English locale:
  • Java: Set Locale.US for number formatting
  • C++: Set locale to "C" or "en_US.UTF-8"
  • C: Set locale to "C" or "en_US.UTF-8"
  • TypeScript: Already uses period for decimals
• Improve buildCheck() to detect when rebuild needed:
  • Check if debug printer executable exists
  • Compare executable timestamp with source file timestamps
  • Rebuild if any source files are newer than executable
• Create tests/README.md documentation:
  • Explain purpose: comparing reference implementation to other runtimes
  • List DebugPrinter locations in each runtime
  • Document how to run individual debug printers
  • Document how to run compare-with-reference-impl.ts
• Automated test: All DebugPrinters produce identical output
  • Note: Minor expected differences remain:
    • time field: Java shows 0.016 after update, C/C++ show 0.0
    • TypeScript: minor precision differences, null vs "" for audioPath
    • These are implementation details, not bugs
• User test: Verify with multiple skeleton files

Notes

C++ Serializer Implementation Strategy (Option 4 - Hybrid Incremental)

• Started with manual port of basic structure from Java
• Plan: Extract patterns into templates for automation
• Benefits: Quick progress while learning challenges, builds toward automation

Phase 2: JSON Serializers and HeadlessTest Rename

Rename DebugPrinter to HeadlessTest

• Rename all DebugPrinter files to HeadlessTest:
  • Java: DebugPrinter.java → HeadlessTest.java
  • C++: DebugPrinter.cpp → HeadlessTest.cpp
  • C: debug-printer.c → headless-test.c
  • TypeScript: DebugPrinter.ts → HeadlessTest.ts
• Update VS Code launch configs to say "headless test ($runtime)":
  • spine-libgdx/.vscode/launch.json
  • spine-cpp/.vscode/launch.json
  • spine-c/.vscode/launch.json
  • spine-ts/.vscode/launch.json
• Rename tests/compare-with-reference-impl.ts to headless-test-runner.ts
• Update build files:
  • CMakeLists.txt for C++ (executable name: headless-test)
  • CMakeLists.txt for C (executable name: headless-test)
  • Gradle for Java (task name: runHeadlessTest, main class: HeadlessTest)
• Update tests/README.md with new names

Implement JSON Serializers in Core Runtimes

• Java (spine-libgdx):
  • Create SkeletonSerializer class in com.esotericsoftware.spine.utils
  • Implement serializeSkeletonData(SkeletonData, Writer/StringBuilder)
  • Implement serializeSkeleton(Skeleton, Writer/StringBuilder)
  • Implement serializeAnimationState(AnimationState, Writer/StringBuilder)
  • Add depth/verbosity options to control output
  • Handle circular references and limit nesting
  • Update HeadlessTest to use SkeletonSerializer
  • Review serializer with user:
    • Test with actual skeleton file to see output format
    • Add cycle detection to handle circular references (outputs "")
    • Verify it compiles and produces JSON output
  • Create comprehensive API analyzer tool:
    • Analyzer discovers all types accessible via SkeletonData, Skeleton, and AnimationState
    • For each type, enumerate all getters including inherited ones
    • Generate Java serializer from analysis data
    • Handle enums, abstract types, inner classes, and type parameters
    • Filter out test classes and non-source files
  • Work on SkeletonSerializer.java generation until it actually compiles.
  • Move Java files to correct location and update
    • Remove SkeletonSerializer.java from spine-libgdx project
    • Both files should be in spine-libgdx-tests project instead
  • Create JsonWriter implementations
    • Create JsonWriter.java in spine-libgdx-tests/src/com/esotericsoftware/spine/utils/
      • Use StringBuffer internally instead of Writer parameter
      • Add getString() method to return the built JSON string
      • No throws IOException declarations
  • Update Java serializer generator
    • Modify tests/generate-java-serializer.ts (see tests/README.md for details)
      • Output to spine-libgdx-tests/src/com/esotericsoftware/spine/utils/SkeletonSerializer.java
      • Ensure NO throws IOException declarations on methods
      • Methods return String instead of taking Writer parameter
      • JsonWriter instantiated without parameters (uses internal StringBuffer)
      • Removed JsonWriter inner class generation (using separate JsonWriter.java)
      • Methods use RuntimeException for error handling
  • Update HeadlessTest.java to use SkeletonSerializer.serializeXXX() and output to stdout
  • Optimize Java serializer generator to use @Null annotations to skip unnecessary null checks
    • Exploit that analysis-result.json already has @Null preserved in return types
    • Methods without @Null are guaranteed non-null, skip null checks for efficiency
    • Verified: getName() calls skip null checks, getSequence() calls include null checks
    • Implement exclusion system with single source of truth in tests/java-exclusions.txt
      • Format: type ClassName, method ClassName methodName(), field ClassName fieldName
      • analyze-java-api.ts loads exclusions and marks PropertyInfo.excluded = true/false
      • Java generator filters excluded types from instanceof chains and skips excluded properties
      • Subsequent generators (C++, C, TypeScript) transform already-filtered Java output
      • Current exclusions: SkeletonAttachment (type), TrackEntry.getListener() (method)
    • Filter excluded types from instanceof chains in abstract type handlers
    • Remove dead code (writeBlendMode) - analyze why it's generated in analyze-java-api.ts but is never called
      • Issue: Enums were getting dedicated write methods AND inline .name() serialization
      • Solution: Skip enum types when generating write methods since they're handled inline
      • Result: writeBlendMode and other enum write methods removed, cleaner code
    • Fix writeXXX() method signatures to take only 1 argument, not 2
      • writeXXX() methods should only take the object to serialize: writeAnimation(Animation obj)
      • NOT: writeAnimation(JsonWriter json, Animation obj)
      • JsonWriter should be accessed via internal instance field
      • This affects ALL writeXXX() methods in the serializer
• C++ (spine-cpp): DETERMINISTIC direct transformation of Java SkeletonSerializer
  • Create spine-cpp/tests/JsonWriter.h as direct port of Java JsonWriter
    • Header-only implementation (no .cpp file)
    • Use spine::String instead of StringBuffer (has append methods)
    • Direct method-for-method port from Java version
  • Create C++ serializer generator
    • Create tests/generate-cpp-serializer.ts that ports SkeletonSerializer.java to C++
      • Output to spine-cpp/tests/SkeletonSerializer.h (header-only, no .cpp)
      • Include <spine/spine.h> for all types (no individual headers)
      • Direct transformation of Java code to C++ with regex rules
      • All method implementations inline in the header
  • Update spine-cpp/tests/HeadlessTest.cpp to use SkeletonSerializer
  • Compile the generated code, derrive additional regex rules or fix inconsistent C++ API, repeat until user says stop
  • Fix writeXXX() method signatures to take only 1 argument, not 2
    • writeXXX() methods should only take the object to serialize: writeAnimation(Animation* obj)
    • NOT: writeAnimation(JsonWriter& json, Animation* obj)
    • JsonWriter should be accessed via internal instance field _json
    • This affects ALL writeXXX() methods in the serializer
  • Fix C++ generator issues and inconsistencies:
    • Remove hardcoded obj->getData() → &obj->getData() rule
    • Fix writeColor/writeTextureRegion: add a & and * version for each
    • Generate both pointer and reference versions for all class types:
      • Reference version has full implementation
      • Pointer version delegates to reference version: writeXXX(json, *obj)
      • likely a post-processing step after generating the reference only version of serializer: find all methods, add pointer version below.
    • Fix all tests/*.ts files to use __dirname instead of process.cwd() for file paths
      • Makes them work from any directory they're invoked from
      • Use path.resolve(__dirname, '..', 'relative-path') pattern
    • Fix obj.field access pattern to use obj->field in C++ (Java uses . for all objects, C++ uses -> for pointers)
      • Transform obj.r/g/b/a to obj->r/g/b/a etc
      • Apply to all field access patterns, not just Color
    • Fix C++ field access for underscore-prefixed fields
      • C++ private fields are prefixed with underscore (e.g. _offset, _to) but Java fields are not
      • Transform obj->field to obj->_field for specific known private fields
      • Example: obj->offset should become obj->_offset in FromRotate class
    • Add hardcoded no-null-check fix for C++-specific methods (these always return &):
      • BoundingBoxAttachment.getBones(), ClippingAttachment.getBones()
      • MeshAttachment.getBones(), MeshAttachment.getEdges()
    • Fix nested arrays null checks: getVertices() for DeformTimeline and getDrawOrders() for DrawOrderTimeline
      • These methods already have special casing but include unnecessary nullptr checks
      • Remove the nullptr checks from the special case handling since they return references to nested arrays
    • Replace enum .name() calls with switch statements in C++
      • Java: json.writeValue(obj.getMixBlend().name()) writes enum as string
      • C++: String::valueOf((int)obj->getMixBlend()) doesn't work (String::valueOf doesn't exist)
      • Solution: Generate switch statements that map C++ enum values to Java string equivalents
      • Example: MixBlend_Setup -> "setup", MixBlend_First -> "first", etc.
      • Use analysis-result.json to find all enums and generate proper mappings
    • Add ability to completely replace writeXXX functions with custom implementations in C++ generator
      • Need custom writeSkin function because C++ getAttachments() returns AttachmentMap::Entries (value type iterator)
      • Need custom writeColor function because Color.r/g/b/a are public fields without _ prefix
      • Need custom writeSkinEntry function that takes AttachmentMap::Entry* instead of Java SkinEntry
      • Create mechanism in tests/generate-cpp-serializer.ts to replace auto-generated functions with hand-written ones
      • Implement custom writeSkin function that properly handles AttachmentMap::Entries iteration
    • Create regenerate-all.sh script in tests/ that runs all generators in sequence
    • Fix C++ JsonWriter output formatting issues (array formatting broken)
    • Test with sample skeleton files
• TypeScript (spine-ts):
  • Follow what we did for spine-cpp wrt to JsonWriter, SkeletonSerializer and the generator
• C#
  • Figure out how we can add the HeadlessTest and run it without adding it to the assembly itself
  • Follow what we did for spine-cpp wrt
• C (spine-c):
  • Follow what we did for spine-cpp wrt to JsonWriter, SkeletonSerializer and the generator (this one will need some ultrathink and discussion with the user before code changs)
• Update tests/README.md to describe the new setup

Phase 3: Intermediate Representation for Cross-Language Serializer Generation

Problem

Current approach requires maintaining separate generator logic for each language (Java, C++, C, TypeScript, C#). The complex analysis, exclusion handling, and serialization logic is duplicated across generators, making maintenance difficult.

Solution: Language-Agnostic Intermediate Representation (IR)

Create a single IR generator that captures all serialization logic in a JSON format that language-specific generators can consume without a lot of post-processing. The IR will still contain Java specific types and names. Language specific generators then just have to translate.

IR Structure

interface SerializerIR {
  publicMethods: PublicMethod[];
  writeMethods: WriteMethod[];
  enumMappings: { [enumName: string]: { [javaValue: string]: string } };
}

type Property = Primitive | Object | Enum | Array | NestedArray;

interface Primitive {
  kind: "primitive";
  name: string; // "duration"
  getter: string; // "getDuration"
  valueType: string; // "float", "int", "boolean", "String"
  isNullable: boolean;
}

interface Object {
  kind: "object";
  name: string; // "color"
  getter: string; // "getColor"
  valueType: string; // "Color"
  writeMethodCall: string; // "writeColor"
  isNullable: boolean;
}

interface Enum {
  kind: "enum";
  name: string; // "mixBlend"
  getter: string; // "getMixBlend"
  enumName: string; // "MixBlend"
  isNullable: boolean;
}

interface Array {
  kind: "array";
  name: string; // "timelines"
  getter: string; // "getTimelines"
  elementType: string; // "Timeline", "int", "String"
  elementKind: "primitive" | "object";
  writeMethodCall?: string; // "writeTimeline" (for objects)
  isNullable: boolean;
}

interface NestedArray {
  kind: "nestedArray";
  name: string; // "vertices"
  getter: string; // "getVertices"
  elementType: string; // "float"
  isNullable: boolean;
}

Implementation Plan

• Create tests/generate-serializer-ir.ts:
  • Reuse logic from tests/generate-java-serializer.ts (analysis, exclusions, property detection)
  • Output SerializerIR as JSON to tests/output/serializer-ir.json
  • All exclusions and filtering pre-applied - no post-processing needed
• Update language generators to consume IR:
  • Replace tests/generate-java-serializer.ts with IR-based version
    • Sort skin entries by slot index before emission in Java
  • Modify tests/generate-cpp-serializer.ts to use IR
  • Create tests/generate-ts-serializer.ts using IR
  • Create tests/generate-cs-serializer.ts using IR
  • Language generators focus purely on syntax transformation
• Update tests/regenerate-all.sh to generate IR first, then all languages

Misc (added by user while Claude worked, need to be refined!)

• HeadlessTest should probably
  • Have a mode that does what we currently do: take files and animation name, and output serialized skeleton data and skeleton. Used for ad-hoc testing of files submitted by users in error reports etc.
  • Have "unit" test like tests, that are easily extensible
    • each test has a name and points to the corresponding function to execute
    • HeadlessTest can take as args a single name, multiple test names, or no args in which case it runs all tests in order
    • Structure and cli handling needs to be the same in all HeadlessTest implementations
    • tests/headless-test-runner.ts should also support these same cli args, run each runtime test, then compare outputs.