Test Suite Coverage Guide

Copy Markdown View Source

This guide describes how JSONSchex runs the official JSON Schema Test Suites, what is included by default, and how to debug or customize coverage.

Overview

JSONSchex includes the JSON Schema Test Suite fixtures and runs Draft 7 and Draft 2020-12 tests by default.

Default suite paths:

  • test/fixtures/JSON-Schema-Test-Suite/tests/draft7
  • test/fixtures/JSON-Schema-Test-Suite/tests/draft2020-12

Default exclusions

The following test sets are excluded by default:

  • optional/cross-draft

Both official test modules exclude this fixture file because cross-draft resolution is outside the current support scope. Passing the included fixtures does not imply complete cross-draft support.

Draft 7 status

The Draft 7 fixture suite passes draft: "draft7" so fixtures without $schema use Draft 7. A canonical Draft 7 declaration also selects that draft automatically. Without a recognized declaration or explicit option, compilation defaults to Draft 2020-12; see the draft-selection rules.

Focused regressions live in test/draft7_compatibility_test.exs. The runner enables assertion options for optional format and content fixtures; optional/cross-draft remains excluded.

How official test suites are run

JSONSchex uses a test runner that scans the test suite directory and executes all eligible JSON files, excluding the paths listed above.

The JSON Schema Test Suite is included as a Git submodule (see .gitmodules in the repository root).

You can run the full test suite with:

mix test

The suite runner is invoked in separate test files:

  • test/official_jsts_test.exs for Draft 2020-12
  • test/official_jsts_draft7_test.exs for Draft 7

Running tests

Run all tests

mix test

This executes the entire test suite, including Draft 7 and Draft 2020-12 (except exclusions).

Run one draft's suite

mix test test/official_jsts_test.exs
mix test test/official_jsts_draft7_test.exs

Run with verbose output

mix test --trace

This shows each test as it runs, which is helpful for identifying slow or problematic tests.

Debugging a single suite file

There are debug test modules that can run a single suite file for focused investigations. These live in test/debug_*.exs.

Example: Running a specific JSON test file

To debug a specific test case from the official suite:

  1. Identify the test file you want to debug (e.g., test/fixtures/JSON-Schema-Test-Suite/tests/draft2020-12/type.json)

  2. Create or edit a debug test file (e.g., test/debug_type_test.exs):

defmodule JSONSchex.DebugTypeTest do
  use ExUnit.Case
  use JSONSchex.Test.SuiteRunner

  # This macro generates an ExUnit test for each case in the fixture.
  run_suite("test/fixtures/JSON-Schema-Test-Suite/tests/draft2020-12/type.json")
end
  1. Run just that test file:
mix test test/debug_type_test.exs --trace

The single-file macro uses the default draft. On Elixir 1.19 or newer, focus on Draft 7 while preserving its configured options by filtering the existing official test module:

mix test test/official_jsts_draft7_test.exs --name-pattern 'JSTS: type -'

The --name-pattern option was introduced in Elixir 1.19. On older supported Elixir versions, run the complete Draft 7 module instead:

mix test test/official_jsts_draft7_test.exs --trace

Example: Testing a single schema from a suite file

For even more focused debugging, you can extract a specific test case:

defmodule JSONSchex.DebugSingleTest do
  use ExUnit.Case

  test "debug specific type test case" do
    schema = %{"type" => "integer"}
    assert {:ok, compiled} = JSONSchex.compile(schema)

    # Check acceptance and the specific rejection rule, not just any error.
    assert :ok == JSONSchex.validate(compiled, 1)
    assert {:error, [%{rule: :type}]} = JSONSchex.validate(compiled, "not an integer")
  end
end

This is helpful for isolating regressions or investigating a failing test case.

Using IEx for Interactive Debugging

You can also use IEx to interactively test schemas:

iex -S mix

Then in the IEx session:

# Load a schema from a test file
{:ok, content} = File.read("test/fixtures/JSON-Schema-Test-Suite/tests/draft2020-12/type.json")
tests = JSONSchex.JSON.decode!(content)

# Get the first test group
[first_group | _] = tests
schema = first_group["schema"]

# Compile and test
{:ok, compiled} = JSONSchex.compile(schema)
JSONSchex.validate(compiled, 1)
JSONSchex.validate(compiled, "not a number")

Notes on optional vocabularies

The runner explicitly enables format_assertion: true for optional format fixtures and content_assertion: true for optional content fixtures. Application code must opt into content assertions separately. Format checks can also be enabled by an active Draft 2020-12 format-assertion vocabulary.

Draft 7 ignores contentSchema, including when content assertions are enabled. Its supported contentEncoding and contentMediaType checks still run. See the content and format guide.

Best practices

  • Keep the default exclusions minimal and explicit.
  • Use debug test files to isolate issues before changing global test coverage.
  • When adding new keyword support, run the relevant subset of the suite first, then the full suite.
  • Use --trace flag to see which tests are slow or hanging.
  • Check the official test suite repository for updates: https://github.com/json-schema-org/JSON-Schema-Test-Suite