doordash-oss/PropertyTestingKit

# PropertyTestingKit Coverage-guided fuzz testing for Swift. ## Overview PropertyTestingKit brings coverage-guided fuzzing to Swift Testing: - **Coverage-guided fuzzing** - Automatically discover inputs that exercise new code paths - **Corpus persistence** - Save and replay interesting inputs across test runs - **Regression testing** - Replay saved corpus to catch regressions - **High throughput** - ~35M iterations/sec with full per-test concurrent coverage isolation ## Requirements - macOS 26+ / iOS 26+ - Swift 6.3+ ## Installation ### Swift Package Manager Add PropertyTestingKit to your `Package.swift`: dependencies: [ .package(url: "https://github.com/alex-reilly-dd/PropertyTestingKit.git", from: "0.0.1"), ], targets: [ .testTarget( name: "YourTests", dependencies: ["PropertyTestingKit"] ), ] ## Usage ### Coverage-Guided Fuzzing The `fuzz` function automatically generates inputs that maximize code coverage: import Testing import PropertyTestingKit @Test func testDatabaseQuery() async throws { try await fuzz(seeds: [ ("users", 0), ("users", 100), ("orders", -1), ]) { table, limit in let query = buildQuery(table: table, limit: limit) let result = database.execute(query) // Properties that should hold for all inputs #expect(result.isValid || result.hasError) if limit < 0 { #expect(result.hasError, "Negative limit should error") } } } **How it works:** 1. Starts with seed values (yours + type defaults from `MutatorProviding.defaultMutator`) 2. Runs each input and captures coverage 3. Inputs that hit new code paths are saved to the corpus 4. Mutates interesting inputs to discover more paths 5. Stops when the time limit is reached 6. Saves minimal corpus to disk for future runs **On subsequent runs:** - Replays saved corpus to check for crashes (regression testing) ### Corpus Storage The corpus is saved alongside your test files: Tests/ MyTests/ ParserTests.swift Corpus/ # Created automatically testParser/ corpus.json # Saved inputs + coverage signatures Commit the `Corpus/` directory to version control for deterministic CI runs. ### Fuzzing vs. regression There are two entry points. `fuzz(...)` explores inputs and maintains a corpus; `regress(...)` only replays a saved corpus to verify it still passes. The split is deliberate: regression takes none of the fuzz-only knobs (`seeds`, `coverageStrategy`, `parallelism`), and its plugins are `AnalysisHandler`s that can only observe (`stop` / `recordIssue`) — so it's impossible, at compile time, to hand a replay a configuration or a plugin that would explore or mutate the corpus. `fuzz(...)` takes a `persistence:` policy controlling how it treats an existing corpus: | `CorpusPersistence` | Behavior | |------|----------| | `.auto` | Replay the corpus if one exists, otherwise fuzz fresh and save (default) | | `.replace` | Delete any existing corpus, fuzz fresh, and save | | `.extend` | Load the existing corpus as seeds, fuzz, and save | | `.ephemeral` | Fuzz in memory only — ignore any existing corpus and don't save (nothing touches disk) | **Per-test control:** @Test func testParser() async throws { // Force re-fuzzing even if a corpus exists try await fuzz(persistence: .replace) { (input: String) in parse(input) } } @Test func testExtendCorpus() async throws { // Build on the existing corpus with a longer duration try await fuzz(persistence: .extend, duration: .seconds(120)) { (input: String) in parse(input) } } @Test func testParserRegression() async throws { // Replay the saved corpus only — fails if any saved input now trips the test try await regress { (input: String) in parse(input) } } **Suite-level control via environment:** Users may want to run background fuzzing campaigns outside of the standard CI loop uising `FUZZ_CORPUS_MODE=refuzzextend`. This allows a balance to be struck between fast deterministic test runs and thorough testing. # Re-fuzz all tests, replacing existing corpora FUZZ_CORPUS_MODE=refuzzreplace swift test # Extend existing corpora with more fuzzing (2 minute duration) FUZZ_CORPUS_MODE=refuzzextend FUZZ_DURATION=120 swift test # CI mode: force every fuzz test to replay-only — no exploration (fast, deterministic). # regress(...) tests always replay regardless of this variable. FUZZ_CORPUS_MODE=regressiononly swift test ### Custom Seeds Provide domain-specific seeds to guide the fuzzer toward edge cases: @Test func testNumberParser() async throws { try await fuzz(seeds: [ "0", "-0", "+0", // Zero variants String(Int.max), // Boundary String(Int.min), // Boundary "1.5", "1e10", // Invalid formats " 42 ", // Whitespace ]) { input in if let n = NumberParser.parse(input) { // Round-trip property #expect(NumberParser.parse(String(n)) == n) } } } ### Custom Mutators Use domain-specific mutation strategies instead of the default `MutatorProviding` conformance: @Test func testInputValidation() async throws { // Single mutator with multiple strategies try await fuzz(using: String.mutators(.sql, .xss)) { input in let sanitized = sanitize(input) #expect(!sanitized.contains("DROP TABLE")) #expect(!sanitized.contains("