Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,12 @@ CheckoutUITests/PaymentTests/testPaysWithCard()
TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE="$PWD/skip.txt" xcodebuild test …
```

- **Names.** Each line is `module/suite/name`, the way `-skip-testing` names a test and the records name it: the test bundle, the class without its Swift module, and the test as the result bundle names it (`testAppliesDiscount()`). The selector works too (`testAppliesDiscount`), including the bridged selectors of throwing and asynchronous Swift tests (`…AndReturnError:`, `…WithCompletionHandler:`). A Swift Testing suite nested in another is named by its whole path, `module/Outer/Inner/name`: the first component is the module, the last the test, and everything between the suite.
- **Only tests.** A line with fewer than three components, or an empty one, is ignored. Skip a whole target with `-skip-testing:CheckoutTests` instead. A suite is never skipped by naming it, because it can gain tests the list doesn't know about.
- **Names.** Each line is `module/suite/name`, the way `-skip-testing` names a test and the records name it: the test bundle, the class without its Swift module, and the test as the result bundle names it (`testAppliesDiscount()`). The selector works too (`testAppliesDiscount`), including the bridged selectors of throwing and asynchronous Swift tests (`…AndReturnError:`, `…WithCompletionHandler:`).
- **Only tests.** A line without exactly three components is ignored. Skip a whole target with `-skip-testing:CheckoutTests` instead. A suite is never skipped by naming it, because it can gain tests the list doesn't know about.
- **Forgiving.** Blank lines and tests that don't exist are ignored. An unreadable file skips nothing. Skipping doesn't depend on `TEST_COVERAGE_ATTRIBUTION_DIR`.
- **Where it applies.** `xcodebuild` passes the variable to the test process with the `TEST_RUNNER_` prefix: the app that hosts unit tests, and the runner of UI tests. It works with `test` and `test-without-building`, with `-only-testing`, and with parallel testing, where Xcode lists the tests, without the removed ones, before it hands them out to its clones. `swift test` reads it from the environment.
- **What a skipped test looks like.** It's removed from the suite XCTest builds for its class, so it doesn't run and isn't in the result bundle. A class whose tests are all removed doesn't run its class `setUp` and `tearDown`. A class with tests left runs them as usual, class `setUp` included.
- **Swift Testing.** In a suite with the `.coverageAttribution` trait, a listed test is cancelled before it starts, every argument of a parameterized test included, and reported as skipped, whether tests run in parallel or not. The line names it as the records do: the Swift module, every suite the test is nested in, outermost first (`CheckoutTests/Pricing/Discounts/appliesDiscount()`), as `-skip-testing` names it, and the test's display name when it declares one (`Applies a discount`), its function otherwise (`appliesDiscount()`). A nested test also matches a line that names it by its innermost suite alone (`CheckoutTests/Discounts/appliesDiscount()`), as Tuist did before it recorded whole paths. This needs Swift 6.3 (Xcode 26.4) or later: before, a trait can skip a test only by not running it, which reports it as passed, so the listed tests run. Listed tests in suites without the trait run too.
- **Swift Testing.** In a suite with the `.coverageAttribution` trait, a listed test is cancelled before it starts, every argument of a parameterized test included, and reported as skipped, whether tests run in parallel or not. The line names it as the records do: the Swift module, the innermost suite, and the test's display name when it declares one (`Applies a discount`), its function otherwise (`appliesDiscount()`). A display name with a `/` can't be listed. This needs Swift 6.3 (Xcode 26.4) or later: before, a trait can skip a test only by not running it, which reports it as passed, so the listed tests run. Listed tests in suites without the trait run too.

### Unskippable tests

Expand Down Expand Up @@ -162,7 +162,7 @@ A scope is one of the following:
Each record is laid out in the host's byte order, little-endian on every Apple platform, as follows:

- `u8` kind (`0` gap, `1` XCTest, `2` Swift Testing), `u8` flags (`1` overlapped), `u8` version (currently `1`), `u8` zero.
- The module, the suite and the test name, each a `u32` length followed by UTF-8 bytes. XCTest records name the test bundle as the module. Swift Testing records name the Swift module, and every suite the test is nested in, outermost first and separated by `/` (`Outer/Inner`), as the result bundle does.
- The module, the suite and the test name, each a `u32` length followed by UTF-8 bytes. XCTest records name the test bundle as the module.
- `u32` count of images whose counters moved. For each image: `u32` image index, `u32` count, then `count` `u32` counter indices and `count` `u64` deltas (how much each counter moved).

The deltas are what turns counters into lines. In the image's coverage mapping (`__llvm_covfun`), a region's count is an expression over counters, so knowing which counters moved isn't enough.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,7 @@ public struct CoverageAttributionTrait: TestTrait, SuiteTrait, TestScoping {
}
let components = test.id.nameComponents
let module = test.id.moduleName
// Every suite the test is nested in, outermost first (`Outer/Inner`), the way the result
// bundle and `-only-testing` name it; the innermost alone can't tell nested suites apart.
let suite = components.dropLast().joined(separator: "/")
let suite = components.count >= 2 ? components[components.count - 2] : ""
// The name the result bundle reports the test under, which is the test's identity
// everywhere else: its display name when it declares one, its function otherwise.
let name = test.displayName ?? components.last ?? test.name
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -445,11 +445,9 @@ static bool is_class_suite(NSString *name) {
// Skip list: one test per line, `module/suite/name`, the way xcodebuild's -skip-testing names a
// test and the records name it: the test bundle, the class without its Swift module, and the test
// as the result bundle names it (`testAdd()`) or as its selector (`testAdd`,
// `testAddAndReturnError:`, `testAddWithCompletionHandler:`). A Swift Testing suite nested in
// another is its whole path, `module/Outer/Inner/name`: the first component is the module, the
// last the test, and everything between the suite. Blank lines, lines with fewer than three
// components (a whole module is skipped with -skip-testing, and a suite can gain tests the list
// doesn't know) and tests that don't exist are ignored. An unreadable file skips nothing.
// `testAddAndReturnError:`, `testAddWithCompletionHandler:`). Blank lines, lines without exactly
// three components (a whole module is skipped with -skip-testing, and a suite can gain tests the
// list doesn't know) and tests that don't exist are ignored. An unreadable file skips nothing.
//
// The listed tests are removed from +[XCTestCase defaultTestSuite], the suite XCTest builds for
// each class, both when it lists the tests (parallel runs hand the listed tests out to their
Expand Down Expand Up @@ -481,25 +479,19 @@ static void load_skip_list(void) {
for (NSString *line in [contents componentsSeparatedByCharactersInSet:NSCharacterSet.newlineCharacterSet]) {
NSArray<NSString *> *components = [[line stringByTrimmingCharactersInSet:NSCharacterSet.whitespaceCharacterSet]
componentsSeparatedByString:@"/"];
if (components.count < 3 || [components containsObject:@""]) continue;
NSString *suite = [[components subarrayWithRange:NSMakeRange(1, components.count - 2)] componentsJoinedByString:@"/"];
[tests addObject:skip_identifier(components.firstObject, suite, components.lastObject)];
if (components.count != 3 || components[0].length == 0 || components[1].length == 0
|| components[2].length == 0)
continue;
[tests addObject:skip_identifier(components[0], components[1], components[2])];
}
if (tests.count > 0) skipped_tests = tests;
}

// The list is read once, by the constructor, and never changes after: no lock is needed.
// A test in a nested suite also matches a line that names it by its innermost suite alone
// (`module/Inner/name`), as Tuist did before it recorded the whole path.
bool test_coverage_attribution_is_skipped(const char *module, const char *suite, const char *name) {
if (!skipped_tests || !module || !suite || !name) return false;
NSString *module_name = @(module), *suite_path = @(suite), *test_name = @(name);
if (!module_name || !suite_path || !test_name) return false;
if ([skipped_tests containsObject:skip_identifier(module_name, suite_path, test_name)]) return true;
NSRange slash = [suite_path rangeOfString:@"/" options:NSBackwardsSearch];
if (slash.location == NSNotFound) return false;
NSString *innermost = [suite_path substringFromIndex:slash.location + 1];
return [skipped_tests containsObject:skip_identifier(module_name, innermost, test_name)];
NSString *identifier = skip_identifier(@(module), @(suite), @(name));
return identifier && [skipped_tests containsObject:identifier];
}

// Unskippable XCTest tests: a class marks them with class methods the observer looks up by name, so
Expand Down
16 changes: 8 additions & 8 deletions Tests/TestCoverageAttributionTests/SkipListTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ extension AttributionTests {
AppTests/SwiftTestingTests/unskippable()
AppTests/UnskippableSwiftTestingTests/runs()
AppTests/UnskippableSwiftTestingTests/skipped()
AppTests/UnskippableSwiftTestingTests/SkippableTests/skipped()
AppTests/UnskippableSwiftTestingTests/SkippableTests/unskippable()
AppTests/SkippableTests/skipped()
AppTests/SkippableTests/unskippable()
AppTests/UntraitedSwiftTestingTests/runs()
AppTests/UnlistedTests
AppTests/MissingTests/testMissing()
Expand Down Expand Up @@ -102,7 +102,7 @@ extension AttributionTests {

/// The lines `skipsTheListedTests` expects in `unskippable.tsv`, every test listed, named as
/// the records name them: XCTest tests as XCTest names them, without `()` or a bridged suffix,
/// and Swift Testing tests under every suite they're nested in.
/// and Swift Testing tests under their innermost suite.
private static func expectedUnskippable(_ run: SkipListRun) -> [String] {
var lines = [
"AppTests\tPartlySkippedTests\ttestUnskippable\t1",
Expand All @@ -113,7 +113,7 @@ extension AttributionTests {
]
if run != .shard {
lines += [
"AppTests\tUnskippableSwiftTestingTests/SkippableTests\tunskippable()\t1",
"AppTests\tSkippableTests\tunskippable()\t1",
"AppTests\tSwiftTestingTests\tunskippable()\t1",
"AppTests\tUnskippableSwiftTestingTests\truns()\t1",
]
Expand All @@ -133,8 +133,8 @@ extension AttributionTests {
skipList: """
ExampleTests/MultiplicationTests/Multiplying by zero
ExampleTests/MultiplicationTests/multiplyByOne()
ExampleTests/MultiplicationTests/UnskippableMultiplicationTests/multiplyNegatives()
ExampleTests/MultiplicationTests/UnskippableMultiplicationTests/square()
ExampleTests/UnskippableMultiplicationTests/multiplyNegatives()
ExampleTests/UnskippableMultiplicationTests/square()
""",
) { unskippable = try Self.unskippableTests(in: $0) }
let names = records.filter { $0.kind == .swiftTesting }.map(\.name)
Expand All @@ -154,8 +154,8 @@ extension AttributionTests {
#endif
#expect(unskippable == [
"ExampleTests\tMultiplicationTests\tmultiplyByOne()\t1",
"ExampleTests\tMultiplicationTests/UnskippableMultiplicationTests\tmultiplyByMinusOne()\t0",
"ExampleTests\tMultiplicationTests/UnskippableMultiplicationTests\tmultiplyNegatives()\t1",
"ExampleTests\tUnskippableMultiplicationTests\tmultiplyByMinusOne()\t0",
"ExampleTests\tUnskippableMultiplicationTests\tmultiplyNegatives()\t1",
])
}

Expand Down
67 changes: 0 additions & 67 deletions Tests/TestCoverageAttributionTests/SuitePathTests.swift

This file was deleted.

Loading