buildkite/test-engine-client

# Buildkite Test Engine Client Buildkite Test Engine Client (bktec) is an open source tool to orchestrate your test suites. It uses your Buildkite Test Engine suite data to intelligently partition and parallelize your tests. bktec supports multiple test runners and offers various features to enhance your testing workflow. Below is a comparison of the features supported by each test runner: | Feature | RSpec | Jest | Playwright | Cypress | pytest | pytest-pants | gotest | Cucumber | NUnit | Custom test runner | | --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | | [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | | Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | | Filter tests by tag | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | Automatically retry failed test | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ## Installation The latest version of bktec can be downloaded from https://github.com/buildkite/test-engine-client/releases ### Supported OS/Architecture ARM and AMD architecture for linux, darwin, and windows The available Go binaries - bktec-darwin-amd64 - bktec-darwin-arm64 - bktec-linux-amd64 - bktec-linux-arm64 - bktec-windows-amd64.exe - bktec-windows-arm64.exe ## Using bktec ### Buildkite Pipeline environment variables bktec uses the following Buildkite Pipeline provided environment variables. | Environment Variable | Description| | -------------------- | ----------- | | `BUILDKITE_BUILD_ID` | The UUID of the Buildkite build. bktec uses this UUID along with `BUILDKITE_STEP_ID` to uniquely identify the test plan. | | `BUILDKITE_JOB_ID` | The UUID of the job in Buildkite build. | | `BUILDKITE_ORGANIZATION_SLUG` | The slug of your Buildkite organization. | | `BUILDKITE_PARALLEL_JOB` | The index number of a parallel job created from a Buildkite parallel build step.
Make sure you configure `parallelism` in your pipeline definition. You can read more about Buildkite parallel build step on this [page](https://buildkite.com/docs/pipelines/controlling-concurrency#concurrency-and-parallelism).| | `BUILDKITE_PARALLEL_JOB_COUNT` | The total number of parallel jobs created from a Buildkite parallel build step.
Make sure you configure `parallelism` in your pipeline definition. You can read more about Buildkite parallel build step on this [page](https://buildkite.com/docs/pipelines/controlling-concurrency#concurrency-and-parallelism). | | `BUILDKITE_STEP_ID` | The UUID of the step group in Buildkite build. bktec uses this UUID along with `BUILDKITE_BUILD_ID` to uniquely identify the test plan. ### Create API access token To use bktec, you need a Buildkite API access token with `read_suites`, `read_test_plan`, and `write_test_plan` scopes. You can generate this token from your [Personal Settings](https://buildkite.com/user/api-access-tokens) in Buildkite. After creating the token, set the `BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN` environment variable with the token value. export BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN=token ### Configure Test Engine suite slug To use bktec, you need to configure the `BUILDKITE_TEST_ENGINE_SUITE_SLUG` environment variable with your Test Engine suite slug. You can find the suite slug in the URL of your suite. For example, in the URL `https://buildkite.com/organizations/my-organization/analytics/suites/my-suite`, the slug is `my-suite`. export BUILDKITE_TEST_ENGINE_SUITE_SLUG=my-slug ### Preview: Test Selection You can pass test selection strategy configuration and additional change context to the test plan API request. This preview is enabled only when `BKTEC_PREVIEW_SELECTION` is truthy (`1`, `true`, `yes`, or `on`). This functionality is under development, and these flags currently have undefined behavior. Environment variables: export BKTEC_PREVIEW_SELECTION=true export BUILDKITE_TEST_ENGINE_SELECTION_STRATEGY=percent Command-line flags: BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --selection-strategy percent \ --selection-param percent=40 #### Automatic git metadata collection When `--selection-strategy` is set, the `plan` command automatically collects git metadata from the current repository and sends it with the API request. This includes commit information (SHA, author, committer, message), diff data (files changed, numstat, full diff), and context fields (branch name, base branch, pipeline slug, build UUID). For pipelines that use `plan` without `--selection-strategy`, you can opt in to metadata collection with the `--collect-git-metadata` flag (or `BUILDKITE_TEST_ENGINE_COLLECT_GIT_METADATA=true`). This collects the same git metadata without requiring selection to be configured: BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --collect-git-metadata The base branch for diff computation is resolved using a fallback chain: 1. Explicit override via `--metadata base_branch=` 2. `BUILDKITE_PULL_REQUEST_BASE_BRANCH` (auto-set by Buildkite on PR builds) 3. Auto-detection via `/HEAD`, then `/main`, then `/master` Most users don't need to configure anything. Override `base_branch` only if your repository uses a non-standard default branch (for example, `develop` or `trunk`) and `/HEAD` isn't configured. The `--remote` flag (default `origin`) controls which git remote is used for base branch detection. You can also set `BUILDKITE_TEST_ENGINE_REMOTE`. Auto-collected values are merged with any explicit `--metadata` flags you provide. Your explicit values always take precedence. #### Manual metadata overrides Use `--metadata key=value` to pass additional metadata or override auto-collected values. Use `--selection-param key=value` to pass strategy parameters. Both flags are repeatable. Values can be large and multiline. BKTEC_PREVIEW_SELECTION=true ./bktec plan --json --selection-strategy percent \ --selection-param percent=40 \ --metadata base_branch=develop `--selection-param` and `--metadata` are only supported as repeatable CLI flags. ### Preview: Commit Metadata Backfill bktec can collect historical git commit metadata from your repository and upload it to Buildkite for training test selection models. This is useful for bootstrapping models with historical changeset data so that test selection can identify which tests are relevant to your code changes. The `tools` subcommands are hidden from `bktec --help` by default. Setting `BKTEC_PREVIEW_SELECTION` to a truthy value (`1`, `true`, `yes`, or `on`) makes them visible in help output. The commands can always be invoked directly regardless of this setting. Two commands are available under `bktec tools`: **Collect and upload commit metadata:** bktec tools backfill-commit-metadata \ --access-token "bkua_..." \ --organization-slug "my-org" \ --suite-slug "my-suite" **Generate the tarball locally for inspection before uploading:** bktec tools backfill-commit-metadata --output commit-metadata.tar.gz # Inspect the contents tar tzf commit-metadata.tar.gz # commit-metadata.jsonl # metadata.json # Upload when ready bktec tools backfill-commit-metadata \ --upload commit-metadata.tar.gz \ --suite-slug "my-suite" The API access token requires `read_suites` and `write_suites` scopes. For detailed usage, flags, and configuration options, see the [Commit Metadata Backfill](./docs/commit-metadata-backfill.md) guide. ### Configure the test runner To configure the test runner for bktec, please refer to the detailed guides provided for each supported test runner. You can find the guides at the following links: - [Jest](./docs/jest.md) - [Playwright](./docs/playwright.md) - [Cypress](./docs/cypress.md) - [pytest](./docs/pytest.md) - [pytest pants](./docs/pytest-pants.md) - [go test](./docs/gotest.md) - [RSpec](./docs/rspec.md) - [Cucumber](./docs/cucumber.md) - [NUnit](./docs/nunit.md) - [Custom Test Runner](./docs/custom-test-runner.md) ### Running bktec Please download the executable and make it available in your testing environment. To parallelize your tests in your Buildkite build, you can amend your pipeline step configuration to: steps: - name: "Rspec" command: ./bktec run parallelism: 10 env: BUILDKITE_TEST_ENGINE_SUITE_SLUG: my-suite BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: your-secret-token BUILDKITE_TEST_ENGINE_TEST_RUNNER: rspec BUILDKITE_TEST_ENGINE_RESULT_PATH: tmp/result.json ### Debugging To enable debug mode, set the `BUILDKITE_TEST_ENGINE_DEBUG_ENABLED` environment variable to `true`. This will print detailed output to assist in debugging bktec. ### Possible exit statuses bktec may exit with a variety of exit statuses, outlined below: - If there is a configuration error, bktec will exit with status 16. - If the test runner (e.g. RSpec) exits cleanly, the exit status of the runner is returned. This will likely be 0 for successful test runs, 1 for failing test runs, but may be any other error status returned by the runner. - If the test runner is terminated by an OS level signal, such as SIGSEGV or SIGABRT, the exit status returned will be equal to 128 plus the signal number. For example, if the runner raises a SIGSEGV, the exit status will be (128 + 11) = 139. ## Development - [Go Installation Guide](https://golang.org/doc/install) - [Ruby Installation Guide](https://www.ruby-lang.org/en/documentation/installation/) - [Node.js Installation Guide](https://nodejs.org/en/download/package-manager/) Once you have these dependencies installed, run `bin/setup` to install dependencies for the sample projects for testing purposes. To test, run: ./bin/test [^1]: NB: Test splitting is not supported for Pants, because Pants will decide which tests to run. For go test, test splitting is by package and not by file.

标签：EVTX分析