Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save oliverbarnes/c277f47dece7bcd858a65fbd5f0d1578 to your computer and use it in GitHub Desktop.
Save oliverbarnes/c277f47dece7bcd858a65fbd5f0d1578 to your computer and use it in GitHub Desktop.
Flaky openai e2e test
➜ tensorzero git:(test-json-mode-off-in-common) ✗ cargo test-e2e ::openai::test_json_mode_off_inference_request
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.28s
────────────
Nextest run ID 478d71ae-ff63-4594-bdf4-01919401e929 with nextest profile: default
Starting 1 test across 17 binaries (1763 tests skipped)
PASS [ 7.133s] tensorzero-internal::e2e providers::openai::test_json_mode_off_inference_request
────────────
Summary [ 7.150s] 1 test run: 1 passed, 1763 skipped
➜ tensorzero git:(test-json-mode-off-in-common) ✗ cargo test-e2e ::openai::test_json_mode_off_inference_request
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.28s
────────────
Nextest run ID bea33b3d-8e79-4cc0-91ed-d5abd13e1ca6 with nextest profile: default
Starting 1 test across 17 binaries (1763 tests skipped)
PASS [ 6.068s] tensorzero-internal::e2e providers::openai::test_json_mode_off_inference_request
────────────
Summary [ 6.086s] 1 test run: 1 passed, 1763 skipped
➜ tensorzero git:(test-json-mode-off-in-common) ✗ cargo test-e2e ::openai::test_json_mode_off_inference_request
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.15s
────────────
Nextest run ID c7e24f74-8243-429d-b536-d88509fa5e0d with nextest profile: default
Starting 1 test across 17 binaries (1763 tests skipped)
TRY 1 FAIL [ 6.487s] tensorzero-internal::e2e providers::openai::test_json_mode_off_inference_request
──── TRY 1 STDOUT: tensorzero-internal::e2e providers::openai::test_json_mode_off_inference_request
running 1 test
Connecting to ClickHouse
Connected to ClickHouse in 151.040916ms
test providers::openai::test_json_mode_off_inference_request ... FAILED
failures:
failures:
providers::openai::test_json_mode_off_inference_request
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1393 filtered out; finished in 6.48s
──── TRY 1 STDERR: tensorzero-internal::e2e providers::openai::test_json_mode_off_inference_request
thread 'providers::openai::test_json_mode_off_inference_request' panicked at tensorzero-internal/tests/e2e/providers/common.rs:10383:5:
assertion `left == right` failed
left: "The capital"
right: "The capital city of Japan is Tokyo."
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment