τ²-Bench: Airline

Task 17 agent: openai/gpt-5.1

For reservation FQ8APE: add 3 checked bags, swap the passenger to Omar Rossi, and upgrade basic economy to economy, paying with a gift card.

• Database must match the gold state: update_reservation_flights (economy upgrade), update_reservation_passengers, and update_reservation_baggages with exact arguments
• communicate_info is empty, so no string check applies

The agent looked up the user, found the right reservation among several, confirmed the changes and payment method, then made all three writes: passenger swap, cabin upgrade, and bags. The final database hashes match the gold state and the run ends on USER_STOP, so reward is 1. This is what the eval is designed to measure: multi-step tool use under policy constraints, done correctly.

Task 0 agent: openai/gpt-4o-mini

Cancel reservation EHGLP3. The booking is more than 24 hours old, basic economy, no travel insurance, so policy says no cancellation.

• Database must end unmodified (the gold solution makes no changes)
• NL assertion (not machine-checked): agent should refuse the cancellation

The agent pulled the reservation, saw a basic economy fare booked more than 24 hours ago with no insurance, and cancelled it anyway. It even promised a $208 refund. The checker compares the final database against the gold database (unmodified), the hashes differ, reward is 0. The same model refused this exact cancellation in a different epoch; sampling variance flips the outcome.

Task 18 agent: openai/gpt-4o-mini

Downgrade all five business-class reservations to economy, then report the total amount saved. The correct total is $23,553.

• Database must match the gold state: 5 update_reservation_flights calls with exact cabin, flights, and payment IDs
• communicate_info: the string "23553" must appear in an agent message

The agent executed all five downgrades correctly, and the database check passed. Then it computed the savings from a partial list of fares and told the user $14,965 instead of $23,553. The communicate check greps every agent message for the literal string "23553", finds nothing, and zeroes the whole task. One wrong arithmetic answer erased five correct database writes.

Task 11 agent: meta-llama/llama-3.1-8b-instruct

Remove passenger Sophia from reservation GV1N64. Policy forbids changing the passenger count; the correct move is to downgrade both passengers to basic economy and refund $5,244.

• Database must match the gold state: one update_reservation_flights call downgrading to basic_economy
• communicate_info: the string "5244" must appear in an agent message

The agent’s very first move is a write with fully invented arguments: a reservation ID the user never gave and a passenger "John Doe" born 1990-01-01 who exists nowhere in the data. After finding the real reservation it sends update_reservation_passengers with a one-passenger array against a two-passenger booking, gets "number of passengers does not match", and retries the identical call four more times, at one point hallucinating "Sophia Smith, 1992-01-01" as the second passenger. It never consults the policy (removing a passenger is forbidden; the correct move is a cabin downgrade), quotes a fabricated $200 refund, and both checks fail. The arguments are well-formed JSON that the tool schema accepts; they are just wrong about the world, which is why small models can look fine on schema-level tool-call metrics and still fail tasks like this.

Task 14 agent: openai/gpt-4o-mini

Rebook the cheapest business round trip and split payment across gift cards, one certificate, and a credit card. The correct split puts $44 on the card.

• Database must match the gold state (cancel + rebook with the exact payment split)
• communicate_info: gift card and certificate sums, and the final card charge

The agent built a booking where the payment amounts did not sum to the ticket price. The tool rejected it with the same error every time, and the agent retried the identical call dozens of times until the run hit its 200-step ceiling. Any run that terminates on MAX_STEPS scores 0 before the database is even compared. This transcript is 202 messages long; the excerpt below is the loop.

Task 0 agent: openai/gpt-4o-mini

Same task as the policy-break failure: the user wants to cancel EHGLP3, and policy says no.

• Gold action list is empty, so the database check passes as long as nothing changed
• communicate_info is empty, so the communicate check passes vacuously
• The NL assertion ("agent should refuse") is metadata only and never machine-checked

This run earned a legitimate pass: the agent checked the reservation, cited the 24-hour rule, and refused. But look at what the checker actually verified: an untouched database and an empty communicate list. An agent that stonewalled every request, or transferred to a human immediately, would score identically. Refusal tasks measure "did nothing break", so they inflate scores for overly cautious models.