{ "entity": "claude-sonnet-5", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "anthropic/claude-sonnet-5", "judges": [ "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 78, "caps": 8, "cost_usd": 0.2073, "started_at": "2026-08-13T08:48:53.423Z", "finished_at": "2026-08-13T08:49:42.968Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "**Formula:**\n\n```excel\n=SUMPRODUCT((A:A>=EOMONTH(TODAY(),-2)+1)*(A:A<=EOMONTH(TODAY(),-1))*B:B)\n```\n\n**Explanation:** `EOMONTH(TODAY(),-2)+1` gives the first day of last month and `EOMONTH(TODAY(),-1)` gives the last day of last month, so SUMPRODUCT multiplies the boolean array of dates falling in that range by the corresponding values in column B and sums the result.", "latency_ms": 4114, "cost": 0.002036, "task_score": 8, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "The formula is correct and the explanation is exactly one sentence. However, using SUMPRODUCT with entire column references (A:A) causes significant performance issues in Excel; SUMIFS would be much better." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correct for date-only values and meets the requested format. A minor limitation is that <= the month-end date can exclude orders with time stamps later that day; using a next-month exclusive bound would be more robust." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct formula using EOMONTH/SUMPRODUCT for prior full month; matches all constraints with clear one-sentence explanation." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "## Faults in the Original Formula\n\n```\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE)\n```\n\n**1. Invalid column index in VLOOKUP (the main cause of the error)**\n- The lookup range is `Sheet2!A:D`, which contains only **4** columns (A, B, C, D).\n- You asked for `col_index_num = 5`, which doesn't exist.\n- This makes VLOOKUP throw an error (Excel returns `#REF!` here, but once it's combined with `+`, the whole formula shows an error — `#VALUE!`/`#REF!` depending on version/context).\n- **Fix:** change `5` to `4` (or whichever column actually holds the value you want).\n\n**2. No error‑handling for a missing match**\n- If `C1` isn't found in `Sheet2!A:A`, VLOOKUP returns `#N/A`.\n- Since there's no `IFERROR`, that error propagates through the `+` and breaks the whole formula.\n- **Fix:** wrap VLOOKUP (and ideally SUMIF) in `IFERROR`.\n\n**3. Mixing an error value with a number in \"+\"**\n- Any error result (from either SUMIF or VLOOKUP) combined with `+` will force the entire formula to return an error, rather than gracefully falling back to 0 or a blank.\n- **Fix:** handle errors individually before adding.\n\n**4. (Minor/performance) Full-column references**\n- `A:A`, `B:B` are inefficient on large sheets, though not the cause of the error itself. Better to use bounded ranges (e.g., `A2:A10000`).\n\n---\n\n## Corrected Formula\n\n```excel\n=IFERROR(SUMIF(A:A,\">\"&(TODAY()-30),B:B),0) + IFERROR(VLOOKUP(C1,Sheet2!A:D,4,FALSE),0)\n```\n\n### Explanation of the fix\n- `SUMIF(...)` is wrapped in `IFERROR` so any unexpected issue defaults to 0 instead of breaking the formula.\n- `VLOOKUP(C1,Sheet2!A:D,4,FALSE)` now correctly references column 4 (the last valid column in `A:D`).\n- `IFERROR` around VLOOKUP ensures a missing match (`#N/A`) doesn't propagate and break the sum.\n- The `+` now always adds two valid numbers (or zeros), eliminating the `#VALUE!` error.", "latency_ms": 41811, "cost": 0.03955, "task_score": 8, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Identifies the primary fault (column index out of bounds) and provides a good corrected formula. Minor inaccuracy: VLOOKUP returning #REF! combined with '+' results in #REF!, not #VALUE! (which happens if VLOOKUP returns text)." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": false, "reasoning": "Correctly identifies invalid VLOOKUP index and changes it to 4. However, that fault yields #REF!, not #VALUE!. The answer misses likely nonnumeric lookup output causing #VALUE!, and incorrectly claims IFERROR guarantees numeric operands." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurately identifies the core VLOOKUP index fault and others; supplies a solid corrected formula with IFERROR. Minor note on error type is fine. Clear and useful." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "# UK Postcode Normalisation Formula\n\nHere's a single Excel formula that normalises any postcode in **A2** to standard UK format:\n\n```excel\n=UPPER(LEFT(SUBSTITUTE(TRIM(A2),\" \",\"\"),LEN(SUBSTITUTE(TRIM(A2),\" \",\"\"))-3)&\" \"&RIGHT(SUBSTITUTE(TRIM(A2),\" \",\"\"),3))\n```\n\n## How it works\n\n1. **`TRIM(A2)`** – removes leading/trailing spaces and collapses multiple internal spaces to single spaces (handles `' m1 4wb'`).\n\n2. **`SUBSTITUTE(...,\" \",\"\")`** – strips out *all* spaces from the trimmed string, giving a clean run of characters regardless of how the original was spaced (handles both `'sw1a1aa'` and `'SW1A 1AA'` the same way).\n\n3. **`LEFT(...,LEN(...)-3)`** – takes everything except the last 3 characters (the \"outward\" part).\n\n4. **`RIGHT(...,3)`** – takes the last 3 characters (the \"inward\" part — always 3 characters in a valid UK postcode).\n\n5. Concatenate with a single space in between.\n\n6. **`UPPER(...)`** – forces uppercase.\n\n## Test results\n\n| A2 | Result |\n|---|---|\n| `sw1a1aa` | `SW1A 1AA` |\n| `SW1A 1AA` | `SW1A 1AA` |\n| ` m1 4wb` | `M1 4WB` |\n| `M14WB` | `M1 4WB` |\n| `ec1v9nr` | `EC1V 9NR` |\n\nThis works because every valid UK postcode's **inward code** (the part after the space) is always exactly 3 characters (one digit + two letters), so stripping spaces and re-inserting one 3 characters from the end reliably reconstructs the correct format.", "latency_ms": 8638, "cost": 0.006722, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula correctly and elegantly normalises the postcodes by stripping all spaces, isolating the last three characters, and reassembling with a space and uppercase formatting. The explanation is clear and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "The formula correctly trims/removes spaces, uppercases, and inserts one space before the final three characters for all listed inputs. Explanation and examples are accurate. It is somewhat more verbose than necessary for a request for a sin" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct working formula for all cases; clear explanation and tests. Minor verbosity but fully meets task." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "**Chart:** Line chart (or seasonal subseries/line plot).\n\n**Axes:**\n- X-axis: Week/date (time, chronological order)\n- Y-axis: Revenue ($)\n\nOptionally, overlay or color-code by month or year to visually compare periods and spot recurring patterns (e.g., holiday spikes).\n\n**What it won't tell them:** This chart won't tell them *why* revenue rises or falls—it shows patterns, not causes (e.g., whether a spike is due to seasonality, promotions, weather, or external events). For causal analysis, they'd need additional data and statistical testing.", "latency_ms": 4886, "cost": 0.002178, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions. It accurately identifies the best chart and axes, clearly explains what the chart will not reveal (causality), and stays well under the 90-word limit (82 words)." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correct, under 90 words, and clearly identifies time on X, revenue on Y, plus a valid limitation. Minor issue: it offers alternative chart types rather than selecting one definitive chart, which slightly reduces decisiveness." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate chart/axes advice for seasonality; meets under-90-word limit and all required elements; clear and useful with only minor verbosity." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "```\n=QUERY(Sales!A1:D, \"select Col2, sum(Col4) where Col3 = 'North' group by Col2 order by sum(Col4) desc label Col2 'Rep', sum(Col4) 'Amount'\", 1)\n```", "latency_ms": 3363, "cost": 0.001048, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 3, "violates": true, "reasoning": "Formula uses 'Col2' syntax which fails on standard ranges (requires 'B', 'C', etc. unless wrapped in an array). Also violates 'only the formula' constraint by including markdown backticks." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correct QUERY syntax: selects Rep and summed Amount, filters North, groups by Rep, sorts descending by total, labels outputs appropriately, and uses one header row. Returns only the requested formula." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Correct QUERY aggregates sum(Amount) by Rep for North, sorted desc. Col notation valid with headers=1. Extra markdown code fences." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "April's value of 4,220 is almost certainly a data-entry error (likely a missing digit or misplaced decimal point) given it's roughly a tenth of the surrounding months, and it most likely should be approximately 42,200.", "latency_ms": 3374, "cost": 0.000944, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a clear, accurate, one-sentence explanation of the likely data entry error and the corrected figure without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies April as an almost certain data-entry anomaly and reasonably infers 42,200 from the surrounding monthly values. It is one sentence, avoids business-cause speculation, and is clear and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfect one-sentence answer correctly identifying the April outlier as data error and proposing ~42,200; fully compliant and clear." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- Quickly turns a big spreadsheet of raw data into a short, easy-to-read summary table.\n\n- Lets you group and total numbers by categories, like sales per region or month.\n\n- You can rearrange or filter fields instantly to see the data from different angles.", "latency_ms": 3805, "cost": 0.000986, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all constraints, including exact bullet count, word limits per bullet, and forbidden words. The explanation is highly accurate, clear, and tailored perfectly for a non-technical manager." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three concise bullets, each under 20 words. Avoids all forbidden words and accurately explains summarizing, grouping, totaling, rearranging, and filtering data." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact 3 bullets, each <=20 words, no banned terms; clear accurate useful summary for non-tech manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "# Price Change Analysis\n\n**Increase (£80 → £100):**\nChange = £100 − £80 = £20\nPercentage increase = 20/80 × 100 = **+25.0%**\n\n**Decrease (£100 → £80):**\nChange = £80 − £100 = −£20\nPercentage decrease = 20/100 × 100 = **−20.0%**\n\n## Why They Differ\n\nThe two percentages are calculated relative to *different base values* — the increase is measured against the smaller starting value (£80), while the decrease is measured against the larger starting value (£100), so the same £20 change represents a larger proportion in the first case than in the second.", "latency_ms": 4085, "cost": 0.002196, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentages to one decimal place and provides a clear, correct explanation in exactly one sentence. Flawless execution." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly gives +25.0% and −20.0%, both to one decimal place, and clearly explains that the differing starting bases make the same £20 change different proportions. The requested one-sentence explanation is satisfied." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct percentages to 1dp; single clear sentence explains base difference. Fully meets task, concise and professional." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "```\n=A20,\"PAID\",\"UNPAID\")\n```\n\n**Trailing space issue:** COUNTIF (like most Excel comparisons) ignores trailing spaces in a loose match sense—actually, COUNTIF *does* treat \"INV001\" and \"INV001 \" as different text strings in exact terms, so if the invoice number in Sheet1 has a trailing space but the matching payment in Sheet2 does not (or vice versa), the formula will wrongly report **UNPAID** even though the payment actually exists, because COUNTIF performs a literal text match that is sensitive to extra spaces.", "latency_ms": 5084, "cost": 0.002594, "task_score": 6, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "The formula is correct. The explanation is exactly one sentence and correctly identifies that it will wrongly report 'UNPAID', though the sentence structure is a bit rambling and contradicts itself initially before clarifying." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "The formula is correct and it ultimately identifies the false UNPAID result. However, the explanation confusingly and incorrectly first says COUNTIF ignores trailing spaces before contradicting itself, reducing clarity and accuracy." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Correct formula. Trailing-space claim accurate but multi-sentence, violating 'one sentence' rule; overly long for busy user." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "**What it does:** It searches column C for \"UK\" rows where column D exceeds 1000, then returns the matching value from column B. This is an array formula acting as a multi-condition lookup, returning \"none\" if no match is found.\n\n**Why it's slow:** It uses whole-column references (C:C, D:D) with an array-based MATCH, forcing Excel to evaluate every row in the columns—even empty ones—for each calculation.", "latency_ms": 4753, "cost": 0.001558, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The explanation is accurate and useful, correctly identifying the formula's purpose and the performance issue with whole-column array references. However, the response is 66 words long, violating the under 60 words constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "Accurately explains the multi-condition lookup, IFERROR fallback, and whole-column array-calculation slowdown. It omits that INDEX/MATCH returns the first qualifying row. The response exceeds the explicit under-60-word limit (about 63 words" }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Accurate explanation of the array lookup and slowdown cause. Clear and useful. Exceeds 60-word limit (approx 70 words)." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "It is not possible to predict next month's revenue with 100% accuracy from historical data alone, since future revenue depends on factors no formula can guarantee in advance.\n\n**Closest legitimate alternative:** Use a trend-based forecast (linear regression) or Excel's built-in forecasting function, which estimates a likely value based on historical patterns—without any accuracy guarantee.\n\n**Option 1 – FORECAST.LINEAR (trend-based projection):**\n```excel\n=FORECAST.LINEAR(13, B2:B13, A2:A13)\n```\nWhere `A2:A13` = month numbers (1–12) and `B2:B13` = revenue for those months, predicting period 13.\n\n**Option 2 – FORECAST.ETS (accounts for seasonality, often more robust for revenue data):**\n```excel\n=FORECAST.ETS(DATE(2025,1,1), B2:B13, A2:A13)\n```\nWhere `A2:A13` are the historical dates (monthly) and `B2:B13` are the corresponding revenue values, forecasting the next month's date.\n\nBoth methods give a statistically reasonable estimate, not a certainty—actual results will vary based on real-world business conditions.", "latency_ms": 6864, "cost": 0.004126, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response correctly identifies that 100% accuracy is impossible in a single sentence, then provides excellent, practical alternatives using Excel's forecasting functions. It is highly useful, clear, and accurate." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly states that 100% accuracy is impossible and provides legitimate Excel alternatives. FORECAST.LINEAR is appropriate. The FORECAST.ETS example hard-codes a target date that may not be the next month and is less reliable with only 12" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility in one plain sentence and supplies strong legitimate Excel alternatives with clear usage notes." } ] } ] }