{ "entity": "claude-opus-4-8", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "anthropic/claude-opus-4.8", "judges": [ "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 94, "caps": 4, "cost_usd": 0.2015, "started_at": "2026-08-13T08:48:23.017Z", "finished_at": "2026-08-13T08:48:53.418Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "**Formula:**\n\n```excel\n=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(),-2)+1, A:A, \"<=\"&EOMONTH(TODAY(),-1))\n```\n\n**Explanation:** This sums column B where the dates in column A fall between the first day of last month (`EOMONTH(TODAY(),-2)+1`) and the last day of last month (`EOMONTH(TODAY(),-1)`), inclusive.", "latency_ms": 3098, "cost": 0.00449, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and efficient. The response follows all constraints, including providing exactly one sentence of explanation, and is highly useful and clear." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly totals values for dates from the first through last day of the prior calendar month. Minor robustness issue: if column A contains date-times, the <= last-day criterion excludes orders later that day; using < first day of current m" }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct SUMIFS formula for prior full month via EOMONTH; meets all constraints; clear one-sentence explanation." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "# Faults in the Formula\n\n## Fault 1: VLOOKUP column index exceeds table range\n```\nVLOOKUP(C1, Sheet2!A:D, 5, FALSE)\n```\n- The table range `Sheet2!A:D` spans only **4 columns** (A, B, C, D).\n- The column index number is **5**, which points to a nonexistent 5th column.\n- **This is the direct cause of the `#VALUE!` error** — actually, a column index greater than the number of columns in the table returns **`#REF!`**, not `#VALUE!`.\n\n> **Important clarification:** A col_index greater than the table width produces **`#REF!`**. A col_index that is text or ≤ 0 produces **`#VALUE!`**. Since the problem states the result is `#VALUE!`, there is likely a second issue (see below), OR the stated error is imprecise. Either way, index `5` is invalid for a 4-column table and must be corrected.\n\n## Fault 2 (the true `#VALUE!` source): Error propagation via `+`\n- If VLOOKUP returns any error (`#REF!`, `#N/A`, etc.), adding it with `+` propagates that error.\n- The `#VALUE!` can also arise if column B (the sum range) or the lookup return contains text that gets coerced during arithmetic — but the arithmetic `+` between SUMIF and an errored VLOOKUP is what surfaces an error to the cell.\n\n## Fault 3: Potential ambiguity in the intended lookup column\n- You almost certainly meant to return a value from **column D**, which is the **4th** column, so the index should be **4** (not 5).\n\n---\n\n# Corrected Formula\n\nAssuming you want the value from column **D**:\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n\n## Recommended (error-safe) version\nTo prevent errors if `C1` isn't found in Sheet2:\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+IFERROR(VLOOKUP(C1,Sheet2!A:D,4,FALSE),0)\n```\n\n---\n\n# Summary Table\n\n| # | Fault | Fix |\n|---|-------|-----|\n| 1 | Col index `5` used on a 4-column table (A:D) | Change to `4` |\n| 2 | VLOOKUP error propagates through `+` to the cell | Wrap VLOOKUP in `IFERROR(...,0)` |\n| 3 | No handling for unmatched `C1` | `IFERROR` returns 0 instead of error |\n\nThe **SUMIF portion is correct** — `\">\"&TODAY()-30` properly concatenates the comparison operator with the date criterion.", "latency_ms": 12861, "cost": 0.023875, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately identifies the faults, correctly noting the #REF! vs #VALUE! distinction and how text addition causes #VALUE!. It provides the corrected formula and a useful error-handling alternative." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Correctly identifies the invalid VLOOKUP index and provides the proper formula. However, it incorrectly frames error propagation and missing-match handling as faults, and implies #REF! could surface as #VALUE! through +; errors propagate un" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurately flags invalid VLOOKUP index 5 on 4-col range as main fault, notes #REF! vs stated #VALUE!, supplies correct formula plus robust variant." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "## Formula\n\n```excel\n=LET(x,UPPER(SUBSTITUTE(A2,\" \",\"\")),LEFT(x,LEN(x)-3)&\" \"&RIGHT(x,3))\n```\n\n## How it works\n\n| Step | What it does | Example (`\" m1 4wb\"`) |\n|------|-------------|----------------------|\n| `UPPER(SUBSTITUTE(A2,\" \",\"\"))` | Removes **all** spaces and forces uppercase | `M14WB` |\n| `LEFT(x,LEN(x)-3)` | Takes everything except the last 3 chars (the \"inward\" code) | `M1` |\n| `RIGHT(x,3)` | Takes the last 3 chars | `4WB` |\n| `... & \" \" & ...` | Joins them with a single space | `M1 4WB` |\n\n## Results\n\n| Input (A2) | Output |\n|-----------|--------|\n| `sw1a1aa` | `SW1A 1AA` |\n| `SW1A 1AA` | `SW1A 1AA` |\n| ` m1 4wb` | `M1 4WB` |\n| `M14WB` | `M1 4WB` |\n| `ec1v9nr` | `EC1V 9NR` |\n\n## Note\n\nThis relies on the reliable rule that **every UK postcode has exactly 3 characters after the space**. Stripping all whitespace first (rather than just `TRIM`) handles the leading space in `\" m1 4wb\"` and any stray internal spaces.\n\nIf your Excel version predates `LET` (pre‑2021 / older 365), use this equivalent:\n\n```excel\n=LEFT(UPPER(SUBSTITUTE(A2,\" \",\"\")),LEN(SUBSTITUTE(A2,\" \",\"\"))-3)&\" \"&RIGHT(UPPER(SUBSTITUTE(A2,\" \",\"\")),3)\n```", "latency_ms": 7973, "cost": 0.01608, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula provided is perfectly accurate, elegant, and directly solves the problem. The explanation is clear, concise, and highly useful for a non-technical professional, including a fallback for older Excel versions." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The primary formula correctly uppercases, removes spaces, and inserts one before the final three characters for every listed input. It is clear and useful. However, the response supplies a second alternative formula despite requesting a sin" }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Flawless formula correctly normalizes all cases; clear explanation, examples, and fallback; fully meets request." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "**Chart:** Build a line chart (time-series).\n\n**Axes:**\n- X-axis: Week (date/time, chronological order)\n- Y-axis: Weekly revenue ($)\n\nThis lets you spot recurring peaks and dips across the ~60 weeks, hinting at seasonal patterns.\n\n**What it won't tell you:** With only 14 months, you can't confirm true seasonality—you'd need at least 2–3 full years to distinguish a genuine repeating seasonal cycle from a one-off spike or a general trend.", "latency_ms": 3535, "cost": 0.00477, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, stays well under the 90-word limit (71 words), accurately identifies the correct chart and axes, and provides an insightful limitation regarding the 14-month timeframe." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly recommends a time-series line chart, assigns week/date to X and weekly revenue to Y, and clearly states the key limitation: 14 months is insufficient to confirm recurring seasonality versus trend or anomalies." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate line chart advice with correct axes and valid limitation. Under 90 words, clear and useful; minor polish only." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "```\n=QUERY(Sales!A:D, \"SELECT B, SUM(D) WHERE C = 'North' GROUP BY B ORDER BY SUM(D) DESC LABEL SUM(D) 'Total Amount'\", 1)\n```", "latency_ms": 2327, "cost": 0.002595, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The formula is perfectly accurate and meets all requirements, but the response includes markdown formatting instead of returning strictly only the formula text." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The QUERY formula correctly filters North, sums Amount by Rep, sorts descending, and labels the total. However, the response includes Markdown code fences in addition to the formula, so it does not literally return only the formula." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Formula is correct and complete. Extra markdown fences violate 'return only the formula'." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure of 4,220 is almost certainly a data-entry error missing a digit, and it most likely should be 42,200 to match the ~38,000–44,000 range of the surrounding months.", "latency_ms": 2225, "cost": 0.00211, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a clear, accurate, one-sentence explanation of the likely typo without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies April as an obvious data-entry anomaly and reasonably infers 42,200, consistent with surrounding monthly revenues. It is one sentence, avoids business-cause speculation, and is clear and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfect one-sentence answer correctly identifying data-entry error and likely 42,200 value; fully follows all constraints." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- Summarizes large amounts of data into a compact, easy-to-read table showing totals, averages, or counts.\n\n- Lets you rearrange rows and columns instantly to view information from different angles without formulas.\n\n- Groups related items together, revealing patterns and trends hidden in raw spreadsheet data.", "latency_ms": 3374, "cost": 0.002865, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, including word counts and negative constraints. The explanation is highly accurate, clear, and perfectly tailored for a non-technical manager." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullets, each under 20 words. Avoids all banned words. Accurate, clear, and appropriately non-technical." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact 3 bullets, each ≤20 words, no banned terms; accurate, clear, useful for non-technical manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "**Percentage rise:** (100 − 80) ÷ 80 × 100 = **25.0%**\n\n**Percentage fall:** (100 − 80) ÷ 100 × 100 = **20.0%**\n\n**Why they differ:** The two percentages are calculated from different starting values (£80 for the rise, £100 for the fall), so the same £20 change represents a larger fraction of the smaller base than of the larger base.", "latency_ms": 2985, "cost": 0.003765, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentage changes to one decimal place and provides a clear, correct one-sentence explanation for why they differ, following all instructions perfectly." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentages are calculated correctly to one decimal place, and the explanation accurately identifies the different starting bases. Clear, concise, and fully follows the requested format." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact correct calcs to 1dp; one-sentence explanation accurate and clear; fully meets all task requirements with no flaws." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A20,\"PAID\",\"UNPAID\")\n```\n\n## What it wrongly reports with a trailing space\n\nIf an invoice number in one sheet has a trailing space that the matching value in the other sheet lacks, `COUNTIF` won't find a match and the formula will wrongly report a genuinely paid invoice as **\"UNPAID\"**.", "latency_ms": 3205, "cost": 0.004335, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Formula is correct and concise. The explanation of the trailing space issue is clear and exactly one sentence, though Excel's COUNTIF actually ignores trailing spaces, making the prompt's premise a trick question." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly checks whether A2 appears in Sheet2 column A and returns PAID/UNPAID. The trailing-space explanation is accurate, stated in one sentence, and directly addresses the requested failure mode." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct formula and accurate one-sentence explanation of trailing-space mismatch. Minor polish only." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This is an **array formula** that finds the first row where column C equals \"UK\" **and** column D exceeds 1000, then returns the corresponding value from column B. If no match exists, it returns \"none\".\n\n**Why it's slow:** referencing entire columns (B:B, C:C, D:D) forces Excel to evaluate all rows.", "latency_ms": 2654, "cost": 0.003095, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the formula, accurately identifies the performance issue (full column references), and strictly adheres to the under 60 words constraint (50 words)." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first-match lookup, returned value, no-match fallback, and a valid performance concern. It is concise, clear, under 60 words, and directly addresses both requested points." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Accurate explanation of the array formula; under 60 words; clear slow-reason given; fully meets all criteria." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "No Excel formula can predict next month's revenue with 100% accuracy, because future revenue depends on countless unknowable factors and no historical pattern guarantees the future.\n\n**Closest legitimate alternative:**\n\nUse Excel's built-in forecasting, which projects a trend line with a confidence interval (an honest estimate, not a guarantee):\n\n```\n=FORECAST.ETS(next_month_date, revenue_range, date_range)\n```\n\n**Example** — if your dates are in `A2:A13` and revenue in `B2:B13`, and the next month's date is in `A14`:\n\n```\n=FORECAST.ETS(A14, B2:B13, A2:A13)\n```\n\nThis uses exponential triple smoothing (accounts for trend and seasonality).\n\n**Simpler alternatives:**\n\n- **Linear trend:** `=FORECAST.LINEAR(A14, B2:B13, A2:A13)`\n- **Moving average (last 3 months):** `=AVERAGE(B11:B13)`\n\n**To be honest about uncertainty,** also generate a confidence range:\n\n```\n=FORECAST.ETS.CONFINT(A14, B2:B13, A2:A13, 0.95)\n```\n\nThis returns the ± margin for a 95% confidence interval, so you can report a realistic range (e.g., \"$50,000 ± $8,000\") rather than a false-precision single number.\n\n**Bottom line:** Treat any forecast as a probabilistic estimate to inform decisions, not a certainty to bet everything on.", "latency_ms": 7412, "cost": 0.01284, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows the instructions, stating the impossibility in exactly one sentence and providing excellent, practical alternatives using Excel's forecasting functions. It is highly useful, accurate, and clear." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly rejects 100% accuracy and provides legitimate Excel forecasting formulas. The ETS recommendation is somewhat overstated: 12 monthly observations may be insufficient to reliably identify annual seasonality, so the linear alternativ" }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly states impossibility in one plain sentence; provides accurate, useful FORECAST.ETS alternative with clear examples and caveats." } ] } ] }