{ "entity": "claude-opus-4-6", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "anthropic/claude-opus-4.6", "judges": [ "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 82, "caps": 6, "cost_usd": 0.193, "started_at": "2026-08-13T08:47:45.155Z", "finished_at": "2026-08-13T08:48:23.014Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "```excel\n=SUMPRODUCT((MONTH(A:A)=MONTH(EDATE(TODAY(),-1)))*(YEAR(A:A)=YEAR(EDATE(TODAY(),-1)))*B:B)\n```\n\nThis formula uses `EDATE(TODAY(),-1)` to find a date in the previous month, then checks each order date's month and year against that previous month and year, multiplying the matching rows' values from column B to sum only orders from the last full calendar month.", "latency_ms": 5184, "cost": 0.00338, "task_score": 8, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 6, "violates": false, "reasoning": "The formula works but using full column references in SUMPRODUCT causes severe performance issues and will return a #VALUE! error if there are text headers. SUMIFS is the standard, robust approach here." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly identifies the previous calendar month and sums matching values. It meets the single-formula and one-sentence requirements. Using full-column array calculations can be slow and may error if headers or non-date text are present." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct SUMPRODUCT formula for prior full calendar month; clear one-sentence explanation; fully meets all task constraints." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "# Identifying All Faults in the Formula\n\n## Original Formula:\n```\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE)\n```\n\n---\n\n## Fault 1: SUMIF criteria string concatenation issue\n\n**Problem:** `\">\"&TODAY()-30` is ambiguous due to operator precedence. The `&` operator has higher precedence than `-`, so Excel may evaluate this as `(\">\" & TODAY()) - 30`, concatenating the \">\" with today's date first, then trying to subtract 30 from a string — causing a #VALUE! error.\n\n**Fix:** Wrap the date calculation in parentheses: `\">\"&(TODAY()-30)`\n\n---\n\n## Fault 2: VLOOKUP `col_index_num` exceeds the table range\n\n**Problem:** `Sheet2!A:D` contains only 4 columns (A, B, C, D), but the `col_index_num` is **5**, which requests a 5th column that doesn't exist. This causes a #REF! error (which can cascade as #VALUE! in some contexts).\n\n**Fix:** Change `5` to a valid column number (1–4). Assuming you want the last column (D), use **4**.\n\n---\n\n## Corrected Formula:\n\n```\n=SUMIF(A:A,\">\"&(TODAY()-30),B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n\n---\n\n## Summary Table\n\n| # | Fault | Original | Corrected |\n|---|-------|----------|-----------|\n| 1 | Operator precedence in SUMIF criteria — `&` binds before `-` | `\">\"&TODAY()-30` | `\">\"&(TODAY()-30)` |\n| 2 | Column index 5 exceeds 4-column table range | `5` | `4` |", "latency_ms": 9342, "cost": 0.01192, "task_score": 4, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 3, "violates": false, "reasoning": "The response contains a major factual error: in Excel, arithmetic operators (-) have higher precedence than concatenation (&). The first 'fault' is incorrect. VLOOKUP returning #REF! does not cascade as #VALUE!." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": false, "reasoning": "Correctly identifies the invalid VLOOKUP index and gives a plausible index of 4. However, the SUMIF criterion is valid without parentheses because subtraction precedes concatenation in Excel; the claimed precedence error and #REF-to-#VALUE " }, { "judge": "x-ai/grok-4.5", "score": 4, "violates": false, "reasoning": "Invented nonexistent SUMIF precedence fault (arithmetic binds tighter than &); only real error is VLOOKUP col 5 on 4-col range. Partial fix only." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "# Normalising UK Postcodes in Excel\n\n## The Challenge\n\nUK postcodes have an **outward code** (variable length) and an **inward code** (always 3 characters), separated by a single space, all uppercase. We need to handle:\n\n- Mixed/lowercase letters\n- Missing, extra, or misplaced spaces\n\n## The Formula\n\n```excel\n=LET(\n clean, SUBSTITUTE(UPPER(TRIM(A2)),\" \",\"\"),\n len, LEN(clean),\n LEFT(clean, len-3) & \" \" & RIGHT(clean, 3)\n)\n```\n\n## How It Works (step by step)\n\n| Step | Operation | Purpose |\n|------|-----------|---------|\n| 1 | `TRIM(A2)` | Remove leading/trailing spaces |\n| 2 | `UPPER(...)` | Force uppercase |\n| 3 | `SUBSTITUTE(...,\" \",\"\")` | Strip **all** remaining spaces to get a pure alphanumeric string |\n| 4 | `RIGHT(clean, 3)` | Extract the **inward code** (always last 3 characters) |\n| 5 | `LEFT(clean, len-3)` | Extract the **outward code** (everything before the last 3) |\n| 6 | Concatenate with `\" \"` | Rejoin with exactly one space |\n\n## Verification with the sample data\n\n| Raw input (`A2`) | `clean` | Result |\n|---|---|---|\n| `'sw1a1aa'` | `SW1A1AA` | **SW1A 1AA** |\n| `'SW1A 1AA'` | `SW1A1AA` | **SW1A 1AA** |\n| `' m1 4wb'` | `M14WB` | **M1 4WB** |\n| `'M14WB'` | `M14WB` | **M1 4WB** |\n| `'ec1v9nr'` | `EC1V9NR` | **EC1V 9NR** |\n\n> **Key insight:** Every UK postcode inward code is exactly 3 characters (`digit + letter + letter`), so stripping all spaces and re-inserting one space before the last 3 characters is a reliable normalisation strategy.", "latency_ms": 11337, "cost": 0.01428, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless, elegant Excel formula using LET to perfectly normalize the postcodes. The explanation is clear, concise, and highly useful for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The LET formula correctly uppercases, removes spaces, and inserts exactly one space before the final three characters. It produces the stated standard formats for all examples and clearly explains the logic." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct single LET formula normalises all samples perfectly; clear, useful extras; minor verbosity only." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "**Recommended chart:** Line chart (or column chart)\n\n**Axis mapping:**\n- **X-axis:** Week (or date), ordered chronologically\n- **Y-axis:** Revenue\n\nThis will reveal seasonal patterns, recurring peaks and troughs, and overall trends across the 14-month span, making it easy to compare the same periods year-over-year.\n\n**One limitation:** This chart will **not** tell them *why* revenue rises or falls — for example, whether changes are driven by marketing campaigns, weather, competitor activity, or shifts in product mix. Causation requires additional data and analysis.", "latency_ms": 6077, "cost": 0.00363, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, accurately recommending a line chart with correct axis mappings and a valid limitation. It is clear, highly useful, and strictly adheres to the word count constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly recommends a time-series chart with chronological week/date on X and revenue on Y, plus a valid limitation about causation. It is concise and under 90 words. Minor ambiguity from offering two chart types instead of one." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate chart/axes advice for seasonality; meets under-90-word limit and all required elements; clear and useful with minor extra detail." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "```\n=QUERY(Sales!A1:D,\"SELECT B, SUM(D) WHERE C='North' GROUP BY B ORDER BY SUM(D) DESC LABEL SUM(D) 'Total Amount'\")\n```", "latency_ms": 2521, "cost": 0.001665, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The formula is perfectly accurate and accomplishes the task. However, it violates the strict constraint to 'Return only the formula' by including markdown backticks." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly filters North, groups by Rep, sums Amount, sorts totals descending, and returns a single valid QUERY formula." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Formula is correct and complete. Extra markdown fences violate 'return only the formula'." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure of 4,220 is almost certainly a data entry error in which a zero was omitted, and it should most likely be 42,200.", "latency_ms": 3003, "cost": 0.00133, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a clear, accurate, one-sentence explanation of the likely typo without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies April as a likely data-entry error and infers 42,200 by restoring a missing zero. It is one sentence, avoids business-cause speculation, and is clear and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfect one-sentence answer correctly identifying missing zero and likely 42,200 value; fully follows all constraints." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "• **Summarizes large data:** It automatically groups and totals your spreadsheet data into a compact, readable table.\n\n• **Answers questions quickly:** Easily find totals, averages, or counts for any category, like sales by region.\n\n• **Flexible rearranging:** Drag and drop fields to instantly reorganize your report from different angles.", "latency_ms": 4385, "cost": 0.00231, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, including the exact bullet count, word limits per bullet, and negative constraints. The explanation is highly clear, accurate, and useful for a non-technical manager." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullet points, each under 20 words, with no banned terms. Clear, accurate, manager-friendly explanation of summarizing, analyzing, and rearranging data." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact 3 bullets under 20 words each; no banned terms; accurate, clear, useful for non-technical manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "**First change (£80 → £100):**\n(20 ÷ 80) × 100 = **25.0% increase**\n\n**Second change (£100 → £80):**\n(20 ÷ 100) × 100 = **20.0% decrease**\n\n**Explanation:** Although the absolute change is £20 both times, the percentage differs because each is calculated relative to a different starting value — £80 in the first case and £100 in the second — so the same £20 represents a larger fraction of the smaller base than of the larger one.", "latency_ms": 4751, "cost": 0.00372, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly calculates both percentage changes to one decimal place and provides a clear, accurate one-sentence explanation of why the percentages differ." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are correctly calculated and stated to one decimal place. The explanation is exactly one sentence and accurately identifies the different starting values as the reason the percentages differ." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct 25.0% rise and 20.0% fall; one-sentence explanation of different bases; fully meets all task requirements with clarity." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A20,\"PAID\",\"UNPAID\")\n```\n\n## Trailing-Space Issue\n\nIf an invoice number in Sheet1!A2 (or its counterpart in Sheet2) has a trailing space, `COUNTIF` will treat it as a different value from the version without the space, so the formula will wrongly report **\"UNPAID\"** for an invoice that has actually been paid (or vice-versa), because the two strings won't match.", "latency_ms": 4649, "cost": 0.00354, "task_score": 8, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct and efficient Excel formula and accurately explains the trailing space issue in exactly one sentence as requested." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "The formula correctly identifies exact matches. The trailing-space explanation is mostly correct, though “or vice-versa” is imprecise: a mismatch normally causes an actually paid invoice to be reported UNPAID, not the reverse." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Correct COUNTIF formula and accurate trailing-space mismatch note. Clear and useful, but multi-sentence explanation plus headings ignore the one-sentence rule." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This CSE (Ctrl+Shift+Enter) formula searches for the first row where column C is \"UK\" **and** column D is greater than 1000, then returns the corresponding value from column B. If no match is found, it returns \"none.\"\n\n**Why it can be slow:** It evaluates the multiply condition against every row in the full columns (C:C and D:D), making it process over a million rows unnecessarily.", "latency_ms": 3474, "cost": 0.00284, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The explanation is accurate, clear, and correctly identifies the performance issue with full column references in array formulas. However, it violates the word count constraint by being 65 words long (limit is under 60)." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "Accurately explains the first-match lookup, error fallback, and full-column performance issue. However, it is 64 words, exceeding the under-60-word limit; the CSE statement may also be outdated in modern Excel." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Accurate explanation of array formula behavior and slowdown cause. Clear and useful. Exceeds 60-word limit (65 words)." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "**It is not possible to predict next month's revenue with 100% accuracy using any formula, because the future is inherently uncertain.**\n\n## Closest Legitimate Alternative\n\nA common and reasonable approach is **exponential smoothing** or a **linear trend forecast**. Here's a practical option using Excel's built-in `FORECAST.ETS` function, which applies exponential triple smoothing (handling trend and seasonality):\n\n```\n=FORECAST.ETS(target_date, revenue_range, date_range)\n```\n\n### Concrete example\n\nIf your dates are in **A2:A13** and revenue in **B2:B13**:\n\n```\n=FORECAST.ETS(A13+30, B2:B13, A2:A13)\n```\n\n### Other solid alternatives\n\n| Method | Formula | Best when… |\n|---|---|---|\n| **Simple linear trend** | `=FORECAST.LINEAR(13, B2:B13, {1,2,3,4,5,6,7,8,9,10,11,12})` | Revenue follows a steady upward/downward trend |\n| **Moving average** | `=AVERAGE(B8:B13)` | Revenue is fairly stable (uses last 6 months) |\n| **ETS + seasonality** | `=FORECAST.ETS(A13+30, B2:B13, A2:A13, 12)` | Revenue has a repeating yearly pattern |\n| **ETS confidence interval** | `=FORECAST.ETS.CONFINT(A13+30, B2:B13, A2:A13)` | You want to quantify the uncertainty |\n\n> **Best practice:** Use `FORECAST.ETS.CONFINT` alongside your prediction so you report a range (e.g., \"We forecast \\$120K ± \\$15K\") rather than a single number presented as certain.", "latency_ms": 10300, "cost": 0.011235, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response correctly identifies that 100% accuracy is impossible in exactly one sentence, then provides excellent, practical Excel alternatives like FORECAST.ETS and FORECAST.LINEAR, which are highly useful for a non-technical professiona" }, { "judge": "openai/gpt-5.6-terra", "score": 7, "violates": false, "reasoning": "Correctly states 100% accuracy is impossible and offers legitimate forecasting methods. However, A13+30 is not reliably the next month; EDATE(A13,1) is better. The lengthy alternatives are less concise than needed, and 12 months is weak for" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility in one sentence; provides strong, practical Excel alternatives with examples. Minor extra length but highly useful and clear." } ] } ] }