{ "entity": "claude-fable-5", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "anthropic/claude-fable-5", "judges": [ "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 89, "caps": 6, "cost_usd": 0.38, "started_at": "2026-08-13T08:46:56.088Z", "finished_at": "2026-08-13T08:47:45.145Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "**Formula for D1:**\n\n```excel\n=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(),-2)+1, A:A, \"<=\"&EOMONTH(TODAY(),-1))\n```\n\n**Explanation:** `EOMONTH(TODAY(),-2)+1` returns the first day of the previous month and `EOMONTH(TODAY(),-1)` returns its last day, so SUMIFS totals only the values in column B whose dates in column A fall within that full prior calendar month.", "latency_ms": 6724, "cost": 0.01448, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and efficient. The explanation is exactly one sentence as requested, and all constraints are strictly followed." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly identifies the prior calendar month and sums matching values. Minor robustness issue: using <= the month-end date can exclude orders with time stamps later that day; using < the first day of the current month is safer." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct SUMIFS with EOMONTH for prior full month; exact format and one clear sentence; fully useful and concise." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "## Faults in the formula\n\n**1. Invalid `col_index_num` in VLOOKUP (the main error)**\n`VLOOKUP(C1, Sheet2!A:D, 5, FALSE)` asks for the **5th column** of a range that is only **4 columns wide** (A:D). This is illegal — Excel returns an error from the VLOOKUP, which then poisons the whole formula through the `+`. The index must be **4 or less** (4 = column D).\n\n*(Pedantic note: an out-of-range index technically produces `#REF!`. If you're literally seeing `#VALUE!`, it means the error is surfacing at the addition step — see fault 2.)*\n\n**2. Adding a non-numeric VLOOKUP result**\nEven with a valid column index, if the value returned by VLOOKUP is **text** (e.g., column D holds names or numbers stored as text), then `number + text` → `#VALUE!`. The return column must contain true numbers.\n\n**3. No error handling for a failed lookup**\nIf `C1` isn't found in Sheet2 column A, VLOOKUP returns `#N/A`, which again breaks the whole sum. Wrap it in `IFERROR`/`IFNA`.\n\n**What is *not* wrong:**\n- `SUMIF(A:A, \">\"&TODAY()-30, B:B)` is syntactically correct — concatenating `\">\"` with `TODAY()-30` is the proper way to build a dynamic date criterion.\n\n## Corrected formula\n\n```excel\n=SUMIF(A:A, \">\"&TODAY()-30, B:B) + IFERROR(VLOOKUP(C1, Sheet2!A:D, 4, FALSE), 0)\n```\n\nChanges made:\n- `5` → `4` (returns column D, the last available column — adjust to 2 or 3 if a different column was intended)\n- Wrapped VLOOKUP in `IFERROR(…, 0)` so a missing lookup value doesn't destroy the SUMIF result\n- Ensure Sheet2 column D contains actual numeric values, not text\n\n**Optional modern alternative** (Excel 2021/365), which avoids column-counting errors entirely:\n\n```excel\n=SUMIF(A:A, \">\"&TODAY()-30, B:B) + IFERROR(XLOOKUP(C1, Sheet2!A:A, Sheet2!D:D), 0)\n```", "latency_ms": 22374, "cost": 0.07925, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately identifies the faults, including the subtle distinction between #REF! and #VALUE! errors in this context. Provides a robust corrected formula and a modern alternative. Clear and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Correctly identifies the invalid VLOOKUP column index and gives a workable A:D-based correction. However, index 5 produces #REF!, not #VALUE!, and error propagation does not convert it. The extra alleged faults are conditional data issues, " }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurately identifies the core VLOOKUP index error and secondary issues; provides solid corrected formulas with clear explanations. Minor pedantry on error type." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "## Formula\n\n**Modern Excel (365 / 2021+), using `LET`:**\n\n```excel\n=LET(p, UPPER(SUBSTITUTE(TRIM(A2),\" \",\"\")),\n LEFT(p, LEN(p)-3) & \" \" & RIGHT(p, 3))\n```\n\n**Older Excel versions (no `LET`):**\n\n```excel\n=UPPER(LEFT(SUBSTITUTE(TRIM(A2),\" \",\"\"),LEN(SUBSTITUTE(TRIM(A2),\" \",\"\"))-3)&\" \"&RIGHT(SUBSTITUTE(TRIM(A2),\" \",\"\"),3))\n```\n\n## How it works\n\n1. **`TRIM(A2)`** — removes leading/trailing spaces (handles `' m1 4wb'`).\n2. **`SUBSTITUTE(...,\" \",\"\")`** — strips *all* internal spaces so every input is reduced to a bare code (e.g. `SW1A1AA`, `M14WB`).\n3. **`UPPER(...)`** — converts to uppercase (handles `'sw1a1aa'`, `'ec1v9nr'`).\n4. **`LEFT(p, LEN(p)-3) & \" \" & RIGHT(p, 3)`** — splits off the last three characters (the inward code, which is always 3 characters in a UK postcode) and re-joins with a single space.\n\n## Results\n\n| Input | Output |\n|---|---|\n| `sw1a1aa` | `SW1A 1AA` |\n| `SW1A 1AA` | `SW1A 1AA` |\n| ` m1 4wb` | `M1 4WB` |\n| `M14WB` | `M1 4WB` |\n| `ec1v9nr` | `EC1V 9NR` |\n\nThis works because all valid UK postcodes end in exactly three characters (digit + two letters) after the space, regardless of outward-code length.", "latency_ms": 14062, "cost": 0.04661, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The prompt explicitly asked for 'a single Excel formula', but the response provided two different formulas (one for modern Excel and one for older versions). The formulas themselves are highly accurate and well-explained." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The formulas correctly uppercase, remove spaces, and insert one space before the final three characters for all examples. However, the task requested a single formula, while the response provides two alternative formulas plus unnecessary ex" }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Correct single-purpose formulas that work on all examples; clear and useful. Violates 'a single Excel formula' by supplying two plus extras." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "**Build a line chart** (time-series plot).\n\n- **X-axis:** week (dates, in order)\n- **Y-axis:** weekly revenue\n\nWith 14 months, the overlapping ~2 months let you compare the same season across years — look for repeating peaks/dips at similar calendar points.\n\n**What it won't tell you:** *why* patterns occur. A spike could be seasonality, a promotion, a holiday, or a one-off event — the chart shows correlation with time, not causes. (Also, 14 months is barely more than one full cycle, so it can't confirm a pattern truly repeats.)", "latency_ms": 9161, "cost": 0.01499, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, accurately identifying the best chart and axes. It clearly explains what the chart won't show, and stays under the 90-word limit (88 words)." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly recommends a time-series line chart with week/date on X and weekly revenue on Y. Clearly states it cannot establish causes, while adding a useful caveat about limited repeat cycles. Under 90 words." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate line chart advice with correct axes and limitation; under 90 words; clear, useful, minor extra detail." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "`=QUERY(Sales!A:D, \"SELECT B, SUM(D) WHERE C = 'North' GROUP BY B ORDER BY SUM(D) DESC LABEL SUM(D) 'Total Amount'\", 1)`", "latency_ms": 7270, "cost": 0.00554, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 9, "violates": false, "reasoning": "The formula is perfectly accurate and meets all requirements. Minor deduction because the markdown backticks technically violate 'only the formula', though standard in formatting." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly filters North, groups by Rep, sums Amount, sorts totals descending, includes an appropriate header count, and returns only the requested formula." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct QUERY syntax and logic for the exact requirements; returns solely the formula as instructed." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure of 4,220 is almost certainly a data entry error (a dropped digit), and it most likely should be 42,200, consistent with the ~38,000–44,000 range of all other months.", "latency_ms": 7229, "cost": 0.00547, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a single sentence that correctly identifies the likely typo and the probable correct figure without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies an apparent data-entry error and infers 42,200 from the surrounding monthly range. It is one sentence, avoids business-cause speculation, and is clear and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfect one-sentence answer correctly identifying data entry error and likely true April value; fully follows all constraints." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- A pivot table quickly summarizes large amounts of data into a small, easy-to-read table.\n- It lets you group and total information, like sales by region or month, without writing formulas.\n- You can rearrange rows and columns instantly to view the same data from different angles.", "latency_ms": 6459, "cost": 0.00668, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Follows all constraints perfectly. The explanation is accurate, highly accessible to a non-technical manager, and perfectly concise within the word limits." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullets, each under 20 words, with no banned words. Accurate, clear explanation of summarizing, grouping, totaling, and rearranging data for a non-technical manager." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfectly meets all constraints with accurate, clear, concise bullets ideal for a non-technical manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "# Percentage Changes\n\n**Rise:** £80 → £100\n$$\\frac{100 - 80}{80} \\times 100 = \\frac{20}{80} \\times 100 = \\textbf{25.0\\% increase}$$\n\n**Fall:** £100 → £80\n$$\\frac{80 - 100}{100} \\times 100 = \\frac{-20}{100} \\times 100 = \\textbf{20.0\\% decrease}$$\n\n**Explanation:** The percentage changes differ because although the change in price (£20) is the same in both cases, each percentage is calculated relative to a different starting value — £80 for the rise but £100 for the fall.", "latency_ms": 6211, "cost": 0.01103, "task_score": 10, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing accurate percentage changes to one decimal place and a clear, one-sentence explanation of why they differ." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both calculations are correct to one decimal place, and the explanation accurately identifies the different starting values as the reason the percentages differ. Clear, complete, and directly responsive." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct calcs to 1dp, precise one-sentence explanation of different bases; fully meets task, clear and useful." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A20,\"PAID\",\"UNPAID\")\n```\n\nDrag it down column B as needed.\n\n**Trailing-space issue:** If the invoice number in Sheet1!A2 (or its match in Sheet2) has a trailing space, the exact-match lookup fails, so the formula will wrongly report **\"UNPAID\"** for an invoice that has actually been paid.\n\n*(Tip: wrap with TRIM to guard against this — e.g. `=IF(SUMPRODUCT(--(TRIM(Sheet2!A1:A1000)=TRIM(A2)))>0,\"PAID\",\"UNPAID\")` — noting TRIM can't be applied directly inside COUNTIF.)*", "latency_ms": 9252, "cost": 0.01842, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 9, "violates": false, "reasoning": "The formula is correct and the explanation of the trailing space issue is accurate and contained in a single sentence. The extra tip is helpful, though slightly reduces concision." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The main COUNTIF formula correctly identifies listed payments, and the trailing-space warning is accurate for mismatched values. However, it supplies a second optional Excel formula despite the request for one formula." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate COUNTIF formula and correct one-sentence trailing-space warning; minor extra tip does not detract." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This array formula finds the first row where column C equals \"UK\" AND column D exceeds 1000, then returns the corresponding value from column B. If no match exists, it returns \"none\".\n\n**Why it can be slow:** It references entire columns (C:C, D:D), forcing Excel to evaluate the array calculation across ~1 million rows instead of just the used range.", "latency_ms": 6683, "cost": 0.00954, "task_score": 5, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The explanation is perfectly accurate, clear, and correctly identifies the performance issue with full-column array references. However, it contains exactly 60 words, which violates the strict 'under 60 words' constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first matching-row lookup, returned value, and fallback. Correctly identifies full-column array evaluation as a large-sheet performance issue. It is clear, non-technical, and 59 words." }, { "judge": "x-ai/grok-4.5", "score": 5, "violates": true, "reasoning": "Accurate explanation of the array formula and valid slowdown reason. Exactly 60 words breaches 'under 60'. Clear and useful." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "Predicting next month's revenue with 100% accuracy is not possible with any formula, because future revenue depends on factors that historical data cannot fully capture.\n\n**Closest legitimate alternative:** a statistical forecast with a confidence interval.\n\n## Option 1: FORECAST.ETS (best for data with seasonality/trends)\n\nAssuming dates in A2:A13 and revenue in B2:B13:\n\n```excel\n=FORECAST.ETS(A14, B2:B13, A2:A13)\n```\n\nAdd a confidence interval to quantify uncertainty:\n\n```excel\n=FORECAST.ETS.CONFINT(A14, B2:B13, A2:A13, 0.95)\n```\n\nThis gives you a range: forecast ± confidence interval, meaning \"we're 95% confident the actual value falls within this range.\"\n\n## Option 2: TREND (simple linear trend)\n\n```excel\n=TREND(B2:B13, A2:A13, A14)\n```\n\n## Option 3: Simple moving average (if revenue is stable)\n\n```excel\n=AVERAGE(B11:B13)\n```\n\n**Practical advice:** Present the forecast as a range, not a single number (e.g., \"$95K–$110K\"), and note that 12 data points is a small sample — accuracy improves with more history and by incorporating known factors like seasonality, pipeline data, or contracted revenue.", "latency_ms": 10416, "cost": 0.02498, "task_score": 9, "judges": [ { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response correctly identifies that 100% accuracy is impossible in a single sentence, then provides excellent, practical Excel alternatives like FORECAST.ETS with clear explanations for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly states 100% accuracy is impossible and provides valid forecasting formulas with uncertainty. FORECAST.ETS is appropriate, though offering multiple options is less focused than the requested closest alternative and the confidence-i" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility in one plain sentence; supplies strong legitimate Excel alternatives with clear usage notes. Minor extra length but highly useful and accurate." } ] } ] }