{ "entity": "qwen3-7-max", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "qwen/qwen3.7-max", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 95, "caps": 0, "cost_usd": 0.2959, "started_at": "2026-08-14T12:19:12.405Z", "finished_at": "2026-08-14T12:31:48.433Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "**Formula:**\n`=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(),-2)+1, A:A, \"<\"&EOMONTH(TODAY(),-1)+1)`\n\n**Explanation:**\nThis formula uses the SUMIFS function to sum the order values in column B where the dates in column A fall on or after the first day of the previous month and strictly before the first day of the current month, both of which are dynamically calculated using the EOMONTH function.", "latency_ms": 32271, "cost": 0.011677575, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, single formula, no helper column/pivot, clear explanation; accurate logic for last full calendar month." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a perfectly accurate formula that dynamically calculates the previous calendar month. It strictly follows all constraints, including the one-sentence explanation limit, and is highly useful and clear." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly sums values for dates from the first day of the previous month through, but excluding, the first day of the current month. It meets all requested format and method constraints." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "## Faults Identified\n\n### Fault 1 — VLOOKUP Column Index Out of Range\nThe table array is `Sheet2!A:D`, which contains exactly **4 columns**:\n\n| Column | Index |\n|--------|-------|\n| A | 1 |\n| B | 2 |\n| C | 3 |\n| D | 4 |\n\nThe formula requests column index **`5`**, which doesn't exist in that range. This is the core fault.\n\n> **Note on the error type:** An out-of-range column index actually causes VLOOKUP to return **`#REF!`**, not `#VALUE!`. A **`#VALUE!`** error would surface if, after fixing the index, VLOOKUP retrieves a **text value** — because the `+` operator cannot add text to the numeric SUMIF result. This secondary failure is likely what you're seeing once the `#REF!` is resolved.\n\n---\n\n### Corrected Formula\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n\n**Change:** `5` → `4` (returns the value from column **D**, the last column in the range).\n\n---\n\n### Recommended (with error handling)\n\nTo guard against `#N/A` (no match found) or `#VALUE!` (text in column D) errors:\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+IFERROR(VLOOKUP(C1,Sheet2!A:D,4,FALSE),0)\n```\n\nThis wraps the VLOOKUP in `IFERROR`, substituting `0` whenever the lookup fails or returns a non-addable value.", "latency_ms": 312525, "cost": 0.09881025, "task_score": 7, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Correctly identifies col index fault and offers fix, but wrongly claims out-of-range index causes #REF! not #VALUE! (actually Excel gives #REF! only if index<1; index>columns also gives #REF!, so claim is accurate actually). Slight redundan" }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately identifies the out-of-range column index and correctly explains that this causes a #REF! error, while the #VALUE! error stems from adding text. Provides a clear, corrected formula and a helpful robust alternative." }, { "judge": "openai/gpt-5.6-terra", "score": 7, "violates": false, "reasoning": "Correctly identifies the invalid VLOOKUP index and gives a valid correction. However, the recommended IFERROR wrapper does not handle a successful text lookup value; the addition still returns #VALUE!. It also cannot confirm text is the rep" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "## The Formula\n\n```excel\n=LET(c, UPPER(SUBSTITUTE(A2,\" \",\"\")), LEFT(c, LEN(c)-3) & \" \" & RIGHT(c, 3))\n```\n\n## How it works\n\nThe formula does three things in sequence:\n\n| Step | Action | Example (`' m1 4wb'`) |\n|---|---|---|\n| **1. Clean** | `UPPER(SUBSTITUTE(A2,\" \",\"\"))` — strips *all* spaces and uppercases | `M14WB` |\n| **2. Outward** | `LEFT(c, LEN(c)-3)` — grabs everything except the last 3 chars | `M1` |\n| **3. Inward** | `RIGHT(c, 3)` — grabs the last 3 chars | `4WB` |\n\nThese are then joined with a single space: **`M1 4WB`**\n\n## Verification against all inputs\n\n| Raw input | Cleaned (`c`) | Result |\n|---|---|---|\n| `sw1a1aa` | `SW1A1AA` | **`SW1A 1AA`** |\n| `SW1A 1AA` | `SW1A1AA` | **`SW1A 1AA`** |\n| ` m1 4wb` | `M14WB` | **`M1 4WB`** |\n| `M14WB` | `M14WB` | **`M1 4WB`** |\n| `ec1v9nr` | `EC1V9NR` | **`EC1V 9NR`** |\n\n> **Note:** `LET` is available in Excel 365 / 2021+. For older versions, use this equivalent (same logic, just repeated):\n> ```excel\n> =LEFT(UPPER(SUBSTITUTE(A2,\" \",\"\")),LEN(UPPER(SUBSTITUTE(A2,\" \",\"\")))-3)&\" \"&RIGHT(UPPER(SUBSTITUTE(A2,\" \",\"\")),3)\n> ```", "latency_ms": 36843, "cost": 0.012831025, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, well-verified formula meeting the single-formula requirement, with helpful fallback for older Excel. Clear and concise, though slightly more verbose than strictly needed." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless, elegant Excel formula using LET, perfectly solving the problem. It includes clear explanations, a breakdown of the steps, and a fallback for older Excel versions. Highly useful and accurate." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The primary LET formula correctly uppercases, removes spaces, and inserts one space before the final three characters for every example. It is clear and well verified. However, the response supplies a second alternative formula despite the " } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a **line chart**. Place **Week** on the X-axis and **Revenue** on the Y-axis to visualize fluctuations. \n\n**What it won’t tell them:** It cannot confirm *annual* seasonality. With only 14 months of data, it lacks a second full year to prove the peaks and dips are a repeating yearly cycle rather than a one-off trend.", "latency_ms": 34939, "cost": 0.0128974, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Reasonable, correct advice under word limit; concise and clear, though could suggest overlaying years for seasonality comparison." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly answers all parts of the prompt within the word limit. It correctly identifies the chart and axes, and provides an insightful, accurate limitation regarding the 14-month dataset." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly recommends a line chart with Week on X and Revenue on Y, and gives a valid limitation: 14 months cannot establish recurring annual seasonality. It is concise, clear, and under 90 words." } ], "machine_checks": { "max_words": 90 }, "machine_violation": false, "machine_reasons": [] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "=QUERY(Sales!A:D, \"SELECT B, SUM(D) WHERE C = 'North' GROUP BY B ORDER BY SUM(D) DESC\", 1)", "latency_ms": 14601, "cost": 0.004568075, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct QUERY formula summing Amount by Rep for North, sorted descending; header row usage slightly debatable but functionally correct and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate, meets all requirements, and provides exactly the requested output with no extraneous text." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correct QUERY syntax: filters North, groups by Rep, sums Amount, sorts descending, and specifies one header row. Returns only the requested formula." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure is almost certainly a typographical error missing a zero, so it should most likely be 42,200.", "latency_ms": 17620, "cost": 0.005467825, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Correctly identifies data entry error; but doesn't justify why 42,200 specifically vs other plausible values like 42,100. Concise, follows one-sentence format, avoids business causes." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a single sentence that correctly identifies the likely typo and the probable correct figure without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "It correctly identifies April as an apparent missing-zero typographical error and gives the most plausible corrected value, 42,200, in one sentence without speculating about business causes." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "* It quickly summarizes massive spreadsheets into simple reports by grouping related information together.\n* It automatically calculates totals, averages, and counts so you can spot trends without manual math.\n* It lets you instantly rearrange rows and columns to view your data from different perspectives.", "latency_ms": 28084, "cost": 0.00936625, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Clear, accurate, concise bullets under 20 words each, avoids forbidden terms, useful for non-technical manager." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all constraints, providing exactly 3 bullet points under 20 words each, avoiding forbidden words, and clearly explaining pivot tables." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains summarization, calculations, and rearranging data. Uses exactly three bullet points, each under 20 words, and avoids all forbidden words." } ], "machine_checks": { "banned": [ "aggregate", "axis" ], "exact_bullets": 3 }, "machine_violation": false, "machine_reasons": [] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "**Percentage changes:**\n\n- **Increase** from £80 to £100: (20 ÷ 80) × 100 = **+25.0%**\n- **Decrease** from £100 to £80: (−20 ÷ 100) × 100 = **−20.0%**\n\n**Why they differ:**\n\nAlthough the absolute change is the same (£20) in both cases, the percentage is calculated relative to the *starting* value, and £20 is a larger fraction of £80 than it is of £100.", "latency_ms": 4929, "cost": 0.001662325, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct percentages to one decimal, clear explanation matches requirement, concise and well-formatted." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentage changes to one decimal place and provides a clear, one-sentence explanation for why they differ, following all instructions perfectly." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are calculated correctly to one decimal place, and the explanation is exactly one clear sentence accurately identifying the different starting-value bases." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A20, \"PAID\", \"UNPAID\")`\n\n**Trailing Space Issue:**\nIf an invoice number has a trailing space, the formula will wrongly report it as 'UNPAID' because the extra space prevents an exact text match with the corresponding payment record.", "latency_ms": 21397, "cost": 0.007373525, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct formula and accurate explanation of trailing space issue; concise and meets task requirements well." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is correct and perfectly answers the prompt. The explanation of the trailing space issue is accurate and strictly adheres to the one-sentence constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly checks whether A2 appears in Sheet2 column A and returns the requested labels. The trailing-space explanation is accurate and given in one sentence. It provides exactly one formula and is clear and concise." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "It returns the first value in column B where column C is \"UK\" and column D exceeds 1000, or \"none\" if unmatched. It is slow on large sheets because using entire column references in array multiplication forces Excel to process over one million rows, heavily taxing memory.", "latency_ms": 35474, "cost": 0.012884125, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, concise explanation under 60 words, correctly identifies array formula and full-column reference as slowness cause." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately explains the formula and correctly identifies entire column references in array operations as the cause of slowness. It perfectly adheres to the word count constraint and is highly clear." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first matching value, fallback result, and a valid large-sheet performance issue from full-column array calculations. It is clear, concise, non-technical enough, and under 60 words." } ], "machine_checks": { "max_words": 60 }, "machine_violation": false, "machine_reasons": [] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "It is impossible to predict next month's revenue with 100% accuracy using only historical data because future business outcomes are influenced by countless unpredictable real-world variables.\n\n**Closest Legitimate Alternative:**\nAssuming your previous 12 months' dates are in `A2:A13`, their corresponding revenues are in `B2:B13`, and next month's target date is in `A14`, use Excel's Exponential Smoothing function to calculate the most statistically probable forecast based on historical trends and seasonality:\n\n`=FORECAST.ETS(A14, B2:B13, A2:A13)`\n\n*(Note: Because no forecast is 100% accurate, it is highly recommended to also calculate your margin of error by finding the 95% confidence interval using `=FORECAST.ETS.CONFINT(A14, B2:B13, A2:A13, 0.95)`).*", "latency_ms": 36352, "cost": 0.01155515, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility, provides legitimate Excel formula alternative with proper syntax and confidence interval, clear and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, stating the impossibility in exactly one sentence and providing an excellent, highly useful alternative formula (FORECAST.ETS) with clear instructions for a non-technical user." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly states that 100% accuracy is impossible and provides a valid Excel forecasting formula with appropriate assumptions. Minor overclaim: FORECAST.ETS is not necessarily the “most statistically probable” forecast, and 12 data points m" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] } ], "recapped_at": "2026-08-16T08:16:35.442Z", "recap_note": "Machine constraint caps applied retroactively under protocol judge-2026-08b; judge verdicts unchanged, task scores capped where a deterministic check failed." }