{ "entity": "gpt-5-6-sol", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "openai/gpt-5.6-sol", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 97, "caps": 0, "cost_usd": 0.1787, "started_at": "2026-08-13T08:56:14.508Z", "finished_at": "2026-08-13T08:56:41.847Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "```excel\n=SUMIFS(B:B,A:A,\">=\"&EOMONTH(TODAY(),-2)+1,A:A,\"<\"&EOMONTH(TODAY(),-1)+1)\n```\n\nThis sums order values where the order date falls from the first day of the previous calendar month up to, but not including, the first day of the current month.", "latency_ms": 4874, "cost": 0.006095, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct formula for last full month, single cell, no pivot/helper column, clear concise explanation." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and efficient. The explanation is exactly one sentence and clearly explains how the formula works. All constraints were followed." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct SUMIFS formula for prior full calendar month; clear one-sentence explanation; fully meets all task requirements." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "**Fault:** `VLOOKUP` requests column **5**, but `Sheet2!A:D` contains only **4 columns**. This normally produces `#REF!`, not `#VALUE!`.\n\nAssuming the intended return column is **D**, use:\n\n```excel\n=SUMIF(A:A,\">\"&(TODAY()-30),B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n\nThe original `SUMIF` criterion is valid; the added parentheses simply make the date calculation explicit. If a fifth return column is actually required, the lookup range must extend to column E instead:\n\n```excel\n=SUMIF(A:A,\">\"&(TODAY()-30),B:B)+VLOOKUP(C1,Sheet2!A:E,5,FALSE)\n```", "latency_ms": 10903, "cost": 0.021015, "task_score": 6, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Identifies correct core fault (column index) but incorrectly claims this yields #REF! not #VALUE!, missing that VLOOKUP error propagates as #VALUE! when added; slightly inaccurate explanation but useful fix." }, { "judge": "google/gemini-3.1-pro-preview", "score": 6, "violates": false, "reasoning": "Identifies the column index error but fails to explain why the formula returns #VALUE! (likely because the VLOOKUP returns text which cannot be added to the SUMIF result)." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly identifies sole VLOOKUP col_index fault vs A:D range; notes error-type mismatch; supplies valid fixes. Minor: assumes intent without more faults." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "```excel\n=LET(x,UPPER(SUBSTITUTE(TRIM(A2),\" \",\"\")),LEFT(x,LEN(x)-3)&\" \"&RIGHT(x,3))\n```\n\nThis removes existing spaces, converts to uppercase, then inserts one space before the final three characters.", "latency_ms": 4725, "cost": 0.00528, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, concise formula meeting requirements; uses modern LET which may not work in older Excel, minor compatibility caveat not mentioned." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is flawless. It correctly removes all spaces, converts to uppercase, and accurately inserts a single space before the final three characters using the LET function for efficiency." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Flawless Excel formula correctly normalizes all given postcodes to standard UK format." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a **time-series line chart**. Put **week/date** on the x-axis and **weekly revenue** on the y-axis; optionally mark months or holidays to reveal repeating patterns.\n\nIt will **not prove seasonality or explain its cause**—14 months includes only one full annual cycle, so apparent patterns may reflect promotions, holidays, or random variation.", "latency_ms": 5359, "cost": 0.005875, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, clear, meets word limit, correct chart/axes recommendation, and correctly notes limitation; slightly technical phrasing for non-technical audience." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, stays well under the word limit, correctly identifies the axes, and provides a highly insightful limitation regarding the 14-month timeframe." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfectly meets all criteria: correct chart advice, under 90 words, fully addresses axes and limitation, clear and useful." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "=QUERY(Sales!A:D,\"select B, sum(D) where C = 'North' group by B order by sum(D) desc label B 'Rep', sum(D) 'Total Amount'\",1)", "latency_ms": 1971, "cost": 0.00168, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct QUERY formula with proper grouping, filtering, sorting, and labels; minor nit: assumes header row handling with 1, generally correct syntax." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides exactly the correct QUERY formula as requested, following all constraints including returning only the formula." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct QUERY formula matching all requirements exactly; returns only the formula as specified." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure almost certainly has a missing zero and should be 42,200.", "latency_ms": 1491, "cost": 0.001, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correctly identifies missing digit issue and gives plausible estimate; slightly odd exact figure (42,200) rather than range, but concise and follows instructions." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a concise, one-sentence answer that correctly identifies the typo without speculating on business causes." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfect one-sentence answer correctly flags missing zero in April and gives likeliest fix; fully meets all constraints." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- A pivot table quickly summarizes large data lists into useful totals, counts, or averages.\n- It rearranges categories to compare results from different perspectives without changing the original data.\n- You can filter details and spot trends, patterns, or exceptions that support better decisions.", "latency_ms": 5361, "cost": 0.00763, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Three bullets, all under 20 words, avoids banned words, clear and accurate explanation for a non-technical manager." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains pivot tables to a non-technical manager. It strictly follows all constraints: exactly 3 bullets, under 20 words each, and avoids the banned words. Highly clear and useful." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact 3 bullets under 20 words each; accurate plain-language explanation; no banned terms; fully meets all constraints." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "The price rose by **25.0%** and then fell by **20.0%**. They differ because each percentage change is calculated from a different starting price (£80 versus £100).", "latency_ms": 1623, "cost": 0.00155, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct percentages, one decimal place, clear one-sentence explanation, concise and accurate." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentage changes to one decimal place and provides a clear, one-sentence explanation for why they differ, following all instructions perfectly." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct 25.0% rise and 20.0% fall; one-sentence explanation accurate; fully meets all task requirements with clarity." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A20,\"PAID\",\"UNPAID\")\n```\n\nIf the invoice number in `A2` has a trailing space but the payment entry does not, it will wrongly report `UNPAID`.", "latency_ms": 5302, "cost": 0.00525, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Formula correct and matches task; explanation accurate about trailing space mismatch causing UNPAID. Concise, meets format, minor lack of TRIM discussion but not required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is correct and efficient. The explanation of the trailing space issue is accurate and provided in exactly one sentence as requested." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct COUNTIF formula; accurate one-sentence note on trailing-space mismatch causing false UNPAID." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "The formula returns the first value from column B where column C equals “UK” and column D is greater than 1000. If no match or another error occurs, it returns “none.” It may be slow because it evaluates entire columns in array calculations.", "latency_ms": 3700, "cost": 0.003135, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate explanation, correct slowness reason, concise, likely under 60 words; minor lack of mention it's an array formula needing Ctrl+Shift+Enter in old Excel." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the formula, correctly identifies why it would be slow (evaluating entire columns in array calculations), and strictly adheres to the word count limit." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Accurate explanation of the array formula; under 60 words; correctly notes full-column array eval as slowdown cause. Flawless." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "Predicting next month’s revenue with 100% accuracy from only 12 prior months is impossible; the closest legitimate alternative is a linear-trend forecast (dates in `A2:A13`, revenue in `B2:B13`): `=FORECAST.LINEAR(EDATE(MAX(A2:A13),1),B2:B13,A2:A13)`.", "latency_ms": 7185, "cost": 0.009775, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility, gives valid alternative formula, concise and clear, meets task requirements well." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, correctly stating the impossibility in a single sentence and providing a highly accurate, practical Excel formula for the closest alternative." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly states impossibility in one sentence; provides accurate, usable FORECAST.LINEAR alternative. Fully meets all criteria." } ] } ] }