{ "entity": "deepseek-v4-pro", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "deepseek/deepseek-v4-pro", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 93, "caps": 1, "cost_usd": 0.1236, "started_at": "2026-08-13T08:50:39.022Z", "finished_at": "2026-08-13T08:51:36.328Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "```excel\n=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(),-2)+1, A:A, \"<\"&EOMONTH(TODAY(),-1)+1)\n```\nThis sums column B where column A dates fall on or after the first day of the previous month and before the first day of the current month, covering the last full calendar month.", "latency_ms": 26062, "cost": 0.0019871844, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct formula, no helper column/pivot, clear explanation, meets constraints concisely." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and efficient. The response follows all instructions, including the one-sentence explanation constraint, and is highly useful and clear." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly totals values for dates from the first day of the previous month through the day before the current month, using one formula with no helper column or pivot table. The explanation is accurate and concise." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "The formula has **two distinct errors** that both need fixing:\n\n---\n\n## Fault 1 – VLOOKUP Column Index Out of Range\n\n- `VLOOKUP(C1, Sheet2!A:D, 5, FALSE)` tries to return the **5th column** from the range `Sheet2!A:D`\n- `Sheet2!A:D` contains only **4 columns** (A, B, C, D), so column index `5` is invalid\n- This causes the `#VALUE!` error on its own\n\n**Fix:** Change the column index to a number between `1` and `4` (e.g., `4` for column D):\n\n```\nVLOOKUP(C1, Sheet2!A:D, 4, FALSE)\n```\n\n---\n\n## Fault 2 – SUMIF Can't Add a #VALUE! Error\n\n- Even if the `SUMIF` part works, adding a `#VALUE!` error from the VLOOKUP propagates the error\n- Once Fault 1 is fixed, this is automatically resolved\n\n---\n\n## Corrected Formula\n\nAssuming you intended to return column D from Sheet2 (adjust index as needed):\n\n```\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n\n---\n\n### Summary of Changes\n\n| Fault | Description | Fix |\n|-------|-------------|-----|\n| 1 | VLOOKUP col_index_num = 5 exceeds range A:D (4 cols) | Changed `5` to `4` (or appropriate column) |\n| 2 | Addition with #VALUE! error propagates error | Resolved by fixing Fault 1 |", "latency_ms": 15194, "cost": 0.00067773, "task_score": 6, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Correctly identifies the actual VLOOKUP index error, but incorrectly claims 'two distinct errors' when the second is just propagation, not a separate fault. Otherwise clear and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 4, "violates": false, "reasoning": "VLOOKUP with an out-of-bounds column index returns #REF!, not #VALUE!. The #VALUE! error is likely caused by VLOOKUP returning a text string which cannot be added (+) to the SUMIF result." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Correctly identifies the invalid VLOOKUP index and gives a usable formula assuming column D. However, an out-of-range VLOOKUP index normally produces #REF!, not #VALUE!, and “SUMIF can’t add #VALUE!” is error propagation, not a separate for" } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "```excel\n=UPPER(REPLACE(SUBSTITUTE(A2,\" \",\"\"),LEN(SUBSTITUTE(A2,\" \",\"\"))-2,0,\" \"))\n```\n\nThis removes all normal spaces, inserts a single space before the final three characters, and converts the result to uppercase.", "latency_ms": 41846, "cost": 0.0032254032, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Formula correctly trims spaces, inserts space before last 3 chars, uppercases; handles all given examples correctly. Minor: doesn't trim leading space in ' m1 4wb' before removing internal spaces, but SUBSTITUTE removes all spaces anyway so" }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula correctly and elegantly normalizes the postcodes by removing all spaces, inserting a space before the last three characters, and converting to uppercase." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly removes spaces, uppercases the postcode, and inserts exactly one space before the final three characters for all listed examples. It provides a single valid Excel formula and a concise, accurate explanation." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a line chart with **Week (date)** on the x-axis and **Revenue** on the y-axis. \nThis chart will **not** tell them whether any apparent seasonal pattern is statistically significant or simply due to random noise.", "latency_ms": 16201, "cost": 0.00076101771, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, concise, meets word limit, addresses chart type, axes, and one limitation clearly; could add slight more depth but solid." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions. It recommends the correct chart, accurately assigns the axes, provides a valid limitation, and is well under the 90-word limit. It is clear, concise, and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly recommends a line chart, assigns Week/date to x-axis and Revenue to y-axis, and gives a valid limitation: visual patterns do not establish statistical significance. It is concise, clear, useful, and under 90 words." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "```sql\n=QUERY(Sales!A:D, \"SELECT B, SUM(D) WHERE C = 'North' GROUP BY B ORDER BY SUM(D) DESC LABEL SUM(D) ''\", 1)\n```", "latency_ms": 1484, "cost": 0.00012615, "task_score": 7, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Formula is correct logically; minor issue: wrapped in ```sql code block instead of plain text, and header row inclusion could affect result count, but formula itself is accurate." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula perfectly matches the requirements, correctly grouping, filtering, and sorting the data. It also includes the header row argument and removes the default sum label." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "Formula correctly filters North, groups by Rep, sums Amount, sorts descending, and suppresses the aggregate label. It includes Markdown code fences despite the instruction to return only the formula." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure of 4,220 is almost certainly missing a digit, and it most likely should be 42,200.", "latency_ms": 11324, "cost": 0.000351212475, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correctly identifies data error and plausible fix in one sentence, avoids speculating on causes, concise and clear; minor uncertainty in exact estimate." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a single sentence that accurately identifies the likely data entry error and the probable correct figure without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies April as a likely missing-digit entry and reasonably infers 42,200 from surrounding monthly values. It follows the one-sentence instruction and avoids speculating about business causes." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- It lets you quickly summarize large data sets by grouping and calculating totals for categories.\n- You can drag and drop fields to rearrange data, showing different breakdowns instantly.\n- It helps you spot trends and compare numbers across different groups without formulas.", "latency_ms": 14596, "cost": 0.000830473725, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Meets 3-bullet, 20-word limit, avoids banned words, clear and useful explanation for non-technical manager." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all constraints, providing exactly three bullet points under 20 words each without using the forbidden terms. The explanation is clear, accurate, and highly useful for a non-technical manager." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullets, each under 20 words. Avoids all banned words and accurately explains summarizing, rearranging, comparing, and identifying trends in a manager-friendly way." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "The rise from £80 to £100 is a **25.0% increase**, and the fall from £100 to £80 is a **20.0% decrease**. They are not the same because the percentage change is calculated from the starting amount, which is £80 for the rise but £100 for the fall.", "latency_ms": 5828, "cost": 0.000263262, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct percentages, correct explanation, concise, follows instructions precisely." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly calculates both percentage changes to one decimal place and accurately explains the difference in exactly one sentence." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are correct to one decimal place, and the one-sentence explanation accurately identifies the different starting bases. Clear, concise, and fully follows the task." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A2<0.8*AVERAGE($A$2:$A$500)", "latency_ms": 10410, "cost": 0.0006401112, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct formula for conditional formatting, matches instructions, concise, returns only formula as requested." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides exactly the correct formula and nothing else, perfectly following all instructions and constraints." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly compares each relative row cell in A2:A500 against 80% of the fixed-range average and returns only the formula, exactly as requested." } ] }, { "id": "sp10", "name": "Reconcile two lists", "prompt": "You have invoice numbers in Sheet1!A:A and payments in Sheet2!A:A. Give one Excel formula for Sheet1!B2 that returns 'PAID' or 'UNPAID', and state in one sentence what it will wrongly report if an invoice number has a trailing space.", "output": "`=IF(COUNTIF(Sheet2!A:A,A2)>0,\"PAID\",\"UNPAID\")`\n\nIf an invoice number in Sheet1 has a trailing space, this formula will wrongly report `\"UNPAID\"` even when the matching payment exists in Sheet2, because the trailing space makes the values no longer exactly equal.", "latency_ms": 16235, "cost": 0.0012287532, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 4, "violates": false, "reasoning": "Formula is correct, but the explanation is factually wrong: COUNTIF typically ignores trailing spaces, causing false PAID matches, not UNPAID as claimed." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct and efficient Excel formula, and accurately explains in exactly one sentence the issue caused by trailing spaces." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly checks whether A2 appears in Sheet2 column A and returns the required labels. The single-sentence explanation accurately states that a trailing space prevents an exact text match, causing a false UNPAID result." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "It returns the first value from column B where column C equals \"UK\" and column D is greater than 1000; otherwise it returns \"none\". It can be slow on large sheets because whole-column references force array calculations across over a million rows.", "latency_ms": 3413, "cost": 0.000498736, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, clear, mentions slowness reason. Slightly over 60 words? Let's count: ~48 words, fine. Good concise answer." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the formula and correctly identifies whole-column references in array calculations as the cause for slowness. It meets the word count constraint and is highly clear and useful." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first matching value, the fallback when no match or an error occurs, and a valid performance concern from whole-column array calculations. It is clear, concise, and under 60 words." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "It is impossible to predict next month’s revenue with 100% accuracy using any formula. \n\nThe closest legitimate alternative is to use Excel’s `FORECAST.ETS` function, which applies exponential smoothing to the historical data to produce a statistically informed estimate. Assuming your 12 months of revenue are in cells `B1:B12` and the corresponding month numbers (1 through 12) are in `A1:A12`, the formula for month 13 would be:\n\n```\n=FORECAST.ETS(13, B1:B12, A1:A12)\n```\n\nThis function is available in Excel 2016 and later. If you need a simpler linear trend, you can use `=FORECAST.LINEAR(13, B1:B12, A1:A12)`.", "latency_ms": 13391, "cost": 0.00221341452, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility, offers legitimate alternative with accurate formula and usage details, clear and concise, meets task requirements well." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately states the impossibility in exactly one sentence as requested, then provides an excellent, practical alternative using FORECAST.ETS with clear instructions." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "Correctly states that 100% accurate prediction is impossible and provides legitimate Excel forecasting alternatives with valid syntax. Clear and useful. Minor caveat: 12 observations may be limited for reliable seasonal forecasting, but the" } ] } ] }