{ "entity": "deepseek-v4-flash", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "deepseek/deepseek-v4-flash", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 93, "caps": 0, "cost_usd": 0.1097, "started_at": "2026-08-13T08:49:42.971Z", "finished_at": "2026-08-13T08:50:39.017Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "`=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(),-2)+1, A:A, \"<=\"&EOMONTH(TODAY(),-1))`\n\nThis formula sums all values in column B where the date in column A is between the first and last day of the previous full calendar month, using `EOMONTH` to dynamically calculate those boundaries based on today's date.", "latency_ms": 7919, "cost": 0.00021252, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct SUMIFS formula with proper EOMONTH boundaries, no helper column/pivot, concise explanation as requested." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and dynamically calculates the previous calendar month. The explanation is exactly one sentence, and all constraints are followed." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correct for date-only entries and clearly meets the requested format. Minor limitation: using <= the prior month’s last date can exclude orders with time values later that day; a < first day of current month criterion is more robust." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "The formula has several faults. Here's a breakdown of each issue and the corrected formula.\n\n## Faults Identified\n\n1. **VLOOKUP column index out of range** \n `VLOOKUP(C1,Sheet2!A:D,5,FALSE)` references column index 5, but Sheet2 only has columns **A to D** (four columns). This will cause `#VALUE!` or `#REF!`.\n\n2. **Potential mismatched data types or array inconsistency** \n The `SUMIF` returns a number, but if `VLOOKUP` fails (e.g., C1 not found with `FALSE`), it returns `#N/A`, and adding `#N/A` to a number gives `#VALUE!`. However, the column index error is the most direct cause here.\n\n3. **No error handling** \n If `VLOOKUP` cannot find the lookup value, the whole formula errors out. Adding error handling prevents this.\n\n## Corrected Formula\n\nAssuming you meant to return a value from column D (the 4th column in range A:D):\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B) + IFERROR(VLOOKUP(C1,Sheet2!A:D,4,FALSE),0)\n```\n\nIf you actually need column **E** (5th column), you must expand the range in Sheet2 to include column E, e.g., `Sheet2!A:E`:\n\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B) + IFERROR(VLOOKUP(C1,Sheet2!A:E,5,FALSE),0)\n```\n\n## Summary of Changes\n\n| Fault | Fix |\n|-------|-----|\n| VLOOKUP column index 5 on A:D | Changed to 4 (if targeting D) or expanded range to A:E (if targeting E) |\n| No error handling | Wrapped VLOOKUP in `IFERROR(..., 0)` to return 0 instead of error |\n\nThe most likely intended column is **D** (index 4), so the first corrected formula above is the recommended fix.", "latency_ms": 46505, "cost": 0.0000805497, "task_score": 7, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correctly identifies main fault (column index out of range) and adds error handling; clear and useful, though slightly verbose and adds unrequested IFERROR change beyond strict fix." }, { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Correctly identifies the primary fault (column index 5 for a 4-column range) and provides a corrected formula with good error handling. Minor inaccuracy: #N/A + number returns #N/A, not #VALUE!." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Correctly identifies the out-of-range VLOOKUP index and provides a workable correction. However, it incorrectly says #N/A added to a number becomes #VALUE! (it remains #N/A), and labels speculative data-type/error-handling issues as formula" } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "=REPLACE(UPPER(TRIM(SUBSTITUTE(A2,\" \",\"\"))),LEN(UPPER(TRIM(SUBSTITUTE(A2,\" \",\"\"))))-2,0,\" \")", "latency_ms": 35894, "cost": 0.00033702074, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Formula correctly trims spaces, removes internal spaces, uppercases, and inserts space before last 3 chars, matching standard UK postcode format for all examples." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula perfectly normalizes the postcodes by removing all spaces, converting to uppercase, and inserting a single space before the last three characters. It directly answers the prompt with no unnecessary text." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly removes spaces, trims surrounding spaces, uppercases the postcode, and inserts one space before the final three characters. It is a single valid Excel formula and handles all listed examples." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a **line chart** with **weeks** (or months) on the x-axis and **revenue** on the y-axis. This shows revenue trends over time, revealing seasonal peaks or troughs. One thing this chart will **not** tell them is whether the seasonality is statistically significant—it only shows visual patterns, not proof that the variation isn’t random.", "latency_ms": 2889, "cost": 0.00001881, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, concise, clear recommendation with axes and limitation stated; word count under 90; well within instructions." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, accurately recommends a line chart with correct axes, provides a valid limitation, and stays well under the 90-word limit." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly recommends a line chart with time on the x-axis and revenue on the y-axis, and accurately notes it cannot establish statistical significance. Minor ambiguity: saying “weeks (or months)” weakens the requested field-specific guidanc" } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "=QUERY('Sales'!A:D, \"select B, sum(D) where C = 'North' group by B order by sum(D) desc\", 1)", "latency_ms": 4652, "cost": 0.000116714, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correct formula logic, but header row 1 assumption with full column A:D and label '1' may cause header misinterpretation issues; minor label formatting could differ but generally functional and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate, meets all requirements, and adheres strictly to the constraint of returning only the formula." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly queries Sales columns, filters Region to North, groups by Rep, sums Amount, sorts descending by total, and specifies one header row. It returns only the requested formula." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure of 4,220 is almost certainly a decimal error (off by a factor of 10), and it most likely should be 42,200.", "latency_ms": 1018, "cost": 0.00001988, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correctly identifies likely decimal/typo error and gives plausible corrected figure in one sentence, avoiding business speculation as instructed; concise and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a clear, accurate, one-sentence explanation without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "It correctly identifies April as a likely factor-of-ten data-entry error and gives 42,200 as the most likely corrected value, in exactly one sentence without speculating about causes." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- Summarizes large tables into a clear, compact summary.\n- Lets you drag and drop fields to rearrange data instantly.\n- Finds totals, averages, and counts across different categories.", "latency_ms": 1219, "cost": 0.00001116, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, avoids banned words, meets bullet count and length, clear for non-technical manager; minor redundancy in first bullet ('summary' twice conceptually)." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Perfectly follows all constraints. Explains pivot tables clearly and concisely for a non-technical manager without using banned jargon, keeping each bullet well under the 20-word limit." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullet points, each under 20 words. Avoids all forbidden terms. Accurately and clearly explains summarization, field rearrangement, and calculations for a non-technical manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "The percentage increase from £80 to £100 is (20 ÷ 80) × 100 = **25.0%**, while the percentage decrease from £100 to £80 is (20 ÷ 100) × 100 = **20.0%**; they differ because each percentage change is calculated using the starting price as the denominator, so the same absolute change of £20 represents a larger proportion of the lower starting price than of the higher one.", "latency_ms": 7394, "cost": 0.0000187796, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 10, "violates": false, "reasoning": "Correct percentages to 1dp, clear single-sentence explanation, concise and accurate." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentage changes to one decimal place and provides a clear, correct explanation in exactly one sentence, perfectly meeting all instructions." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are calculated correctly to one decimal place, and the single-sentence explanation accurately states that different starting values produce different percentage changes." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "`=A20, \"PAID\", \"UNPAID\")` \n\nIf an invoice number in Sheet1 has a trailing space (e.g., \"INV-001 \"), the formula will wrongly report `UNPAID` because it will not match the clean version in Sheet2 (e.g., \"INV-001\") due to the extra space.", "latency_ms": 5885, "cost": 0.000112232, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, simple formula; accurate explanation of trailing space issue; concise and clear for non-technical user." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is correct and efficient. The explanation is exactly one sentence and accurately describes the issue with trailing spaces causing false 'UNPAID' results." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Provides a correct COUNTIF-based formula for Sheet1!B2 and accurately explains that a trailing space in the invoice number prevents a match, causing an incorrect UNPAID result." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This formula finds the first row where column C is \"UK\" and column D > 1000, then returns the value from column B. If no match, it returns \"none\". It is slow on large sheets because it references entire columns (C:C, D:D), forcing Excel to evaluate millions of cells instead of a defined range.", "latency_ms": 4300, "cost": 0.000067678, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, concise explanation with correct reason for slowness, within 60-word limit (54 words)." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately explains the formula, correctly identifies the performance issue with whole-column references, and stays under the 60-word limit." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first matching row, returned value, fallback result, and a valid performance issue from whole-column references. It is concise, clear, and under 60 words." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "It is impossible to predict next month's revenue with 100% accuracy using any Excel formula, because revenue depends on unknown future events, market changes, and random variation.\n\n**Closest legitimate alternative:** \nUse the `FORECAST.ETS` function, which applies exponential smoothing to historical data and provides a statistically reasonable prediction based on trends and seasonality.\n\nExample formula (assuming monthly revenue data is in cells B2:B13 for the past 12 months, and you want the forecast for the next period):\n\n```\n=FORECAST.ETS(13, B2:B13, {1,2,3,4,5,6,7,8,9,10,11,12}, 1, 1)\n```\n\n- `13` = the next time period (after month 12) \n- `B2:B13` = the 12 months of revenue \n- `{1...12}` = the timeline (months 1 through 12) \n- `1` = seasonality length (assumes yearly pattern; adjust if needed) \n- `1` = data completion (handles missing data automatically) \n\nThis formula will not be 100% accurate, but it is the best built-in Excel method for forecasting from 12 months of data.", "latency_ms": 6726, "cost": 0.00005148, "task_score": 7, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correctly states impossibility and gives valid alternative, but seasonality parameter explanation is slightly inaccurate (1 means auto-detect, not length-1 season)." }, { "judge": "google/gemini-3.1-pro-preview", "score": 9, "violates": false, "reasoning": "Accurately states the impossibility in exactly one sentence and provides a highly relevant alternative (FORECAST.ETS) with a clear, usable example for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 7, "violates": false, "reasoning": "Correctly states 100% accuracy is impossible and offers a legitimate Excel forecasting method. However, it incorrectly says seasonality value 1 assumes a yearly pattern; 1 means no seasonality (12 would represent annual monthly seasonality)" } ] } ] }