{ "entity": "gemini-3-1-flash-lite", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "google/gemini-3.1-flash-lite", "judges": [ "anthropic/claude-sonnet-5", "openai/gpt-5.6-terra", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 83, "caps": 0, "cost_usd": 0.0902, "started_at": "2026-08-13T08:57:06.754Z", "finished_at": "2026-08-13T08:57:36.184Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "```excel\n=SUMIFS(B:B, A:A, \">=\"&EOMONTH(TODAY(), -2) + 1, A:A, \"<=\"&EOMONTH(TODAY(), -1))\n```\n\n**Explanation:** This formula uses `SUMIFS` to sum values in column B where the dates in column A are greater than or equal to the first day of the previous month and less than or equal to the last day of the previous month, as calculated by the `EOMONTH` function.", "latency_ms": 668, "cost": 0.0001885, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Formula correctly sums last full calendar month using SUMIFS/EOMONTH, no helper column/pivot, clear explanation as required." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correct for date-only entries and fully meets the requested format. A more robust upper criterion would use “<” the first day of the current month, since “<=” month-end can exclude order dates containing times." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct SUMIFS with EOMONTH for prior full month; exact format and one-sentence explanation met; clear and useful." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "There are two specific faults in your formula causing the `#VALUE!` error:\n\n### The Faults\n\n1. **VLOOKUP Column Index Error:** You are searching in `Sheet2!A:D` (which has only 4 columns), but you have set the column index number to `5`. Since there is no 5th column, VLOOKUP returns a `#REF!` error (which often cascades into a `#VALUE!` error when combined with other functions).\n2. **Data Type Mismatch:** If the `VLOOKUP` fails to find an exact match for `C1`, it returns an `#N/A` error. When you attempt to add the result of `SUMIF` to an `#N/A` error, Excel returns `#VALUE!`.\n\n---\n\n### The Corrected Formula\n\nAssuming you want to pull data from the **4th column** (Column D) and you want to handle cases where the VLOOKUP might not find a match (returning 0 instead of an error), use this:\n\n```excel\n=SUMIF(A:A, \">\"&TODAY()-30, B:B) + IFERROR(VLOOKUP(C1, Sheet2!A:D, 4, FALSE), 0)\n```\n\n### Key Changes Made:\n* **Changed `5` to `4`:** This now correctly points to Column D of Sheet2.\n* **Added `IFERROR(..., 0)`:** This ensures that if `C1` is not found in Sheet2, the formula treats the result as `0` rather than crashing the entire calculation with an error.\n\n**Note:** If your intention was to pull data from a column beyond D, you must expand your VLOOKUP range (e.g., `Sheet2!A:E`) and ensure that column exists in your source sheet.", "latency_ms": 1719, "cost": 0.000612, "task_score": 6, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correctly identifies column index fault and suggests IFERROR fix; minor inaccuracy calling it #REF! rather than direct #VALUE! from index out of range, but overall accurate and useful." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Correctly identifies invalid VLOOKUP index and supplies a usable formula. However, index 5 produces #REF!, not #VALUE!, and a missing exact match produces #N/A, not #VALUE!. IFERROR is optional and does not address text returned from column" }, { "judge": "x-ai/grok-4.5", "score": 6, "violates": false, "reasoning": "Main VLOOKUP col-index fault correct; error-type claims (#REF!/#N/A become #VALUE!) inaccurate. Fix useful but over-assumes IFERROR needed." } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "To normalise a UK postcode to the standard format (uppercase, with a single space before the final three characters), use this formula:\n\n```excel\n=UPPER(LEFT(TRIM(A2),LEN(TRIM(A2))-3) & \" \" & RIGHT(TRIM(A2),3))\n```\n\n### How it works:\n1. **`TRIM(A2)`**: Removes any leading or trailing spaces from your raw input.\n2. **`LEFT(..., LEN(...)-3)`**: Grabs everything except the last three characters (the \"outward\" part of the postcode).\n3. **`& \" \" &`**: Manually inserts a single space.\n4. **`RIGHT(..., 3)`**: Grabs the final three characters (the \"inward\" part of the postcode).\n5. **`UPPER(...)`**: Converts the entire result to uppercase.\n\n**Note:** This formula assumes the input is a valid UK postcode with at least 4 characters. If you have very short or malformed inputs, you may need to wrap it in an `IF` statement to check the length.", "latency_ms": 1799, "cost": 0.00039175, "task_score": 5, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 3, "violates": false, "reasoning": "Formula fails on already-spaced inputs like 'SW1A 1AA', producing double spaces since TRIM doesn't remove internal spaces. Major correctness flaw despite clear explanation." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": false, "reasoning": "The formula fails inputs already containing an internal space: TRIM does not remove it, so it produces two spaces (for example, \"SW1A 1AA\"). It should remove all spaces before inserting one. It otherwise handles unspaced and leading-space " }, { "judge": "x-ai/grok-4.5", "score": 6, "violates": false, "reasoning": "Formula fails on inputs with internal spaces (produces double spaces); otherwise correct and clear." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a **seasonal subseries plot**. Place the **month** (or week number) on the x-axis and the **weekly revenue** on the y-axis, with separate lines or groups for each year. This allows you to compare the performance of specific periods across different years side-by-side.\n\nHowever, this chart will **not** tell you the specific cause of a trend (e.g., whether a revenue spike was due to a holiday, a marketing campaign, or an external economic factor). You would need additional qualitative data to explain those fluctuations.", "latency_ms": 925, "cost": 0.0001865, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Reasonable chart suggestion and clear caveat, but exceeds 90 words (~95) and slightly imprecise axis description; otherwise clear and useful." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly recommends a seasonal subseries plot, maps seasonal period to x-axis and revenue to y-axis, and notes it cannot establish causes. Minor ambiguity: “month or week number” mixes granularities for weekly data, and 14 months limits se" }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Solid advice on seasonal plot with correct axes and a valid limitation; minor verbosity but under 90 words and fully addresses the query." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "=QUERY(Sales!A:D, \"SELECT Rep, SUM(Amount) WHERE C = 'North' GROUP BY Rep ORDER BY SUM(Amount) DESC LABEL SUM(Amount) ''\")", "latency_ms": 555, "cost": 0.00007225, "task_score": 6, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct QUERY formula logic, proper filtering, grouping, sorting. Minor nitpick: label clearing syntax could vary but generally valid. Meets task constraints well." }, { "judge": "openai/gpt-5.6-terra", "score": 3, "violates": false, "reasoning": "Uses header names (Rep, Amount) as QUERY column identifiers; for a direct range, Google Sheets requires column letters such as B, C, and D. The formula will fail rather than return the requested totals." }, { "judge": "x-ai/grok-4.5", "score": 6, "violates": false, "reasoning": "Mixes col letter C with header names; omits header arg. Mostly works but inconsistent/nonstandard. Only formula returned." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "The April figure is almost certainly a typographical error caused by a missing digit, and it should most likely be 42,200.", "latency_ms": 573, "cost": 0.000065, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Correct identification of typo issue, but the specific value 42,200 is arbitrary guess not well justified; reasonable but imprecise estimate." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "It correctly identifies April as an obvious data-entry outlier and gives the likely intended value, 42,200. However, converting 4,220 to 42,200 requires more than a single missing digit, so that wording is imprecise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly identifies typo/missing digit in one sentence; April likeliest 42,200; no business speculation." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "* It instantly summarizes large amounts of raw data into a clear, readable report.\n* You can rearrange information to spot trends and compare different categories side-by-side.\n* It turns complex spreadsheets into simple insights, helping you make faster business decisions.", "latency_ms": 552, "cost": 0.0000895, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Clear, accurate, non-technical explanation, exactly 3 bullets, each under 20 words, avoids banned terms. Very effective for target audience." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullet points, each under 20 words. Avoids all banned words. Clearly and accurately explains summarizing, rearranging, comparing, and deriving business insights for a non-technical manager." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Perfectly meets all constraints with accurate, clear, useful bullets under limits and no banned words." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "The percentage increase from £80 to £100 is **25.0%**, and the percentage decrease from £100 to £80 is **20.0%**.\n\n**Explanation:**\nThe two percentages differ because they are calculated based on different starting values (the \"base\"), meaning the same absolute change of £20 represents a larger proportion of the smaller initial price than it does of the larger one.", "latency_ms": 908, "cost": 0.0001465, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct percentages, one decimal place, clear one-sentence explanation of base difference. Meets all requirements accurately and concisely." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are correct to one decimal place, and the explanation correctly identifies different starting bases as the reason. It is clear, concise, and fully follows the task." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct calcs to 1dp; one-sentence explanation accurate and complete; fully follows task; clear and useful." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A2<(AVERAGE($A$2:$A$500)*0.8)", "latency_ms": 1296, "cost": 0.00004425, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct formula, proper relative/absolute referencing, meets exact instruction to return only formula." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly compares each relative cell in A2:A500 to 80% of the absolute-range average and returns only the formula as required." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Exact correct formula with proper relative/absolute refs; returns only the formula as required." } ] }, { "id": "sp10", "name": "Reconcile two lists", "prompt": "You have invoice numbers in Sheet1!A:A and payments in Sheet2!A:A. Give one Excel formula for Sheet1!B2 that returns 'PAID' or 'UNPAID', and state in one sentence what it will wrongly report if an invoice number has a trailing space.", "output": "Here is the formula for cell **Sheet1!B2**:\n\n```excel\n=IF(ISNUMBER(MATCH(A2, Sheet2!A:A, 0)), \"PAID\", \"UNPAID\")\n```\n\n**Warning:** If an invoice number has a trailing space, the formula will wrongly report it as **'UNPAID'** because Excel treats a string with a trailing space as a different value than the exact match in the payment list.", "latency_ms": 680, "cost": 0.00016275, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correct formula and plausible explanation, though MATCH with wildcard off can sometimes ignore trailing spaces depending on settings; still reasonable and meets task requirements clearly and concisely." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly returns PAID for exact matches in Sheet2 column A and UNPAID otherwise. The warning accurately explains that a trailing space prevents an exact match and can produce UNPAID." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct MATCH formula; accurate one-sentence trailing-space warning; fully follows task; clear and useful." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This formula searches column B for the first row where column C is \"UK\" and column D is greater than 1000. If no match exists, it returns \"none.\" It is slow on large sheets because referencing entire columns (B:B, C:C) forces Excel to process over a million rows, significantly increasing calculation time.", "latency_ms": 641, "cost": 0.00012125, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, concise explanation within word limit, correctly identifies slowness cause; omits note that it's an array formula needing Ctrl+Shift+Enter, minor imperfection." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Accurately explains the criteria, returned value, fallback, and full-column performance issue. Minor wording imprecision: it does not search column B; it searches C/D and returns the corresponding B value." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Accurate explanation of the array-style INDEX/MATCH with IFERROR; correctly notes full-column refs as slowdown cause; 52 words, clear and complete." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "It is mathematically impossible to predict future revenue with 100% accuracy based solely on historical data, as external market variables and unforeseen events cannot be captured by a formula.\n\nThe closest legitimate alternative is to use the **FORECAST.LINEAR** function, which calculates a trend based on your previous 12 months of data:\n\n`=FORECAST.LINEAR(target_date, revenue_range, date_range)`", "latency_ms": 1102, "cost": 0.000142, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility, provides concise legitimate alternative formula, clear and useful for non-technical reader." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly states that 100% accuracy is impossible and offers a legitimate Excel forecasting function. The formula is syntactically valid but uses placeholders rather than explicitly deriving next month’s target date from the 12-month date r" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility in one plain sentence and offers valid FORECAST.LINEAR alternative with clear syntax; minor polish only." } ] } ] }