{
 "entity": "grok-4-5",
 "entity_type": "model",
 "category": "Spreadsheets",
 "suite": "suite-2026-08-tasks/spreadsheets",
 "model": "x-ai/grok-4.5",
 "judges": [
  "anthropic/claude-sonnet-5",
  "google/gemini-3.1-pro-preview",
  "openai/gpt-5.6-terra"
 ],
 "judge_protocol": "judge-2026-08",
 "anchors": "solo-2026-08-a",
 "score": 94,
 "caps": 2,
 "cost_usd": 0.1533,
 "started_at": "2026-08-13T08:59:05.106Z",
 "finished_at": "2026-08-13T09:00:34.197Z",
 "tasks": [
  {
   "id": "sp1",
   "name": "Excel formula from a description",
   "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.",
   "output": "**Formula:**  \n`=SUMIFS(B:B,A:A,\">=\"&EOMONTH(TODAY(),-2)+1,A:A,\"<=\"&EOMONTH(TODAY(),-1))`\n\nThis sums order values in column B only for dates in column A that fall on or after the first day of the prior calendar month and on or before the last day of that same month (relative to today).",
   "latency_ms": 8144,
   "cost": 0.0031844,
   "task_score": 9,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 9,
     "violates": false,
     "reasoning": "Formula correctly sums last full calendar month using SUMIFS with EOMONTH, no helper column/pivot, with clear explanation."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The formula is perfectly accurate and efficient. The explanation is exactly one sentence as requested, and all constraints are met. Highly useful and clear."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "Correct for date-only values and follows all requested format constraints. Minor robustness issue: using <= the prior month’s final date excludes orders on that date if column A contains date-time values; a < first day of current month crit"
    }
   ]
  },
  {
   "id": "sp2",
   "name": "Fix a broken formula",
   "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.",
   "output": "**Faults in the formula:**\n- `VLOOKUP(C1,Sheet2!A:D,5,FALSE)` requests column **5**, but `Sheet2!A:D` contains only **4** columns (A=1, B=2, C=3, D=4). This is invalid and produces a `#REF!` error (which then propagates through the `+`).\n- No other syntax/logic errors are present in the `SUMIF` (the criteria `\">\"&TODAY()-30` is valid for date serials) or the overall structure. (If the looked-up value is text, the `+` could separately yield `#VALUE!`, but the column index is the clear break given the stated sheet layout.)\n\n**Corrected formula:**\n```excel\n=SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,4,FALSE)\n```\n(If the value you actually need is not in column D, adjust the range to include the desired column and set the index accordingly; with only A–D available the maximum valid index is 4.)",
   "latency_ms": 80081,
   "cost": 0.0266484,
   "task_score": 9,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 4,
     "violates": false,
     "reasoning": "Misidentifies error type: invalid VLOOKUP col_index gives #REF!, not #VALUE!. Task states #VALUE! occurs, so real cause (likely text mismatch) is missed. Correction incomplete/inaccurate despite clear formatting."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 9,
     "violates": false,
     "reasoning": "Accurately identifies the out-of-bounds column index in VLOOKUP. Correctly notes that this causes a #REF! error, and explains that the #VALUE! error mentioned in the prompt likely comes from adding text."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 9,
     "violates": false,
     "reasoning": "Correctly identifies the invalid VLOOKUP column index and supplies the appropriate four-column correction. It accurately notes this normally causes #REF!, not #VALUE!. Minor caveat: choosing column 4 assumes D is the intended return column,"
    }
   ]
  },
  {
   "id": "sp3",
   "name": "Clean messy data",
   "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.",
   "output": "```excel\n=LET(s,UPPER(SUBSTITUTE(A2,\" \",\"\")),LEFT(s,LEN(s)-3)&\" \"&RIGHT(s,3))\n```",
   "latency_ms": 17620,
   "cost": 0.0060144,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 8,
     "violates": false,
     "reasoning": "Correct, concise formula handling all cases via LET; trims spaces, uppercases, inserts space before last 3 chars. Minor: no TRIM for leading/trailing spaces beyond internal ones, but SUBSTITUTE handles that."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The formula perfectly and concisely solves the problem using LET to remove spaces, convert to uppercase, and insert exactly one space before the final three characters."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly removes spaces, uppercases, and inserts exactly one space before the final three characters for every listed input. It is a single Excel formula, uses A2 as required, and is concise and clear."
    }
   ]
  },
  {
   "id": "sp4",
   "name": "Choose the right chart",
   "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.",
   "output": "**Line chart.**  \nPut week (or date) on the X-axis and revenue on the Y-axis.  \n\nIt will not tell them the *cause* of any seasonality.",
   "latency_ms": 5238,
   "cost": 0.0017884,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 6,
     "violates": false,
     "reasoning": "Reasonable answer under 90 words, but weak: line chart with 14 months doesn't clearly show seasonality; better would be overlaying years or heatmap. Meets format constraints though."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly follows all instructions, stays well under the word limit, correctly identifies the axes, and provides a valid limitation of the chart."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly recommends a line chart, assigns time to the X-axis and revenue to the Y-axis, and gives a valid limitation: it cannot identify causes of observed seasonality. Clear, concise, and well under 90 words."
    }
   ]
  },
  {
   "id": "sp5",
   "name": "Google Sheets QUERY",
   "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.",
   "output": "```\n=QUERY(Sales!A:D,\"select B,sum(D) where C='North' group by B order by sum(D) desc\",1)\n```",
   "latency_ms": 5341,
   "cost": 0.0018864,
   "task_score": 5,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 7,
     "violates": false,
     "reasoning": "Correct QUERY logic; header row argument should be 1 but data starts row1 with headers, using 'A:D' with header 1 is fine. Minor label 'group by B' may need 'label sum(D)'."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 5,
     "violates": true,
     "reasoning": "The formula is perfectly accurate and meets all logical requirements, but the response includes markdown backticks, violating the constraint to return ONLY the formula."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 5,
     "violates": true,
     "reasoning": "The formula correctly filters North, totals Amount by Rep, and sorts descending. It includes Markdown code fences, so it is not literally only the formula."
    }
   ]
  },
  {
   "id": "sp6",
   "name": "Spot the error in the numbers",
   "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.",
   "output": "The April figure is almost certainly a data-entry error missing a trailing zero and should most likely be 42,200.",
   "latency_ms": 5922,
   "cost": 0.0022804,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 6,
     "violates": false,
     "reasoning": "Correctly identifies likely data entry error; suggested value 42,200 is plausible but arbitrary rather than exactly derivable; concise, meets one-sentence and no-speculation constraint."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly follows all instructions, accurately identifying the likely typo and the corrected figure in a single, clear sentence without speculating on business causes."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly identifies April as an apparent data-entry error and infers 42,200 by adding a trailing zero; it is one sentence and avoids business-cause speculation."
    }
   ]
  },
  {
   "id": "sp7",
   "name": "Pivot logic in words",
   "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.",
   "output": "- A pivot table rearranges your spreadsheet data to show summaries in a clear grid.\n- You choose what goes across the top and down the side to group related items.\n- It quickly adds up numbers or counts items based on your groupings for easy insights.",
   "latency_ms": 7822,
   "cost": 0.0032164,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 8,
     "violates": false,
     "reasoning": "Clear, accurate, avoids banned words, meets bullet count and word limits, useful for non-technical manager; slightly generic but solid."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly follows all instructions, including the exact bullet count, word limits per bullet, and negative constraints. The explanation is highly accurate, clear, and perfectly tailored for a non-technical manager."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Exactly 3 bullet points, each under 20 words, with no banned terms. Clear, accurate, manager-friendly explanation of rearranging, grouping, and summarizing data."
    }
   ]
  },
  {
   "id": "sp8",
   "name": "Percentage change trap",
   "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.",
   "output": "**25.0% rise, 20.0% fall.**  \n\nThey differ because each percentage is taken relative to a different base value (£80 vs £100).",
   "latency_ms": 4696,
   "cost": 0.0020084,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct percentages, clear one-sentence explanation, concise and accurate; meets all requirements."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response accurately calculates both percentages to one decimal place and provides a clear, one-sentence explanation of why they differ, following all instructions perfectly."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Both percentage changes are correct to one decimal place, and the single-sentence explanation accurately identifies the different base values. Clear, concise, and fully follows the task."
    }
   ]
  },
  {
   "id": "sp9",
   "name": "Conditional formatting rule",
   "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.",
   "output": "=A2<0.8*AVERAGE($A$2:$A$500)",
   "latency_ms": 4850,
   "cost": 0.0020424,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct formula for CF applied to A2:A500 with relative row, absolute range; matches 20% below average condition, concise and exact as requested."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides the exact, correct Excel formula and follows the negative constraint to return nothing else."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "The formula correctly compares each cell in A2:A500 to 80% of the absolute-range average, using a relative cell reference and fixed average range. It returns only the requested formula."
    }
   ]
  },
  {
   "id": "sp10",
   "name": "Reconcile two lists",
   "prompt": "You have invoice numbers in Sheet1!A:A and payments in Sheet2!A:A. Give one Excel formula for Sheet1!B2 that returns 'PAID' or 'UNPAID', and state in one sentence what it will wrongly report if an invoice number has a trailing space.",
   "output": "=IF(COUNTIF(Sheet2!A:A,A2)>0,\"PAID\",\"UNPAID\")\n\nIt will wrongly report \"UNPAID\" for a paid invoice whose number has a trailing space (exact-match failure).",
   "latency_ms": 4934,
   "cost": 0.0021484,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 8,
     "violates": false,
     "reasoning": "Correct formula and accurate explanation of trailing space issue; concise, meets task requirements well."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides a perfectly accurate Excel formula and clearly explains the trailing space issue in exactly one sentence, meeting all constraints and requirements."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "The formula correctly tests whether A2 appears in Sheet2 column A and returns the required labels. The single sentence accurately explains that a trailing space prevents an exact match and can yield UNPAID."
    }
   ]
  },
  {
   "id": "sp11",
   "name": "Explain a formula back",
   "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.",
   "output": "It returns the first value from column B where C=\"UK\" and D>1000 (array match), or \"none\" on error/no match. Slow on large sheets as it scans full columns.",
   "latency_ms": 6209,
   "cost": 0.0022424,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 8,
     "violates": false,
     "reasoning": "Accurate, concise, under 60 words, mentions slowness cause; could be slightly more precise on array formula nature."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly explains the formula's function and correctly identifies full-column array processing as the cause of slowness, all well within the 60-word limit."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Accurately explains the first matching value in column B, the fallback on errors/no match, and the performance issue from scanning full-column references. It is concise, clear, and under 60 words."
    }
   ]
  },
  {
   "id": "sp12",
   "name": "Refuse the impossible",
   "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.",
   "output": "**It is not possible to predict next month's revenue with 100% accuracy from the previous 12 months using any Excel formula.**\n\nClosest legitimate alternative (linear trend forecast):  \nAssume months 1–12 are in `A1:A12` and corresponding revenues in `B1:B12`. Next month’s forecast is:\n\n```excel\n=FORECAST.LINEAR(13, B1:B12, A1:A12)\n```\n\n(This fits a straight-line trend to the historical data and extrapolates one step; accuracy is not guaranteed and depends entirely on how well the past linear pattern continues.)",
   "latency_ms": 7355,
   "cost": 0.0025284,
   "task_score": 10,
   "judges": [
    {
     "judge": "anthropic/claude-sonnet-5",
     "score": 9,
     "violates": false,
     "reasoning": "Correctly states impossibility in one sentence, provides valid formula and alternative, clear and concise."
    },
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response correctly identifies that 100% accuracy is impossible in a single sentence, then provides a highly appropriate and practical alternative formula (FORECAST.LINEAR) with clear instructions."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly states that 100% accurate prediction is impossible, then provides a valid Excel linear forecasting formula with clear assumptions and an appropriate accuracy caveat. It follows the requested structure and is concise."
    }
   ]
  }
 ]
}