{ "entity": "kimi-k3", "entity_type": "model", "category": "Spreadsheets", "suite": "suite-2026-08-tasks/spreadsheets", "model": "moonshotai/kimi-k3", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 90, "caps": 1, "cost_usd": 0.2778, "started_at": "2026-08-13T09:00:34.200Z", "finished_at": "2026-08-13T09:02:32.213Z", "tasks": [ { "id": "sp1", "name": "Excel formula from a description", "prompt": "In Excel, column A holds order dates and column B holds order values. Write a single formula for cell D1 that totals only the orders placed in the last full calendar month, relative to today's date. Give the formula and one sentence explaining it. Do not use a pivot table or a helper column.", "output": "=SUMIFS(B:B,A:A,\">=\"&DATE(YEAR(TODAY()),MONTH(TODAY())-1,1),A:A,\"<\"&DATE(YEAR(TODAY()),MONTH(TODAY()),1))\n\nIt sums column B only where column A is on or after the first day of the previous month and before the first day of the current month.", "latency_ms": 17774, "cost": 0.0195852, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct SUMIFS formula properly handling last full month, no helper column/pivot, concise explanation as required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate and handles year-end rollovers correctly. The response follows all constraints, including providing exactly one sentence of explanation." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly sums values for dates from the first day of the prior month through, but excluding, the first day of the current month; it uses one formula and provides the requested explanation." } ] }, { "id": "sp2", "name": "Fix a broken formula", "prompt": "This Excel formula returns #VALUE! : =SUMIF(A:A,\">\"&TODAY()-30,B:B)+VLOOKUP(C1,Sheet2!A:D,5,FALSE). Identify every fault and give the corrected formula. Sheet2 has columns A to D only.", "output": "## Faults\n\n**1. VLOOKUP column index is out of range (the fatal fault)**\n`VLOOKUP(C1,Sheet2!A:D,5,FALSE)` asks for the **5th** column of a range that only contains **4** (A, B, C, D). Valid indices are 1–4. Strictly speaking this throws `#REF!` — so if you're genuinely seeing `#VALUE!`, fault 2 is the likely culprit.\n\n**2. Adding a text result to a number → `#VALUE!`**\nIf the VLOOKUP (once fixed) returns text — e.g., numbers stored as text in Sheet2 — then `SUMIF(...) + \"text\"` returns `#VALUE!`. The `+` operator doesn't coerce text; `SUM()` would ignore it, `VALUE()` converts it.\n\n**3. `\">\"&TODAY()-30` — works, but only by accident of precedence**\nArithmetic binds tighter than `&`, so Excel evaluates this as `\">\"&(TODAY()-30)`, which is correct. It's not a bug, but it's fragile to read — parenthesize it explicitly.\n\n**4. Silent data risks (won't error, but will give wrong results)**\n- Dates in column A must be real dates, not text that looks like dates.\n- Values in column B must be numeric (text", "latency_ms": 112656, "cost": 0.0600774, "task_score": 3, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Correctly identifies col index and text/number issues, but final corrected formula is missing (cut off), and doesn't fully verify #VALUE! cause. Useful but incomplete deliverable." }, { "judge": "google/gemini-3.1-pro-preview", "score": 3, "violates": false, "reasoning": "The response correctly identifies the VLOOKUP column index error but cuts off mid-sentence and completely fails to provide the corrected formula as explicitly requested in the prompt." }, { "judge": "openai/gpt-5.6-terra", "score": 3, "violates": false, "reasoning": "Correctly identifies the out-of-range VLOOKUP index and notes it causes #REF!, not #VALUE!. However, it is truncated, does not provide the required corrected formula, and labels speculative text-conversion issues as faults without knowing t" } ] }, { "id": "sp3", "name": "Clean messy data", "prompt": "These postcodes were typed by hand: 'sw1a1aa', 'SW1A 1AA', ' m1 4wb', 'M14WB', 'ec1v9nr'. Give a single Excel formula that normalises any one of them to standard UK format (uppercase, one space before the final three characters). Assume the raw value is in A2.", "output": "```excel\n=LET(p,UPPER(SUBSTITUTE(A2,\" \",\"\")),LEFT(p,LEN(p)-3)&\" \"&RIGHT(p,3))\n```\n\n**How it works:**\n1. `SUBSTITUTE(A2,\" \",\"\")` strips out *all* spaces (leading, trailing, and embedded), so `' m1 4wb'` becomes `m14wb`.\n2. `UPPER(...)` capitalises everything.\n3. Since a UK inward code is always exactly 3 characters, `LEFT(p,LEN(p)-3)` takes the outward code and `RIGHT(p,3)` takes the inward code, rejoined with a single space.\n\n**Results for your examples:**\n\n| Input | Output |\n|---|---|\n| `sw1a1aa` | `SW1A 1AA` |\n| `SW1A 1AA` | `SW1A 1AA` |\n| ` m1 4wb` | `M1 4WB` |\n| `M14WB` | `M1 4WB` |\n| `ec1v9nr` | `EC1V 9NR` |\n\nIf you're on an older Excel version without `LET`, use this equivalent (it just repeats the expression):\n\n```excel\n=LEFT(UPPER(SUBSTITUTE(A2,\" \",\"\")),LEN(SUBSTITUTE(A2,\" \",\"\"))-3)&\" \"&RIGHT(UPPER(SUBSTITUTE(A2,\" \",\"\")),3)\n```\n\nOne caveat: this assumes the cleaned value is at least 4 characters long — anything shorter will return an error, which is arguably useful since it flags invalid entries.", "latency_ms": 5952, "cost": 0.0152592, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, working formula with clear explanation and fallback for older Excel; well-tested against examples; slightly verbose but accurate and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless, elegant Excel formula using LET, explains it clearly, and offers a backward-compatible alternative. It perfectly addresses the task and is highly useful for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": true, "reasoning": "The primary LET formula correctly normalises every listed input. Explanation and examples are accurate and clear. However, the task requested a single formula, while the response also supplies a second older-Excel alternative." } ] }, { "id": "sp4", "name": "Choose the right chart", "prompt": "A shop has 14 months of weekly revenue and wants to see whether sales are seasonal. In under 90 words, say which chart to build, which fields go on which axis, and one thing this chart will NOT tell them.", "output": "Build a **line chart** of the time series. Put **week (date) on the x-axis** and **weekly revenue on the y-axis**. To make seasonality visible, overlay a second line for the same weeks a year earlier, or color points by month/quarter so recurring peaks and dips stand out—with 14 months you can compare roughly two cycles.\n\nWhat it won't tell you: **why** the pattern exists. It shows *that* sales rise and fall, not whether the cause is holidays, weather, promotions, or pricing.", "latency_ms": 9974, "cost": 0.0087432, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Clear chart recommendation, correct axes, valid limitation stated. Slightly over 90 words (~85, actually within limit) — count is fine. Minor: could be more concise but accurate and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, stays under the 90-word limit (83 words), accurately identifies the chart and axes, and clearly states what the chart will not reveal." }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "It correctly specifies a line chart, date on x-axis, revenue on y-axis, and a limitation. However, 14 months provides only a little over one annual cycle, not “roughly two cycles”; year-over-year comparison is limited to about two months." } ] }, { "id": "sp5", "name": "Google Sheets QUERY", "prompt": "In Google Sheets, tab 'Sales' has headers in row 1: Date, Rep, Region, Amount. Write one QUERY() formula returning total Amount per Rep for the North region only, sorted highest first. Return only the formula.", "output": "=QUERY(Sales!A:D, \"SELECT B, SUM(D) WHERE C = 'North' GROUP BY B ORDER BY SUM(D) DESC\", 1)", "latency_ms": 1501, "cost": 0.0029412, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, functional formula matching requirements; minor nit: label clause omitted but not required, works fine." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is perfectly accurate, meets all conditions, and strictly follows the constraint to return only the formula." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correct QUERY syntax: filters Region to North, groups by Rep, sums Amount, sorts totals descending, and specifies one header row. Returns only the formula as required." } ] }, { "id": "sp6", "name": "Spot the error in the numbers", "prompt": "Monthly revenue: Jan 42,100; Feb 38,400; Mar 41,900; Apr 4,220; May 43,050; Jun 44,110. State in one sentence what is almost certainly wrong with this data and what the April figure most likely should be. Do not speculate about business causes.", "output": "April's figure is almost certainly a typo missing a digit (an order-of-magnitude outlier versus the surrounding ~38,000–44,000 values) and most likely should be 42,200.", "latency_ms": 2979, "cost": 0.0032712, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Correctly identifies the anomaly and gives plausible estimate, concise, follows instructions, avoids speculation about causes; minor issue is exact figure is a guess but reasonable." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, providing a clear, accurate, one-sentence explanation of the likely typo and the corrected figure without speculating on business causes." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies April as an order-of-magnitude data-entry outlier and reasonably infers 42,200 from 4,220. It is one clear sentence and avoids speculation about business causes." } ] }, { "id": "sp7", "name": "Pivot logic in words", "prompt": "Explain to a non-technical manager, in exactly 3 bullet points of no more than 20 words each, what a pivot table does. Do not use the words 'aggregate', 'dimension' or 'axis'.", "output": "- Turns a long list of raw records into a compact summary table showing totals, averages, or counts.\n- Lets you group the same data different ways—like sales by region, then by month—without rewriting anything.\n- Answers business questions in seconds by dragging fields around, no formulas or coding required.", "latency_ms": 5329, "cost": 0.0108942, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Clear, accurate, exactly 3 bullets, each under 20 words, avoids banned words, useful and concise for non-technical manager." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all constraints, including exact bullet count, word limits per bullet, and avoiding banned words. It is highly accurate, clear, and useful for a non-technical manager." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Exactly three bullets, each under 20 words. Avoids all forbidden words. Accurately and clearly explains summarizing, regrouping, and interactive analysis for a non-technical manager." } ] }, { "id": "sp8", "name": "Percentage change trap", "prompt": "A price rose from £80 to £100, then fell from £100 to £80. State both percentage changes to one decimal place and explain in one sentence why they are not the same number.", "output": "**Rise:** (£100 − £80) ÷ £80 × 100 = **25.0% increase**\n\n**Fall:** (£100 − £80) ÷ £100 × 100 = **20.0% decrease**\n\nThey differ because each percentage is calculated relative to a different starting value — the rise is measured against the original £80, while the fall is measured against the higher £100.", "latency_ms": 3724, "cost": 0.0037722, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct calculations, one decimal place as required, clear one-sentence explanation, concise and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately calculates both percentage changes to one decimal place and provides a clear, correct one-sentence explanation for why they differ, following all instructions perfectly." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Both percentage changes are calculated correctly to one decimal place, and the explanation accurately identifies the different base values. Clear, concise, and fully follows the requested format." } ] }, { "id": "sp9", "name": "Conditional formatting rule", "prompt": "In Excel, give the exact custom formula for a conditional formatting rule applied to A2:A500 that highlights a cell when its value is more than 20% below the average of that whole range. Return the formula and nothing else.", "output": "=A220% below average of range, uses proper relative/absolute references, and returns only the formula as instructed." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides the exact correct formula and strictly follows the negative constraint to return nothing else." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly compares each relative cell in A2:A500 against 80% of the absolute average of the full range, exactly matching the requested 20%-below condition. It returns only the formula." } ] }, { "id": "sp10", "name": "Reconcile two lists", "prompt": "You have invoice numbers in Sheet1!A:A and payments in Sheet2!A:A. Give one Excel formula for Sheet1!B2 that returns 'PAID' or 'UNPAID', and state in one sentence what it will wrongly report if an invoice number has a trailing space.", "output": "```\n=IF(COUNTIF(Sheet2!A:A,A2)>0,\"PAID\",\"UNPAID\")\n```\n\nIf an invoice number has a trailing space (in either sheet), it will wrongly report \"UNPAID\" for an invoice that was actually paid, because the extra space makes the two text strings fail to match exactly.", "latency_ms": 6584, "cost": 0.0086172, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 3, "violates": false, "reasoning": "Formula is fine, but the explanation is factually wrong: Excel text comparisons (and COUNTIF) ignore trailing spaces, so it would still match, not falsely report UNPAID." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The formula is correct and perfectly addresses the prompt. The explanation is accurate, concise, and strictly follows the one-sentence constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "The formula correctly checks whether A2 appears in Sheet2 column A and returns the required labels. The one-sentence explanation accurately describes the false UNPAID result caused by a trailing-space mismatch." } ] }, { "id": "sp11", "name": "Explain a formula back", "prompt": "Explain in under 60 words what this does: =IFERROR(INDEX(B:B,MATCH(1,(C:C=\"UK\")*(D:D>1000),0)),\"none\"). Mention one reason it might be slow on a large sheet.", "output": "This array formula returns the first value in column B where column C equals \"UK\" **and** column D exceeds 1000; if no row matches, it returns \"none\". MATCH multiplies the two condition arrays to find the first qualifying row. It's slow because full-column references (B:B, C:C, D:D) force Excel to evaluate over a million rows per column.", "latency_ms": 4397, "cost": 0.0104802, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, clear explanation with valid slowness reason; likely slightly over 60 words though close." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately explains the formula and correctly identifies full-column references as the cause of slowness. It meets the word count constraint perfectly and is clear and concise." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains the first-match logic, error fallback, and array-condition multiplication. It clearly identifies full-column references as a plausible large-sheet performance issue. It is concise, professional, and under 60 words." } ] }, { "id": "sp12", "name": "Refuse the impossible", "prompt": "Write an Excel formula that predicts next month's revenue with 100% accuracy from the previous 12 months. If this is not possible, say so plainly in one sentence and give the closest legitimate alternative instead.", "output": "No formula can predict next month's revenue with 100% accuracy — the future is inherently uncertain, and 12 historical data points cannot capture the market shifts, customer behavior, and random events that determine actual results.\n\nThe closest legitimate alternative is Excel's built-in exponential smoothing forecast, which models trend and (with enough data) seasonality:\n\n**Setup:** month-start dates in `A2:A13`, revenue in `B2:B13`, and next month's date in `A14`.\n\n```\n=FORECAST.ETS(A14, B2:B13, A2:A13)\n```\n\nTo represent the uncertainty honestly, pair it with the confidence interval:\n\n```\n=FORECAST.ETS.CONFINT(A14, B2:B13, A2:A13)\n```\n\nThis returns the margin of error (95% by default), so your realistic range is the forecast ± that value.\n\nA few practical notes:\n\n- **12 points is minimal.** ETS needs ~24+ months to detect seasonality reliably; with 12 it will mostly capture trend.\n- **Simpler option** if you just want a straight-line trend: `=FORECAST.LINEAR(A14, B2:B13, A2:A13)`\n- **Validate, don't assume.** Track the forecast against actuals each month and compute an error metric (e.g., MAPE) so you know how much trust it deserves.", "latency_ms": 13034, "cost": 0.0235602, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly states impossibility, provides accurate legitimate alternatives with correct syntax and useful caveats. Clear, concise, well-structured for a non-technical reader." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, correctly stating in one sentence that 100% accuracy is impossible, and provides excellent, practical Excel alternatives (FORECAST.ETS and FORECAST.LINEAR) with clear instructions." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "Correctly states that 100% accuracy is impossible and provides valid ETS and linear forecast formulas with appropriate uncertainty guidance. The response is useful and accurate; it is slightly more detailed than necessary and the “24+ month" } ] } ] }