{ "entity": "github-copilot", "entity_type": "tool", "category": "Coding", "suite": "suite-2026-08-skills/coding", "model": "openai/gpt-5.3-codex", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 90, "caps": 0, "cost_usd": 0.3415, "started_at": "2026-08-10T10:58:32.646Z", "finished_at": "2026-08-10T10:59:30.590Z", "tasks": [ { "id": "c1", "name": "Duration parser", "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.", "output": "```python\nimport re\n\ndef parse_duration(s: str) -> int:\n \"\"\"\n Convert duration strings like:\n '1h30m', '45s', '2h', '90m', '1h2m3s'\n into total seconds.\n\n Valid format:\n - Optional hours, then optional minutes, then optional seconds\n - Each unit appears at most once, in that order (h, m, s)\n - Integers only, lowercase unit letters only\n \"\"\"\n if not isinstance(s, str):\n raise ValueError(\"Duration must be a string\")\n\n m = re.fullmatch(r'(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?', s)\n if not m:\n raise ValueError(f\"Malformed duration: {s!r}\")\n\n h, mn, sec = m.groups()\n if h is None and mn is None and sec is None:\n raise ValueError(f\"Malformed duration: {s!r}\") # empty string\n\n total = (int(h) if h else 0) * 3600\n total += (int(mn) if mn else 0) * 60\n total += (int(sec) if sec else 0)\n return total\n\n\n# --- assert-based tests ---\nassert parse_duration(\"1h30m\") == 5400\nassert parse_duration(\"1h2m3s\") == 3723\n\ntry:\n parse_duration(\"1m2h\") # malformed: wrong unit order\n assert False, \"Expected ValueError for malformed input\"\nexcept ValueError:\n pass\n```\n\n", "latency_ms": 9467, "cost": 0.00766675, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, handles edge cases well, includes three tests including malformed. Empty string check for m.groups() is slightly redundant with fullmatch but works fine." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct Python function, handles malformed inputs properly by raising ValueError, and includes exactly three assert-based tests as requested." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Solid correct impl with regex; three assert-style tests incl. malformed. Minor: only two positive cases, slightly verbose docstring." } ] }, { "id": "c2", "name": "Find the bug", "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) sum += x;\n n++;\n }\n return sum / n;\n}", "output": "**Bug:** `n` is incremented for every element, so you divide the sum of only positive values by the total count (including non-positives), which makes the average too low.\n\n```js\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) {\n sum += x;\n n++;\n }\n }\n return n === 0 ? 0 : sum / n; // or NaN if you prefer when no positives\n}\n```", "latency_ms": 2737, "cost": 0.0026915, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct bug identification, clear one-sentence explanation, fixed code with sensible edge-case handling for zero positives." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the bug, explains it in exactly one sentence as requested, and provides a correct, improved version of the function that also handles division by zero." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly identifies bug, one-sentence explanation, and proper fix including edge case. Fully matches task." } ] }, { "id": "c3", "name": "SQL without window functions", "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.", "output": "```sql\nSELECT\n u.email,\n COUNT(o.id) AS order_count\nFROM users u\nJOIN orders o\n ON o.user_id = u.id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY u.id, u.email\nHAVING COUNT(o.id) > 3\nORDER BY order_count DESC;\n```\n\nThis returns each user’s email plus how many orders they placed in the last 30 days, only for users with more than 3 orders, sorted by highest count first.", "latency_ms": 5629, "cost": 0.00253225, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, valid Postgres SQL, no window functions, meets all requirements, clear and concise explanation." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The SQL query is perfectly accurate for PostgreSQL, follows all constraints (no window functions), and correctly filters, groups, and sorts the data as requested. The explanation is concise and clear." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct Postgres SQL meeting all requirements exactly; clear, concise, and immediately useful." } ] }, { "id": "c4", "name": "Infinite useEffect", "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);", "output": "It loops because the effect **depends on `items`** and also **updates `items`**.\n\n### What happens step-by-step\n\n```jsx\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);\n```\n\n1. Component renders with some `items` value.\n2. Effect runs (because effects run after render, and on mount dependencies are considered “changed”).\n3. Fetch resolves, `setItems(newData)` is called.\n4. `setItems` updates state → component re-renders.\n5. Dependency check sees `items` changed (new array reference).\n6. Effect runs again, fetches again, calls `setItems` again...\n7. Repeats forever.\n\nEven if the fetched contents are “same”, `r.json()` typically produces a **new array/object reference**, so React treats it as changed.\n\n---\n\n### Fixed version\n\nIf you only want to fetch once on mount, use an empty dependency array:\n\n```jsx\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n fetch('/api/items')\n .then(r => r.json())\n .then(setItems);\n}, []);\n```\n\n(If you’re in React Strict Mode in development, you may see two fetches on mount—this is expected dev behavior, not an infinite loop.)", "latency_ms": 4864, "cost": 0.00506975, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct explanation and fix, clear, concise, accurate about strict mode nuance. Minor omission of cleanup/abort but not required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the infinite loop caused by updating a dependency within the effect, and provides the correct fix using an empty dependency array. It is clear, accurate, and concise." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate step-by-step explanation of the infinite loop cause and solid fix with empty deps. Minor polish only; highly clear and useful." } ] }, { "id": "c5", "name": "Typed debounce", "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.", "output": "```ts\ntype Debounced unknown> =\n ((this: ThisParameterType, ...args: Parameters) => void) & {\n cancel: () => void;\n };\n\nfunction debounce unknown>(\n fn: F,\n waitMs: number\n): Debounced {\n let timer: ReturnType | undefined;\n\n const debounced = function (\n this: ThisParameterType,\n ...args: Parameters\n ): void {\n if (timer !== undefined) {\n clearTimeout(timer);\n }\n\n const context = this;\n timer = setTimeout(() => {\n fn.call(context, ...args);\n }, waitMs);\n };\n\n const cancel = (): void => {\n if (timer !== undefined) {\n clearTimeout(timer);\n timer = undefined;\n }\n };\n\n return Object.assign(debounced, { cancel });\n}\n```\n\nThe naive generic form loses type information because it constrains the callback with a broad function type (instead of reusing the same generic parameter in the callback position), so TypeScript widens the parameter list and cannot preserve the original argument tuple.", "latency_ms": 11999, "cost": 0.01410675, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, meets constraints, no 'any', includes cancel, gives concise explanation. Minor: explanation slightly awkward but factually reasonable; type-safe implementation is solid." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless TypeScript debounce implementation that meets all constraints, avoids 'any', and includes a clear, single-sentence explanation of the type widening issue." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct typed debounce with cancel; preserves args via Parameters/ThisParameterType; no any; accurate one-sentence explanation. Minor polish only." } ] }, { "id": "c6", "name": "Leftmost binary search", "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.", "output": "Here’s a binary search that returns the **first occurrence** of `target` in a sorted array (with duplicates), or `-1` if not found.\n\n```python\ndef first_occurrence(nums, target):\n left, right = 0, len(nums) - 1\n ans = -1\n\n while left <= right:\n mid = (left + right) // 2\n\n if nums[mid] == target:\n ans = mid # record match\n right = mid - 1 # keep searching left part for first occurrence\n elif nums[mid] < target:\n left = mid + 1\n else:\n right = mid - 1\n\n return ans\n```\n\n### Complexity\n- **Time:** `O(log n)` \n- **Space:** `O(1)`\n\n### Distinguishing test case (vs ordinary binary search)\n```python\nnums = [1, 2, 2, 2, 3]\ntarget = 2\nprint(first_occurrence(nums, target)) # 1\n```\n\n- A standard binary search might return index `2` (or `1`/`3` depending on implementation).\n- This version must return **`1`**, the first occurrence.", "latency_ms": 4494, "cost": 0.005243, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct implementation, correct complexity, good distinguishing test case, clear and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all requirements: accurate code for finding the first occurrence, correct time and space complexity, and a clear distinguishing test case." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct first-occurrence binary search, accurate O(log n) complexity, clear distinguishing test case. Fully meets task." } ] }, { "id": "c7", "name": "Security review", "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n const p = req.query.name;\n db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n res.sendFile(__dirname + '/uploads/' + p);\n });\n});", "output": "Here are the security issues, **most severe first**, with fixes:\n\n1. **SQL Injection (Critical)**\n ```js\n db.query(`SELECT * FROM files WHERE name = '${p}'`, ...)\n ```\n User input is directly interpolated into SQL.\n\n **Fix:** Use parameterized queries/prepared statements.\n ```js\n db.query('SELECT * FROM files WHERE name = ?', [p], ...)\n ```\n\n2. **Path Traversal / Arbitrary File Read (Critical)**\n ```js\n res.sendFile(__dirname + '/uploads/' + p);\n ```\n An attacker can pass `../` sequences (or similar tricks) to escape `/uploads` and read other files.\n\n **Fix:** Never concatenate paths with user input. Use a safe root and strict filename validation.\n ```js\n res.sendFile(p, { root: path.join(__dirname, 'uploads'), dotfiles: 'deny' });\n ```\n Plus validate `p` against an allowlist regex (e.g. only `[a-zA-Z0-9._-]`).\n\n3. **Broken Access Control / IDOR (High)**\n Anyone who can call `/file` can request any filename; no authentication/authorization checks are shown.\n\n **Fix:** Require authentication and enforce ownership/permissions in the DB query, e.g. `WHERE name = ? AND owner_id = ?`.\n\n4. **Authorization Logic Bypass via Ignored DB Result/Error (High)**\n The query result (`rows`) and error (`e`) are ignored, and file is sent regardless. \n If the DB check was intended for access control or existence checks, it is completely bypassed.\n\n **Fix:** Handle `e`, verify `rows.length`, and only send file when authorized.\n ```js\n if (e) return next(e);\n if (!rows.length) return res.sendStatus(404); // or 403\n ```\n\n5. **Insufficient Input Validation (Medium)**\n `p` is unvalidated. This enables traversal tricks, weird encodings, overly long values, and edge-case parser issues.\n\n **Fix:** Enforce strict validation:\n - type is string\n - length limit\n - allowlist characters only\n - optionally map logical file IDs to server-side paths (best approach)\n\n6. **Potential Information Leakage via Error Handling (Medium)**\n No error callback on `sendFile`; errors may bubble to default handlers and leak internals in misconfigured environments.\n\n **Fix:** Provide `sendFile` callback and centralized safe error handling.\n ```js\n res.sendFile(..., (err) => { if (err) return next(err); });\n ```\n\n---\n\nA safer pattern is: **auth user → validate input → parameterized DB authorization check → sendFile with fixed root**.", "latency_ms": 14823, "cost": 0.01701875, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, comprehensive coverage of SQLi, path traversal, access control, error handling, with concrete fixes. Well-organized and clear, minimal excess." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately identifies all security issues, orders them by severity, and provides clear, actionable fixes. It is highly useful, concise, and follows all instructions perfectly." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate prioritized vulns with solid fixes; fully follows task; clear and useful though slightly technical for non-dev." } ] }, { "id": "c8", "name": "Safe migration", "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.", "output": "Use a **multi-step migration** so the only `ACCESS EXCLUSIVE` locks are very short.\n\n### Safe migration steps (in order)\n\n```sql\n-- 1) Add the column as nullable (fast metadata change)\nALTER TABLE orders ADD COLUMN status text;\n```\n\n```sql\n-- 2) Set default for new rows only (fast metadata change)\nALTER TABLE orders ALTER COLUMN status SET DEFAULT 'pending';\n```\n\n```sql\n-- 3) Backfill existing 40M rows in batches (repeat until 0 rows)\n-- Run this many times from app/script, each in its own transaction:\nWITH batch AS (\n SELECT ctid\n FROM orders\n WHERE status IS NULL\n LIMIT 10000\n)\nUPDATE orders o\nSET status = 'pending'\nFROM batch\nWHERE o.ctid = batch.ctid;\n```\n\n```sql\n-- 4) Add NOT NULL semantics without immediate full validation\nALTER TABLE orders\n ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\n```\n\n```sql\n-- 5) Validate constraint (scans table, but avoids long ACCESS EXCLUSIVE lock)\nALTER TABLE orders\n VALIDATE CONSTRAINT orders_status_not_null;\n```\n\n```sql\n-- 6) Flip to real NOT NULL (quick if already proven)\nALTER TABLE orders\n ALTER COLUMN status SET NOT NULL;\n```\n\n```sql\n-- 7) Optional cleanup (if you want only the column-level NOT NULL)\nALTER TABLE orders\n DROP CONSTRAINT orders_status_not_null;\n```\n\n---\n\n### Dangerous step (and why)\n\nThe dangerous one is **setting `NOT NULL` directly on a large table without prevalidation** (or doing `ADD COLUMN ... NOT NULL DEFAULT ...` in one shot on older PG), because Postgres may need to scan/rewrite lots of data while holding a strong table lock, causing long blocking.\n\nIn this plan, step 5 does the heavy scan under a weaker lock, so step 6 is typically quick. \n\n> Note: On PostgreSQL 11+, `ADD COLUMN ... DEFAULT 'pending'` (constant default) is optimized and avoids table rewrite, but `SET NOT NULL` is still the part to treat carefully.", "latency_ms": 13238, "cost": 0.0161455, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, well-structured multi-step migration with correct use of NOT VALID/VALIDATE pattern, correctly identifies dangerous step. Minor: batching approach via ctid isn't ideal but not wrong. Good clarity and usefulness." }, { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Provides the classic safe multi-step approach and correctly identifies the locking dangers. However, for Postgres 11+, this entire process is unnecessary as adding a column with a constant default is a fast metadata-only operation." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate multi-step PG migration avoiding long locks; correctly flags danger. Minor note on PG11+ is helpful polish. Fully follows task." } ] }, { "id": "c9", "name": "Fix the code not the test", "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify(' Hello World! ')).toBe('hello-world');", "output": "You're right — current implementation only lowercases and replaces single spaces, so it leaves leading/trailing dashes, multiple dashes, and punctuation.\n\nA fix that matches the test (and is generally robust) is:\n\n```js\nexport const slugify = (s) =>\n s\n .toLowerCase()\n .trim()\n .replace(/[^a-z0-9\\s-]/g, '') // remove punctuation/symbols\n .replace(/\\s+/g, '-') // collapse whitespace to single -\n .replace(/-+/g, '-'); // collapse repeated dashes\n```\n\nNow:\n\n```js\nslugify(' Hello World! ') // \"hello-world\"\n```", "latency_ms": 2730, "cost": 0.0029505, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly fixes implementation to pass the test, explains issue clearly, concise and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct and robust implementation that passes the test. It clearly explains the issue and the solution in a concise manner." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct fix that passes the test; clear explanation and code. Minor: extra collapse step unnecessary here but harmless." } ] }, { "id": "c10", "name": "Race condition", "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n if (cache) return cache;\n const r = await fetch('/config');\n cache = await r.json();\n return cache;\n}", "output": "You have a **check-then-act race** on `cache`.\n\n### What races?\nTwo (or more) callers can enter `getConfig()` before `cache` is set.\n\n### Problem interleaving\nAssume `cache = null` initially:\n\n1. Call A enters `getConfig()`, sees `cache` is null.\n2. A starts `fetch('/config')` and awaits.\n3. Call B enters before A finishes, also sees `cache` is null.\n4. B also starts `fetch('/config')`.\n\nNow you have duplicate requests. \nWorse, if responses differ over time:\n\n5. B finishes first, sets `cache = configNew`.\n6. A finishes later, sets `cache = configOld` (stale overwrite).\n\n---\n\n### Fix: cache the in-flight Promise\nSerialize concurrent callers onto one request.\n\n```js\nlet cache; // undefined = not loaded\nlet inFlight = null; // Promise while loading\n\nasync function getConfig() {\n if (cache !== undefined) return cache;\n if (inFlight) return inFlight;\n\n inFlight = fetch('/config')\n .then(r => {\n if (!r.ok) throw new Error(`HTTP ${r.status}`);\n return r.json();\n })\n .then(cfg => {\n cache = cfg;\n return cfg;\n })\n .finally(() => {\n inFlight = null; // allow retry if fetch failed\n });\n\n return inFlight;\n}\n```\n\nThis guarantees:\n- only one fetch at a time,\n- all concurrent callers await the same result,\n- no stale overwrite from racing requests.", "latency_ms": 8394, "cost": 0.0092435, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly identifies check-then-act race, gives clear interleaving example, and provides a solid fix with in-flight promise deduplication. Accurate, clear, useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately identifies the race condition, clearly explains the interleaving, and provides a robust fix by caching the promise. Flawless and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correctly identifies check-then-act race, clear interleaving, solid promise-cache fix. Minor extras (error handling) do not detract." } ] }, { "id": "c11", "name": "Retry with backoff", "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.", "output": "```ts\ntype RetryOpts = { maxAttempts?: number; baseMs?: number; maxMs?: number };\n\nexport async function retryAsync(\n fn: () => Promise,\n { maxAttempts = 5, baseMs = 200, maxMs = 10_000 }: RetryOpts = {}\n): Promise {\n let lastErr: any;\n for (let attempt = 1; attempt <= maxAttempts; attempt++) {\n try {\n return await fn();\n } catch (err: any) {\n lastErr = err;\n const status = err?.status ?? err?.response?.status; // e.g., fetch/axios-style\n const is4xx = typeof status === \"number\" && status >= 400 && status < 500;\n const retryable = !is4xx || status === 429;\n if (!retryable || attempt === maxAttempts) break;\n\n const exp = Math.min(maxMs, baseMs * 2 ** (attempt - 1)); // exponential backoff\n const jitter = Math.random() * exp; // full jitter: [0, exp)\n await new Promise((r) => setTimeout(r, jitter));\n }\n }\n throw lastErr;\n}\n```\n\nOn the final failure, it **throws the last encountered error** (no further retries).", "latency_ms": 4378, "cost": 0.0055825, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct logic, meets constraints, concise (~23 lines code), states final failure behavior clearly. Minor: no explicit line count check but under 30." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly implements the requested TypeScript retry wrapper with exponential backoff, jitter, attempt limits, and correct 4xx handling. It stays under the 30-line limit and clearly states the final failure behavior." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct TS retry with exp backoff+jitter, max attempts, skips non-429 4xx; under 30 lines; states final throw. Minor: assumes err.status shape." } ] }, { "id": "c12", "name": "Recursive type", "prompt": "Write a TypeScript type DeepPartial that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.", "output": "```ts\ntype DeepPartial =\n T extends Function\n ? T\n : T extends readonly (infer U)[]\n ? readonly DeepPartial[]\n : T extends object\n ? { [K in keyof T]?: DeepPartial }\n : T;\n```\n\nArrays must be treated as collections whose **elements** are recursively deep-partialized (`U[] -> DeepPartial[]`), rather than as plain objects where you’d map over keys like `\"length\"` or array methods.", "latency_ms": 4446, "cost": 0.00283675, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, functional implementation with clear one-sentence explanation of array handling; minor nuance: forcing readonly on arrays may not match original mutability, slight imperfection." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct and robust DeepPartial type and a clear, accurate one-sentence explanation of array handling." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Correct DeepPartial type and one-sentence array explanation. Minor polish possible; highly technical for non-tech audience but matches task." } ] }, { "id": "c13", "name": "Bash one-liner", "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.", "output": "```sh\nfind . -type f -printf '%s\\t%p\\0' | sort -z -t $'\\t' -k1,1nr | head -z -n 10 | numfmt -z --to=iec --field=1 --delimiter=$'\\t' | tr '\\0' '\\n'\n```\n\nBrief breakdown:\n\n- `find . -type f -printf '%s\\t%p\\0'` \n Recursively finds regular files under `.` and prints: `sizepath`.\n\n- `sort -z -t $'\\t' -k1,1nr` \n Sorts NUL-delimited records (`-z`) by field 1 (size), numeric, reverse (largest first).\n\n- `head -z -n 10` \n Keeps only the top 10 NUL-delimited records.\n\n- `numfmt -z --to=iec --field=1 --delimiter=$'\\t'` \n Converts field 1 (bytes) to human-readable units (KiB, MiB, …), still NUL-delimited.\n\n- `tr '\\0' '\\n'` \n Converts final NUL separators to newlines for display.\n\nThis safely handles filenames with spaces (and other special characters).", "latency_ms": 8606, "cost": 0.009107, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Command is technically sound, handles spaces via NUL-delimiting, and explanation is clear and accurate. Minor edge case (filenames with tabs/newlines) unhandled, but overall solid and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless, highly robust shell command using NUL delimiters to safely handle all filenames. The explanation is clear, concise, and perfectly addresses all constraints." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct, robust null-delimited pipeline; handles spaces; clear brief explanations. Minor: GNU-specific flags assumed." } ] }, { "id": "c14", "name": "Explain and cost", "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n for i in range(len(xs)):\n for j in range(i+1, len(xs)):\n if xs[i] == xs[j]:\n return True\n return False", "output": "`has_dup(xs)` checks whether a list has **any duplicate value**.\n\n### What it does\n- It compares each element `xs[i]` with every later element `xs[j]`.\n- If any pair matches, it returns `True`.\n- If no match is found after all comparisons, it returns `False`.\n\n### Complexity of the given version\n- **Time:** `O(n^2)` (nested loops over pairs)\n- **Space:** `O(1)` (uses only a few variables)\n\n---\n\nTo make it **O(n)** time, use a `set` to track seen values:\n\n```python\ndef has_dup(xs):\n seen = set()\n for x in xs:\n if x in seen:\n return True\n seen.add(x)\n return False\n```\n\n### Complexity of rewritten version\n- **Time:** `O(n)` average-case (each set lookup/insert is average `O(1)`)\n- **Space:** `O(n)` for the `seen` set", "latency_ms": 4442, "cost": 0.00421925, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate explanation, correct complexities, correct O(n) rewrite with tradeoff noted (space O(n)). Clear, concise, meets all instructions." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the function, correctly identifies the original time and space complexity, and provides an optimal O(n) rewrite with its corresponding complexities. It is clear, concise, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Accurate explanation, correct complexities, clean O(n) rewrite with set; fully meets task, clear and useful." } ] }, { "id": "c15", "name": "Regex with limits", "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.", "output": "A commonly used **“pretty strict but not perfect”** UK postcode regex is:\n\n```regex\n/^(?:GIR ?0AA|(?:[A-PR-UWYZ][0-9][0-9A-HJKPSTUW]?|[A-PR-UWYZ][A-HK-Y][0-9][0-9ABEHMNPRVWXY]?)[ ]?[0-9][ABD-HJLNP-UW-Z]{2})$/i\n```\n\n### One-line explanation of each part\n- `^` — start of string (so nothing can appear before the postcode). \n- `(?: ... )` — non-capturing wrapper for the full postcode alternatives. \n- `GIR ?0AA` — explicitly allows the historic special postcode `GIR 0AA` (space optional). \n- `|` — or: either the special `GIR` format or a normal postcode format. \n- `[A-PR-UWYZ]` — first area letter (excludes letters not used in this position, like `Q`, `V`, `X`). \n- `[0-9][0-9A-HJKPSTUW]?` — one outward-code shape: digit, then optional digit/allowed letter. \n- `|` — or the other outward-code shape below. \n- `[A-PR-UWYZ][A-HK-Y][0-9][0-9ABEHMNPRVWXY]?` — two-letter area outward-code shape with constrained final optional char. \n- `[ ]?` — optional single space between outward and inward parts. \n- `[0-9]` — inward sector digit. \n- `[ABD-HJLNP-UW-Z]{2}` — final two inward letters, excluding disallowed letters like `C`, `I`, `K`, `M`, `O`, `V`. \n- `$` — end of string (so nothing can appear after the postcode). \n- `/i` — case-insensitive matching.\n\n### Two examples of imperfection (invalid postcodes it would accept)\n- `AA1 1AA` — matches the pattern, but `AA` is not a real postcode area. \n- `ZA1 1AA` — also matches pattern, but `ZA` is not a real postcode area.\n\nSo this is useful for format validation, but not a complete authoritative check of every live UK postcode allocation.", "latency_ms": 22324, "cost": 0.02479925, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Solid regex, clear per-part explanation, gives two invalid-accepted examples, explicitly states imperfection. Task asked for valid-rejected OR invalid-accepted; met via one option satisfactorily." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all instructions, providing a highly accurate regex, clear one-line explanations, correct examples of accepted invalid postcodes, and explicitly avoiding any claim of perfection." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate common UK postcode regex with precise one-line part explanations and valid imperfection examples; fully follows task without claiming perfection." } ] }, { "id": "c16", "name": "Callback to async", "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n if (e) return cb(e);\n getOrders(user.id, (e2, orders) => {\n if (e2) return cb(e2);\n getTotals(orders, (e3, totals) => cb(e3, totals));\n });\n});", "output": "Here’s a clean `async/await` refactor with **proper error propagation**.\n\n### If you can return a Promise (preferred)\n\n```js\nconst { promisify } = require('node:util');\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id) {\n // Any error thrown here automatically rejects the Promise\n const user = await getUserAsync(id);\n const orders = await getOrdersAsync(user.id);\n const totals = await getTotalsAsync(orders);\n return totals;\n}\n```\n\nUsage:\n\n```js\ngetUserTotals(id)\n .then(totals => cb(null, totals))\n .catch(cb); // propagates errors, nothing swallowed\n```\n\n---\n\n### If you must keep callback API directly\n\n```js\nconst { promisify } = require('node:util');\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nfunction getUserTotals(id, cb) {\n (async () => {\n const user = await getUserAsync(id);\n const orders = await getOrdersAsync(user.id);\n return await getTotalsAsync(orders);\n })()\n .then(totals => cb(null, totals))\n .catch(err => cb(err)); // explicit error propagation\n}\n```\n\nNo `try/catch` is required unless you want to transform/log errors; uncaught errors in `await` chain reject and flow to `.catch(cb)`.", "latency_ms": 6457, "cost": 0.00729925, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, idiomatic async/await refactor with proper error propagation via promisify and .catch(cb); no swallowing. Clear, concise, offers two valid approaches. Minor: no try/catch example shown despite mentioning it, slightly verbose." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly refactors the code to async/await using util.promisify, provides both Promise-returning and callback-wrapping options, and correctly handles error propagation without swallowing errors. Clear and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Correct async/await refactor with promisify and proper error propagation via catch; clear dual options, minor Node assumption." } ] }, { "id": "c17", "name": "Diagnose from a trace", "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n at renderRows (/app/src/table.js:42:19)\n at Table (/app/src/table.js:12:5)\n at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.", "output": "Most likely root cause: **`renderRows` is calling `.map` on data that is `undefined` during the initial render** (e.g., `props.rows`, `state.rows`, or derived data isn’t initialized yet).\n\nBecause it only fails on **production first paint**, the likely trigger is a **timing/initialization difference** (SSR/hydration or async data arriving after first render), where dev happens to mask it.\n\n**First thing I’d check:** \nAt `/app/src/table.js` line 42, inspect the value being mapped and confirm it has a safe default on first render, e.g.:\n\n- state initialized as `[]` (not `undefined`)\n- prop default set (`rows = []`)\n- guard before map (`(rows ?? []).map(...)`)\n\nSo: verify what variable at line 42 is undefined in prod first render, and ensure it’s always an array before render.", "latency_ms": 4232, "cost": 0.0041335, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, plausible root cause and concrete first check given limited info; clear and concise, appropriately hedged given ambiguity of prod-only failure." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately identifies the root cause (mapping over undefined data on initial render) and provides the exact first step to check (line 42 for safe defaults). Clear, concise, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate root cause and check for the undefined map; clear, actionable, fits all criteria with only tiny polish gaps." } ] }, { "id": "c18", "name": "Idempotency", "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.", "output": "Use a client-supplied **Idempotency-Key** header (UUID), scoped by merchant/customer, e.g. unique on `(account_id, idempotency_key)`.\n\n**Storage:** durable DB table `payment_idempotency` with:\n- `account_id`, `key` (unique)\n- `request_hash` (amount/currency/order)\n- `status` (`IN_PROGRESS|SUCCEEDED|FAILED`)\n- `payment_id`\n- `http_status`, `response_body`\n- timestamps/TTL\n\n**Flow:**\n1. On first POST, atomically `INSERT ... status=IN_PROGRESS`. \n2. If insert succeeds, process charge, then update row with final status + exact response payload.\n3. If key already exists:\n - If `request_hash` differs, return `422` (key reuse with different payload).\n - If existing row is `SUCCEEDED/FAILED`, return the **same stored HTTP status/body** as original.\n - If `IN_PROGRESS` (concurrent duplicate), either wait briefly and replay final stored response, or return `409/202` with retry hint; never create a second charge.\n\n**Second time return:** byte-for-byte same result (including payment_id) once first attempt completed.\n\n**Remaining failure mode:** the classic **external side-effect ambiguity** (charge committed at PSP, but service crashes before persisting idempotency result). Without PSP-side idempotency/reconciliation, “exactly once” is not guaranteed.", "latency_ms": 10862, "cost": 0.00832475, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, thorough design covering key, storage, concurrency handling, replay response, and correctly names the crash-before-persist failure mode. Slightly exceeds typical conciseness but within 200 words; well-structured and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly addresses all prompts within the word limit. It accurately describes the idempotency key, storage, concurrent duplicate handling, second-time return, and correctly identifies the remaining failure mode." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate idempotency design covering all required points; under 200 words; clear and useful with minor polish possible." } ] } ] }