{ "entity": "grok-4-5", "entity_type": "model", "category": "Coding", "suite": "suite-2026-08-skills/coding", "model": "x-ai/grok-4.5", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "openai/gpt-5.6-terra" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 88, "caps": 0, "cost_usd": 0.257, "started_at": "2026-08-16T06:34:19.358Z", "finished_at": "2026-08-16T06:35:35.123Z", "tasks": [ { "id": "c1", "name": "Duration parser", "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.", "output": "```python\nimport re\n\ndef parse_duration(s):\n if not isinstance(s, str) or not s:\n raise ValueError(\"malformed duration\")\n match = re.fullmatch(r'(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?', s)\n if not match or not any(match.groups()):\n raise ValueError(\"malformed duration\")\n h, m, sec = match.groups()\n total = 0\n if h:\n total += int(h) * 3600\n if m:\n total += int(m) * 60\n if sec:\n total += int(sec)\n return total\n\n# Assert-based tests\nassert parse_duration('1h30m') == 5400\nassert parse_duration('1h2m3s') == 3723\ntry:\n parse_duration('1x')\n assert False, \"expected ValueError\"\nexcept ValueError:\n pass\n```", "latency_ms": 13986, "cost": 0.0070824, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct logic, handles malformed input, includes required tests. Minor: regex allows empty string edge case handled separately; overall solid and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The function correctly parses the duration strings into seconds using regex, raises ValueError for malformed inputs, and includes the three required assert-based tests." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly parses ordered hour/minute/second components, rejects empty, non-string, and malformed inputs via ValueError, and computes totals accurately. Includes three assert-based tests, including malformed input handling." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c2", "name": "Find the bug", "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) sum += x;\n n++;\n }\n return sum / n;\n}", "output": "**Bug:** `n++` runs for every element, so `n` is the total length rather than the count of positives, and the result is (sum of positives) / (all elements).\n\n```javascript\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) {\n sum += x;\n n++;\n }\n }\n return sum / n;\n}\n```", "latency_ms": 5837, "cost": 0.0025144, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly identifies bug, explains clearly in one sentence, provides correct fix. Minor omission: doesn't mention division by zero edge case." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the bug, explains it in exactly one sentence, and provides the correctly fixed function." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly identifies that the counter includes non-positive elements, explains the resulting incorrect denominator in one sentence, and provides the proper fix by incrementing the count only for positive values." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c3", "name": "SQL without window functions", "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.", "output": "```sql\nSELECT u.email, COUNT(o.id) AS order_count\nFROM users u\nJOIN orders o ON o.user_id = u.id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY u.id, u.email\nHAVING COUNT(o.id) > 3\nORDER BY order_count DESC;\n```", "latency_ms": 4325, "cost": 0.0019584, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, idiomatic Postgres SQL meeting all requirements: no window functions, proper filtering, grouping, having, ordering. Clear and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The SQL query perfectly matches all requirements, uses standard Postgres syntax for intervals, correctly filters and groups the data, and avoids window functions as requested." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correct Postgres query: filters to the last 30 days, groups orders per user, keeps counts above 3, returns email and count, and sorts descending. No window functions used." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c4", "name": "Infinite useEffect", "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);", "output": "## Why it loops forever\n\n`useEffect` runs after render whenever a value in its dependency array changes.\n\n1. First render: `items` is `[]`.\n2. Effect runs → `fetch` → `setItems(data)`.\n3. `setItems` updates state → React re-renders with a **new** `items` array.\n4. Dependency `[items]` sees a changed reference → effect runs again → fetch → `setItems` → …\n5. Repeat forever.\n\nPutting the state you’re setting in the same effect’s deps guarantees a loop (unless the setter is a no-op, which it isn’t here—each response is a new array).\n\n## Fixed version\n\nRun the fetch once on mount (empty deps):\n\n```js\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n fetch('/api/items')\n .then((r) => r.json())\n .then(setItems);\n}, []); // only on mount\n```\n\n**Notes:**\n- If you need to refetch when something else changes (e.g. a filter id), put **that** value in the deps—not `items`.\n- In Strict Mode (dev), React may run mount effects twice on purpose; that’s not the same as this infinite loop.\n- Prefer aborting/canceling the fetch on unmount if the component can unmount mid-request.", "latency_ms": 7216, "cost": 0.0032924, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, clear explanation of the loop cause and correct fix with empty deps, plus useful bonus notes. Minor nit: could mention cleanup/AbortController code, but not required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately and concisely explains the infinite loop caused by the dependency array, provides the correct fix, and offers useful context for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Precisely explains the reference-change feedback loop, provides the correct empty-dependency fix, and appropriately notes Strict Mode behavior and parameter-based refetching. Clear, concise, and accurate." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c5", "name": "Typed debounce", "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.", "output": "```typescript\nfunction debounce(\n fn: (...args: TArgs) => void,\n delay: number\n): ((...args: TArgs) => void) & { cancel(): void } {\n let timeoutId: ReturnType | undefined;\n\n const debounced = (...args: TArgs) => {\n if (timeoutId !== undefined) clearTimeout(timeoutId);\n timeoutId = setTimeout(() => {\n fn(...args);\n timeoutId = undefined;\n }, delay);\n };\n\n debounced.cancel = () => {\n if (timeoutId !== undefined) {\n clearTimeout(timeoutId);\n timeoutId = undefined;\n }\n };\n\n return debounced;\n}\n```\n\n**Why the naive generic loses type information:** A signature like `debounce any>(fn: F): F` collapses parameters to `any` (and cannot attach `.cancel`) because the `any[]` constraint erases the concrete argument tuple instead of capturing it.", "latency_ms": 14596, "cost": 0.0059044, "task_score": 5, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, meets constraints, no 'any', includes cancel method and clear explanation. Minor: explanation uses 'any' in text discussing naive signature, but that's necessary context, not code." }, { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The code is excellent and meets all requirements, but the response violates the negative constraint by using the word 'any' in the explanation text." }, { "judge": "openai/gpt-5.6-terra", "score": 3, "violates": true, "reasoning": "The implementation preserves tuple arguments conceptually but does not type-check: `debounced` is inferred without a `.cancel` property. It also uses the forbidden term `any` in the explanation. The explanation is inaccurate: `F` generally " } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c6", "name": "Leftmost binary search", "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.", "output": "```python\ndef binary_search_first(arr, target):\n \"\"\"Return the index of the first occurrence of target in a sorted array\n (with possible duplicates), or -1 if not found.\n \"\"\"\n left, right = 0, len(arr) - 1\n result = -1\n while left <= right:\n mid = (left + right) // 2\n if arr[mid] == target:\n result = mid\n right = mid - 1 # keep looking for an earlier occurrence\n elif arr[mid] < target:\n left = mid + 1\n else:\n right = mid - 1\n return result\n```\n\n**Time complexity:** \\(O(\\log n)\\) \n**Space complexity:** \\(O(1)\\)\n\n**Distinguishing test case**\n\n```python\narr = [1, 2, 2, 2, 3]\ntarget = 2\n# ordinary binary search may return 1, 2 or 3\n# this version must return 1\nassert binary_search_first(arr, target) == 1\n```", "latency_ms": 9090, "cost": 0.0036464, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct implementation, correct complexity, clear distinguishing test case demonstrating difference from ordinary binary search. Concise and accurate." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all requirements: accurate code for finding the first occurrence, correct complexity analysis, and a clear distinguishing test case." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly implements first-occurrence binary search by continuing left after a match. Complexity is accurate, and the duplicate-containing test case clearly distinguishes it from ordinary binary search." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c7", "name": "Security review", "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n const p = req.query.name;\n db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n res.sendFile(__dirname + '/uploads/' + p);\n });\n});", "output": "## Security issues (most severe first)\n\n### 1. Path traversal → arbitrary file read\n`p` is concatenated into the filesystem path and passed to `sendFile`. A request like \n`/file?name=../../../../etc/passwd` (or `..\\\\..\\\\...` on Windows) can read any file the process can open.\n\n**Fix:** Treat the upload dir as a fixed root, resolve safely, and reject escapes. Prefer `res.sendFile` with a `root` option and a basename (or other strict allowlist):\n\n```javascript\nconst path = require('path');\n\napp.get('/file', (req, res) => {\n const name = path.basename(String(req.query.name || ''));\n if (!name) return res.status(400).send('Bad request');\n\n res.sendFile(name, { root: path.join(__dirname, 'uploads') }, (err) => {\n if (err) res.status(err.statusCode || 404).end();\n });\n});\n```\n\n(Only serve names you have already validated against the DB / an allowlist.)\n\n---\n\n### 2. SQL injection\n`p` is interpolated into the SQL string:\n\n```javascript\n`SELECT * FROM files WHERE name = '${p}'`\n```\n\nAn attacker can send `name=' OR '1'='1` (or worse) and run arbitrary SQL.\n\n**Fix:** Parameterized query only:\n\n```javascript\ndb.query('SELECT * FROM files WHERE name = ?', [name], (e, rows) => { /* ... */ });\n```\n\n(Use whatever placeholder style your driver requires: `?`, `$1`, etc.)\n\n---\n\n### 3. DB result ignored (authZ / existence bypass)\nThe handler always calls `sendFile` with user input, regardless of `e` or `rows`. The query does not gate access, so “only files in `files`” is not enforced.\n\n**Fix:** Only serve after a successful lookup; use the DB value (not raw query input) as the file id/name:\n\n```javascript\ndb.query('SELECT * FROM files WHERE name = ?', [name], (e, rows) => {\n if (e) return res.status(500).end();\n if (!rows || !rows.length) return res.status(404).end();\n\n const safeName = path.basename(rows[0].name);\n res.sendFile(safeName, { root: path.join(__dirname, 'uploads') }, (err) => {\n if (err) res.status(err.statusCode || 404).end();\n });\n});\n```\n\n---\n\n### 4. No authentication or authorization\nAny client who can hit `GET /file` can attempt to read uploads (and, via bugs above, more). There is no session/role check or per-file ACL.\n\n**Fix:** Require a logged-in user (session/JWT/etc.) and enforce that they may access that row (owner, shared-with, role). Return 401/403 before touching disk.\n\n---\n\n### 5. Errors swallowed / weak failure handling\n`e` from `db.query` is ignored; failed queries can still hit `sendFile`. `sendFile` errors are not handled in the original code. That yields 500s, info leaks, or odd behavior.\n\n**Fix:** Handle DB and `sendFile` errors explicitly; don’t proceed on failure; avoid leaking paths/SQL in responses.\n\n---\n\n### 6. Missing input validation\n`req.query.name` may be missing, non-string (array), empty, or absurdly long. That feeds both SQLi and path issues and can cause unstable behavior.\n\n**Fix:** Normalize early:\n\n```javascript\nconst raw = req.query.name;\nconst name = typeof raw === 'string' ? path.basename(raw).slice(0, 255) : '';\nif (!name || !/^[\\w.\\-]+$/.test(name)) return res.status(400).send('Bad request');\n```\n\n(Tune the allowlist to your real filename rules.)\n\n---\n\n### 7. Minor: path concatenation instead of `path.join` / `root`\n`__dirname + '/uploads/' + p` is easy to get wrong across platforms and makes traversal easier to miss.\n\n**Fix:** Always use `path.join` / `path.resolve` and `sendFile`’s `root` (as above); never concatenate user strings into paths.\n\n---\n\n## Hardened shape (illustrative)\n\n```javascript\nconst path = require('path');\n\napp.get('/file', requireAuth, (req, res) => {\n const raw = req.query.name;\n const name = typeof raw === 'string' ? path.basename(raw) : '';\n if (!name || !/^[\\w.\\-]+$/.test(name)) {\n return res.status(400).send('Bad request');\n }\n\n db.query(\n 'SELECT name FROM files WHERE name = ? AND owner_id = ?',\n [name, req.user.id],\n (e, rows) => {\n if (e) return res.status(500).end();\n if (!rows?.length) return res.status(404).end();\n\n res.sendFile(rows[0].name, {\n root: path.join(__dirname, 'uploads'),\n dotfiles: 'deny',\n }, (err) => {\n if (err) res.status(err.statusCode || 404).end();\n });\n }\n );\n});\n```\n\n**Priority order to fix:** path traversal → SQL injection → enforce DB/auth checks → validation and error handling.", "latency_ms": 35239, "cost": 0.0107224, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Thorough, accurate identification of path traversal, SQLi, missing auth checks, error handling, with correct fixes and code. Well-organized, slightly verbose but clear and highly useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Excellent response. Accurately identifies all major and minor security flaws, orders them logically by severity, and provides clear, practical fixes. Highly useful and well-formatted." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly identifies the core SQL injection, path traversal, authorization bypass, and error-handling issues with practical fixes. Minor overstatement: arbitrary SQL depends on driver configuration; authentication requirements are contextua" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c8", "name": "Safe migration", "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.", "output": "## Safe migration (Postgres 11+)\n\nOn PG 11+, a constant `DEFAULT` is a catalog-only change (no rewrite). The long lock risk is almost always the **NOT NULL** validation scan. Do it in stages so the heavy work does not hold `AccessExclusiveLock`.\n\n### Steps (in order)\n\n**1. Add the column nullable, with default** (brief `AccessExclusiveLock`)\n\n```sql\nALTER TABLE orders\n ADD COLUMN status text DEFAULT 'pending';\n```\n\nNew rows get `'pending'`. Existing rows are not rewritten; the default is applied on read.\n\n**2. Backfill existing rows in batches** (row-level locks only)\n\nEven with a PG 11+ default, physically writing values avoids surprises (e.g. later `NOT NULL`, logical decoding, or `DEFAULT` drops). Batch by PK:\n\n```sql\n-- Repeat until 0 rows updated\nUPDATE orders\nSET status = 'pending'\nWHERE id IN (\n SELECT id FROM orders\n WHERE status IS NULL\n ORDER BY id\n LIMIT 10000\n);\n```\n\nUse larger/smaller batches depending on load; add `pg_sleep` between batches if needed. App should also write `status` on create/update going forward.\n\n**3. Install a NOT VALID check** (brief `AccessExclusiveLock`, no scan)\n\n```sql\nALTER TABLE orders\n ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\n```\n\n**4. Validate the constraint** (long scan, but **not** exclusive)\n\n```sql\nALTER TABLE orders\n VALIDATE CONSTRAINT orders_status_not_null;\n```\n\nUses `ShareUpdateExclusiveLock`: allows concurrent `SELECT` / `INSERT` / `UPDATE` / `DELETE`. Blocks other `VACUUM`/DDL that need stronger locks.\n\n**5. Attach real NOT NULL** (brief `AccessExclusiveLock`)\n\n```sql\nALTER TABLE orders\n ALTER COLUMN status SET NOT NULL;\n```\n\nWith a validated `CHECK (status IS NOT NULL)`, Postgres does not re-scan the whole table.\n\n**6. Drop the redundant check** (brief `AccessExclusiveLock`)\n\n```sql\nALTER TABLE orders\n DROP CONSTRAINT orders_status_not_null;\n```\n\n**7. Analyze**\n\n```sql\nANALYZE orders;\n```\n\n---\n\n### Which step is dangerous, and why?\n\n| Step | Lock | Risk |\n|------|------|------|\n| **Naive `SET NOT NULL`** (if you skip 3–4 and run `ALTER COLUMN status SET NOT NULL` alone) | **`AccessExclusiveLock` for the full table scan** | **Dangerous on 40M rows.** Blocks all reads and writes until every row is checked. That is the classic outage. |\n| **Step 4 `VALIDATE CONSTRAINT`** | `ShareUpdateExclusiveLock` + full scan | Slow, but **not** exclusive; normal DML continues. Still avoid peak traffic; watch replication lag. |\n| Steps 1, 3, 5, 6 | Brief `AccessExclusiveLock` | Safe if they don’t wait behind long transactions. Run when the lock can be acquired quickly (`lock_timeout` recommended). |\n| Step 2 backfill | Row locks / I/O | Can cause bloat, replication lag, and hot-row contention if batches are too large—not an exclusive lock, but operationally the heaviest step. |\n\n**Do not** run a one-shot:\n\n```sql\n-- Avoid on large tables (especially pre-11; still riskier habit on 11+)\nALTER TABLE orders\n ADD COLUMN status text NOT NULL DEFAULT 'pending';\n```\n\nOn PG 11+ that is often metadata-fast, but you still take `AccessExclusiveLock` and couple add + null-check in one shot; the staged path above keeps the only long scan non-exclusive and is the usual production pattern for tens of millions of rows.\n\n**Extras:** set `lock_timeout` (e.g. `'2s'`) on short DDL so you fail fast instead of queueing behind a long tx; retry. Prefer a maintenance window for validate/backfill if replicas lag. On **PG ≤ 10**, `ADD COLUMN ... DEFAULT` rewrites the table—always use add-without-default + batch backfill there.", "latency_ms": 44499, "cost": 0.0144204, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Technically accurate, correct lock behavior, clear ordered steps, explicitly identifies the dangerous step (naive SET NOT NULL / full-scan validate), well formatted and thorough." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a highly accurate, step-by-step guide for adding a NOT NULL column safely in Postgres. It correctly identifies the dangerous steps and explains the locking mechanisms clearly. Excellent advice for a non-technical profe" }, { "judge": "openai/gpt-5.6-terra", "score": 6, "violates": false, "reasoning": "Mostly sound PG11+ staged approach and lock discussion. However, after a fast default, legacy rows read as 'pending', so `WHERE status IS NULL` generally updates none; claimed physical backfill does not occur. It also overstates the need/ri" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c9", "name": "Fix the code not the test", "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify(' Hello World! ')).toBe('hello-world');", "output": "```js\nexport const slugify = (s) =>\n s\n .toLowerCase()\n .trim()\n .replace(/[^\\w\\s-]/g, '')\n .replace(/\\s+/g, '-');\n```", "latency_ms": 6778, "cost": 0.0028884, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct fix: trims, removes punctuation, collapses whitespace to single hyphen, passes the given test. Clean, concise, no extraneous explanation needed." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The provided implementation correctly fixes the code to make the test pass by handling trimming, removing punctuation, and collapsing multiple spaces into a single dash." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correctly trims leading/trailing whitespace, removes punctuation, lowercases, and collapses repeated whitespace into single hyphens. It changes only the implementation and produces the expected output." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c10", "name": "Race condition", "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n if (cache) return cache;\n const r = await fetch('/config');\n cache = await r.json();\n return cache;\n}", "output": "## Race condition\n\n`cache` is only set **after** `await fetch` / `await r.json()` finish. Concurrent calls that enter before that all see `cache === null` and each start their own request.\n\n### Interleaving that triggers it\n\n1. `getConfig()` #1 runs, `cache` is `null`, starts `fetch('/config')`, yields at `await`.\n2. `getConfig()` #2 runs (same tick or before #1 finishes), `cache` is still `null`, starts a **second** `fetch('/config')`.\n3. Both responses complete; each assigns `cache` (duplicate work; last write wins).\n\nAny number of overlapping callers can stampede the network the same way.\n\n## Fix\n\nCache the **in-flight promise**, not only the final value, so every caller shares one request:\n\n```javascript\nlet cache = null;\n\nasync function getConfig() {\n if (!cache) {\n cache = fetch('/config').then((r) => r.json());\n }\n return cache;\n}\n```\n\nNotes:\n- First caller creates and stores the promise; later callers await the same one.\n- On success, later calls still resolve to the same parsed config (promise is settled).\n- This snippet does not clear `cache` on failure; if you want retries after a failed fetch, reset `cache` in a `.catch` before rethrowing.", "latency_ms": 7181, "cost": 0.0038144, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correctly identifies race condition, explains interleaving accurately, provides a valid fix caching the promise. Clear and concise; minor omission of explicit error-handling code but notes it." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the race condition, clearly explains the interleaving, and provides an elegant, correct fix by caching the promise. It is concise and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "Correctly identifies the check-then-await race, gives a valid overlapping-call interleaving, and fixes it by memoizing the in-flight promise. Clear and concise. Minor omission: it notes but does not implement retry behavior after a failed r" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c11", "name": "Retry with backoff", "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.", "output": "```typescript\ntype RetryOpts = { maxAttempts?: number; baseMs?: number; maxMs?: number };\n\nfunction isNonRetryableClientError(err: unknown): boolean {\n const status = (err as { status?: number })?.status ?? (err as { response?: { status?: number } })?.response?.status;\n return typeof status === \"number\" && status >= 400 && status < 500 && status !== 429;\n}\n\n/** Retries `fn` with exp. backoff + full jitter. On final failure, rethrows the last error. */\nexport async function withRetry(fn: () => Promise, opts: RetryOpts = {}): Promise {\n const maxAttempts = opts.maxAttempts ?? 5;\n const baseMs = opts.baseMs ?? 100;\n const maxMs = opts.maxMs ?? 10_000;\n let lastErr: unknown;\n for (let attempt = 1; attempt <= maxAttempts; attempt++) {\n try {\n return await fn();\n } catch (err) {\n lastErr = err;\n if (attempt === maxAttempts || isNonRetryableClientError(err)) throw err;\n const exp = Math.min(maxMs, baseMs * 2 ** (attempt - 1));\n const delay = Math.random() * exp; // full jitter\n await new Promise((r) => setTimeout(r, delay));\n }\n }\n throw lastErr; // unreachable; satisfies TS\n}\n```\n\n**Final failure:** the last error is rethrown (same rejection the caller would have seen on that attempt). Non-429 4xx errors are thrown immediately without further retries.", "latency_ms": 5270, "cost": 0.0028384, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, meets constraints, within line limit, clearly states final failure behavior. Minor: line count close to 30 but within limit; solid quality." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all requirements, including the constraints on line count, exponential backoff with jitter, max attempts, and 4xx logic. It clearly states what happens on final failure." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "Correct exponential backoff with full jitter, attempt limit, and immediate handling of non-429 4xx errors. It clearly states final-failure behavior and stays within 30 code lines. Minor edge case: invalid or zero maxAttempts is not validate" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c12", "name": "Recursive type", "prompt": "Write a TypeScript type DeepPartial that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.", "output": "```typescript\ntype DeepPartial = T extends readonly any[]\n ? { [K in keyof T]: DeepPartial }\n : T extends object\n ? { [K in keyof T]?: DeepPartial }\n : T;\n```\n\nArrays must be mapped element-wise (keeping array structure) instead of turning them into plain objects with optional keys, which would break length, indices, and array methods.", "latency_ms": 5380, "cost": 0.0026764, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct DeepPartial implementation with array handling explained in one sentence as required; minor nit: array elements could be optional too but acceptable, well-clarified answer." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless TypeScript implementation of DeepPartial and accurately explains the array handling in exactly one sentence." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Correct recursive conditional type: arrays/tuples are mapped without optionalizing their structural keys, while plain-object properties become optional. The explanation is exactly one sentence and accurately states why arrays require separa" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c13", "name": "Bash one-liner", "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.", "output": "```bash\nfind . -type f -exec du -h {} + | sort -rh | head -n 10\n```\n\n- `find . -type f`: recursively lists only regular files under the current directory.\n- `-exec du -h {} +`: runs `du -h` (human-readable sizes) on the found files in efficient batches; `{} +` ensures filenames with spaces/newlines are passed safely.\n- `sort -rh`: sorts the output in reverse order using human-readable size comparison (largest first).\n- `head -n 10`: prints only the first 10 lines.", "latency_ms": 14282, "cost": 0.0047604, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, handles spaces via find -exec +, sort -rh works with human-readable sizes, clear concise explanation. Minor nitpick: du batches could group sizes oddly but generally fine." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The command perfectly fulfills all requirements, correctly handling spaces and human-readable sorting. The explanation is concise, accurate, and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 8, "violates": false, "reasoning": "Correctly finds regular files recursively, handles spaces, sorts human-readable sizes descending, and limits to ten. Minor caveat: `du` reports disk usage rather than exact file size, and newline-containing filenames would not be safely sor" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c14", "name": "Explain and cost", "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n for i in range(len(xs)):\n for j in range(i+1, len(xs)):\n if xs[i] == xs[j]:\n return True\n return False", "output": "### Explanation\nThe function `has_dup(xs)` determines whether a list `xs` contains any duplicate elements. It does this via brute-force pairwise comparison: the outer loop picks each index `i`, and the inner loop checks all later indices `j > i`. If any `xs[i] == xs[j]`, it immediately returns `True`. If no such pair exists, it returns `False`.\n\n### Complexity\n- **Time:** \\(O(n^2)\\) — in the worst case (no duplicates) every pair is examined, i.e. \\(\\frac{n(n-1)}{2}\\) comparisons.\n- **Space:** \\(O(1)\\) — only a handful of index variables are used; no auxiliary data structures grow with the input size.\n\n### \\(O(n)\\) rewrite\nUse a hash set to record elements already seen. Each lookup/insert is expected \\(O(1)\\), so a single linear pass yields expected linear time (and \\(O(n)\\) extra space).\n\n```python\ndef has_dup(xs):\n seen = set()\n for x in xs:\n if x in seen:\n return True\n seen.add(x)\n return False\n```", "latency_ms": 9000, "cost": 0.0034824, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate explanation, correct complexities, correct O(n) rewrite with clear tradeoff (space increase). Clear, concise, meets task fully." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the function, correctly identifies the time and space complexity, and provides an optimal O(n) rewrite using a set. It is clear, concise, and highly useful." }, { "judge": "openai/gpt-5.6-terra", "score": 10, "violates": false, "reasoning": "Accurately explains pairwise duplicate detection, gives correct worst-case O(n^2) time and O(1) auxiliary space, and provides a clear hash-set rewrite with appropriately qualified expected O(n) time and O(n) extra space." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c15", "name": "Regex with limits", "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.", "output": "**Regex:** `/^[A-Z]{1,2}\\d[A-Z\\d]? ?\\d[A-Z]{2}$/i`\n\n- `^` — start of string \n- `[A-Z]{1,2}` — 1–2 letters (area) \n- `\\d` — one digit (district start) \n- `[A-Z\\d]?` — optional letter or digit (rest of district) \n- ` ?` — optional space \n- `\\d` — one digit (sector) \n- `[A-Z]{2}` — two letters (unit) \n- `$` — end of string \n- `i` flag — case-insensitive \n\nTwo valid UK postcodes this would reject: `EC1A 1BB` (the `A` after the digit in the outward code is only partially covered by the optional class in some edge positions), `W1A 0AX` (same outward-code pattern limitation). \n\n(The pattern is deliberately imperfect.)", "latency_ms": 10410, "cost": 0.0034304, "task_score": 3, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 3, "violates": false, "reasoning": "The regex actually matches EC1A 1BB and W1A 0AX, so the claimed rejections are factually wrong; explanation inaccurate despite disclaimer." }, { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": false, "reasoning": "The response follows the format but is factually incorrect: the provided regex actually *accepts* both 'EC1A 1BB' and 'W1A 0AX', contradicting the claim that it rejects them." }, { "judge": "openai/gpt-5.6-terra", "score": 3, "violates": false, "reasoning": "The regex and component explanations are clear, but both cited postcodes are actually accepted by the pattern. Thus it fails the required accurate examples of valid rejections or invalid acceptances, despite correctly noting imperfection." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c16", "name": "Callback to async", "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n if (e) return cb(e);\n getOrders(user.id, (e2, orders) => {\n if (e2) return cb(e2);\n getTotals(orders, (e3, totals) => cb(e3, totals));\n });\n});", "output": "```js\nasync function getUserTotals(id) {\n const user = await getUser(id);\n const orders = await getOrders(user.id);\n return await getTotals(orders);\n}\n```\n\nErrors reject the returned promise naturally—no swallowing, no extra `try/catch` unless you need to map or handle them at this layer. If `getUser` / `getOrders` / `getTotals` are still callback-based, promisify them first (e.g. `util.promisify` or an equivalent wrapper).", "latency_ms": 12120, "cost": 0.0044264, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct async/await refactor with proper error propagation via promise rejection; notes promisification need. Minor lack of usage example with try/catch for callers, but solid and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly refactors the callback hell into clean async/await syntax, correctly explaining that errors propagate naturally as rejected promises, and notes the need to promisify the original functions." }, { "judge": "openai/gpt-5.6-terra", "score": 5, "violates": false, "reasoning": "The async/await flow and propagation principle are correct for promise-returning APIs. However, the supplied functions are callback-based in the prompt, so the code as written will not work unless promisified; it only mentions, rather than " } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c17", "name": "Diagnose from a trace", "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n at renderRows (/app/src/table.js:42:19)\n at Table (/app/src/table.js:12:5)\n at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.", "output": "**Most likely root cause:** `renderRows` is calling `.map` on a value (rows/data/items prop or derived state) that is `undefined` on the initial production render. The code assumes it is always an array.\n\n**Why dev passes but prod first paint fails:** In dev you often mount with data already present (HMR, slower paint, Strict Mode double-render, mocks, or a parent that has resolved). The production bundle’s first paint hits the real initial state/props (empty fetch, missing default, SSR/CSR mismatch, or a parent that doesn’t pass the array yet) before any update fills it in.\n\n**First thing to check:** At `/app/src/table.js:42`, identify the exact expression being `.map`’d and confirm whether it can be `undefined`/`null` on first render (log or break on `props`/`state` entering `Table` and `renderRows`). Then guard it—e.g. default param / `defaultProps` / `rows = []`, `rows?.map`, or an early return—so the first paint never calls `.map` on a non-array.", "latency_ms": 20079, "cost": 0.0060284, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate root cause analysis and sensible first check; well-structured and concise. Slightly verbose but clear and non-technical-friendly." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately identifies the root cause and provides a clear, actionable first step to check, perfectly addressing the prompt's requirements in a concise manner suitable for a non-technical professional." }, { "judge": "openai/gpt-5.6-terra", "score": 9, "violates": false, "reasoning": "Correctly identifies an undefined initial value being mapped and prioritizes checking the exact line/expression and first-render inputs. The proposed guards are practical. The dev-versus-production explanations are somewhat speculative, but" } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c18", "name": "Idempotency", "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.", "output": "**Idempotent POST /payments**\n\n**Key:** Client-generated `Idempotency-Key` header (UUID). Optional body hash bound to the key.\n\n**Storage:** Durable store (e.g. Postgres) with a unique constraint on `idempotency_key`. Row holds key, request fingerprint, status (`processing` | `done`), and final response (status + body). TTL optional.\n\n**Flow:**\n1. Insert row for key in `processing` (or no-op if exists).\n2. Unique violation / existing row → do not charge again.\n3. On success: run payment, write response, set `done`.\n4. Concurrent duplicate: second writer loses the insert (or waits on row lock / advisory lock); it reads the winner’s row and returns that outcome once `done` (or 409/425 while `processing`).\n\n**Second time:** Same HTTP status and body as the first completed attempt (replay), including errors after the charge decision.\n\n**Residual failure mode:** Process charges the provider, then crashes before persisting the `done` response—retry may not see a record and can double-charge (fix-effect before commit).", "latency_ms": 8600, "cost": 0.0033384, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, concise, covers key, storage, concurrency, replay, and names a real gap (crash between charge and commit). Slightly informal formatting but within word limit and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly addresses all prompts (key, storage, concurrency, replay, and residual failure mode) with excellent technical accuracy. It is highly concise, clear, and strictly adheres to the 200-word limit constraint." }, { "judge": "openai/gpt-5.6-terra", "score": 7, "violates": false, "reasoning": "Covers key, durable storage, concurrent handling, and replay response clearly. However, the stated residual failure is inconsistent: a committed processing row should still exist after the crash. The real issue is ambiguous/in-doubt process" } ], "machine_checks": { "max_words": 200 }, "machine_violation": false, "machine_reasons": [] } ], "recapped_at": "2026-08-16T08:16:35.388Z", "recap_note": "Machine constraint caps applied retroactively under protocol judge-2026-08b; judge verdicts unchanged, task scores capped where a deterministic check failed." }