{ "entity": "gpt-5-3-codex", "entity_type": "model", "category": "Coding", "suite": "suite-2026-08-skills/coding", "model": "openai/gpt-5.3-codex", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 92, "caps": 0, "cost_usd": 0.3425, "started_at": "2026-08-16T06:25:52.659Z", "finished_at": "2026-08-16T06:26:35.629Z", "tasks": [ { "id": "c1", "name": "Duration parser", "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.", "output": "```python\nimport re\n\ndef parse_duration(s: str) -> int:\n \"\"\"\n Convert duration strings like:\n '1h30m', '45s', '2h', '90m', '1h2m3s'\n into total seconds.\n\n Rules:\n - Units must be in h -> m -> s order if present.\n - Each unit can appear at most once.\n - At least one unit must be present.\n - No extra characters/spaces allowed.\n \"\"\"\n if not isinstance(s, str):\n raise ValueError(\"Duration must be a string\")\n\n m = re.fullmatch(r'(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?', s)\n if not m:\n raise ValueError(f\"Malformed duration: {s!r}\")\n\n h, m_, s_ = m.groups()\n if h is None and m_ is None and s_ is None:\n raise ValueError(f\"Malformed duration: {s!r}\")\n\n total = 0\n if h is not None:\n total += int(h) * 3600\n if m_ is not None:\n total += int(m_) * 60\n if s_ is not None:\n total += int(s_)\n\n return total\n\n\n# --- assert-based tests ---\nassert parse_duration(\"1h30m\") == 5400\nassert parse_duration(\"1h2m3s\") == 3723\n\ntry:\n parse_duration(\"1h70x\") # malformed\n assert False, \"Expected ValueError for malformed input\"\nexcept ValueError:\n pass\n```", "latency_ms": 4607, "cost": 0.00769475, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct regex-based parser, handles malformed input via ValueError, includes 3 tests with one malformed case. Clean and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a correct and robust Python function, handles malformed inputs by raising ValueError, and includes exactly three assert-based tests (including one for malformed input) as requested." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Solid correct function and three assert tests including malformed. Minor: only two positive cases, try/except style slightly indirect." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c2", "name": "Find the bug", "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) sum += x;\n n++;\n }\n return sum / n;\n}", "output": "The bug is that `n` is incremented for **every** element instead of only positive ones, so the sum of positives is divided by the total count.\n\n```js\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) {\n sum += x;\n n++;\n }\n }\n return n === 0 ? 0 : sum / n; // or NaN/null depending on desired behavior\n}\n```", "latency_ms": 1935, "cost": 0.0026775, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly identifies bug, explains clearly in one sentence, provides fixed code with sensible zero-division handling." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the bug, explains it in exactly one sentence as requested, and provides a correct and robust implementation of the function." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly identifies bug in one sentence and supplies fixed code with edge-case handling; fully meets task." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c3", "name": "SQL without window functions", "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.", "output": "```sql\nSELECT\n u.email,\n COUNT(*) AS order_count\nFROM users u\nJOIN orders o\n ON o.user_id = u.id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY u.id, u.email\nHAVING COUNT(*) > 3\nORDER BY order_count DESC;\n```\n\nThis returns each user’s email and how many orders they placed in the last 30 days, only for users with more than 3 orders, sorted from most orders to least.", "latency_ms": 1910, "cost": 0.00279825, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, meets constraints, no window functions, Postgres syntax valid, clear and concise explanation." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The SQL query is perfectly accurate, follows all constraints including avoiding window functions, targets PostgreSQL correctly, and is concise and easy to understand." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct Postgres SQL meeting all requirements exactly; clear, concise, and immediately useful." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c4", "name": "Infinite useEffect", "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);", "output": "`useEffect` runs **after every render where any dependency changed**.\n\nIn your code:\n\n```jsx\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);\n```\n\n`items` is in the dependency array, and inside the effect you call `setItems(...)`.\n\nSo the cycle is:\n\n1. Initial render (`items = []`)\n2. Effect runs (because first mount)\n3. Fetch resolves, `setItems(newData)` updates state\n4. Component re-renders with new `items`\n5. React sees `items` changed (new array reference), so effect runs again\n6. Fetch again → `setItems` again → repeat forever\n\nEven if data content is “same”, it’s usually a **new array/object reference**, so React treats it as changed.\n\n---\n\n### Fixed version (fetch once on mount)\n\n```jsx\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n fetch('/api/items')\n .then(r => r.json())\n .then(setItems);\n}, []); // no items dependency\n```\n\n---\n\nIf you want refetching based on something, put that “something” (e.g. `userId`, `query`) in deps—not `items` itself.", "latency_ms": 3466, "cost": 0.00516775, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate explanation of infinite loop cause, correct fixed code, clear and concise, useful for professional. Minor verbosity but solid." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the infinite loop caused by updating a dependency inside the effect, and provides the correct fix. It is clear, accurate, and concise." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate explanation of infinite loop via deps and setState; correct empty-deps fix; clear, concise, highly useful." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c5", "name": "Typed debounce", "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.", "output": "```ts\ntype Debounced unknown> =\n ((...args: Parameters) => void) & { cancel: () => void };\n\nfunction debounce unknown>(\n fn: F,\n waitMs: number\n): Debounced {\n let timer: ReturnType | undefined;\n\n const debounced = ((...args: Parameters) => {\n if (timer !== undefined) {\n clearTimeout(timer);\n }\n timer = setTimeout(() => {\n fn(...args);\n }, waitMs);\n }) as Debounced;\n\n debounced.cancel = () => {\n if (timer !== undefined) {\n clearTimeout(timer);\n timer = undefined;\n }\n };\n\n return debounced;\n}\n```\n\nExample usage (fully typed):\n\n```ts\nconst save = (id: string, force: boolean) => {\n console.log(id, force);\n};\n\nconst debouncedSave = debounce(save, 300);\n\ndebouncedSave(\"abc\", true); // ✅ typed as (id: string, force: boolean)\ndebouncedSave.cancel(); // ✅ available\n```\n\nA naive signature like `(fn: (...args: T[]) => void)` loses type information because it turns a function’s specific parameter tuple into a homogeneous array type, so per-position argument types are no longer preserved.", "latency_ms": 3821, "cost": 0.00621075, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, avoids 'any', includes cancel(), types preserved, uses one 'as' cast but no 'any' keyword; explanation sentence given as required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly implements the debounce function with strict typing, avoids 'any', includes the cancel method, and provides an accurate one-sentence explanation." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Flawless typed debounce with cancel, no any; accurate one-sentence explanation; clear example." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c6", "name": "Leftmost binary search", "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.", "output": "Here’s a binary search that returns the **first occurrence** of `target` in a sorted array (with possible duplicates), or `-1` if not found.\n\n```python\ndef first_occurrence(arr, target):\n left, right = 0, len(arr) - 1\n ans = -1\n\n while left <= right:\n mid = (left + right) // 2\n\n if arr[mid] == target:\n ans = mid # potential answer\n right = mid - 1 # keep searching left half for earlier occurrence\n elif arr[mid] < target:\n left = mid + 1\n else:\n right = mid - 1\n\n return ans\n```\n\n### Complexity\n- **Time:** `O(log n)` \n- **Space:** `O(1)`\n\n### Test case that distinguishes it from ordinary binary search\n```python\narr = [1, 2, 2, 2, 3, 4]\ntarget = 2\nprint(first_occurrence(arr, target)) # 1\n```\n\nWhy this distinguishes it:\n- A standard binary search might return index `2` or `3` (any matching index).\n- This version must return **`1`**, the first occurrence.", "latency_ms": 2804, "cost": 0.004585, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct, efficient implementation with proper complexity stated and a clear distinguishing test case demonstrating first-occurrence behavior vs ordinary binary search." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly implements the requested binary search, correctly states the time and space complexity, and provides a clear test case that demonstrates the difference from standard binary search." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct first-occurrence binary search, accurate O(log n) complexity, distinguishing test case included; clear and complete." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c7", "name": "Security review", "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n const p = req.query.name;\n db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n res.sendFile(__dirname + '/uploads/' + p);\n });\n});", "output": "Here are the security issues, **most severe first**, with fixes.\n\n1. **Arbitrary file read (Path Traversal) — Critical**\n - `res.sendFile(__dirname + '/uploads/' + p)` uses untrusted input directly in a path.\n - Attackers can request things like `?name=../../../../etc/passwd` (or encoded variants) to read files outside `/uploads`.\n - **Fix:**\n - Never concatenate paths from user input.\n - Use a safe filename/id model (e.g., UUID stored in DB), or strict allowlist validation.\n - Use `path.resolve` + boundary check, or `sendFile` with `root` and validated basename only.\n\n2. **SQL Injection — Critical**\n - `db.query(\\`SELECT * FROM files WHERE name = '${p}'\\`)` is injectable.\n - Attackers can alter query logic (`' OR 1=1 --`) or potentially exfiltrate/modify data depending on DB config.\n - **Fix:**\n - Use parameterized queries/prepared statements:\n - `db.query('SELECT * FROM files WHERE name = ?', [p], ...)`\n\n3. **Broken access control / IDOR — High**\n - No authentication or authorization check is shown.\n - Anyone who knows/guesses a filename can retrieve it.\n - **Fix:**\n - Require auth middleware.\n - Check that requested file belongs to the authenticated user (or user has permission) before serving.\n\n4. **Fail-open logic (ignores DB result and DB errors) — High**\n - The DB query result is ignored; file is sent regardless of whether record exists or query failed.\n - This bypasses any intended DB-based control.\n - **Fix:**\n - If DB error: return 500 safely.\n - If no matching/authorized row: return 404/403.\n - Only call `sendFile` after successful authorization decision.\n\n5. **Insufficient input validation — Medium**\n - `name` is unrestricted (can include separators, control chars, huge values, weird encodings).\n - **Fix:**\n - Enforce strict schema (length, charset, extension, no slashes, etc.).\n - Prefer opaque file IDs rather than raw names.\n\n6. **Weak error handling / potential info leakage — Medium**\n - No `sendFile` callback; errors may leak behavior differences or internals (especially in non-production configs).\n - **Fix:**\n - Add explicit error handling and generic responses.\n - Ensure production disables stack traces to clients.\n\n---\n\n### Minimal safer pattern (example)\n\n```js\nconst path = require('path');\n\napp.get('/file', requireAuth, async (req, res) => {\n const name = req.query.name;\n if (!/^[a-zA-Z0-9._-]{1,100}$/.test(name)) {\n return res.status(400).send('Invalid file name');\n }\n\n db.query(\n 'SELECT stored_name FROM files WHERE name = ? AND owner_id = ?',\n [name, req.user.id],\n (err, rows) => {\n if (err) return res.status(500).send('Server error');\n if (!rows || rows.length === 0) return res.status(404).send('Not found');\n\n const stored = rows[0].stored_name; // ideally server-generated safe name\n res.sendFile(stored, { root: path.join(__dirname, 'uploads') }, (e) => {\n if (e) res.status(e.statusCode || 404).send('Not found');\n });\n }\n );\n});\n```\n\nIf you want, I can provide a hardened version using file IDs (best practice) instead of filenames.", "latency_ms": 10284, "cost": 0.01689275, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, thorough, well-ordered by severity, actionable fixes, includes safer code example. Minor verbosity but overall excellent and correct." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately identifies all security issues, orders them by severity, provides clear fixes, and includes a helpful, secure code example. It perfectly meets all criteria." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate prioritized list of key vulns with solid fixes and example; minor polish only, highly useful and clear." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c8", "name": "Safe migration", "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.", "output": "Use the **expand / backfill / enforce** pattern.\n\n```sql\n-- 1) Add the column as nullable first (fast metadata change)\nALTER TABLE orders\n ADD COLUMN status text;\n\n-- 2) Set default for new rows (fast metadata change)\nALTER TABLE orders\n ALTER COLUMN status SET DEFAULT 'pending';\n```\n\nThen backfill old rows in small batches from your app/script/job runner:\n\n```sql\n-- Repeat until 0 rows updated\nWITH batch AS (\n SELECT id\n FROM orders\n WHERE status IS NULL\n ORDER BY id\n LIMIT 10000\n FOR UPDATE SKIP LOCKED\n)\nUPDATE orders o\nSET status = 'pending'\nFROM batch\nWHERE o.id = batch.id;\n```\n\n(Commit each batch.)\n\nAfter backfill is complete:\n\n```sql\n-- 3) Add a NOT VALID check (quick lock, no full scan yet)\nALTER TABLE orders\n ADD CONSTRAINT orders_status_not_null_chk\n CHECK (status IS NOT NULL) NOT VALID;\n\n-- 4) Validate it (full table scan, but not long ACCESS EXCLUSIVE lock)\nALTER TABLE orders\n VALIDATE CONSTRAINT orders_status_not_null_chk;\n\n-- 5) Flip to real NOT NULL\nALTER TABLE orders\n ALTER COLUMN status SET NOT NULL;\n\n-- Optional cleanup\nALTER TABLE orders\n DROP CONSTRAINT orders_status_not_null_chk;\n```\n\n### Dangerous step (and why)\n\nThe dangerous step is **`ALTER COLUMN status SET NOT NULL` if done naively** (or doing everything in one big ALTER on older setups), because Postgres may need to verify all rows and can require an **ACCESS EXCLUSIVE** lock long enough to block writes (and possibly reads depending contention). \nBy backfilling first and validating via `CHECK ... NOT VALID` + `VALIDATE CONSTRAINT`, you move the expensive scan to a safer step with weaker locking, making the final `SET NOT NULL` short.\n\nIf you’re on PG11+, adding a column with constant default is much cheaper than older versions, but the phased approach is still the safest for a 40M-row table.", "latency_ms": 12443, "cost": 0.0198415, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Technically accurate, correct pattern, identifies backfill as risky (though final NOT NULL is actually safe in PG12+, minor inaccuracy). Clear steps, useful, slightly verbose but well organized." }, { "judge": "google/gemini-3.1-pro-preview", "score": 9, "violates": false, "reasoning": "Excellent, accurate explanation of the safe backfill pattern. It correctly identifies the locking danger of a naive NOT NULL constraint, though it could emphasize more that PG11+ makes a single ALTER statement safe for constant defaults." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Accurate phased migration for large table; correctly flags lock risk. Minor version/edge notes could be tighter; clear ordered steps." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c9", "name": "Fix the code not the test", "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify(' Hello World! ')).toBe('hello-world');", "output": "You need to normalize whitespace and strip punctuation, not just replace single spaces.\n\n```js\nexport const slugify = (s) =>\n s\n .toLowerCase()\n .trim()\n .replace(/[^a-z0-9\\s-]/g, '') // remove punctuation/symbols\n .replace(/\\s+/g, '-') // collapse whitespace to single -\n .replace(/-+/g, '-'); // collapse repeated -\n```\n\nWith this:\n\n```js\nslugify(' Hello World! ') // \"hello-world\"\n```", "latency_ms": 1793, "cost": 0.0026705, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correctly fixes implementation to handle trimming, multiple spaces, and punctuation, passing the test. Clear, concise, accurate code with explanation." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response correctly identifies the issue and provides a robust implementation that passes the test, handling leading/trailing spaces, multiple spaces, and punctuation. It is clear and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct fix for slugify; matches test exactly. Clear, concise, useful code with explanation." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c10", "name": "Race condition", "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n if (cache) return cache;\n const r = await fetch('/config');\n cache = await r.json();\n return cache;\n}", "output": "The race is a classic **check-then-act** around an `await`.\n\n### What goes wrong\n\n`getConfig()` does:\n\n1. Check `if (cache)`.\n2. If empty, start `fetch` and `await`.\n\nBecause `await` yields control, another caller can run in between.\n\n### Problematic interleaving\n\nAssume two callers: A and B.\n\n1. A calls `getConfig()`, sees `cache === null`.\n2. A starts `fetch('/config')`, hits `await` (pauses).\n3. B calls `getConfig()`, still sees `cache === null`.\n4. B also starts `fetch('/config')`, hits `await`.\n\nNow you have **two in-flight requests**.\n\nIf responses differ over time, you can also get overwrite:\n\n5. B finishes first, sets `cache = configNew`.\n6. A finishes later, sets `cache = configOld` (stale overwrite).\n\n---\n\n## Fix: cache the in-flight Promise (single-flight)\n\n```js\nlet cache = null;\nlet cachePromise = null;\n\nasync function getConfig() {\n if (cache !== null) return cache; // already resolved value\n if (cachePromise) return cachePromise; // request in flight\n\n cachePromise = fetch('/config')\n .then(r => r.json())\n .then(cfg => {\n cache = cfg;\n return cfg;\n })\n .finally(() => {\n // Clear in-flight marker after completion.\n // Keep `cache` for successful calls.\n cachePromise = null;\n });\n\n return cachePromise;\n}\n```\n\n### Why this works\n\n- First caller creates `cachePromise`.\n- Concurrent callers reuse that same Promise instead of starting new fetches.\n- Only one request is made at a time.\n- All callers get the same result.\n\nIf you want retries after failure, this already supports it (`cache` stays `null`, `cachePromise` is cleared in `finally`).", "latency_ms": 5874, "cost": 0.0085015, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate diagnosis of race condition, correct interleaving explanation, and a solid single-flight fix. Clear, well-structured, concise for the technical depth required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the race condition, clearly explains the problematic interleaving, and provides a robust, standard fix by caching the promise. It is highly useful, accurate, and concise." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate race ID, clear interleaving, solid single-flight fix. Minor polish only; highly usable and clear." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c11", "name": "Retry with backoff", "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.", "output": "```ts\ntype RetryOptions = {\n maxAttempts: number;\n baseDelayMs?: number;\n maxDelayMs?: number;\n jitterRatio?: number; // 0.2 => ±20%\n};\n\nexport async function retryWithBackoff(\n op: () => Promise,\n { maxAttempts, baseDelayMs = 200, maxDelayMs = 10_000, jitterRatio = 0.2 }: RetryOptions\n): Promise {\n for (let attempt = 1; attempt <= maxAttempts; attempt++) {\n try {\n return await op();\n } catch (err: any) {\n const status = err?.status ?? err?.response?.status;\n const nonRetriable4xx = status >= 400 && status < 500 && status !== 429;\n if (nonRetriable4xx || attempt === maxAttempts) throw err;\n\n const exp = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));\n const jitter = 1 + (Math.random() * 2 - 1) * jitterRatio; // [1-j, 1+j]\n const delay = Math.max(0, Math.floor(exp * jitter));\n await new Promise((r) => setTimeout(r, delay));\n }\n }\n throw new Error(\"Unreachable\");\n}\n```\n\nOn final failure (max attempts reached, or a non-retriable 4xx except 429), it **throws the last/original error** to the caller; no further retries occur.", "latency_ms": 5550, "cost": 0.0096565, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct logic, meets constraints, concise (~26 lines code), states final failure behavior clearly. Minor: unreachable throw line slightly odd but harmless." }, { "judge": "google/gemini-3.1-pro-preview", "score": 5, "violates": true, "reasoning": "The response provides a correct and well-implemented TypeScript retry wrapper meeting all functional requirements. However, it violates the maximum 30 lines constraint, as the total response length is 31 lines." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct TS impl of backoff+jitter+max+4xx/429 rules; under 30 lines; clearly states final throw. Minor: assumes err.status shape." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c12", "name": "Recursive type", "prompt": "Write a TypeScript type DeepPartial that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.", "output": "```ts\ntype DeepPartial =\n T extends (...args: any[]) => any\n ? T\n : T extends ReadonlyArray\n ? ReadonlyArray>\n : T extends Array\n ? Array>\n : T extends object\n ? { [K in keyof T]?: DeepPartial }\n : T;\n```\n\nArrays must be treated by recursively partializing their **element type** while preserving the array/readonly-array wrapper, instead of treating arrays like plain objects with optional numeric keys and methods.", "latency_ms": 2336, "cost": 0.00364875, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, functional implementation; explanation clear and accurate; slightly verbose but concise overall." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless TypeScript implementation of DeepPartial and perfectly explains the array handling in exactly one sentence, meeting all requirements." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct DeepPartial impl handling arrays/functions/objects; one-sentence explanation accurate and clear. Minor: slightly verbose for non-tech audience." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c13", "name": "Bash one-liner", "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.", "output": "```sh\nfind . -type f -exec du -h {} + | sort -rh | head -n 10\n```\n\n**What each part does:**\n- `find . -type f` — recursively find all regular files under the current directory (`.`).\n- `-exec du -h {} +` — run `du -h` on those files in batches (`+`), producing human-readable sizes.\n- `| sort -rh` — sort results by size (`-h` = human numeric, `-r` = largest first).\n- `| head -n 10` — keep only the top 10 largest files.\n\nThis handles filenames with spaces because `find ... -exec ...` passes each filename as a proper argument (not split on spaces).", "latency_ms": 5115, "cost": 0.007623, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, handles spaces properly, clear explanation. Minor note: du -h with batched files could produce slight sort inconsistencies for very large files across different units, but generally works well." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a perfectly accurate, single shell command that meets all requirements, including handling spaces correctly. The explanations are clear, concise, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct, space-safe command with accurate brief explanations; minor polish possible but expert-level." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c14", "name": "Explain and cost", "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n for i in range(len(xs)):\n for j in range(i+1, len(xs)):\n if xs[i] == xs[j]:\n return True\n return False", "output": "`has_dup(xs)` checks whether a list contains any duplicate value.\n\n### What it does\n```python\ndef has_dup(xs):\n for i in range(len(xs)):\n for j in range(i+1, len(xs)):\n if xs[i] == xs[j]:\n return True\n return False\n```\n\n- It compares each element with every element after it.\n- If any pair is equal, it returns `True`.\n- If no equal pair is found, it returns `False`.\n\n### Complexity of this version\n- **Time:** `O(n^2)` (nested loops over pairs)\n- **Space:** `O(1)` extra space\n\n---\n\n### O(n) rewrite\nUse a `set` to track values already seen:\n\n```python\ndef has_dup(xs):\n seen = set()\n for x in xs:\n if x in seen:\n return True\n seen.add(x)\n return False\n```\n\n### Complexity of rewritten version\n- **Time:** `O(n)` average case (set lookup/add is average `O(1)`)\n- **Space:** `O(n)` extra space for the set", "latency_ms": 2665, "cost": 0.00448525, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate explanation, correct complexities, correct O(n) rewrite with tradeoff noted. Clear, concise, well-structured. Minor nuance: average-case O(1) set ops noted appropriately." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the function, correctly identifies the time and space complexity of the original code, and provides an accurate O(n) rewrite using a set with its corresponding complexities." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Accurate explanation, complexities, and correct O(n) rewrite using set; clear, complete, no flaws." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c15", "name": "Regex with limits", "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.", "output": "A practical (but **not perfect**) regex is:\n\n```regex\n^(?:[A-Z]{1,2}\\d[A-Z\\d]?\\s?\\d[A-Z]{2})$\n```\n\nOne-line breakdown of each part:\n\n- `^` — start of string (so nothing can appear before the postcode). \n- `(?: ... )` — non-capturing group for the whole postcode pattern. \n- `[A-Z]{1,2}` — 1 or 2 letters for the postcode area. \n- `\\d` — one digit for the district. \n- `[A-Z\\d]?` — optional extra district character (letter or digit). \n- `\\s?` — optional space between outward and inward parts. \n- `\\d` — one digit for the sector. \n- `[A-Z]{2}` — final two letters for the unit. \n- `$` — end of string (so nothing can appear after the postcode). \n\nTwo examples this regex would **accept even though they’re invalid**:\n\n1. `QV1 1AA` (uses postcode area letters that are not assigned). \n2. `A0 0AA` (invalid area/district combination in real UK allocations). \n\nSo this is useful for format-checking, but it does **not** fully validate against all real UK postcode rules/special cases.", "latency_ms": 11431, "cost": 0.01756125, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 6, "violates": false, "reasoning": "Regex reasonable, explanation clear, states imperfection as required. But task asked for invalid rejection OR acceptance examples; gave two 'accepted but invalid' rather than showing rejected valid ones, less varied than ideal." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly follows all instructions, provides a solid regex with accurate one-line explanations, correctly identifies two invalid postcodes it would accept, and explicitly avoids claiming perfection." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate simplified regex with clear one-line parts; correctly gives two invalid accepts; admits imperfection; fully follows task; clear and useful." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c16", "name": "Callback to async", "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n if (e) return cb(e);\n getOrders(user.id, (e2, orders) => {\n if (e2) return cb(e2);\n getTotals(orders, (e3, totals) => cb(e3, totals));\n });\n});", "output": "Here’s a clean `async/await` refactor with proper error propagation (no swallowed errors).\n\n### If you can return a Promise (preferred)\n\n```js\nconst { promisify } = require('node:util');\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id) {\n const user = await getUserAsync(id);\n const orders = await getOrdersAsync(user.id);\n const totals = await getTotalsAsync(orders);\n return totals;\n}\n```\n\n- Any failure in `getUser`, `getOrders`, or `getTotals` will reject the returned Promise automatically.\n- No `catch` block needed unless you want to transform/log and rethrow.\n\n---\n\n### If you must keep callback API (`cb`)\n\n```js\nconst { promisify } = require('node:util');\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id, cb) {\n try {\n const user = await getUserAsync(id);\n const orders = await getOrdersAsync(user.id);\n const totals = await getTotalsAsync(orders);\n cb(null, totals);\n } catch (err) {\n cb(err); // propagate error, do not swallow\n }\n}\n```\n\nThis preserves the original callback behavior while using `async/await` internally.", "latency_ms": 3334, "cost": 0.00573125, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, clear, functional refactor with proper error propagation via try/catch and promisify; offers two useful variants; slightly verbose but appropriate for clarity." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly refactors the code to async/await, correctly handles error propagation without swallowing errors, and provides both promise-returning and callback-preserving options. It is clear, accurate, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct async/await refactor with proper error propagation via promises or try/catch; clear dual options, minor Node assumption." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c17", "name": "Diagnose from a trace", "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n at renderRows (/app/src/table.js:42:19)\n at Table (/app/src/table.js:12:5)\n at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.", "output": "**Most likely root cause:** \n`renderRows` is being called with an **undefined array-like value** (probably `rows`), and then `.map()` is executed on it during the initial production render.\n\nBecause it fails only on **production first paint**, the usual culprit is an **initial render timing/data availability issue** (e.g., async data not loaded yet, hydration/SSR initial props missing, or prod API payload shape differs).\n\n**First thing I’d check:** \nAt `/app/src/table.js` lines 12 and 42, check exactly what variable is being mapped and what its value is on first render in prod. \nSpecifically verify:\n\n- initial state/prop default is `[]` (not `undefined`)\n- any fetched data path exists before mapping (e.g., `data?.rows ?? []`)\n- SSR/initial props include that field in production\n\nIn short: confirm the mapped value is always an array on first render.", "latency_ms": 5607, "cost": 0.0071435, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, plausible root cause and reasonable checks; concise and useful despite generic env-specific detail lacking." }, { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Accurate and directly answers the prompt. However, it includes some technical jargon (SSR, hydration, optional chaining) which slightly reduces usefulness for a strictly non-technical professional, though it remains clear." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate root cause and practical first check; clear, useful, fully addresses the query with only minor polish gaps." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c18", "name": "Idempotency", "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.", "output": "Use a client-supplied **Idempotency-Key** (UUID) on `POST /payments`, scoped by `merchant_id` (or customer+endpoint).\n\nStore it in a durable `payment_idempotency` table:\n\n- `(merchant_id, idempotency_key)` **UNIQUE**\n- `request_hash` (amount/currency/source fingerprint)\n- `status` (`processing|succeeded|failed`)\n- `payment_id`\n- full `http_status` + `response_body`\n- timestamps/TTL\n\nFlow:\n\n1. Start DB transaction.\n2. `INSERT ... ON CONFLICT DO NOTHING` a row with `processing`.\n3. If insert succeeds, this request “owns” execution; call processor, create payment, then persist final status + response, commit.\n4. If conflict:\n - If `request_hash` differs, return **409 Conflict** (key reuse with different payload).\n - If same and existing row is `succeeded/failed`, return **exactly the same** `http_status` and `response_body` as first call.\n - If same and `processing`, return **202 Accepted** (or 409 + Retry-After) indicating in-flight duplicate.\n\nSo a retry cannot create a second local payment record/charge attempt.\n\n**Remaining failure mode:** **external side-effect ambiguity** (“charged-but-not-recorded”): processor charges successfully, but service crashes/network partitions before saving final idempotency result. This is mitigated with reconciliation but not eliminated.", "latency_ms": 5920, "cost": 0.00723275, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, thorough design covering key, storage, concurrency handling, response replay, and a valid failure mode. Slightly exceeds concise clarity but stays under 200 words; well-structured and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly addresses all prompts: defines the key, storage, concurrent handling, return values, and accurately identifies the remaining failure mode. It is highly accurate, concise, and strictly adheres to the 200-word limit." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Flawless idempotency design covering key, storage, concurrency, replay response, and residual failure; under 200 words, clear and complete." } ], "machine_checks": { "max_words": 200 }, "machine_violation": false, "machine_reasons": [] } ], "recapped_at": "2026-08-16T08:16:35.300Z", "recap_note": "Machine constraint caps applied retroactively under protocol judge-2026-08b; judge verdicts unchanged, task scores capped where a deterministic check failed." }