{ "entity": "gpt-5-6-luna", "entity_type": "model", "category": "Coding", "suite": "suite-2026-08-skills/coding", "model": "openai/gpt-5.6-luna", "judges": [ "anthropic/claude-sonnet-5", "google/gemini-3.1-pro-preview", "x-ai/grok-4.5" ], "judge_protocol": "judge-2026-08", "anchors": "solo-2026-08-a", "score": 93, "caps": 0, "cost_usd": 0.2005, "started_at": "2026-08-16T06:27:46.953Z", "finished_at": "2026-08-16T06:28:35.928Z", "tasks": [ { "id": "c1", "name": "Duration parser", "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.", "output": "```python\nimport re\n\n\ndef parse_duration(s):\n \"\"\"Convert a duration such as '1h30m' to seconds.\"\"\"\n if not isinstance(s, str):\n raise ValueError(\"duration must be a string\")\n\n match = re.fullmatch(\n r\"(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?\",\n s,\n )\n\n if match is None or not any(match.groups()):\n raise ValueError(f\"malformed duration: {s!r}\")\n\n hours, minutes, seconds = (\n int(value or 0) for value in match.groups()\n )\n return hours * 3600 + minutes * 60 + seconds\n\n\n# Tests\nassert parse_duration(\"1h30m\") == 5400\nassert parse_duration(\"1h2m3s\") == 3723\ntry:\n parse_duration(\"1hour\")\nexcept ValueError:\n malformed = True\nelse:\n malformed = False\nassert malformed\n```", "latency_ms": 6585, "cost": 0.0003639, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct implementation, handles malformed input properly, includes three tests as required. Minor: uses try/except instead of pure assert style, but functionally correct and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a robust, accurate Python function using regex, correctly raises ValueError on malformed inputs, and includes exactly three assert-based tests as requested." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct parsing and error handling for all examples; three solid assert tests including malformed. Minor verbosity in malformed test setup." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c2", "name": "Find the bug", "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n for (const x of xs) {\n if (x > 0) sum += x;\n n++;\n }\n return sum / n;\n}", "output": "The bug is that `n` counts every element instead of only the positive ones.\n\n```js\nfunction avgPositive(xs) {\n let sum = 0, n = 0;\n\n for (const x of xs) {\n if (x > 0) {\n sum += x;\n n++;\n }\n }\n\n return sum / n;\n}\n```", "latency_ms": 3725, "cost": 0.0002256, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct bug identification, concise explanation, and correct fixed function. Minor: doesn't handle n=0 division edge case, but not required." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the bug, explains it in exactly one sentence as requested, and provides the correctly fixed function." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct bug ID and one-sentence explanation; fixed code is accurate and complete." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c3", "name": "SQL without window functions", "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.", "output": "```sql\nSELECT\n u.email,\n COUNT(o.id) AS order_count\nFROM users AS u\nJOIN orders AS o\n ON o.user_id = u.id\nWHERE o.created_at >= CURRENT_TIMESTAMP - INTERVAL '30 days'\nGROUP BY u.id, u.email\nHAVING COUNT(o.id) > 3\nORDER BY order_count DESC;\n```", "latency_ms": 3075, "cost": 0.0001209, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct Postgres SQL, no window functions, proper filtering, grouping, having clause, and ordering as requested. Clean and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The SQL query is perfectly accurate, follows all instructions, avoids window functions, and correctly targets PostgreSQL syntax to return the requested data." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct Postgres SQL; filters, groups, having, order exact; no windows; clear and complete." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c4", "name": "Infinite useEffect", "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);", "output": "`items` is both:\n\n1. A dependency of the effect, and\n2. Updated by the effect.\n\nThe sequence is:\n\n1. The component renders with the initial `items` array.\n2. The effect runs and fetches the data.\n3. `setItems` stores the parsed JSON array.\n4. The component re-renders.\n5. The `items` dependency has changed by reference. React compares dependencies with `Object.is`; it does not deep-compare array contents.\n6. The effect runs again, fetches again, and `response.json()` creates another new array.\n7. `setItems` receives that new array, repeating the cycle.\n\nEven if the arrays contain identical items, they are different array objects.\n\nRun the fetch on mount instead:\n\n```jsx\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n fetch('/api/items')\n .then(response => {\n if (!response.ok) {\n throw new Error(`Request failed: ${response.status}`);\n }\n return response.json();\n })\n .then(setItems)\n .catch(console.error);\n}, []);\n```\n\n`setItems` is a stable React state setter, so it does not need to be included in the dependency array. In development, React Strict Mode may intentionally run the effect twice during mounting, but that is separate from the infinite loop.", "latency_ms": 6966, "cost": 0.0004869, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate, clear explanation of infinite loop cause with correct fix; minor extra content (error handling) but not harmful; concise and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response accurately and precisely explains the infinite loop caused by reference inequality and provides the correct fix with an empty dependency array. It is clear, concise, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate explanation of reference equality loop; solid fixed code with extras. Minor polish only." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c5", "name": "Typed debounce", "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.", "output": "```ts\ntype Debounced unknown> =\n ((...args: Parameters) => void) & {\n cancel(): void;\n };\n\nfunction debounce unknown>(\n fn: F,\n delay: number\n): Debounced {\n let timer: ReturnType | undefined;\n let pendingArgs: Parameters | undefined;\n\n const debounced = ((...args: Parameters): void => {\n pendingArgs = args;\n\n if (timer !== undefined) {\n clearTimeout(timer);\n }\n\n timer = setTimeout(() => {\n const argsToUse = pendingArgs;\n\n timer = undefined;\n pendingArgs = undefined;\n\n if (argsToUse !== undefined) {\n fn(...argsToUse);\n }\n }, delay);\n }) as Debounced;\n\n debounced.cancel = (): void => {\n if (timer !== undefined) {\n clearTimeout(timer);\n timer = undefined;\n }\n\n pendingArgs = undefined;\n };\n\n return debounced;\n}\n\n// Example:\nconst greet = debounce((name: string, age: number) => {\n console.log(`${name} is ${age}`);\n}, 300);\n\ngreet(\"Ada\", 36); // Type-checked\ngreet.cancel();\n```\n\nA naive signature such as `(...args: T[])` loses type information because it models all arguments as a homogeneous array instead of preserving the wrapped function’s positional tuple type.", "latency_ms": 7125, "cost": 0.0005051, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, no 'any', preserves types, has .cancel(), includes required explanation sentence. Minor: uses 'unknown' return, 'as' cast, but overall solid and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless TypeScript debounce implementation that preserves types, includes a cancel method, avoids 'any', and accurately explains the type loss in exactly one sentence." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct typed debounce with cancel, no any; precise one-sentence explanation of naive generic loss." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c6", "name": "Leftmost binary search", "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.", "output": "```python\ndef first_occurrence(arr, target):\n left, right = 0, len(arr) - 1\n result = -1\n\n while left <= right:\n mid = left + (right - left) // 2\n\n if arr[mid] == target:\n result = mid\n right = mid - 1 # Continue searching to the left\n elif arr[mid] < target:\n left = mid + 1\n else:\n right = mid - 1\n\n return result\n```\n\n### Test case\n\n```python\narr = [1, 2, 2, 2, 3, 4]\n\nprint(first_occurrence(arr, 2)) # 1\nprint(first_occurrence(arr, 5)) # -1\n```\n\nThe target `2` occurs at indices `1`, `2`, and `3`; the function returns the first index, `1`. An ordinary binary search might return index `2` or `3`.\n\n**Complexity:**\n\n- Time: `O(log n)`\n- Extra space: `O(1)`", "latency_ms": 6845, "cost": 0.0002092, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct implementation, correct complexity, valid distinguishing test case, clear and concise. Minor: could explain distinguishing aspect more explicitly but overall solid." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all requirements: provides correct code for finding the first occurrence, states time and space complexity, and includes a clear test case demonstrating the difference from standard binary search." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct first-occurrence binary search code, proper complexity, and distinguishing test case with clear explanation." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c7", "name": "Security review", "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n const p = req.query.name;\n db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n res.sendFile(__dirname + '/uploads/' + p);\n });\n});", "output": "1. **Arbitrary file read / path traversal — critical**\n\n `p` is concatenated directly into a filesystem path:\n\n ```js\n __dirname + '/uploads/' + p\n ```\n\n An attacker can request values such as:\n\n ```text\n ../../../../etc/passwd\n ..\\..\\..\\windows\\win.ini\n ```\n\n This can expose files outside `uploads`, including application source, configuration, credentials, or secrets. Symlinks inside `uploads` can also escape the directory.\n\n **Fix:**\n\n - Prefer an opaque file ID rather than a user-supplied path.\n - Resolve the path and verify it remains within the intended directory.\n - Validate that `name` is a string and, if appropriate, restrict it to a basename or an allowlisted character set.\n - Consider preventing symlink escapes or storing uploads outside the application directory.\n\n Example:\n\n ```js\n const path = require('node:path');\n const uploadDir = path.resolve(__dirname, 'uploads');\n\n const name = req.query.name;\n if (typeof name !== 'string' || name.length === 0) {\n return res.status(400).send('Invalid name');\n }\n\n const filePath = path.resolve(uploadDir, name);\n const relative = path.relative(uploadDir, filePath);\n\n if (relative.startsWith('..' + path.sep) || path.isAbsolute(relative)) {\n return res.status(400).send('Invalid path');\n }\n ```\n\n For filenames only, use `path.basename(name)` and reject any value whose basename differs from the supplied value.\n\n2. **SQL injection — critical**\n\n User input is interpolated into SQL:\n\n ```js\n `SELECT * FROM files WHERE name = '${p}'`\n ```\n\n A value such as `' OR '1'='1` can alter the query. Depending on the database driver and configuration, this may permit data disclosure, query manipulation, or additional statements.\n\n **Fix:** use a parameterized query:\n\n ```js\n db.query(\n 'SELECT * FROM files WHERE name = ?',\n [name],\n callback\n );\n ```\n\n Use the placeholder syntax appropriate for the database driver, and validate the input type before querying.\n\n3. **No authorization or access-control check — high**\n\n The handler allows any caller to request a file. There is no authentication, ownership check, tenant check, or permission check. If these are private files, this is an unrestricted file-download/IDOR vulnerability.\n\n **Fix:** authenticate the request and authorize access to the specific file before sending it:\n\n ```js\n if (!req.user) return res.sendStatus(401);\n\n // Verify that the requested file belongs to req.user or is otherwise accessible.\n ```\n\n Ideally, retrieve the file’s stored path and owner from the database and authorize that database record.\n\n4. **The database result is ignored, so the query does not protect file access — high**\n\n The code sends the requested path regardless of whether:\n\n - the query failed,\n - no matching database row exists, or\n - the caller is authorized to access the row.\n\n Thus, even a nonexistent or unauthorized database record does not prevent the file from being served.\n\n **Fix:** handle the error and require exactly one valid, authorized row before calling `sendFile`:\n\n ```js\n db.query('SELECT path, owner_id FROM files WHERE name = ?', [name],\n (err, rows) => {\n if (err) return next(err);\n if (rows.length !== 1) return res.sendStatus(404);\n\n // Perform authorization before sending.\n res.sendFile(verifiedPath, err => {\n if (err && !res.headersSent) next(err);\n });\n });\n ```\n\n5. **Missing error handling — medium**\n\n The database error `e` is ignored, and errors from `sendFile` are not handled. This can cause incorrect responses, confusing behavior, or error-handling failures. Depending on logging and framework configuration, database errors may also expose sensitive information.\n\n **Fix:** use Express error middleware via `next`, return appropriate status codes, and avoid sending a response after an error:\n\n ```js\n app.get('/file', async (req, res, next) => {\n try {\n // validate, query, authorize, and send\n } catch (err) {\n next(err);\n }\n });\n ```\n\n6. **Insufficient input/type validation — medium**\n\n `req.query.name` is not guaranteed to be a simple string. Depending on the query parser, inputs such as `?name[]=x` or structured query values may produce arrays or objects. Very long values can also cause unnecessary database and filesystem work.\n\n **Fix:**\n\n ```js\n if (typeof req.query.name !== 'string' ||\n req.query.name.length === 0 ||\n req.query.name.length > 255) {\n return res.status(400).send('Invalid name');\n }\n ```\n\n Apply an appropriate filename policy, such as rejecting control characters, separators, NUL characters, and disallowed extensions.\n\n7. **Potential denial of service — low to medium**\n\n Unbounded attacker-controlled input is inserted into a SQL query and passed to filesystem handling. Repeated requests can also cause database and disk I/O. This is worsened by the lack of authentication, rate limiting, and input limits.\n\n **Fix:** enforce request/query-size limits, validate and bound the filename, use database timeouts/pooling limits, and apply authentication and rate limiting where appropriate.\n\nA safer design is to accept an opaque file identifier, query the corresponding database record with a parameterized query, authorize that record, use a server-side stored path, verify the resolved path remains within the upload directory, and only then call `res.sendFile`.", "latency_ms": 12890, "cost": 0.0010543, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Comprehensive, accurate, well-prioritized security review with correct fixes for path traversal, SQL injection, missing authz/error handling. Slightly verbose but appropriate given depth requested." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response flawlessly identifies all security vulnerabilities, orders them correctly by severity, and provides clear, actionable fixes. It is highly accurate and extremely useful." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate prioritized list of real issues with solid fixes; minor extra length but highly useful and clear." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c8", "name": "Safe migration", "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.", "output": "For a large table, do this in phases so that the only `ACCESS EXCLUSIVE` locks are short catalog changes.\n\nAssume `orders.id` is an indexed primary key.\n\n### 1. Add the column as nullable\n\n```sql\nBEGIN;\n\nALTER TABLE orders\n ADD COLUMN status text;\n\nALTER TABLE orders\n ALTER COLUMN status SET DEFAULT 'pending';\n\nCOMMIT;\n```\n\nThis does not populate the 40 million existing rows. The default applies to new inserts, while existing rows initially have `NULL`.\n\nThe catalog changes are fast, but they still require a table lock. Run them with a short `lock_timeout` so they fail rather than wait behind a long-running transaction:\n\n```sql\nSET lock_timeout = '2s';\n```\n\nRetry if necessary.\n\n### 2. Backfill existing rows in batches\n\nRun many small transactions rather than one transaction updating 40 million rows.\n\nFor example:\n\n```sql\nWITH batch AS (\n SELECT id\n FROM orders\n WHERE status IS NULL\n ORDER BY id\n LIMIT 10000\n)\nUPDATE orders AS o\nSET status = 'pending'\nFROM batch\nWHERE o.id = batch.id;\n```\n\nCommit after each batch and repeat until no rows remain.\n\nA primary-key range loop is often more efficient:\n\n```sql\nUPDATE orders\nSET status = 'pending'\nWHERE id > :last_id\n AND id <= :next_id\n AND status IS NULL;\n```\n\nThe backfill takes row-level locks and generates substantial WAL and I/O, but it does not require an exclusive table lock.\n\n### 3. Add a `NOT VALID` check constraint\n\n```sql\nALTER TABLE orders\n ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\n```\n\n`NOT VALID` avoids scanning the existing table at this point. The constraint is enforced for new and modified rows.\n\n### 4. Validate the constraint\n\n```sql\nALTER TABLE orders\n VALIDATE CONSTRAINT orders_status_not_null;\n```\n\nThis scans the table, but validation uses a lock mode that allows normal reads and writes. It can still consume significant I/O and may conflict with some DDL.\n\nIf this fails, some existing row is still `NULL`; fix those rows and retry.\n\n### 5. Convert the column to a real PostgreSQL `NOT NULL` column\n\nOn PostgreSQL versions that can use the validated check constraint to prove non-nullability:\n\n```sql\nSET lock_timeout = '2s';\n\nALTER TABLE orders\n ALTER COLUMN status SET NOT NULL;\n```\n\nThen the check constraint is redundant and can be removed:\n\n```sql\nALTER TABLE orders\n DROP CONSTRAINT orders_status_not_null;\n```\n\n### Dangerous step\n\nThe dangerous step is:\n\n```sql\nALTER TABLE orders\n ALTER COLUMN status SET NOT NULL;\n```\n\n`ALTER TABLE` requires an `ACCESS EXCLUSIVE` lock for this operation. On modern PostgreSQL versions, the validated check constraint lets PostgreSQL avoid rescanning the table, so the lock should be brief—but it can still wait behind an old transaction and then block reads and writes once acquired. Use `lock_timeout`, monitor blockers, and retry during a low-traffic period.\n\nOn older PostgreSQL versions that cannot use the validated check constraint as proof, `SET NOT NULL` may scan the entire table while holding the exclusive lock. In that case, there is no way to guarantee a short exclusive lock for a true `NOT NULL` metadata change; upgrade PostgreSQL or schedule that final step for a maintenance window.", "latency_ms": 19122, "cost": 0.001333, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Technically accurate, correct modern PG approach (NOT VALID + validate + SET NOT NULL), clear steps, identifies dangerous step correctly. Slightly verbose for non-technical audience but thorough and correct." }, { "judge": "google/gemini-3.1-pro-preview", "score": 8, "violates": false, "reasoning": "Accurate and safe for older Postgres versions, but misses that since Postgres 11, adding a column with a constant default and NOT NULL is instant and doesn't require this complex backfill process." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate phased migration for large Postgres tables; correctly identifies brief ACCESS EXCLUSIVE risk on SET NOT NULL. Clear, complete, minor version caveats." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c9", "name": "Fix the code not the test", "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify(' Hello World! ')).toBe('hello-world');", "output": "```js\nexport const slugify = (s) =>\n s\n .toLowerCase()\n .trim()\n .replace(/[^a-z0-9]+/g, '-')\n .replace(/^-|-$/g, '');\n```\n\nThis trims surrounding whitespace, collapses whitespace and punctuation into a single hyphen, and removes leading or trailing hyphens.", "latency_ms": 2690, "cost": 0.000122, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Correct fix: handles trimming, punctuation removal, collapsing spaces, produces 'hello-world'. Concise, clear explanation, meets task constraint of fixing impl not test." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response correctly fixes the implementation to pass the provided test case. The code is concise, accurate, and accompanied by a clear explanation of the changes." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct fix passes the test; trims, normalizes non-alnum to hyphens, strips edges. Clear, complete, useful." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c10", "name": "Race condition", "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n if (cache) return cache;\n const r = await fetch('/config');\n cache = await r.json();\n return cache;\n}", "output": "The race is caused by the `await` yielding control before `cache` is assigned.\n\nFor example:\n\n1. Call A checks `cache` — it is `null`.\n2. Call A starts `fetch()` and pauses at `await`.\n3. Call B runs before A finishes and also checks `cache` — still `null`.\n4. Call B starts a second `fetch()`.\n5. Both requests eventually assign to `cache`; the one that finishes last determines the final value.\n\nThus, concurrent callers can cause duplicate requests and may even receive different results.\n\nCache the in-flight promise instead:\n\n```js\nlet configPromise = null;\n\nfunction getConfig() {\n if (!configPromise) {\n configPromise = fetch('/config')\n .then(response => {\n if (!response.ok) {\n throw new Error(`Config request failed: ${response.status}`);\n }\n return response.json();\n })\n .catch(error => {\n // Allow a later call to retry after a failure.\n configPromise = null;\n throw error;\n });\n }\n\n return configPromise;\n}\n```\n\nThe promise is assigned synchronously before the first `await` can yield, so subsequent callers receive the same in-flight promise instead of starting another request.", "latency_ms": 10829, "cost": 0.0007836, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate diagnosis, clear interleaving explanation, correct standard fix caching the promise synchronously, with added error retry logic. Clear and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly identifies the race condition, clearly explains the interleaving that causes it, and provides an excellent, robust fix by caching the promise. It is highly useful and concise." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correctly identifies race, details interleaving, and provides solid promise-caching fix with error handling." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c11", "name": "Retry with backoff", "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.", "output": "```ts\nexport async function retry(\n operation: () => Promise,\n maxAttempts = 5,\n baseDelayMs = 200\n): Promise {\n if (maxAttempts < 1) throw new RangeError(\"maxAttempts must be >= 1\");\n\n for (let attempt = 1; ; attempt++) {\n try {\n return await operation();\n } catch (error) {\n const e = error as { status?: number; response?: { status?: number } };\n const status = e.status ?? e.response?.status;\n const nonRetryable4xx =\n status !== undefined && status >= 400 && status < 500 && status !== 429;\n\n if (attempt >= maxAttempts || nonRetryable4xx) throw error;\n\n const backoff = baseDelayMs * 2 ** (attempt - 1);\n const jitteredDelay = Math.random() * backoff;\n await new Promise(resolve => setTimeout(resolve, jitteredDelay));\n }\n }\n}\n```\n\nOn final failure, the wrapper rethrows the last error. Non-429 4xx errors are rethrown immediately without retrying.", "latency_ms": 6379, "cost": 0.0004644, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct logic, meets constraints, within line limit, clearly states final failure behavior. Minor: no cap on max delay, but functionally solid and concise." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly fulfills all requirements. It provides a robust, well-typed TypeScript implementation under 30 lines, correctly handles the 4xx/429 logic, includes exponential backoff with jitter, and explicitly states the final fail" }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct TS retry with exp backoff+jitter, max attempts, skips non-429 4xx; under 30 lines; states final rethrow. Minor style nits only." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c12", "name": "Recursive type", "prompt": "Write a TypeScript type DeepPartial that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.", "output": "```ts\ntype DeepPartial =\n T extends (...args: any[]) => any ? T :\n T extends (infer U)[] ? DeepPartial[] :\n T extends readonly (infer U)[] ? readonly DeepPartial[] :\n T extends object ? { [K in keyof T]?: DeepPartial } :\n T;\n```\n\nArrays must be handled by recursively partializing their element type while preserving the array container, rather than treating array methods and indices as ordinary object properties.", "latency_ms": 5560, "cost": 0.0003805, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, valid TypeScript with array handling explained in one sentence as requested; minor redundancy since object branch would catch arrays too, but functionally fine and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a highly accurate and robust DeepPartial type, and correctly explains the array handling in exactly one sentence as requested." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Correct DeepPartial impl and accurate one-sentence array note; minor over-handling of functions/readonly is fine and clear." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c13", "name": "Bash one-liner", "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.", "output": "```sh\nfind . -type f -printf '%s\\t%p\\n' | sort -nr -k1,1 | head -n 10 | numfmt --field=1 --to=iec-i --suffix=B\n```\n\n- `find . -type f` — recursively finds regular files below the current directory.\n- `-printf '%s\\t%p\\n'` — prints each file’s size in bytes, a tab, then its pathname. Using a delimiter means spaces in filenames are preserved.\n- `sort -nr -k1,1` — sorts numerically, in reverse order, by the first field (file size).\n- `head -n 10` — keeps the ten largest files.\n- `numfmt --field=1 --to=iec-i --suffix=B` — formats the size field using binary units such as `KiB`, `MiB`, and `GiB`.\n\nThis uses GNU `find`, `sort`, `head`, and `numfmt`.", "latency_ms": 5610, "cost": 0.0004412, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, handles spaces via tab delimiter, clear explanation. Minor issue: filenames with tabs/newlines could break, but that's edge case. Solid, concise, GNU-specific noted." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a flawless, single-line pipeline that perfectly handles spaces, sorts correctly, and formats output as human-readable. The explanations are concise and accurate." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate GNU pipeline handling spaces via tab delimiter; clear brief explanations; minor note on portability only." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c14", "name": "Explain and cost", "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n for i in range(len(xs)):\n for j in range(i+1, len(xs)):\n if xs[i] == xs[j]:\n return True\n return False", "output": "`has_dup(xs)` checks whether the sequence contains any repeated value.\n\n- The outer loop selects each element.\n- The inner loop compares it with every later element.\n- It returns `True` as soon as a matching pair is found.\n- If no pair matches, it returns `False`.\n\nLet \\(n = \\text{len(xs)}\\).\n\n### Complexity of the original\n\n- **Worst-case time:** \\(O(n^2)\\) \n If there are no duplicates, it performs roughly \\(n(n-1)/2\\) comparisons.\n- **Best-case time:** \\(O(1)\\) \n If the first two elements are equal.\n- **Space:** \\(O(1)\\) extra space.\n\n### O(n) rewrite\n\nUse a set to remember values already seen:\n\n```python\ndef has_dup(xs):\n seen = set()\n\n for x in xs:\n if x in seen:\n return True\n seen.add(x)\n\n return False\n```\n\n### Complexity of the rewrite\n\n- **Expected time:** \\(O(n)\\), since set lookup and insertion are expected \\(O(1)\\).\n- **Space:** \\(O(n)\\) in the worst case, for the `seen` set.\n\nThis version requires the elements of `xs` to be hashable, such as integers, strings, and tuples containing hashable values.", "latency_ms": 4281, "cost": 0.0002713, "task_score": 9, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 9, "violates": false, "reasoning": "Accurate explanation, correct complexity analysis, correct O(n) rewrite with valid caveat about hashability. Clear and concise, fully meets task requirements." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly explains the function, accurately states its time and space complexity, and provides a correct O(n) rewrite using a set with its corresponding complexities." }, { "judge": "x-ai/grok-4.5", "score": 9, "violates": false, "reasoning": "Accurate explanation, complexities, and correct O(n) rewrite using a set. Minor note on hashability is helpful; fully meets all criteria." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c15", "name": "Regex with limits", "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.", "output": "```regex\n(?i)^[A-Z]{1,2}\\d[A-Z\\d]?[ ]?\\d[A-Z]{2}$\n```\n\n- `(?i)` — makes matching case-insensitive.\n- `^` — requires the postcode to start at the beginning of the string.\n- `[A-Z]{1,2}` — matches the one- or two-letter postcode area.\n- `\\d` — matches the first digit of the outward code.\n- `[A-Z\\d]?` — optionally matches the second outward-code character.\n- `[ ]?` — permits an optional single space.\n- `\\d[A-Z]{2}` — matches the inward code: one digit followed by two letters.\n- `$` — requires the postcode to end there.\n\nThis checks the general format only; it does not verify that the postcode is officially allocated or that all UK postcode letter restrictions are obeyed.\n\nFor example, it would incorrectly accept these invalid postcodes:\n\n- `AA1 1AA` — `AA` is not a valid UK postcode area.\n- `Q1 1AA` — `Q1` is not a valid UK postcode area.", "latency_ms": 11880, "cost": 0.0007713, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 7, "violates": false, "reasoning": "Regex reasonable, explanation clear, provides two invalid-accepted examples as required, and disclaims imperfection. Minor: didn't provide rejected-valid examples, just accepted-invalid, but task allowed either option." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response provides a solid regex, accurate one-line explanations, and correctly identifies two invalid postcodes the regex would accept. It follows all instructions perfectly without claiming the regex is flawless." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Solid regex and accurate part explanations; correctly notes imperfections with two invalid accepts. Minor: slightly technical for non-experts but fully meets task." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c16", "name": "Callback to async", "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n if (e) return cb(e);\n getOrders(user.id, (e2, orders) => {\n if (e2) return cb(e2);\n getTotals(orders, (e3, totals) => cb(e3, totals));\n });\n});", "output": "If the functions return promises:\n\n```js\nasync function getUserTotals(id) {\n const user = await getUser(id);\n const orders = await getOrders(user.id);\n return getTotals(orders);\n}\n```\n\nAny error from `getUser`, `getOrders`, or `getTotals` automatically propagates as a rejected promise:\n\n```js\ntry {\n const totals = await getUserTotals(id);\n console.log(totals);\n} catch (err) {\n console.error(err);\n}\n```\n\nIf the existing functions are callback-based, promisify them first:\n\n```js\nconst { promisify } = require(\"node:util\");\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id) {\n const user = await getUserAsync(id);\n const orders = await getOrdersAsync(user.id);\n return getTotalsAsync(orders);\n}\n```\n\nThere is intentionally no `catch` inside `getUserTotals`; errors are allowed to propagate to its caller.", "latency_ms": 4310, "cost": 0.0002911, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Correct, addresses both promise and callback cases, error propagation preserved. Minor: could integrate with cb() usage from original more directly, but overall accurate and clear." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly refactors the code to async/await, correctly handles error propagation without swallowing errors, and provides both promise-based and callback-promisified solutions. It is clear, concise, and highly useful." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Correct async/await refactor; errors propagate fully without swallowing; covers both promise and callback cases clearly." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c17", "name": "Diagnose from a trace", "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n at renderRows (/app/src/table.js:42:19)\n at Table (/app/src/table.js:12:5)\n at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.", "output": "The likely root cause is that the rows collection is `undefined` during the initial production render—usually because data is loaded asynchronously, an API response has a different shape in production, or the initial state/prop was not initialized.\n\nAt `table.js:42`, code is probably doing something like:\n\n```js\nrows.map(...)\n```\n\nThe first thing to check is the actual value and shape of `rows` immediately before that line in the production build, along with the production network/API response:\n\n```js\nconsole.log('rows:', rows);\n```\n\nThen ensure the initial value is an array and handle the loading state explicitly:\n\n```js\nconst [rows, setRows] = useState([]);\n\nreturn rows.map(/* ... */);\n```\n\nor, defensively:\n\n```js\n(rows ?? []).map(/* ... */)\n```\n\nAlso verify that production is returning the expected field, such as `response.rows`, rather than `response.data.rows` or no data at all.", "latency_ms": 5110, "cost": 0.000277, "task_score": 8, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, sensible root cause and diagnostic steps, addresses dev/prod discrepancy well; slightly verbose but clear and useful." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "Accurately identifies the root cause (undefined array/data on initial render) and provides the correct first step to check, tailored well for a non-technical professional managing a project." }, { "judge": "x-ai/grok-4.5", "score": 8, "violates": false, "reasoning": "Accurate root cause for prod-only first-paint undefined data; solid first check and fixes. Minor verbosity but clear and useful." } ], "machine_checks": null, "machine_violation": false, "machine_reasons": [] }, { "id": "c18", "name": "Idempotency", "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.", "output": "Use a client-generated `Idempotency-Key`, scoped to the merchant/account. Store it in a durable database table with:\n\n- unique `(account_id, idempotency_key)`\n- request-body hash\n- payment state (`processing`, `succeeded`, `failed`)\n- the final HTTP status and response body\n\nThe first request atomically inserts the row. Only the request that successfully inserts it may initiate the payment, using the same idempotency key with the payment provider. A concurrent request that hits the unique constraint reads the existing row and waits while it is `processing` (or returns `202`), but never charges independently. Once completed, it returns the stored status and body.\n\nA later retry with the same key and identical payload receives exactly the stored response—typically the original `201`/`200` and payment ID. Reusing the key with a different payload returns `409 Conflict`.\n\nThe remaining failure mode is the **external side-effect ambiguity window**: if the provider charges successfully but crashes or loses the idempotency record before the result is persisted, the client may not know whether payment succeeded. Provider-side durable idempotency (and reconciliation) is required to prevent that from becoming a duplicate charge.", "latency_ms": 6132, "cost": 0.0003973, "task_score": 10, "judges": [ { "judge": "anthropic/claude-sonnet-5", "score": 8, "violates": false, "reasoning": "Accurate, well-structured, covers key/storage/concurrency/response, names a real failure mode. Slightly over ideal concision but likely within 200 words; solid technical design." }, { "judge": "google/gemini-3.1-pro-preview", "score": 10, "violates": false, "reasoning": "The response perfectly addresses all requirements within the 200-word limit. It clearly explains the idempotency key, storage, concurrent handling, retry response, and accurately identifies the remaining failure mode." }, { "judge": "x-ai/grok-4.5", "score": 10, "violates": false, "reasoning": "Covers key, storage, concurrency, second response, and residual failure mode accurately and completely within the 200-word limit." } ], "machine_checks": { "max_words": 200 }, "machine_violation": false, "machine_reasons": [] } ], "recapped_at": "2026-08-16T08:16:35.334Z", "recap_note": "Machine constraint caps applied retroactively under protocol judge-2026-08b; judge verdicts unchanged, task scores capped where a deterministic check failed." }