{
 "entity": "cursor",
 "entity_type": "tool",
 "category": "Coding",
 "suite": "suite-2026-08-skills/coding",
 "model": "anthropic/claude-sonnet-5",
 "judges": [
  "google/gemini-3.1-pro-preview",
  "openai/gpt-5.6-terra",
  "x-ai/grok-4.5"
 ],
 "judge_protocol": "judge-2026-08",
 "anchors": "solo-2026-08-a",
 "score": 87,
 "caps": 3,
 "cost_usd": 0.3881,
 "started_at": "2026-08-10T10:57:04.418Z",
 "finished_at": "2026-08-10T10:58:15.758Z",
 "tasks": [
  {
   "id": "c1",
   "name": "Duration parser",
   "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.",
   "output": "```python\nimport re\n\ndef parse_duration(s):\n    \"\"\"\n    Convert a duration string like '1h30m', '45s', '2h', '90m', '1h2m3s'\n    into the total number of seconds (int).\n\n    Supported units, in this order only: h (hours), m (minutes), s (seconds).\n    Each unit may appear at most once, and at least one unit must be present.\n    Raises ValueError on malformed input.\n    \"\"\"\n    if not isinstance(s, str) or s == \"\":\n        raise ValueError(f\"Invalid duration string: {s!r}\")\n\n    pattern = r'^(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?$'\n    match = re.match(pattern, s)\n\n    if not match or not any(match.groups()):\n        raise ValueError(f\"Invalid duration string: {s!r}\")\n\n    hours, minutes, seconds = (int(g) if g else 0 for g in match.groups())\n\n    return hours * 3600 + minutes * 60 + seconds\n\n\n# ---- Tests ----\n\n# Test 1: combination of hours and minutes\nassert parse_duration('1h30m') == 5400  # 1*3600 + 30*60\n\n# Test 2: single unit values\nassert parse_duration('45s') == 45\nassert parse_duration('2h') == 7200\nassert parse_duration('90m') == 5400\nassert parse_duration('1h2m3s') == 3723  # 3600 + 120 + 3\n\n# Test 3: malformed input should raise ValueError\ntry:\n    parse_duration('1h30')  # missing unit for '30'\n    assert False, \"Expected ValueError for malformed input\"\nexcept ValueError:\n    pass\n```\n\n**Notes:**\n- The regex enforces unit order `h → m → s` and disallows duplicate or out-of-order units (e.g., `'30m1h'` or `'1h1h'` will fail).\n- Empty string, non-numeric input, or missing units (like `'1h30'`) raise `ValueError`.\n- Each part is optional individually, but at least one must be present (`''` alone is invalid).",
   "latency_ms": 10309,
   "cost": 0.008536,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides a correct Python function, handles all specified formats, raises ValueError on malformed input, and includes exactly three well-structured assert-based tests as requested."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "Correctly parses intended formats, rejects most malformed inputs, and includes malformed-input assertion coverage. Minor issue: regex `$` can accept a trailing newline (for example, `\"1h\\n\"`), conflicting with rejecting anything malformed. "
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct robust parser with regex; raises ValueError properly. Three grouped assert tests incl. malformed. Minor excess asserts in test 2; otherwise excellent."
    }
   ]
  },
  {
   "id": "c2",
   "name": "Find the bug",
   "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n  let sum = 0, n = 0;\n  for (const x of xs) {\n    if (x > 0) sum += x;\n    n++;\n  }\n  return sum / n;\n}",
   "output": "## Bug\n\nThe counter `n` is incremented for **every** element, not just the positive ones, so the sum of positive numbers gets divided by the total count of all numbers (including negatives and zeros) instead of just the count of positive numbers.\n\n## Corrected function\n\n```javascript\nfunction avgPositive(xs) {\n  let sum = 0, n = 0;\n  for (const x of xs) {\n    if (x > 0) {\n      sum += x;\n      n++;\n    }\n  }\n  return sum / n;\n}\n```",
   "latency_ms": 3901,
   "cost": 0.001906,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly identifies the bug, explains it in exactly one sentence, and provides the correctly fixed function. It is accurate, concise, and follows all instructions."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly identifies that n counts all elements rather than only positives, gives a one-sentence explanation, and supplies the properly corrected function. Clear, concise, and fully follows the requested format."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 10,
     "violates": false,
     "reasoning": "Perfect bug ID, one-sentence explanation, and correct fix. Matches all task requirements exactly with no flaws."
    }
   ]
  },
  {
   "id": "c3",
   "name": "SQL without window functions",
   "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.",
   "output": "# SQL Query\n\n```sql\nSELECT\n    u.email,\n    COUNT(o.id) AS order_count\nFROM users u\nJOIN orders o\n    ON o.user_id = u.id\n    AND o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY\n    u.id,\n    u.email\nHAVING\n    COUNT(o.id) > 3\nORDER BY\n    order_count DESC;\n```\n\n## Explanation\n\n- **JOIN condition includes the date filter**: Placing `o.created_at >= NOW() - INTERVAL '30 days'` in the `JOIN` clause (rather than a `WHERE` clause) ensures we're using an inner join semantics correctly while filtering orders *before* aggregation — functionally equivalent to `WHERE` here since it's an inner join, but keeps the intent clear (filter orders, not users).\n\n- **GROUP BY u.id, u.email**: Grouping by `u.id` (the primary key) is best practice even though `email` is included, since it avoids issues if emails aren't guaranteed unique per row-level assumptions, and it's cheaper/more precise for grouping.\n\n- **HAVING COUNT(o.id) > 3**: Filters groups *after* aggregation to keep only users with more than 3 orders in the window.\n\n- **ORDER BY order_count DESC**: Returns highest order counts first, per the \"most orders first\" requirement.\n\n- **No window functions used** — just a standard `JOIN` + `GROUP BY` + `HAVING`, compliant with the constraint.\n\n### Note on `NOW()`\nIf `created_at` is stored as `timestamptz`, `NOW() - INTERVAL '30 days'` works directly. If it's a plain `timestamp` (no time zone) representing local time, you may want `CURRENT_TIMESTAMP` or explicit casting — but `NOW()` is standard Postgres and works in the vast majority of cases.",
   "latency_ms": 8902,
   "cost": 0.006634,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides a perfectly accurate Postgres SQL query that meets all requirements, avoids window functions, and includes a clear, concise explanation suitable for the target audience."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "SQL correctly filters last-30-day orders, aggregates per user, applies HAVING > 3, and sorts descending without window functions. Minor explanatory claims about grouping cost and timestamp handling are imprecise but do not affect the query."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct Postgres SQL meeting all requirements without windows. Clear, well-explained, minor verbosity in notes."
    }
   ]
  },
  {
   "id": "c4",
   "name": "Infinite useEffect",
   "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n  fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);",
   "output": "## Why it loops forever\n\nThe `useEffect` has `items` in its dependency array. The sequence of events:\n\n1. Component mounts, `items` is `[]`.\n2. Effect runs (since it's the first render), fetches data, calls `setItems(newData)`.\n3. `setItems` updates state → component re-renders.\n4. Because `items` changed (it's a **new array reference**, even if contents were somehow the same), React sees the dependency changed and runs the effect **again**.\n5. That triggers another fetch → another `setItems` → another change to `items` → effect runs again.\n6. This repeats indefinitely, since `setItems` always produces a new array reference, which always differs from the previous `items` value in the dependency check (`Object.is` comparison).\n\nEssentially: **the effect depends on the very state it sets**, so every update to that state re-triggers the effect that updates it — an infinite fetch loop.\n\n## Fixed version\n\nThe effect should only run once (on mount), since it's just fetching initial data and doesn't actually need to react to `items` changing:\n\n```jsx\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n  fetch('/api/items')\n    .then(r => r.json())\n    .then(setItems);\n}, []); // empty dependency array — runs once on mount\n```\n\nIf you *do* need the effect to re-run based on some other value (e.g., a search query or page number), depend on that value instead of `items`:\n\n```jsx\nconst [items, setItems] = useState([]);\nconst [query, setQuery] = useState('');\n\nuseEffect(() => {\n  fetch(`/api/items?q=${query}`)\n    .then(r => r.json())\n    .then(setItems);\n}, [query]); // re-fetch only when query changes, not when items changes\n```\n\nThe key rule: **never put state in the dependency array if the effect's own job is to set that same state**, unless you're intentionally gating on a different trigger.",
   "latency_ms": 8836,
   "cost": 0.006404,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly explains the infinite loop caused by updating a state variable listed in the dependency array. It provides the correct fix (empty dependency array) and a helpful alternative. Flawless, clear, and highly useful."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "Correctly identifies the self-triggering dependency cycle and provides the standard empty-dependency fix. Minor overstatement: setItems does not inherently always produce a new array, and state can sometimes appear in dependencies safely wi"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Precise correct explanation of reference equality loop; clean fixed code with empty deps plus useful variant. Minor extra length but highly clear and complete."
    }
   ]
  },
  {
   "id": "c5",
   "name": "Typed debounce",
   "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.",
   "output": "# TypeScript Debounce with Cancel Support\n\n```typescript\ntype DebouncedFunction<Args extends unknown[]> = ((...args: Args) => void) & {\n  cancel: () => void;\n};\n\nfunction debounce<Args extends unknown[]>(\n  fn: (...args: Args) => unknown,\n  delay: number\n): DebouncedFunction<Args> {\n  let timeoutId: ReturnType<typeof setTimeout> | undefined;\n\n  const debounced = (...args: Args): void => {\n    if (timeoutId !== undefined) {\n      clearTimeout(timeoutId);\n    }\n    timeoutId = setTimeout(() => {\n      timeoutId = undefined;\n      fn(...args);\n    }, delay);\n  };\n\n  debounced.cancel = (): void => {\n    if (timeoutId !== undefined) {\n      clearTimeout(timeoutId);\n      timeoutId = undefined;\n    }\n  };\n\n  return debounced;\n}\n```\n\n## Usage Example\n\n```typescript\nconst search = debounce((query: string, page: number) => {\n  console.log(`Searching \"${query}\" on page ${page}`);\n}, 300);\n\nsearch(\"typescript\", 1); // scheduled\nsearch(\"typescript\", 2); // cancels previous, reschedules\nsearch.cancel();         // cancels pending call\n```\n\n- `Args extends unknown[]` captures the exact parameter tuple of `fn`, so calls to the debounced function are type-checked against the original signature (correct arity, correct types, correct optional/rest params).\n- The intersection type `... & { cancel: () => void }` lets us attach `cancel` while keeping the function's call signature intact.\n- Using `unknown[]` instead of `any[]` avoids disabling type checking on the arguments.\n\n## Why the Naive Signature Loses Type Information\n\nA naive signature like `function debounce(fn: (...args: any[]) => any, delay: number): (...args: any[]) => void` erases the specific parameter types into `any[]`, so TypeScript can no longer check that calls to the debounced function supply the correct number or types of arguments matching the original `fn`.",
   "latency_ms": 8345,
   "cost": 0.007186,
   "task_score": 5,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 5,
     "violates": true,
     "reasoning": "The response provides a correct TypeScript debounce function with a cancel method and preserves types. However, it violates the negative constraint by using the forbidden word 'any' in the explanation."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 5,
     "violates": true,
     "reasoning": "The implementation correctly preserves parameter tuples, provides cancel(), and avoids any in its implementation. However, the response literally uses the forbidden term and type `any` in the naive-signature explanation."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Excellent typed debounce with cancel; preserves args via tuple generic; no any; one-sentence naive explanation is accurate and clear."
    }
   ]
  },
  {
   "id": "c6",
   "name": "Leftmost binary search",
   "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.",
   "output": "# Binary Search — First Occurrence\n\n## Approach\nStandard binary search, but when we find a match, we **don't stop**. We record the index and keep searching the **left half** to see if an earlier occurrence exists.\n\n## Code (Python)\n\n```python\ndef find_first_occurrence(arr, target):\n    lo, hi = 0, len(arr) - 1\n    result = -1\n\n    while lo <= hi:\n        mid = lo + (hi - lo) // 2\n\n        if arr[mid] == target:\n            result = mid       # candidate answer found\n            hi = mid - 1       # keep searching left for an earlier one\n        elif arr[mid] < target:\n            lo = mid + 1\n        else:\n            hi = mid - 1\n\n    return result\n```\n\n## Complexity\n- **Time:** O(log n) — still halves the search space each iteration.\n- **Space:** O(1) — iterative, no extra structures.\n\n## Distinguishing Test Case\n\n```python\narr = [1, 2, 4, 4, 4, 4, 5, 6, 8]\ntarget = 4\n\nprint(find_first_occurrence(arr, target))  # Expected output: 2\n```\n\n**Why this distinguishes it from ordinary binary search:**\n\nAn ordinary binary search would compute `mid = 4` on the first iteration (`lo=0, hi=8`), find `arr[4] == 4`, and **immediately return index 4** — a correct-but-arbitrary match, since it stops at the first match it stumbles upon during the halving process (not necessarily the leftmost one).\n\nThe \"first occurrence\" version, upon finding `arr[4] == 4`, sets `result = 4` but continues searching `[lo=0, hi=3]`. It then finds `arr[2] == 4`, updates `result = 2`, and continues searching `[lo=0, hi=1]`, which contains no more 4's, so the loop ends and it correctly returns **2**.\n\nThis test case is essential because with an array containing only one occurrence of the target, both algorithms behave identically — the divergence only shows up when there are **multiple duplicates and the middle element found during the search is not the leftmost one**.",
   "latency_ms": 8648,
   "cost": 0.007108,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly fulfills all requirements. It provides correct code, accurately states the time and space complexity, and includes a clear test case with an excellent explanation of why it distinguishes the algorithm from standard bi"
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correct first-occurrence binary search, accurate O(log n) time and O(1) space, and a valid duplicate-containing test case that demonstrably differs from ordinary early-return binary search. Clear and complete."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 10,
     "violates": false,
     "reasoning": "Correct first-occurrence binary search, accurate O(log n) complexity, clear distinguishing test with explanation. Flawless and useful."
    }
   ]
  },
  {
   "id": "c7",
   "name": "Security review",
   "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n  const p = req.query.name;\n  db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n    res.sendFile(__dirname + '/uploads/' + p);\n  });\n});",
   "output": "# Security Issues (Most Severe First)\n\n## 1. SQL Injection (Critical)\n```js\ndb.query(`SELECT * FROM files WHERE name = '${p}'`, ...)\n```\nThe `name` parameter is directly interpolated into the SQL string, allowing attackers to inject arbitrary SQL (e.g., `' OR '1'='1`, or stacked queries depending on the driver).\n\n**Fix:** Use parameterized queries / prepared statements.\n```js\ndb.query('SELECT * FROM files WHERE name = ?', [p], (e, rows) => { ... });\n```\n\n## 2. Path Traversal / Arbitrary File Read (Critical)\n```js\nres.sendFile(__dirname + '/uploads/' + p);\n```\n`p` is user-controlled and concatenated directly into a filesystem path. An attacker can supply `../../etc/passwd` or an absolute path to read files outside the `uploads` directory. `sendFile` does not sanitize this for you when you build the path manually.\n\n**Fix:** Sanitize/validate the filename and use `path.join` + `path.resolve`, then verify the resolved path stays within the intended directory. Also pass the `root` option to `sendFile`, which enforces containment.\n```js\nconst path = require('path');\n\napp.get('/file', (req, res) => {\n  const name = req.query.name;\n\n  // Reject anything that isn't a simple filename\n  if (!/^[a-zA-Z0-9._-]+$/.test(name)) {\n    return res.status(400).send('Invalid filename');\n  }\n\n  db.query('SELECT * FROM files WHERE name = ?', [name], (err, rows) => {\n    if (err) return res.status(500).end();\n    if (!rows.length) return res.status(404).end();\n\n    res.sendFile(name, { root: path.join(__dirname, 'uploads') }, (sendErr) => {\n      if (sendErr) res.status(404).end();\n    });\n  });\n});\n```\n\n## 3. No Authorization Check (High)\nThe handler returns *any* file whose name is requested, with no check that the requesting user is allowed to access it (e.g., ownership, ACL). Even after fixing injection/traversal, this endpoint is an open file-disclosure oracle for anything in `uploads/`.\n\n**Fix:** Verify the authenticated user has permission to access the specific file record returned from the DB (e.g., check `rows[0].owner_id === req.user.id`) before calling `sendFile`.\n\n## 4. Unhandled/Ignored DB Error (Medium)\nThe callback ignores `e` (the DB error) entirely and still calls `sendFile` regardless of whether the query succeeded or the file exists in the DB. This can leak files not tracked in the database, cause stack traces to leak in some setups, or crash the process on unhandled errors depending on the driver.\n\n**Fix:** Check `err`, check `rows.length`, and only serve the file if a matching, authorized record was found (see fixed code above).\n\n## 5. No Error Handling on `sendFile` (Low/Medium)\n`res.sendFile` can fail (file missing, permission denied) and, without an error callback, Express's default error handling may leak stack traces or internal paths depending on environment configuration.\n\n**Fix:** Always pass an error callback to `sendFile` and respond with a generic error (as shown above), and ensure `NODE_ENV=production` to suppress stack traces globally as defense-in-depth.\n\n---\n\n### Summary Fix\nThe corrected handler should: parameterize SQL, whitelist/validate the filename format, constrain file resolution to the `uploads` root via `sendFile`'s `root` option, check DB result existence and authorization, and handle all error paths explicitly.",
   "latency_ms": 14884,
   "cost": 0.0126,
   "task_score": 8,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly identifies all security issues, orders them by severity, provides clear explanations and accurate fixes, and is highly useful for a professional."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "Correctly identifies SQL injection, traversal, missing authorization, and error handling, with workable fixes. Minor issues: DB-error claims overstate crash risk, and the proposed sendFile callback treats all errors as 404 without checking "
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 8,
     "violates": false,
     "reasoning": "Accurate severity-ordered issues with solid fixes; minor polish gaps only. Clear and useful for a busy professional."
    }
   ]
  },
  {
   "id": "c8",
   "name": "Safe migration",
   "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.",
   "output": "# Migration: Adding NOT NULL column with default to a 40M row table\n\nThe naive approach (`ADD COLUMN status TEXT NOT NULL DEFAULT 'pending'` in one statement) is actually fast on PG 11+, since defaults are stored as metadata, not rewritten immediately. **But** the real danger is elsewhere: the `NOT NULL` constraint validation and any subsequent full table rewrite/backfill. Here's the safe sequence:\n\n## Step 1: Add the column as nullable, no default rewrite needed\n```sql\nALTER TABLE orders ADD COLUMN status TEXT;\n```\nThis is instant (metadata-only change) on PG 11+, takes a brief `ACCESS EXCLUSIVE` lock but only for milliseconds since no rewrite occurs.\n\n## Step 2: Set the default for new rows\n```sql\nALTER TABLE orders ALTER COLUMN status SET DEFAULT 'pending';\n```\nAlso instant — only affects future inserts, no table rewrite.\n\n## Step 3: Backfill existing rows in small batches ⚠️ **THIS IS THE DANGEROUS STEP**\n```sql\nDO $$\nDECLARE\n  rows_updated INT;\nBEGIN\n  LOOP\n    UPDATE orders\n    SET status = 'pending'\n    WHERE status IS NULL\n      AND ctid IN (\n        SELECT ctid FROM orders\n        WHERE status IS NULL\n        LIMIT 5000\n      );\n    GET DIAGNOSTICS rows_updated = ROW_COUNT;\n    EXIT WHEN rows_updated = 0;\n    COMMIT; -- if run via a procedure in psql / non-transactional session\n    PERFORM pg_sleep(0.05); -- throttle to reduce replication lag / lock contention\n  END LOOP;\nEND $$;\n```\n**Why this is dangerous:** Updating 40M rows in a single `UPDATE orders SET status = 'pending' WHERE status IS NULL` would:\n- Rewrite every row (since Postgres UPDATEs create new row versions)\n- Hold long-running locks and generate a massive amount of WAL/bloat\n- Bloat the table and its indexes, requiring a big autovacuum/VACUUM FULL afterward\n- Risk replication lag or filling disk with WAL on busy systems\n- Potentially deadlock with concurrent writes waiting on locks for rows in-flight\n\nDoing it in **small batches with commits between them** avoids holding one giant transaction/lock, lets autovacuum keep up, and allows you to pause/throttle if replication lag or load spikes. This is the step that requires the most care and monitoring in production.\n\n## Step 4: Add the NOT NULL constraint safely (two-phase)\nDon't just run `ALTER TABLE orders ALTER COLUMN status SET NOT NULL;` directly — on older Postgres versions this does a full table scan while holding `ACCESS EXCLUSIVE`. Instead:\n\n```sql\n-- Add as a CHECK constraint first, NOT VALID (instant, no scan)\nALTER TABLE orders\n  ADD CONSTRAINT orders_status_not_null CHECK (status IS NOT NULL) NOT VALID;\n```\n\n```sql\n-- Validate separately — this scans the table but only takes SHARE UPDATE EXCLUSIVE,\n-- allowing concurrent reads/writes\nALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;\n```\n\n## Step 5 (PG 12+): Convert to a real NOT NULL without re-scanning\n```sql\nALTER TABLE orders ALTER COLUMN status SET NOT NULL;\n```\nOn PG 12+, Postgres is smart enough to skip the redundant validation scan **if it can see a valid CHECK constraint proving NOT NULL already holds** — so this becomes metadata-only. Then drop the now-redundant check constraint:\n\n```sql\nALTER TABLE orders DROP CONSTRAINT orders_status_not_null;\n```\n\n## Summary of lock behavior\n| Step | Lock type | Duration | Risk |\n|---|---|---|---|\n| 1. Add nullable column | ACCESS EXCLUSIVE | milliseconds | low |\n| 2. Set default | ACCESS EXCLUSIVE | milliseconds | low |\n| **3. Backfill in batches** | **ROW EXCLUSIVE per batch** | **long overall, but chunked** | **HIGH — bloat, WAL, lock contention if not throttled/batched** |\n| 4. Add NOT VALID constraint | ACCESS EXCLUSIVE | milliseconds | low |\n| 5. Validate constraint | SHARE UPDATE EXCLUSIVE | long, but non-blocking for reads/writes | moderate |\n| 6. Set NOT NULL (PG12+) | ACCESS EXCLUSIVE | milliseconds (skips scan) | low |\n\n**The batched backfill (Step 3) is the dangerous part of this whole migration** — everything else is either instant metadata changes or non-blocking validations. Get the batch size, throttling, and monitoring (replication lag, lock waits, table bloat) right there.",
   "latency_ms": 18943,
   "cost": 0.018074,
   "task_score": 5,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 5,
     "violates": false,
     "reasoning": "The response incorrectly claims the PG11+ fast path still requires a backfill/validation. Adding a NOT NULL column with a default in PG11+ is an instant metadata operation. The provided steps are only necessary for PG10 and older."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 4,
     "violates": false,
     "reasoning": "The batched backfill code is invalid: COMMIT cannot run inside a DO block. More importantly, on PG 11+ the requested single ADD COLUMN ... NOT NULL DEFAULT is metadata-only, so this needless 40M-row backfill creates the main risk."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 8,
     "violates": false,
     "reasoning": "Accurate modern PG steps; correctly flags batched backfill as dangerous due to bloat/WAL. Minor: COMMIT in DO is invalid without PROCEDURE."
    }
   ]
  },
  {
   "id": "c9",
   "name": "Fix the code not the test",
   "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify('  Hello   World!  ')).toBe('hello-world');",
   "output": "```js\nexport const slugify = (s) =>\n  s\n    .trim()\n    .toLowerCase()\n    .replace(/[^a-z0-9\\s-]/g, '') // strip punctuation like \"!\"\n    .replace(/\\s+/g, '-');        // collapse & convert whitespace to single \"-\"\n```\n\n**Why the original failed:**  \n`slugify('  Hello   World!  ')` with the old code only lower-cased and replaced *single* spaces with `-`, giving `\"--hello---world!--\"` (leading/trailing dashes, multiple dashes for multiple spaces, and the `!` left in) instead of `\"hello-world\"`.\n\nThe fix:\n1. `trim()` removes leading/trailing whitespace.\n2. `replace(/[^a-z0-9\\s-]/g, '')` strips punctuation (e.g., `!`).\n3. `replace(/\\s+/g, '-')` collapses any run of whitespace into a single hyphen.\n\nThis yields `\"hello-world\"` as expected.",
   "latency_ms": 7432,
   "cost": 0.005758,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The implementation correctly fixes the function to pass the test by handling leading/trailing spaces, multiple spaces, and punctuation. The explanation is clear, concise, and highly useful."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly fixes the implementation to trim surrounding whitespace, remove punctuation, and collapse whitespace into hyphens. It satisfies the given test and clearly explains the original failure without altering the test."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct fix passes the test; clear explanation of changes. Minor verbosity in comments for non-tech users, else excellent."
    }
   ]
  },
  {
   "id": "c10",
   "name": "Race condition",
   "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n  if (cache) return cache;\n  const r = await fetch('/config');\n  cache = await r.json();\n  return cache;\n}",
   "output": "# Race Condition Analysis\n\n## The Bug\n\nIf `getConfig()` is called multiple times concurrently (before the first call resolves), **every call will pass the `if (cache)` check** (since `cache` is still `null`) and **trigger its own separate `fetch('/config')`** request.\n\n## The Interleaving\n\n```\nCall A: if (cache) → false (cache is null)\nCall B: if (cache) → false (cache is null)     // A hasn't set cache yet\nCall A: await fetch('/config')  ┐\nCall B: await fetch('/config')  ┘  // duplicate network request\nCall A: cache = await r.json()\nCall B: cache = await r.json()     // overwrites A's result\n```\n\nBecause `await fetch(...)` yields control back to the event loop, any other call to `getConfig()` made in that window sees a stale `cache` value (`null`) and starts its own fetch. This causes:\n- **Redundant network requests** (wasted bandwidth, server load)\n- **Wasted work** — later responses overwrite earlier ones\n- Potential inconsistency if responses differ (e.g., config changes between requests)\n\n## The Fix\n\nCache the **in-flight promise**, not just the resolved value. This way, concurrent callers await the *same* promise instead of starting new fetches.\n\n```js\nlet cachePromise = null;\n\nasync function getConfig() {\n  if (!cachePromise) {\n    cachePromise = fetch('/config').then(r => r.json());\n  }\n  return cachePromise;\n}\n```\n\n### Why this works\n- The synchronous check `if (!cachePromise)` and the assignment happen **without any `await` in between**, so there's no window for another call to interleave and see a stale `null`.\n- All concurrent callers get the **same promise reference** and thus the same resolved data once it completes.\n- Only **one** `fetch` call is ever made.\n\n### Optional: handle failures\nIf the fetch fails, you may want to reset `cachePromise` so future calls can retry:\n\n```js\nlet cachePromise = null;\n\nasync function getConfig() {\n  if (!cachePromise) {\n    cachePromise = fetch('/config')\n      .then(r => r.json())\n      .catch(err => {\n        cachePromise = null; // allow retry on next call\n        throw err;\n      });\n  }\n  return cachePromise;\n}\n```",
   "latency_ms": 9496,
   "cost": 0.007802,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response accurately identifies the race condition, clearly explains the exact interleaving that causes it, and provides a robust, standard fix by caching the promise. It is highly useful and concise."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 9,
     "violates": false,
     "reasoning": "Correctly identifies the check-then-await race, gives a valid interleaving, and fixes it by memoizing the in-flight promise. The optional retry handling is useful. Minor technical imprecision: an async function may not return the identical "
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 10,
     "violates": false,
     "reasoning": "Flawless identification, interleaving explanation, and correct promise-caching fix; clear and complete."
    }
   ]
  },
  {
   "id": "c11",
   "name": "Retry with backoff",
   "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.",
   "output": "```typescript\nasync function withRetry<T>(\n  fn: () => Promise<T>,\n  maxAttempts = 5,\n  baseDelayMs = 200,\n  maxDelayMs = 30_000\n): Promise<T> {\n  for (let attempt = 1; ; attempt++) {\n    try {\n      return await fn();\n    } catch (err: any) {\n      const status = err?.status ?? err?.response?.status;\n      const isNonRetryableClientError =\n        status >= 400 && status < 500 && status !== 429;\n\n      if (isNonRetryableClientError || attempt >= maxAttempts) {\n        throw err; // give up: rethrow original error\n      }\n\n      const backoff = Math.min(baseDelayMs * 2 ** (attempt - 1), maxDelayMs);\n      const jitter = Math.random() * backoff; // full jitter\n      await new Promise((r) => setTimeout(r, jitter));\n    }\n  }\n}\n```\n\n**Behavior on final failure:** once `attempt` reaches `maxAttempts` (or a non-429 4xx error occurs at any point), the function does **not** swallow or wrap the error — it simply `throw err`s the *last* error it received from `fn()`. There is no fallback value; the caller must handle the rejection (e.g., via `try/catch` or `.catch()`), just as if retry logic weren't present at all.\n\n**Assumption:** `fn` throws on failure, and the thrown error exposes an HTTP status via `err.status` or `err.response.status` (matches common HTTP client conventions like `axios` or a custom fetch wrapper). Adjust the status extraction if your error shape differs.",
   "latency_ms": 17064,
   "cost": 0.016966,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly fulfills all requirements, including the constraints. The code is correct, implements exponential backoff with jitter, handles the 4xx logic accurately, and clearly explains the final failure behavior within the 30-li"
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correct async retry loop with capped exponential backoff and full jitter. It retries 429 and non-4xx failures, immediately stops on other 4xx errors, respects the attempt limit, and clearly states that the original final error is rethrown."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct TS impl with exp backoff+jitter, max attempts, 4xx/429 logic; states final failure clearly; under 30 lines; minor style polish only."
    }
   ]
  },
  {
   "id": "c12",
   "name": "Recursive type",
   "prompt": "Write a TypeScript type DeepPartial<T> that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.",
   "output": "## DeepPartial<T>\n\n```typescript\ntype DeepPartial<T> = T extends (infer U)[]\n  ? DeepPartial<U>[]\n  : T extends readonly (infer U)[]\n  ? readonly DeepPartial<U>[]\n  : T extends Function\n  ? T\n  : T extends object\n  ? { [K in keyof T]?: DeepPartial<T[K]> }\n  : T;\n```\n\n### Breakdown\n\n```typescript\ntype DeepPartial<T> =\n  T extends (infer U)[]                    // 1. Check for mutable arrays\n    ? DeepPartial<U>[]\n  : T extends readonly (infer U)[]         // 2. Check for readonly arrays/tuples\n    ? readonly DeepPartial<U>[]\n  : T extends Function                     // 3. Leave functions untouched\n    ? T\n  : T extends object                       // 4. Recurse into plain objects\n    ? { [K in keyof T]?: DeepPartial<T[K]> }\n    : T;                                   // 5. Primitives stay as-is\n```\n\n### Example usage\n\n```typescript\ninterface User {\n  id: number;\n  profile: {\n    name: string;\n    tags: string[];\n  };\n  login: () => void;\n}\n\ntype PartialUser = DeepPartial<User>;\n\n// Equivalent to:\n// {\n//   id?: number;\n//   profile?: {\n//     name?: string;\n//     tags?: string[];\n//   };\n//   login?: () => void;\n// }\n\nconst u: PartialUser = {\n  profile: { tags: ['admin'] } // name and id can be omitted safely\n};\n```\n\n### The one-sentence explanation\n\n**Arrays must be matched and recursed into via their element type (`T extends (infer U)[] ? DeepPartial<U>[] : ...`) *before* falling through to the generic object-mapping branch, because mapping over an array with `{ [K in keyof T]?: ... }` would incorrectly try to make its numeric indices and built-in methods (like `push`, `length`) optional, corrupting the array's type instead of preserving it as an array of partial elements.**",
   "latency_ms": 8123,
   "cost": 0.007126,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides a highly accurate and robust DeepPartial type and perfectly follows the constraint to explain the array handling in exactly one sentence."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "The implementation correctly recurses through objects and array elements while preserving functions and readonly-array status. The required explanation is one accurate sentence. Minor limitation: tuple shape is widened to arrays, and non-pl"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct DeepPartial with proper array handling; clear one-sentence explanation; highly useful and polished with extras that aid understanding."
    }
   ]
  },
  {
   "id": "c13",
   "name": "Bash one-liner",
   "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.",
   "output": "# Command\n\n```bash\nfind . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10\n```\n\n## Breakdown\n\n| Part | Purpose |\n|---|---|\n| `find . -type f -print0` | Recursively finds all regular files starting from the current directory. `-print0` separates results with a **null byte** (`\\0`) instead of a newline/space, which safely handles filenames containing spaces, tabs, or newlines. |\n| `xargs -0 du -h` | Reads the null-separated list (`-0` matches `find`'s `-print0`) and passes filenames as arguments to `du -h`, which prints the disk usage of each file in **human-readable** form (e.g., `4.0K`, `1.2M`, `3.5G`). |\n| `sort -rh` | Sorts the output. `-h` tells `sort` to understand human-readable suffixes (K, M, G, ...) correctly, and `-r` reverses the order so the **largest** files come first. |\n| `head -n 10` | Keeps only the **top 10** lines, i.e., the ten largest files. |\n\n## Why not simpler alternatives?\n\n- `find . -type f | xargs du -h | sort -rh | head` — **breaks** on filenames with spaces, since `xargs` (without `-0`) splits on whitespace.\n- `du -ah . | sort -rh | head` — works, but includes directories too unless you filter; using `find -type f` restricts to files only, which is usually the intent for \"largest files.\"\n\n## Notes / caveats\n\n- This assumes GNU `du`, `sort`, `find`, and `xargs` (standard on Linux). On macOS/BSD, `sort -h` is also supported in modern versions, but very old BSD tools might differ slightly.\n- If you have extremely many files, `xargs` may invoke `du` multiple times in batches — this is safe and handled automatically.\n- An alternative using `-exec` avoids `xargs` entirely:\n\n```bash\nfind . -type f -exec du -h {} + | sort -rh | head -n 10\n```\n\nThis uses `find`'s `-exec ... +` to batch filenames directly into `du`, without needing `xargs` or worrying about `-print0`/`-0` pairing — often the cleanest option.",
   "latency_ms": 9789,
   "cost": 0.007582,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response provides a perfectly accurate and robust shell command that meets all requirements, including handling spaces and human-readable sorting. The explanation is clear, concise, and highly useful."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 5,
     "violates": true,
     "reasoning": "Primary command works for spaces and generally sorts human-readable sizes correctly. However, it supplies a second alternative command despite requesting a single command; `xargs` may run `du` with no files, and newline-containing filenames"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Correct, space-safe command with clear breakdown; minor extra notes don't hurt usefulness or concision."
    }
   ]
  },
  {
   "id": "c14",
   "name": "Explain and cost",
   "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n    for i in range(len(xs)):\n        for j in range(i+1, len(xs)):\n            if xs[i] == xs[j]:\n                return True\n    return False",
   "output": "## What it does\n\n`has_dup(xs)` checks whether a list contains any duplicate elements. It does this by comparing every pair of elements `(i, j)` with `i < j`, and returns `True` as soon as it finds two equal elements. If no pair matches after checking all combinations, it returns `False`.\n\n## Complexity\n\n- **Time:** O(n²) — for each element `i`, it scans all subsequent elements `j`, giving roughly n(n-1)/2 comparisons in the worst case (no duplicates).\n- **Space:** O(1) — only uses a couple of loop variables, no extra data structures.\n\n## O(n) rewrite\n\nUse a hash set to track seen elements — each lookup/insert is O(1) on average.\n\n```python\ndef has_dup(xs):\n    seen = set()\n    for x in xs:\n        if x in seen:\n            return True\n        seen.add(x)\n    return False\n```\n\n**Complexity of new version:**\n- **Time:** O(n) average case (hash set operations are O(1) amortized).\n- **Space:** O(n) — in the worst case (no duplicates), the set stores all n elements.\n\nThis trades space for time: we go from O(n²)/O(1) to O(n)/O(n), which is the classic time-space tradeoff for duplicate detection using hashing.",
   "latency_ms": 7371,
   "cost": 0.004424,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly explains the function, correctly identifies the O(n^2) time and O(1) space complexity, and provides an accurate O(n) rewrite using a set, along with its updated complexities."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Accurately explains pairwise duplicate detection, correctly gives worst-case O(n²) time and O(1) auxiliary space, and provides a clear hash-set rewrite with appropriate average-case O(n) time and O(n) space caveats."
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 10,
     "violates": false,
     "reasoning": "Accurate explanation, correct complexities, proper O(n) rewrite with set. Clear, complete, follows all task parts exactly."
    }
   ]
  },
  {
   "id": "c15",
   "name": "Regex with limits",
   "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.",
   "output": "# UK Postcode Regex\n\n```regex\n^([A-Z]{1,2}\\d[A-Z\\d]?) ?(\\d[A-Z]{2})$\n```\n\n*(Assumes input is uppercase; add the `i` flag or a `.toUpperCase()` preprocessing step if not.)*\n\n## Explanation\n\n| Part | Meaning |\n|---|---|\n| `^` | Start of string |\n| `([A-Z]{1,2}` | 1–2 letters — the postcode area (e.g., `M`, `SW`, `EC`) |\n| `\\d` | A digit — start of the district number |\n| `[A-Z\\d]?)` | Optional letter or digit — handles districts like `W1A` or `SW1A` |\n| ` ?` | An optional space between outward and inward codes |\n| `(\\d[A-Z]{2})` | A digit followed by exactly 2 letters — the inward code (sector + unit) |\n| `$` | End of string |\n\n## Known Failure Cases\n\n**Valid postcodes it would reject:**\n- None that I'm aware of within standard formatting — the regex is deliberately loose to accept most real patterns.\n\n**Invalid postcodes it would incorrectly accept:**\n- `ZZ9 9ZZ` — structurally valid but not a real postcode (letters `Q`, `V`, `X` are never used in certain positions, and this regex doesn't enforce those exclusions).\n- `AI1 1AA` — passes the pattern but `AI` is not an allocated postcode area.\n\n## Caveat\n\nThis regex checks **structural/format validity only** — it does not verify that the postcode corresponds to a real, currently allocated UK postcode area. Full validation would require cross-referencing against Royal Mail's official postcode database (PAF), since not all letter/digit combinations permitted by the format are actually in use. I don't claim this is perfect or exhaustive.",
   "latency_ms": 9178,
   "cost": 0.006398,
   "task_score": 6,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly follows all instructions, providing a solid regex, clear one-line explanations, two accurate examples of invalid postcodes it accepts, and explicitly states it is not perfect."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 6,
     "violates": false,
     "reasoning": "The regex covers common postcode shapes and gives two plausible invalid examples it accepts, satisfying the requested alternative. However, claiming no valid standard-format postcodes are rejected is false: it rejects the valid special post"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 6,
     "violates": false,
     "reasoning": "Solid regex and structure; errs by claiming no valid rejects (misses GIR 0AA etc.); table ok but not pure one-liners; useful caveat."
    }
   ]
  },
  {
   "id": "c16",
   "name": "Callback to async",
   "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n  if (e) return cb(e);\n  getOrders(user.id, (e2, orders) => {\n    if (e2) return cb(e2);\n    getTotals(orders, (e3, totals) => cb(e3, totals));\n  });\n});",
   "output": "## Refactored with `async/await`\n\nSince `getUser`, `getOrders`, and `getTotals` are Node‑style callback APIs (`(err, result) => …`), promisify them first, then let `await` handle propagation — any rejection automatically skips to the `catch`, so nothing gets silently dropped.\n\n```js\nconst { promisify } = require('util');\n\nconst getUserAsync   = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id) {\n  const user   = await getUserAsync(id);\n  const orders = await getOrdersAsync(user.id);\n  const totals = await getTotalsAsync(orders);\n  return totals;\n}\n```\n\nIf you need to keep the original callback-based call signature (`fn(id, cb)`), wrap it — but don't swallow errors, just forward them:\n\n```js\nfunction getUserTotalsCb(id, cb) {\n  getUserTotals(id)\n    .then((totals) => cb(null, totals))\n    .catch((err) => cb(err));\n}\n```\n\n### If the caller uses `async/await` too (preferred)\n\n```js\ntry {\n  const totals = await getUserTotals(id);\n  // use totals\n} catch (err) {\n  // handle or rethrow — never leave this block empty\n  throw err;\n}\n```\n\n### Key points on error propagation\n- No manual `if (err) return cb(err)` checks needed — a rejected promise anywhere in the `await` chain jumps straight to `catch`.\n- Don't add an empty `catch {}` — always rethrow, log, or forward the error explicitly.\n- If `getUser`, `getOrders`, `getTotals` are already Promise-based (not callback-style), skip `promisify` entirely and just `await` them directly.\n- Avoid mixing `.then()` chains with `async/await` in the same function — pick one style to keep error flow predictable.",
   "latency_ms": 17018,
   "cost": 0.015982,
   "task_score": 10,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly refactors the callback hell into async/await using `promisify`. It correctly handles error propagation by letting promises reject naturally, ensuring errors are not swallowed, and provides clear, useful context for a "
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 10,
     "violates": false,
     "reasoning": "Correctly promisifies Node-style callbacks, uses sequential await calls, and preserves all errors as promise rejections. The optional callback wrapper forwards errors properly. Clear, practical, and accurately notes the Promise-based altern"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 8,
     "violates": false,
     "reasoning": "Correct async/await refactor with promisify and proper error propagation; clear and useful, minor verbosity."
    }
   ]
  },
  {
   "id": "c17",
   "name": "Diagnose from a trace",
   "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n    at renderRows (/app/src/table.js:42:19)\n    at Table (/app/src/table.js:12:5)\n    at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.",
   "output": "## Root Cause\n\nThe `Table` component is calling `.map()` on a prop/state variable (likely `rows` or `data`) that is `undefined` on the **initial render**. This strongly suggests the data is coming from an **async source** (API call, context, Redux store, etc.) and the component doesn't have a default value or loading guard for the pre-fetch state.\n\nThe fact that it **only fails in production on first paint** — not in dev — points to a **timing/race condition**, not a logic bug. Common causes:\n\n1. **Dev server masks the issue**: React dev builds run more synchronous checks/warnings and often have slower initial mounts (extra logging, unminified code, no aggressive bundling), which can inadvertently let async data resolve before first render in dev, hiding the race.\n2. **Production build is faster and more aggressively optimized**, so the component mounts and renders *before* the async data (fetch, hydration, context provider) has resolved — exposing the missing default/guard.\n3. Alternatively: **SSR/hydration mismatch** — the server sends HTML built with data, but the client's first paint happens before hydration data is attached (common with Next.js, Remix, etc. in production mode where streaming/chunking behaves differently).\n\n## First Thing to Check\n\n**Look at line 42 in `table.js` and identify what's being mapped:**\n\n```js\nfunction renderRows(rows) {\n  return rows.map(...)  // <- rows is undefined here\n}\n```\n\nThen trace **where `rows` comes from** in the `Table` component (line 12) — is it:\n- A prop passed from a parent that fetches data asynchronously?\n- State initialized as `undefined` instead of `[]`?\n- Context value that isn't populated yet on first client render?\n\n**Quick fix to confirm the theory** — add a guard and see if the error disappears (even if data appears empty on first paint):\n\n```js\nfunction renderRows(rows = []) {\n  return rows.map(...)\n}\n```\n\nor in the component:\n\n```js\nif (!rows) return null; // or a loading state\n```\n\nIf this resolves the crash, it confirms the component is rendering before its data dependency resolves — and the real fix is ensuring the data-fetching logic (useEffect, suspense boundary, initial state) properly guards the render path until data is ready, rather than just patching the symptom.",
   "latency_ms": 12034,
   "cost": 0.007848,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 9,
     "violates": false,
     "reasoning": "Accurately identifies the root cause as mapping an undefined variable due to async timing differences in production. Correctly suggests checking line 42 in table.js as the first step. Highly useful and clear."
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 8,
     "violates": false,
     "reasoning": "Correctly identifies an undefined array-like input on initial render and appropriately prioritizes inspecting line 42 and data initialization/source. The production-versus-dev timing explanation is plausible but somewhat speculative and mor"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 9,
     "violates": false,
     "reasoning": "Accurate root cause analysis of undefined map target and prod/dev race; clear first check and fix. Highly useful and concise enough."
    }
   ]
  },
  {
   "id": "c18",
   "name": "Idempotency",
   "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.",
   "output": "# Idempotent POST /payments\n\n**Key:** Client-generated `Idempotency-Key` header (UUID v4), scoped to the merchant/API key. Client must reuse the same key for retries of the same logical payment.\n\n**Storage:** A dedicated `idempotency_keys` table (same DB/transaction as payments for atomicity):\n- `key`, `merchant_id`, `request_hash`, `status` (`processing`/`completed`), `response_body`, `response_code`, `payment_id`, `created_at`.\n- Unique constraint on `(merchant_id, key)`.\n\n**Flow:**\n1. On request, `INSERT` the key row with status `processing` inside the same transaction as payment creation.\n2. If insert succeeds → process payment → update row to `completed` with the response payload, commit.\n3. If insert violates unique constraint (duplicate key) → another request owns it.\n\n**Concurrent duplicate:** If status is `processing`, return `409 Conflict` (or block briefly with polling) — do not create a second charge. If status is `completed`, return the **stored response** with the original status code (e.g., `200`) and same `payment_id`, without re-charging.\n\n**Second identical retry (after completion):** Same stored response replayed verbatim — idempotent by construction.\n\n**Remaining failure mode:** If the process crashes *after* charging the payment gateway but *before* committing the `completed` status/response, the retry sees `processing` (or a rolled-back row) and may still double-charge — the gap between external side-effect and local commit isn't atomic (requires reconciliation via gateway idempotency keys too).",
   "latency_ms": 8666,
   "cost": 0.00549,
   "task_score": 9,
   "judges": [
    {
     "judge": "google/gemini-3.1-pro-preview",
     "score": 10,
     "violates": false,
     "reasoning": "The response perfectly addresses all requirements: describes the key, storage, concurrent duplicate handling, and second retry response. It correctly identifies the remaining failure mode (crash between external call and local commit). Word"
    },
    {
     "judge": "openai/gpt-5.6-terra",
     "score": 9,
     "violates": false,
     "reasoning": "Meets all requested elements clearly and stays under 200 words. Correctly prevents duplicate local payment creation and replays completed responses. Minor omission: it stores a request hash but does not say to reject same-key, different-pay"
    },
    {
     "judge": "x-ai/grok-4.5",
     "score": 8,
     "violates": false,
     "reasoning": "Solid design covering key, storage, concurrency, replay, and failure mode. Clear and useful; minor verbosity but under 200 words."
    }
   ]
  }
 ]
}