{
  "entity": "qwen3-7-max",
  "entity_type": "model",
  "category": "Coding",
  "suite": "suite-2026-08-skills/coding",
  "model": "qwen/qwen3.7-max",
  "judges": [
    "anthropic/claude-sonnet-5",
    "google/gemini-3.1-pro-preview",
    "openai/gpt-5.6-terra"
  ],
  "judge_protocol": "judge-2026-08",
  "anchors": "solo-2026-08-a",
  "score": 90,
  "caps": 0,
  "cost_usd": 0.4233,
  "started_at": "2026-08-16T06:39:12.804Z",
  "finished_at": "2026-08-16T06:43:14.469Z",
  "tasks": [
    {
      "id": "c1",
      "name": "Duration parser",
      "prompt": "Write a Python function parse_duration(s) that converts strings like '1h30m', '45s', '2h', '90m', '1h2m3s' into total seconds. Raise ValueError on anything malformed. Include three assert-based tests, one of which covers a malformed input.",
      "output": "```python\nimport re\n\ndef parse_duration(s: str) -> int:\n    \"\"\"\n    Converts a duration string (e.g., '1h30m', '45s', '1h2m3s') into total seconds.\n    Raises ValueError on malformed inputs.\n    \"\"\"\n    if not isinstance(s, str):\n        raise ValueError(f\"Expected a string, got {type(s).__name__}\")\n        \n    # Regex explanation:\n    # ^             : Start of string\n    # (?:(\\d+)h)?   : Optional non-capturing group for hours, capturing the digits\n    # (?:(\\d+)m)?   : Optional non-capturing group for minutes, capturing the digits\n    # (?:(\\d+)s)?   : Optional non-capturing group for seconds, capturing the digits\n    # $             : End of string\n    pattern = re.compile(r'^(?:(\\d+)h)?(?:(\\d+)m)?(?:(\\d+)s)?$')\n    match = pattern.fullmatch(s)\n    \n    # If it doesn't match the strict pattern, or if the string is empty\n    if not match or not s:\n        raise ValueError(f\"Malformed duration string: '{s}'\")\n        \n    hours = int(match.group(1)) if match.group(1) else 0\n    minutes = int(match.group(2)) if match.group(2) else 0\n    seconds = int(match.group(3)) if match.group(3) else 0\n    \n    return hours * 3600 + minutes * 60 + seconds\n\n\n# ==========================================\n# Assert-based Tests\n# ==========================================\n\n# Test 1: Standard combinations and single units\nassert parse_duration('1h30m') == 5400\nassert parse_duration('45s') == 45\nassert parse_duration('2h') == 7200\nassert parse_duration('90m') == 5400\n\n# Test 2: Full combination of hours, minutes, and seconds\nassert parse_duration('1h2m3s') == 3723  # (1*3600) + (2*60) + 3\n\n# Test 3: Malformed inputs correctly raise ValueError\ndef test_malformed_inputs():\n    malformed_cases = [\n        '',          # Empty string\n        '1h2h',      # Duplicate units\n        'abc',       # No numbers/invalid characters\n        '1.5h',      # Decimals not supported by \\d+\n        '1h 30m',    # Spaces not allowed in strict format\n        '60',        # Missing unit suffix\n        '30m1h',     # Out of order (must be h -> m -> s)\n    ]\n    for case in malformed_cases:\n        try:\n            parse_duration(case)\n            # If we reach this line, the function failed to raise an error\n            assert False, f\"Expected ValueError for malformed input: '{case}'\"\n        except ValueError:\n            pass  # Expected behavior\n\ntest_malformed_inputs()\n\nprint(\"All tests passed successfully!\")\n```",
      "latency_ms": 73126,
      "cost": 0.017545125,
      "task_score": 10,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correct, robust regex-based implementation with proper ValueError handling; includes required asserts plus extra malformed cases; clear and well-documented."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response provides a correct and robust Python function, handles malformed inputs properly, and includes the requested assert-based tests (including one for malformed inputs). Code is clear and well-commented."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 10,
          "violates": false,
          "reasoning": "Correctly parses ordered h/m/s combinations, rejects empty and malformed formats with ValueError, and handles required examples including 90m. It includes three clearly organized assert-based test sections, including malformed-input coverag"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c2",
      "name": "Find the bug",
      "prompt": "This is meant to return the average of the positive numbers but returns the wrong value. Identify the bug, explain it in one sentence, and give the corrected function.\n\nfunction avgPositive(xs) {\n  let sum = 0, n = 0;\n  for (const x of xs) {\n    if (x > 0) sum += x;\n    n++;\n  }\n  return sum / n;\n}",
      "output": "## Bug\n\nThe `n++` statement is **outside** the `if` block, so it counts *every* element in the array rather than only the positive ones, making the denominator too large and the average too small.\n\n## Corrected Function\n\n```javascript\nfunction avgPositive(xs) {\n  let sum = 0, n = 0;\n  for (const x of xs) {\n    if (x > 0) {\n      sum += x;\n      n++;\n    }\n  }\n  return sum / n;\n}\n```\n\nThe fix is simply moving `n++` inside the `if` block (by adding braces) so that the counter only increments when a positive number is found and added to the sum.",
      "latency_ms": 8276,
      "cost": 0.0020532,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correctly identifies bug, explains clearly, provides correct fix. Slightly verbose but accurate and useful."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response correctly identifies the bug, explains it in a single sentence as requested, and provides the corrected function. It is accurate, clear, and highly useful."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 5,
          "violates": true,
          "reasoning": "Correctly identifies that n increments for every element and provides the proper fix. However, it gives a second explanatory sentence after the corrected function, exceeding the requested one-sentence explanation."
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c3",
      "name": "SQL without window functions",
      "prompt": "Given tables users(id, email) and orders(id, user_id, created_at, total), write SQL returning the email and order count of every user with more than 3 orders in the last 30 days, most orders first. Do not use window functions. Target Postgres.",
      "output": "```sql\nSELECT \n    u.email, \n    COUNT(o.id) AS order_count\nFROM users u\nJOIN orders o \n    ON u.id = o.user_id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY \n    u.id, \n    u.email\nHAVING COUNT(o.id) > 3\nORDER BY \n    order_count DESC, \n    u.email ASC;\n```\n\n### Explanation:\n* **`JOIN`**: An `INNER JOIN` is used because we only care about users who actually have orders.\n* **`WHERE`**: Filters the orders to only those created in the last 30 days. `NOW() - INTERVAL '30 days'` is the standard PostgreSQL syntax for timestamp/interval arithmetic.\n* **`GROUP BY u.id, u.email`**: Grouping by the primary key (`u.id`) alongside the email is a PostgreSQL best practice. It ensures accurate grouping even if there are edge cases where emails aren't strictly unique, and prevents functional dependency errors.\n* **`HAVING`**: Filters the grouped results to only include users with strictly more than 3 orders.\n* **`ORDER BY`**: Sorts the results by the highest order count first. A secondary sort by `email` is added just to ensure deterministic ordering in the event of a tie. \n* **No Window Functions**: Relies purely on standard aggregate functions (`COUNT`) and `GROUP BY` / `HAVING` clauses.",
      "latency_ms": 26596,
      "cost": 0.007375,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correct, meets all constraints, no window functions, Postgres syntax, clear explanation. Minor stylistic tie-break addition not required but harmless."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The SQL query is perfectly accurate, targets PostgreSQL, avoids window functions, and correctly implements all filtering and sorting requirements. The explanation is clear and useful."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 8,
          "violates": false,
          "reasoning": "SQL correctly filters recent orders, groups per user, applies the >3 threshold, and orders by count descending without window functions. The explanation is clear; only the claim that grouping by email prevents functional-dependency errors i"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c4",
      "name": "Infinite useEffect",
      "prompt": "Explain precisely why this React effect loops forever, then give the fixed version.\n\nconst [items, setItems] = useState([]);\nuseEffect(() => {\n  fetch('/api/items').then(r => r.json()).then(setItems);\n}, [items]);",
      "output": "### Why it loops forever\n\nThe infinite loop is caused by a combination of **circular dependency** and **JavaScript referential inequality**. \n\nHere is the precise step-by-step breakdown of the loop:\n\n1. **Initial Render:** The component mounts. `items` is initialized as an empty array `[]`.\n2. **Effect Runs:** The `useEffect` runs after the first render. It fires the `fetch` request.\n3. **State Update:** The API responds, and `.then(setItems)` is called with the parsed JSON data (a new array of objects).\n4. **Re-render:** Calling `setItems` updates the state, triggering a re-render of the component.\n5. **Dependency Check (The Trap):** React checks the dependency array `[items]` to see if it needs to run the effect again. It compares the *previous* `items` array with the *new* `items` array using strict equality (`===`).\n6. **Referential Inequality:** Because `fetch` and `.json()` create a **brand new array in memory**, the new array does not share the same memory reference as the old array. In JavaScript, `[1, 2] !== [1, 2]` if they are different instances. Therefore, React sees that `items` has \"changed\".\n7. **Loop:** Because the dependency changed, React runs the `useEffect` again. This triggers another fetch, which calls `setItems` with another new array reference, triggering another re-render, ad infinitum.\n\n*(Note: Even if the API returns an empty array `[]` every time, `[] !== []` in JavaScript, so the loop will still happen).*\n\n***\n\n### The Fixed Version\n\nIf your goal is to fetch the data **once** when the component mounts, you should use an empty dependency array. This tells React to only run the effect on the initial mount, ignoring subsequent state changes.\n\n```javascript\nconst [items, setItems] = useState([]);\n\nuseEffect(() => {\n  fetch('/api/items')\n    .then(r => r.json())\n    .then(setItems);\n}, []); // Empty array means \"run only on mount\"\n```\n\n#### Alternative Fix (If you actually need to refetch)\nIf your intent was to refetch the data whenever a *different* variable changes (like a `userId` or `filter`), you must put **that** variable in the dependency array, never the state variable that the effect itself updates:\n\n```javascript\nconst [items, setItems] = useState([]);\nconst [userId, setUserId] = useState(1);\n\nuseEffect(() => {\n  fetch(`/api/items?userId=${userId}`)\n    .then(r => r.json())\n    .then(setItems);\n}, [userId]); // Refetches only when userId changes, not when items changes\n```",
      "latency_ms": 27985,
      "cost": 0.0064251,
      "task_score": 10,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Accurate, thorough explanation and correct fix with useful alternative. Clear, well-structured, minor verbosity but appropriate for clarity."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response accurately and clearly explains the infinite loop caused by referential inequality and state updates in the dependency array. It provides the correct fix and a helpful alternative. Flawless execution."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 10,
          "violates": false,
          "reasoning": "Precisely explains the state-update/dependency cycle and referential identity issue, then provides the correct empty-dependency fixed version. The optional refetch example is accurate and useful. No explicit checkable constraints were breac"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c5",
      "name": "Typed debounce",
      "prompt": "Write a debounce function in TypeScript that preserves the argument types of the wrapped function, returns a function with a .cancel() method, and does not use 'any'. Explain in one sentence why the naive generic signature loses type information.",
      "output": "```typescript\ninterface DebouncedFunction<Args extends unknown[]> {\n  (...args: Args): void;\n  cancel(): void;\n}\n\nfunction debounce<Args extends unknown[]>(\n  func: (...args: Args) => unknown,\n  wait: number\n): DebouncedFunction<Args> {\n  let timeoutId: ReturnType<typeof setTimeout> | undefined;\n\n  const debounced = (...args: Args): void => {\n    if (timeoutId !== undefined) {\n      clearTimeout(timeoutId);\n    }\n    \n    timeoutId = setTimeout(() => {\n      func(...args);\n      timeoutId = undefined;\n    }, wait);\n  };\n\n  debounced.cancel = (): void => {\n    if (timeoutId !== undefined) {\n      clearTimeout(timeoutId);\n      timeoutId = undefined;\n    }\n  };\n\n  return debounced;\n}\n```\n\nA naive generic signature constrained merely to the built-in `Function` type loses type information because that base interface lacks specific parameter and return type definitions, causing TypeScript to treat all arguments and return values as implicitly untyped.",
      "latency_ms": 62204,
      "cost": 0.01423375,
      "task_score": 8,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Correct, meets constraints, no 'any', includes cancel method and explanation sentence. Minor: debounced typed as arrow function then assigned cancel, technically needs cast but works loosely; TS may flag it."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly implements the debounce function in TypeScript without using 'any', includes the required .cancel() method, preserves argument types, and provides a clear, one-sentence explanation as requested."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 5,
          "violates": false,
          "reasoning": "The generic argument tuple correctly preserves call arguments and avoids `any`, but the code does not type-check: `debounced` is inferred as a plain function, so assigning `.cancel` causes a property-does-not-exist error. The explanation is"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c6",
      "name": "Leftmost binary search",
      "prompt": "Implement binary search that returns the index of the FIRST occurrence of a target in a sorted array with duplicates, or -1. Give the code and state the complexity. Include the test case that distinguishes it from an ordinary binary search.",
      "output": "# Binary Search — First Occurrence\n\n## The Key Insight\n\nIn a standard binary search, you return immediately when `arr[mid] == target`. To find the **first** occurrence, you must **keep searching left** even after a match, because an earlier duplicate might exist.\n\n## Implementation (Python)\n\n```python\ndef binary_search_first(arr, target):\n    \"\"\"\n    Returns the index of the FIRST occurrence of target in sorted array arr.\n    Returns -1 if target is not found.\n    \"\"\"\n    low = 0\n    high = len(arr) - 1\n    result = -1\n\n    while low <= high:\n        mid = low + (high - low) // 2\n\n        if arr[mid] == target:\n            result = mid        # record this match\n            high = mid - 1      # but KEEP searching left for an earlier one\n        elif arr[mid] < target:\n            low = mid + 1\n        else:\n            high = mid - 1\n\n    return result\n```\n\n## Complexity\n\n| | Complexity |\n|---|---|\n| **Time** | **O(log n)** — the search space is halved every iteration |\n| **Space** | **O(1)** — only a few integer variables are used |\n\n## Distinguishing Test Case\n\nThis is the case that separates a first-occurrence search from a naive binary search:\n\n```python\ndef test_first_occurrence():\n    arr = [1, 2, 2, 2, 2, 2, 3, 4]\n    #       0  1  2  3  4  5  6  7\n\n    target = 2\n\n    # A standard binary search computes mid = (0+7)//2 = 3\n    # arr[3] == 2, so it would return index 3.\n    #\n    # Our function must return index 1 (the FIRST occurrence).\n\n    result = binary_search_first(arr, target)\n    assert result == 1, f\"Expected 1, got {result}\"\n    print(f\"PASS: first occurrence of {target} is at index {result}\")\n\n    # --- Additional edge cases ---\n\n    # Target not present\n    assert binary_search_first([1, 3, 5], 2) == -1\n\n    # All elements are the target\n    assert binary_search_first([4, 4, 4, 4], 4) == 0\n\n    # Target is only at the end\n    assert binary_search_first([1, 2, 3, 5, 5], 5) == 3\n\n    # Single element — match\n    assert binary_search_first([7], 7) == 0\n\n    # Single element — no match\n    assert binary_search_first([7], 3) == -1\n\n    # Empty array\n    assert binary_search_first([], 1) == -1\n\n    print(\"All tests passed.\")\n\ntest_first_occurrence()\n```\n\n## Why Standard Binary Search Fails Here\n\n```\nArray:  [1, 2, 2, 2, 2, 2, 3, 4]\n         0  1  2  3  4  5  6  7\n\nStandard binary search for target = 2:\n  ┌─ low=0, high=7 → mid=3 → arr[3]=2 == target → RETURN 3  ✗ WRONG\n\nFirst-occurrence binary search for target = 2:\n  ├─ low=0, high=7 → mid=3 → arr[3]=2 == target → result=3, high=2\n  ├─ low=0, high=2 → mid=1 → arr[1]=2 == target → result=1, high=0\n  ├─ low=0, high=0 → mid=0 → arr[0]=1 <  target → low=1\n  └─ low=1, high=0 → loop ends → RETURN 1  ✓ CORRECT\n```\n\nThe single line `high = mid - 1` on a match is what forces the search to continue narrowing leftward until it locks onto the very first duplicate.",
      "latency_ms": 23478,
      "cost": 0.005402925,
      "task_score": 10,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correct implementation, correct complexity, clear distinguishing test case, well explained and formatted."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly implements the requested algorithm, correctly states the time and space complexity, and provides a clear test case that distinguishes it from a standard binary search."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 10,
          "violates": false,
          "reasoning": "Correct Python implementation reliably returns the leftmost target index or -1. States O(log n) time and O(1) space. Includes a clear duplicate-containing test that demonstrates ordinary immediate-return binary search can return a later occ"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c7",
      "name": "Security review",
      "prompt": "Review this Express handler and list every security problem you find, most severe first, each with the fix.\n\napp.get('/file', (req, res) => {\n  const p = req.query.name;\n  db.query(`SELECT * FROM files WHERE name = '${p}'`, (e, rows) => {\n    res.sendFile(__dirname + '/uploads/' + p);\n  });\n});",
      "output": "Here is a comprehensive security review of the provided Express handler, ordered from most severe (Critical) to least severe (Low). \n\n### 1. SQL Injection (SQLi)\n* **Severity:** Critical\n* **The Problem:** The user-supplied query parameter `p` is directly interpolated into the SQL string using template literals. An attacker can supply a malicious payload (e.g., `' OR '1'='1'; DROP TABLE files;--`) to read, modify, or destroy the entire database.\n* **The Fix:** Never concatenate user input into SQL strings. Always use parameterized queries (prepared statements) so the database driver treats the input strictly as data, not executable code.\n  ```javascript\n  // Fix: Use '?' placeholder and pass variables in an array\n  db.query('SELECT * FROM files WHERE name = ?', [p], (e, rows) => { ... });\n  ```\n\n### 2. Path Traversal / Directory Traversal (Local File Inclusion)\n* **Severity:** Critical\n* **The Problem:** The user-supplied parameter `p` is concatenated directly into the file path. An attacker can supply a payload like `../../../etc/passwd` or `..\\..\\..\\windows\\system32\\config\\sam`. The server will resolve this path and serve sensitive operating system or application source files outside the intended `/uploads/` directory.\n* **The Fix:** Resolve the absolute path of the requested file and verify that it strictly remains inside the intended base directory before serving it.\n  ```javascript\n  const path = require('path');\n  const uploadsDir = path.resolve(__dirname, 'uploads');\n  const requestedFilePath = path.resolve(uploadsDir, p);\n\n  // Fix: Ensure the resolved path starts with the allowed directory\n  if (!requestedFilePath.startsWith(uploadsDir + path.sep) && requestedFilePath !== uploadsDir) {\n      return res.status(400).send('Invalid file path');\n  }\n  res.sendFile(requestedFilePath);\n  ```\n\n### 3. Broken Access Control / Insecure Direct Object Reference (IDOR)\n* **Severity:** High\n* **The Problem:** The endpoint lacks any authentication or authorization checks. Any anonymous user can request and download any file simply by guessing or knowing its name. If these files are meant to be private or user-specific, this is a massive data leak.\n* **The Fix:** Implement authentication middleware to identify the user, and authorization logic to ensure the user owns or has permission to access the requested file.\n  ```javascript\n  // Fix: Add auth middleware and check ownership\n  app.get('/file', authenticateUser, (req, res) => {\n      // ... later in the DB callback:\n      if (rows[0].owner_id !== req.user.id) {\n          return res.status(403).send('Forbidden');\n      }\n  });\n  ```\n\n### 4. Logic Flaw: Ignoring Database Results (Security Bypass)\n* **Severity:** High\n* **The Problem:** The code queries the database to check if the file exists, but completely ignores the `rows` result. It calls `res.sendFile` regardless of whether the file actually exists in the database. This renders the database check useless and allows attackers to download files that exist on the disk but were never registered in the database (bypassing whatever implicit access control the DB lookup was supposed to provide).\n* **The Fix:** Check the database results and abort the request if the file record is not found.\n  ```javascript\n  // Fix: Validate the DB result before serving the file\n  if (!rows || rows.length === 0) {\n      return res.status(404).send('File not found');\n  }\n  ```\n\n### 5. Unhandled Exceptions and Information Disclosure\n* **Severity:** Medium\n* **The Problem:** \n  1. If the database query fails (`e` is truthy), the error is ignored, and the code attempts to send the file anyway. \n  2. If `res.sendFile` fails (e.g., file missing on disk), it throws an unhandled exception. \n  Unhandled errors can crash the Node.js process (Denial of Service) or leak internal stack traces and file paths to the attacker if a global error handler isn't configured properly.\n* **The Fix:** Explicitly handle database errors and pass a callback to `sendFile` to handle file system errors.\n  ```javascript\n  db.query('...', [p], (e, rows) => {\n      if (e) {\n          console.error(e); // Log internally\n          return res.status(500).send('Internal server error'); // Generic message to user\n      }\n      // ...\n      res.sendFile(requestedFilePath, (err) => {\n          if (err && !res.headersSent) {\n              res.status(404).send('File not found on disk');\n          }\n      });\n  });\n  ```\n\n### 6. Missing Input Validation and Type Confusion\n* **Severity:** Medium\n* **The Problem:** The code assumes `req.query.name` is a string. \n  1. If the user omits the parameter (`/file`), `p` is `undefined`, resulting in the SQL query `WHERE name = 'undefined'` and the file path `/uploads/undefined`.\n  2. If the user passes an array (`/file?name[]=foo&name[]=bar`), `p` becomes an array. This can cause unexpected behavior, type coercion bugs, or bypasses in downstream security checks.\n* **The Fix:** Validate that the input exists and is strictly a string before processing.\n  ```javascript\n  // Fix: Validate input type and presence\n  const p = req.query.name;\n  if (!p || typeof p !== 'string' || p.trim() === '') {\n      return res.status(400).send('Invalid or missing file name');\n  }\n  ```\n\n### 7. Lack of Rate Limiting / Resource Exhaustion\n* **Severity:** Low\n* **The Problem:** The endpoint performs a database query and streams a file from the disk on every request. An attacker can spam this endpoint to exhaust database connection pools, consume server bandwidth, or max out CPU/disk I/O, leading to a Denial of Service (DoS).\n* **The Fix:** Apply rate limiting to the route to restrict how many requests a single IP or user can make in a given timeframe.\n  ```javascript\n  const rateLimit = require('express-rate-limit');\n  const fileLimiter = rateLimit({ windowMs: 15 * 60 * 1000, max: 20 });\n  \n  // Fix: Apply limiter to the route\n  app.get('/file', fileLimiter, (req, res) => { ... });\n  ```\n\n***\n\n### The Combined Secure Implementation\n\nHere is what the fully patched, production-ready code should look like:\n\n```javascript\nconst path = require('path');\nconst rateLimit = require('express-rate-limit');\n\n// 1. Rate limiting to prevent DoS\nconst fileDownloadLimiter = rateLimit({\n  windowMs: 15 * 60 * 1000, // 15 minutes\n  max: 30, // limit each IP to 30 requests per windowMs\n  message: 'Too many download requests, please try again later.'\n});\n\n// 2. Authentication middleware (implementation depends on your auth strategy)\nconst { authenticateUser } = require('./middleware/auth'); \n\napp.get('/file', fileDownloadLimiter, authenticateUser, (req, res) => {\n  // 3. Input validation\n  const p = req.query.name;\n  if (!p || typeof p !== 'string' || p.trim() === '') {\n    return res.status(400).send('Invalid or missing file name.');\n  }\n\n  // 4. Path Traversal protection\n  const uploadsDir = path.resolve(__dirname, 'uploads');\n  const requestedFilePath = path.resolve(uploadsDir, p);\n  \n  if (!requestedFilePath.startsWith(uploadsDir + path.sep) && requestedFilePath !== uploadsDir) {\n    return res.status(400).send('Invalid file path.');\n  }\n\n  // 5. SQL Injection protection (Parameterized query)\n  db.query('SELECT * FROM files WHERE name = ?', [p], (e, rows) => {\n    \n    // 6. Error handling (Database)\n    if (e) {\n      console.error('Database error:', e);\n      return res.status(500).send('An internal error occurred.');\n    }\n\n    // 7. Logic flaw fix (Check if file exists in DB)\n    if (!rows || rows.length === 0) {\n      return res.status(404).send('File not found.');\n    }\n\n    const fileRecord = rows[0];\n\n    // 8. Access Control / IDOR protection\n    if (fileRecord.owner_id !== req.user.id && !req.user.isAdmin) {\n      return res.status(403).send('You do not have permission to access this file.');\n    }\n\n    // 9. Serve file with error handling\n    res.sendFile(requestedFilePath, (err) => {\n      if (err && !res.headersSent) {\n        console.error('File send error:', err);\n        res.status(404).send('File not found on disk.');\n      }\n    });\n  });\n});\n```",
      "latency_ms": 80761,
      "cost": 0.018497975,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Thorough, accurate, well-prioritized coverage of SQLi, path traversal, IDOR, logic flaws, error handling, validation, rate limiting, with correct fixes and combined secure code. Slightly verbose for a non-technical reader but highly useful "
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response flawlessly identifies all security vulnerabilities, orders them by severity, and provides clear, accurate fixes for each. The inclusion of a combined secure implementation adds significant value."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 7,
          "violates": false,
          "reasoning": "Correctly identifies and fixes the core SQL injection, traversal, ignored DB result, and likely authorization issues. However, some claims are overstated or conditional: sendFile errors are ordinarily forwarded by Express, and anonymous acc"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c8",
      "name": "Safe migration",
      "prompt": "Write the Postgres migration to add a NOT NULL column 'status' with default 'pending' to an orders table with 40 million rows, without taking a long exclusive lock. Give the steps in order and say which step is the dangerous one and why.",
      "output": "To safely add a `NOT NULL` column with a default value to a 40-million-row table without causing a prolonged outage, you must avoid long `ACCESS EXCLUSIVE` locks and massive single transactions. \n\n*Note: The strategy below assumes **PostgreSQL 12 or newer**. (See the version warning at the bottom if you are on an older version).*\n\nHere are the steps in order for a zero-downtime migration.\n\n### The Migration Steps\n\n**Step 1: Add the column as nullable (No default yet)**\n```sql\nALTER TABLE orders ADD COLUMN status TEXT;\n```\n*Why:* This is a metadata-only operation. It takes an `ACCESS EXCLUSIVE` lock, but because it doesn't rewrite the table, the lock is held for only milliseconds.\n\n**Step 2: Set the default value for future rows**\n```sql\nALTER TABLE orders ALTER COLUMN status SET DEFAULT 'pending';\n```\n*Why:* This ensures any new `INSERT`s get the default value. In Postgres 11+, setting a constant default is a metadata-only operation and does not rewrite the table. The lock is held for milliseconds.\n\n**Step 3: Backfill existing rows in batches**\n```sql\n-- Do NOT run a single UPDATE. Run this in a script/loop in small batches:\nUPDATE orders SET status = 'pending' \nWHERE id >= {last_processed_id} AND id < {last_processed_id + 10000};\n\n-- Add a small pause between batches (e.g., pg_sleep(0.1)) to let the DB breathe.\n```\n*Why:* This updates the 40M existing rows using only row-level locks, allowing concurrent reads and writes to continue.\n\n**Step 4: Add a `NOT VALID` check constraint**\n```sql\nALTER TABLE orders ADD CONSTRAINT chk_status_not_null CHECK (status IS NOT NULL) NOT VALID;\n```\n*Why:* This adds the constraint to the catalog without scanning the table to verify existing rows. The `ACCESS EXCLUSIVE` lock is held for milliseconds.\n\n**Step 5: Validate the check constraint**\n```sql\nALTER TABLE orders VALIDATE CONSTRAINT chk_status_not_null;\n```\n*Why:* This scans the 40M rows to ensure no nulls exist, but it only requires a `SHARE UPDATE EXCLUSIVE` lock. **This allows normal reads and writes to continue** while the scan happens in the background.\n\n**Step 6: Set the column to NOT NULL**\n```sql\nALTER TABLE orders ALTER COLUMN status SET NOT NULL;\n```\n*Why:* Because of the validated check constraint from Step 5, Postgres 12+ is smart enough to skip the full table scan. It simply updates the catalog. The `ACCESS EXCLUSIVE` lock is held for milliseconds.\n\n**Step 7: Drop the check constraint (Cleanup)**\n```sql\nALTER TABLE orders DROP CONSTRAINT chk_status_not_null;\n```\n*Why:* The `NOT NULL` constraint now handles this natively. Dropping the redundant check constraint is a fast metadata operation.\n\n***\n\n### Which step is the dangerous one and why?\n\n**Step 3 (The Backfill) is the most dangerous step.** \n\nWhile the other steps carry *locking* risks if done incorrectly, the backfill carries severe **operational and resource risks** that routinely cause production outages. Here is why:\n\n1. **Table Bloat:** PostgreSQL uses MVCC (Multi-Version Concurrency Control). Updating 40 million rows creates 40 million \"dead tuples\" (old row versions). If done too quickly, this can literally double the physical size of your table on disk, potentially filling up your disk space and crashing the database.\n2. **WAL Generation and Replication Lag:** Every update generates Write-Ahead Log (WAL) entries. A massive, rapid backfill will generate gigabytes of WAL data, which can overwhelm your disk I/O and cause streaming replicas to lag behind the primary by hours, breaking read-replicas and high-availability setups.\n3. **Autovacuum Starvation:** If you update the rows faster than `autovacuum` can clean up the dead tuples, the table will become severely bloated, degrading query performance across the entire application.\n4. **Transaction Timeouts/Memory Exhaustion:** If a developer accidentally runs `UPDATE orders SET status = 'pending';` as a single query instead of batching it, it will lock all 40M rows, consume massive amounts of RAM, and likely result in an out-of-memory (OOM) kill or a transaction timeout.\n\n**Honorable Mention (The \"Gotcha\" Danger): Step 6 (`SET NOT NULL`)**\nIf you skip Steps 4 and 5 (the check constraint trick) and just run `ALTER TABLE ... SET NOT NULL`, Postgres will perform a full table scan to verify there are no nulls. It will do this while holding an **`ACCESS EXCLUSIVE` lock**, which will completely block all reads and writes to the `orders` table for several minutes, causing a total application outage.\n\n***\n\n### ⚠️ Crucial Version Warnings\n\n* **If you are on Postgres 11:** Step 6 (`SET NOT NULL`) *will* still take a long exclusive lock even with the check constraint trick. The optimization to skip the scan was introduced in Postgres 12. On PG 11, you must leave the column nullable and rely solely on the `CHECK` constraint to enforce non-nullability at the application/database level.\n* **If you are on Postgres 10 or older:** Step 2 (`SET DEFAULT`) will trigger a full table rewrite to apply the default to existing rows, taking a long exclusive lock. On older versions, you must skip Step 2, backfill the data (Step 3), and *then* set the default. \n* **The Postgres 11+ \"One-Liner\":** If you are on PG 11+ and the default is a **constant** (like `'pending'`), you can actually just run `ALTER TABLE orders ADD COLUMN status TEXT NOT NULL DEFAULT 'pending';`. Postgres 11+ optimizes this specific one-liner to avoid a table rewrite. However, the multi-step approach above remains the industry standard because it is universally safe, works if the default is a dynamic function (like `NOW()`), and gives you granular control over the backfill I/O.",
      "latency_ms": 87201,
      "cost": 0.019350525,
      "task_score": 8,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Technically accurate, well-ordered steps, correctly identifies dangerous step with justification, includes useful version caveats. Slightly verbose for a non-technical professional but overall excellent and correct."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 8,
          "violates": false,
          "reasoning": "Provides a thorough, accurate explanation. While the 7-step process is unnecessary in PG11+ for constant defaults (which the response correctly notes at the end), the detailed breakdown and risk analysis are highly valuable and correct."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 6,
          "violates": false,
          "reasoning": "Core lock-minimizing sequence is largely sound for PG12+. However, backfill can overwrite concurrent status changes because it lacks `AND status IS NULL`; batching assumes numeric IDs. The PG10 claim that ALTER COLUMN SET DEFAULT rewrites e"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c9",
      "name": "Fix the code not the test",
      "prompt": "This test fails. Fix the implementation, not the test.\n\n// impl\nexport const slugify = (s) => s.toLowerCase().replace(/ /g, '-');\n// test\nexpect(slugify('  Hello   World!  ')).toBe('hello-world');",
      "output": "```javascript\nexport const slugify = (s) =>\n  s\n    .toLowerCase()\n    .replace(/[^a-z0-9]+/g, '-')\n    .replace(/^-+|-+$/g, '');\n```\n\n### Explanation of Changes\n\nThe original implementation failed because it only replaced single spaces with hyphens. It did not handle:\n1.  **Leading/Trailing Whitespace:** `'  Hello...'` resulted in leading hyphens.\n2.  **Multiple Consecutive Spaces:** `'   '` resulted in multiple hyphens (`---`).\n3.  **Punctuation:** The exclamation mark `!` remained in the string.\n\nThe fixed implementation uses a pipeline of replacements:\n1.  **`.toLowerCase()`**: Converts the string to lowercase.\n2.  **`.replace(/[^a-z0-9]+/g, '-')`**: Replaces any sequence of characters that are **not** lowercase letters or numbers (including spaces and punctuation) with a single hyphen.\n    *   *Example:* `'  hello   world!  '` becomes `'-hello-world-'`.\n3.  **`.replace(/^-+|-+$/g, '')`**: Removes any hyphens that ended up at the very start or very end of the string.\n    *   *Example:* `'-hello-world-'` becomes `'hello-world'`.",
      "latency_ms": 76961,
      "cost": 0.01737255,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correct fix passes the test, clear explanation, concise and well-structured."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response provides a correct implementation that passes the test case, along with a clear, concise explanation of the changes and how the regex works."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 9,
          "violates": false,
          "reasoning": "Implementation correctly produces the expected slug by lowercasing, collapsing whitespace/punctuation to hyphens, and trimming edge hyphens. It fixes implementation rather than the test. Minor wording issue: the original global replacement "
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c10",
      "name": "Race condition",
      "prompt": "Identify the race condition in this code, explain what interleaving causes it, and fix it.\n\nlet cache = null;\nasync function getConfig() {\n  if (cache) return cache;\n  const r = await fetch('/config');\n  cache = await r.json();\n  return cache;\n}",
      "output": "### The Race Condition: \"Cache Stampede\" (Redundant Fetches)\n\nThe race condition in this code is a classic **cache stampede** (or thundering herd) problem. \n\nBecause the `cache` variable is only populated *after* the asynchronous `fetch` and `json()` operations complete, any concurrent calls to `getConfig()` that occur while the first request is still in flight will see `cache` as `null`. This causes the function to initiate multiple, redundant network requests to `/config`.\n\n---\n\n### The Interleaving\n\nHere is a step-by-step timeline showing how the race condition occurs when `getConfig()` is called twice in rapid succession (e.g., `Promise.all([getConfig(), getConfig()])`):\n\n1. **Call A** starts. It evaluates `if (cache)`. `cache` is `null`, so it proceeds.\n2. **Call A** executes `await fetch('/config')`. Because of the `await`, it yields control back to the event loop while waiting for the network.\n3. **Call B** starts. It evaluates `if (cache)`. Because Call A hasn't finished yet, **`cache` is still `null`**. Call B proceeds.\n4. **Call B** executes `await fetch('/config')`. It yields control, having initiated a **second, redundant network request**.\n5. **Call A's** fetch resolves. It parses the JSON, sets `cache = ...`, and returns the data.\n6. **Call B's** fetch resolves. It parses the JSON, overwrites `cache` with the exact same data, and returns.\n\n**Result:** Two network requests were made, wasting bandwidth and server resources, even though the intent was to only fetch the data once.\n\n---\n\n### The Fix\n\nTo fix this, you should **cache the Promise itself**, rather than the resolved value. Because assigning the promise to the variable happens *synchronously*, subsequent calls will immediately see that a request is already in flight and will simply await the existing promise.\n\n```javascript\nlet cachePromise = null;\n\nasync function getConfig() {\n  // If we already have a promise (pending or resolved), return it\n  if (cachePromise) return cachePromise;\n\n  // Synchronously assign the promise to the cache variable\n  cachePromise = fetch('/config').then(r => r.json());\n\n  return cachePromise;\n}\n```\n\n#### Why this works:\nWhen Call A runs, it synchronously assigns the pending Promise to `cachePromise` *before* hitting the first `await`. When Call B runs milliseconds later, `cachePromise` is no longer `null`. Call B simply returns the exact same pending Promise, and both calls will resolve simultaneously when the single network request finishes.\n\n---\n\n### Pro-Tip: Handling Errors\n\nThe basic fix above has one flaw: if the `fetch` fails (e.g., network error or 500 status), `cachePromise` will hold a **rejected promise**. All future calls to `getConfig()` will instantly fail because they will keep returning that same rejected promise. \n\nTo make this production-ready, you should clear the cache if the request fails so it can retry on the next call:\n\n```javascript\nlet cachePromise = null;\n\nfunction getConfig() {\n  if (cachePromise) return cachePromise;\n\n  cachePromise = fetch('/config')\n    .then(r => {\n      if (!r.ok) throw new Error('Network response was not ok');\n      return r.json();\n    })\n    .catch(err => {\n      // Reset the cache on failure so subsequent calls can retry\n      cachePromise = null; \n      throw err;\n    });\n\n  return cachePromise;\n}\n```\n*(Note: The `async` keyword was removed from the function in this final version because we are directly returning the Promise chain, which is slightly more efficient, but keeping `async` works perfectly fine too).*",
      "latency_ms": 37161,
      "cost": 0.008630225,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Accurately identifies race condition, explains interleaving clearly, provides correct fix with promise caching and bonus error handling. Thorough and useful, well-structured."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response accurately identifies the race condition, clearly explains the interleaving, and provides a robust fix (including an excellent pro-tip for error handling). It is highly useful and well-structured."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 9,
          "violates": false,
          "reasoning": "Correctly identifies the concurrent cache-miss interleaving and fixes it by synchronously memoizing the in-flight Promise. The retry-on-failure enhancement is useful. Minor imprecision: fetch does not reject merely for HTTP 500 without an e"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c11",
      "name": "Retry with backoff",
      "prompt": "Write an async retry wrapper in TypeScript: exponential backoff with jitter, a maximum attempt count, and it must NOT retry on 4xx responses other than 429. Maximum 30 lines. State what happens on the final failure.",
      "output": "```typescript\nasync function retry<T>(fn: () => Promise<T>, maxAttempts: number): Promise<T> {\n  let lastError: any;\n  for (let i = 0; i < maxAttempts; i++) {\n    try { return await fn(); }\n    catch (e: any) {\n      lastError = e;\n      const s = e?.status ?? e?.response?.status;\n      if (s >= 400 && s < 500 && s !== 429) throw e;\n      if (i === maxAttempts - 1) break;\n      const delay = Math.min(10000, 100 * 2 ** i * (0.5 + Math.random()));\n      await new Promise(r => setTimeout(r, delay));\n    }\n  }\n  throw lastError;\n}\n```\n**Final failure:** The wrapper throws the last encountered error to the caller.",
      "latency_ms": 64238,
      "cost": 0.017421225,
      "task_score": 10,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Correct exponential backoff with jitter, max attempts, skips retry on 4xx except 429, within line limit, clearly states final failure behavior."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly fulfills all requirements, including the constraints on line count, 4xx handling, and stating the final failure behavior. The code is concise, accurate, and highly useful."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 10,
          "violates": false,
          "reasoning": "Implements async retries with bounded exponential backoff and multiplicative jitter, respects maximum attempts, excludes 4xx except 429, and clearly states final failure behavior. It is concise and under 30 lines."
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c12",
      "name": "Recursive type",
      "prompt": "Write a TypeScript type DeepPartial<T> that makes every nested property optional, and explain in one sentence how it must handle arrays differently from plain objects.",
      "output": "```typescript\ntype DeepPartial<T> = T extends Array<infer U>\n  ? Array<DeepPartial<U>>\n  : T extends object\n  ? { [P in keyof T]?: DeepPartial<T[P]> }\n  : T;\n```\n\nArrays must be handled by recursively applying the partial transformation to their element types rather than iterating over their keys, because treating an array like a plain object would incorrectly make built-in properties like `length` and methods like `push` optional, destroying the array's structural integrity.",
      "latency_ms": 49335,
      "cost": 0.0113516,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Correct, concise DeepPartial type with proper array handling explained in one sentence as requested; minor nit: array elements not made optional, but overall solid."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response provides a correct TypeScript implementation of DeepPartial and includes a clear, accurate one-sentence explanation regarding array handling."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 9,
          "violates": false,
          "reasoning": "The implementation correctly recurses through mutable arrays and object properties, and the single-sentence explanation accurately explains why arrays need special handling. Minor limitations include not preserving readonly arrays/tuples an"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c13",
      "name": "Bash one-liner",
      "prompt": "Give me a single shell command that finds the ten largest files under the current directory recursively and prints them human-readable, largest first. It must handle filenames containing spaces. Explain each part briefly.",
      "output": "```bash\nfind . -type f -exec du -h {} + | sort -rh | head -n 10\n```\n\n### Explanation:\n\n*   **`find .`**: Starts a recursive search in the current directory (`.`).\n*   **`-type f`**: Restricts the results to regular files only, ignoring directories.\n*   **`-exec du -h {} +`**: Runs the `du` (disk usage) command on the found files. \n    *   `-h` outputs sizes in a **h**uman-readable format (e.g., K, M, G). \n    *   `{}` is replaced by the found filenames, and `+` batches them into as few `du` executions as possible. This natively and safely handles filenames with spaces or special characters without needing quotes.\n*   **`|`**: Pipes the output to the next command.\n*   **`sort -rh`**: Sorts the output. \n    *   `-h` tells sort to understand **h**uman-readable numbers (so it knows `1G` is larger than `500M`). It only evaluates the first column (the size), so spaces in the filenames don't break the sorting logic. \n    *   `-r` **r**everses the sort order so the largest files appear at the top.\n*   **`head -n 10`**: Truncates the output to only show the top **10** lines.",
      "latency_ms": 47871,
      "cost": 0.010040325,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Correct, working command handling spaces; clear concise explanation of each part. Minor nuance: du -exec with + batches files, sort by size still works fine. Solid, useful answer."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response provides a flawless, optimal shell command that perfectly meets all requirements, including handling spaces and human-readable sorting. The explanation is clear, accurate, and concise."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 9,
          "violates": false,
          "reasoning": "Correctly finds regular files recursively, safely handles spaces via find -exec batching, sorts human-readable du output largest-first, and limits to ten. Minor caveat: du reports disk usage rather than exact apparent file size, and behavio"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c14",
      "name": "Explain and cost",
      "prompt": "Explain what this does and give its time and space complexity, then rewrite it to be O(n).\n\ndef has_dup(xs):\n    for i in range(len(xs)):\n        for j in range(i+1, len(xs)):\n            if xs[i] == xs[j]:\n                return True\n    return False",
      "output": "### What it does\nThis function checks if a list (or any sequence) `xs` contains any **duplicate values**. \nIt does this by comparing every element in the list to every other element that comes after it. If it finds a match, it immediately returns `True`. If it checks all possible pairs without finding a match, it returns `False`.\n\n---\n\n### Complexity of the Original Code\n\n*   **Time Complexity: $O(n^2)$**\n    The function uses nested loops. The outer loop runs $n$ times, and the inner loop runs $n-1, n-2, \\dots, 1$ times. The total number of comparisons in the worst case (no duplicates) is $\\frac{n(n-1)}{2}$, which simplifies to a quadratic time complexity of **$O(n^2)$**.\n*   **Space Complexity: $O(1)$**\n    The function only uses two integer variables (`i` and `j`) to keep track of indices. It does not create any new data structures that scale with the input size, so it uses constant extra space.\n\n---\n\n### The $O(n)$ Rewrite\n\nTo achieve $O(n)$ time complexity, we can use a **Hash Set**. Sets in Python have an average $O(1)$ time complexity for both lookups and insertions.\n\n```python\ndef has_dup(xs):\n    seen = set()\n    for x in xs:\n        if x in seen:\n            return True\n        seen.add(x)\n    return False\n```\n\n#### Complexity of the Rewritten Code:\n*   **Time Complexity: $O(n)$**\n    We iterate through the list exactly once. For each element, checking if it is in the set (`x in seen`) and adding it to the set (`seen.add(x)`) both take $O(1)$ time on average. Therefore, the overall time complexity is **$O(n)$**. *(Note: It also benefits from an \"early exit\", meaning it stops the moment it finds a duplicate, making the best-case time $O(1)$).*\n*   **Space Complexity: $O(n)$**\n    This is a classic **time-space tradeoff**. In the worst-case scenario (where there are no duplicates), the `seen` set will store all $n$ elements from the list, requiring **$O(n)$** extra memory.\n\n---\n\n### Alternative \"Pythonic\" One-Liner\nYou will often see this written in Python as a one-liner:\n\n```python\ndef has_dup(xs):\n    return len(xs) != len(set(xs))\n```\n**Why the `for` loop version is generally better:** \nWhile the one-liner is shorter and also $O(n)$ time and space, `set(xs)` forces Python to process the *entire* list into a set before checking the length. The `for` loop version above is more efficient in practice because it **short-circuits** (returns `True` immediately) the moment it finds the first duplicate, without processing the rest of the list.",
      "latency_ms": 34801,
      "cost": 0.008083,
      "task_score": 9,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 9,
          "violates": false,
          "reasoning": "Accurate explanation, correct complexities, correct O(n) rewrite with bonus alternative. Clear, thorough, useful, well-organized."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly explains the function, accurately states the original time and space complexity, and provides a correct O(n) rewrite with a clear explanation. It is highly useful and well-structured."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 8,
          "violates": false,
          "reasoning": "Accurately explains duplicate detection, quadratic worst-case time, constant auxiliary space, and provides an expected O(n)-time set-based rewrite with O(n) space. Minor caveat: the rewrite requires hashable elements, unlike the original."
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c15",
      "name": "Regex with limits",
      "prompt": "Write a regex that validates UK postcodes, give a one-line explanation of each part, and then state two valid UK postcodes your regex would reject or two invalid ones it would accept. Do not claim it is perfect.",
      "output": "**Regex:**\n`^[A-Za-z]{1,2}[0-9][A-Za-z0-9]?\\s?[0-9][A-Za-z]{2}$`\n\n**Part-by-part explanation:**\n* `^` : Asserts the start of the string to prevent matching leading invalid characters.\n* `[A-Za-z]{1,2}` : Matches one or two letters representing the outward postcode area.\n* `[0-9]` : Matches the single mandatory digit representing the outward postcode district.\n* `[A-Za-z0-9]?` : Optionally matches a letter or digit to complete the outward code.\n* `\\s?` : Optionally matches a single whitespace character separating the outward and inward codes.\n* `[0-9]` : Matches the single mandatory digit representing the inward postcode sector.\n* `[A-Za-z]{2}` : Matches exactly two letters representing the inward postcode unit.\n* `$` : Asserts the end of the string to prevent matching trailing invalid characters.\n\n**Two invalid UK postcodes this regex would incorrectly accept:**\n1. **`ZZ1 1AA`** - Invalid because the letter 'Z' is never permitted as the first character in a real UK postcode, but this regex allows any letter.\n2. **`AB1 1CM`** - Invalid because the letters 'C' and 'M' (along with I, K, O, and V) are officially excluded from the inward code to prevent OCR/reading confusion with numbers, but this regex accepts any letters in those positions.\n\n*Disclaimer: As requested, I do not claim this regex is perfect. It is a simplified structural check and does not enforce the strict, position-specific letter restrictions defined in the official BS7666 standard, nor does it account for the unique, non-standard `GIR 0AA` postcode.*",
      "latency_ms": 78579,
      "cost": 0.0183077,
      "task_score": 8,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Solid regex, clear per-part explanation, correctly gives two invalid postcodes it would accept, and disclaims imperfection as required. Minor: didn't also give example of valid rejected, but task allows either type."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly follows all instructions, providing a functional regex, accurate one-line explanations for each part, and correctly identifying two invalid postcodes the regex would accept. It explicitly avoids claiming perfection."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 8,
          "violates": false,
          "reasoning": "It fulfills the requested regex, per-part explanations, and two genuinely invalid examples. However, its rationale for ZZ1 1AA is factually wrong: Z can begin real postcodes (for example, ZE). The regex is otherwise a useful simplified stru"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c16",
      "name": "Callback to async",
      "prompt": "Refactor this to async/await with correct error propagation. Errors must not be swallowed.\n\ngetUser(id, (e, user) => {\n  if (e) return cb(e);\n  getOrders(user.id, (e2, orders) => {\n    if (e2) return cb(e2);\n    getTotals(orders, (e3, totals) => cb(e3, totals));\n  });\n});",
      "output": "Here's the refactored version using `async/await` with proper error propagation:\n\n```js\nconst { promisify } = require('util');\n\nconst getUserAsync = promisify(getUser);\nconst getOrdersAsync = promisify(getOrders);\nconst getTotalsAsync = promisify(getTotals);\n\nasync function getUserTotals(id) {\n  const user = await getUserAsync(id);\n  const orders = await getOrdersAsync(user.id);\n  const totals = await getTotalsAsync(orders);\n  return totals;\n}\n```\n\n**Why this is correct:**\n\n- **No swallowed errors** — There is no `try/catch` block. If any `await` rejects, the error propagates up and `getUserTotals` returns a rejected promise. The caller handles it:\n  ```js\n  try {\n    const totals = await getUserTotals(id);\n  } catch (err) {\n    // handle err\n  }\n  ```\n- **`promisify`** converts Node-style `(err, result)` callbacks into proper promises, so rejections carry the original error objects faithfully.\n- **Sequential execution** is preserved — each step waits for the previous one, matching the original nesting.\n\n**If you can't use `promisify`** (e.g., browser environment), wrap manually:\n\n```js\nfunction toPromise(fn, ...args) {\n  return new Promise((resolve, reject) => {\n    fn(...args, (err, result) => {\n      if (err) reject(err);\n      else resolve(result);\n    });\n  });\n}\n\nasync function getUserTotals(id) {\n  const user = await toPromise(getUser, id);\n  const orders = await toPromise(getOrders, user.id);\n  const totals = await toPromise(getTotals, orders);\n  return totals;\n}\n```\n\nBoth versions guarantee that an error at **any** step (`getUser`, `getOrders`, or `getTotals`) immediately aborts the chain and surfaces to the caller — nothing is silently ignored.",
      "latency_ms": 17819,
      "cost": 0.0036285,
      "task_score": 10,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Correct, thorough, well-explained refactor with proper error propagation and fallback option; slightly verbose for a busy professional but clear and accurate."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response perfectly refactors the callback hell into async/await, correctly handles error propagation by letting promises reject naturally, and provides a useful alternative for non-Node environments. Clear and concise."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 10,
          "violates": false,
          "reasoning": "Correctly promisifies all Node-style callbacks and uses sequential awaits. Rejections from any operation propagate through the async function without catch-and-swallow behavior. The manual Promise alternative is also correct and preserves o"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c17",
      "name": "Diagnose from a trace",
      "prompt": "Given this Node stack trace, state the most likely root cause and the first thing you would check:\n\nTypeError: Cannot read properties of undefined (reading 'map')\n    at renderRows (/app/src/table.js:42:19)\n    at Table (/app/src/table.js:12:5)\n    at renderWithHooks (/app/node_modules/react-dom/cjs/react-dom.development.js:16305:18)\n\nThe component works in dev and fails only on the production build's first paint.",
      "output": "## Most Likely Root Cause\n\n**Unguarded `.map()` on async data that is `undefined` on initial render.**\n\nThe `Table` component calls `.map()` on a data array (likely a prop like `rows` or a state variable) that hasn't been populated yet. On the **production first paint** (cold load), React renders the component *before* any data fetching resolves, so the variable is still `undefined`.\n\n### Why it works in dev but not production\n\nIn development, one or more of these typically masks the bug:\n\n- **HMR / state preservation** — navigating during dev often means the data is already cached in memory from a previous render cycle.\n- **React Strict Mode double-render** — can inadvertently allow a second render after data arrives, hiding the broken first render.\n- **Dev tools / cached API responses** — the browser may serve a cached response instantly in dev, making the race condition invisible.\n\nOn a production cold load, none of these safety nets exist. The very first render fires with `undefined` data and crashes.\n\n---\n\n## First Thing to Check\n\n**Open `/app/src/table.js` at line 42** and identify the variable being `.map()`'d. You'll almost certainly find something like:\n\n```js\n// line ~42\nreturn rows.map(row => <tr>...</tr>);  // rows is undefined\n```\n\nThen apply one of these fixes:\n\n```js\n// Option A: Default to empty array\nconst rows = data ?? [];\nreturn rows.map(row => ...);\n\n// Option B: Optional chaining\nreturn rows?.map(row => ...);\n\n// Option C: Early return guard\nif (!rows) return null; // or a loading skeleton\nreturn rows.map(row => ...);\n```\n\nAlso verify the **prop/state initialization** at the top of the `Table` component (line 12) to ensure the default value is `[]` and not `undefined`:\n\n```js\nfunction Table({ rows = [] }) { ... }  // ✅\nfunction Table({ rows }) { ... }        // ❌ rows is undefined if not passed\n```",
      "latency_ms": 21770,
      "cost": 0.004842425,
      "task_score": 8,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Accurate, plausible root cause and clear actionable checks; slightly verbose but well organized and correct technically."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 10,
          "violates": false,
          "reasoning": "The response accurately identifies the root cause (unguarded map on undefined data during initial render) and provides the exact first step to check (line 42 in table.js), along with clear, actionable fixes."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 8,
          "violates": false,
          "reasoning": "Correctly identifies an undefined collection on initial render and appropriately directs checking line 42 and initialization. The dev-versus-production explanations are somewhat speculative; Strict Mode double rendering does not reliably hi"
        }
      ],
      "machine_checks": null,
      "machine_violation": false,
      "machine_reasons": []
    },
    {
      "id": "c18",
      "name": "Idempotency",
      "prompt": "Design an idempotent POST /payments endpoint so a client retry cannot charge twice. Describe the key, where it is stored, what happens on a concurrent duplicate, and what you return the second time. Maximum 200 words. Name the failure mode your design still has.",
      "output": "**Key:** A client-generated UUID passed in the `Idempotency-Key` header.\n\n**Storage:** A relational database table with a `UNIQUE` constraint on the key, storing the request payload and final response.\n\n**Concurrent Duplicate:** The second request attempts an `INSERT`. The `UNIQUE` constraint forces it to wait on the row-level lock until the first transaction commits. It then reads the existing record instead of charging again.\n\n**Second Return:** The server returns the exact cached HTTP status code (e.g., `201 Created`) and response body (the payment receipt) from the original successful request.\n\n**Failure Mode:** *Semantic Key Reuse.* If a buggy client reuses the same `Idempotency-Key` for a logically different payment (e.g., a different amount or cart), the server will blindly return the original cached success response without processing the new transaction, causing a silent failure.",
      "latency_ms": 74967,
      "cost": 0.01835195,
      "task_score": 8,
      "judges": [
        {
          "judge": "anthropic/claude-sonnet-5",
          "score": 8,
          "violates": false,
          "reasoning": "Accurate, clear, concise design covering key, storage, concurrency, response, and failure mode; under 200 words. Minor: lock mechanics slightly oversimplified."
        },
        {
          "judge": "google/gemini-3.1-pro-preview",
          "score": 8,
          "violates": false,
          "reasoning": "Follows all instructions and constraints. The explanation of concurrent duplicates relying purely on a UNIQUE constraint to 'wait and read' glosses over catching the constraint violation exception, but the overall design is solid."
        },
        {
          "judge": "openai/gpt-5.6-terra",
          "score": 8,
          "violates": false,
          "reasoning": "Covers every requested element accurately and concisely. The uniqueness/locking description is broadly sound. Minor omission: it does not address in-progress responses or the crash window around an external payment charge; payload mismatch "
        }
      ],
      "machine_checks": {
        "max_words": 200
      },
      "machine_violation": false,
      "machine_reasons": []
    }
  ],
  "recapped_at": "2026-08-16T08:16:35.437Z",
  "recap_note": "Machine constraint caps applied retroactively under protocol judge-2026-08b; judge verdicts unchanged, task scores capped where a deterministic check failed."
}