A team of 24 researchers, including Arvind Narayanan and Sayash Kapoor (known for the AI Snake Oil project) and collaborators from labs including METR-adjacent and academic groups, ran an unusually direct test of a big claim in the AI industry: that AI agents are close to automating AI research itself. Instead of using narrow, easily-graded benchmarks or submitting AI-written papers to peer review (both of which have known flaws), they invented what they call "shadow evaluations." They took two real, unpublished NeurIPS 2026 submissions, gave frontier AI agents six days and thousands of dollars of compute to independently pursue the same open-ended research question the human authors were working on, and then had the original authors grade the agent's output using their own expert judgment. The result: the agents executed all the engineering work competently and without help, but neither agent produced work that cleared the bar for a publishable contribution, and both would-be papers were unambiguously rejected. The paper identifies five recurring failure patterns: poor judgment about what actually counts as a publishable result, an inability to creatively revise a research design when it hits an obvious flaw, weak backtracking out of dead ends, poor awareness of compute/time budgets, and "instruction drift" where the agent slowly wanders from the original goal over a long task. For developers building or evaluating coding and research agents, this is a useful data point precisely because it isolates engineering competence from judgment: an agent can be a completely capable engineer — writing code, running experiments, iterating — while still lacking the taste and self-correction skill to know whether the work is actually good. That distinction matters directly for anyone building long-horizon autonomous agents for their own domain, since it suggests the harder problem to solve isn't task execution but building in mechanisms for the agent to recognize when it's stuck, wrong, or aimed at the wrong target.