Recommendation: start with bounded repository tasks that have explicit acceptance criteria, reproducible checks, and a reviewable change boundary. Treat agent capability demonstrations as evidence of what may be possible, not as evidence of improved end-to-end delivery. Require isolation for tool execution, retain human approval for consequential actions, and measure outcomes through review, rework, stability, and delivery performance.
This living whitepaper brings together task selection, repository context, execution environments, testing, review, security, and measurement. It distinguishes documented product behaviour, benchmark results, causal research, observational evidence, and practical engineering recommendations.
Start with work your team can assess
Choose an initial task with a clear expected behaviour and a manageable review surface. A narrowly scoped defect, a documented interface change, or a small migration provides a more useful trial than an open instruction to improve an entire application. Record the acceptance criteria before the agent starts.
Coding agents are now being positioned for repository maintenance, incremental features, test development, and asynchronous pull-request work. GitHub’s current application card describes these capabilities alongside limitations including missed problems, false positives, insecure suggestions, limited language coverage, and command execution risk. This is documented product behaviour, not independent evidence that the resulting changes are ready to merge. See GitHub’s Copilot Agents application card, current page consulted 22 September 2026.
Select representative work, including the awkward parts of your repository. Include existing conventions, incomplete documentation, integration boundaries, and tests that are slow or difficult to run. A trial made entirely of isolated examples will leave unanswered questions about maintenance and review.
Make the development environment reproducible
Give the agent the same documented setup and verification commands that a new engineer would need. Specify where implementation rules live, which generated files should be left alone, and how the relevant test suite runs. A failed environment setup should be reported as a failed setup, rather than hidden inside a claim that a change was tested.
Execution boundaries now deserve explicit treatment. GitHub’s sandbox documentation describes local sandboxing as experimental and cloud sandboxing as a public preview. It also states that, without local sandboxing, shell commands can run with the user’s filesystem, network, and credential access. Local sandboxing restricts these areas through policy, but its isolation is lighter than a virtual machine or container, and remote MCP servers are not sandboxed. See GitHub’s cloud and local sandbox documentation, current page consulted 22 September 2026.
For an initial trial, use an isolated checkout, non-production test data, short-lived credentials, and a network policy appropriate to the task. Separate access needed to develop the change from authority to release it. Require an explicit maintainer decision before changing public interfaces, adding runtime dependencies, modifying deployment configuration, or accessing production systems.
Keep the agent’s task and evidence together
A useful handover includes the requested behaviour, the changed files, the checks that ran, and remaining uncertainty. Ask the agent to distinguish commands it executed from checks it merely recommends. The reviewer should be able to reproduce the relevant evidence without reading a full conversation.
Anthropic’s guidance, published 19 December 2024, distinguishes workflows with predefined code paths from agents that dynamically choose processes and tool use. It recommends starting with the simplest design and adding agentic complexity only when flexibility is necessary. The page also warns that its tooling examples have changed since publication, so use the architectural distinction and decision guidance rather than treating the named tools as current recommendations. See Building effective agents.
For repository work, start with a simple sequence such as inspect, plan, change, verify, and review. Add delegation or parallel work only when the team can observe the resulting state and recover from partial completion. Keep session logs, tool calls, test output, and commit history attached to the change where possible.
Define expected behaviour before reading the patch
Write down what must become true, what must remain true, and which failure cases matter. For a change to access control, that includes requests that should be rejected as well as requests that should succeed. For a data transformation, it includes missing, malformed, and boundary inputs.
Use those expectations to review both the implementation and its tests. If the same misunderstanding appears in both, a passing test suite may only confirm internal consistency. Derive at least some checks from requirements, known defects, invariants, or externally observable behaviour instead of asking for tests based solely on the generated implementation.
Keep the examples specific. A regression test should show the reported failure before the fix and the expected behaviour after it. That makes its purpose clear to future maintainers, whether a person or an agent wrote it.
The evidence base now includes capability results that are more ambitious than ordinary pull-request tasks. METR’s April 10, 2026 MirrorCode update reports preliminary results in which agents completed some weeks-long coding tasks, including reimplementing a 16,000-line codebase. The work concerns a specific benchmark and software-reproduction task type, so it should not be treated as representative evidence for general software delivery. See METR’s MirrorCode update.
This strengthens the case for testing task boundaries rather than assuming that agents are limited to small edits. It does not remove the need for requirements, independent checks, and maintainability review.
Give automated review a defined role
An automated reviewer can provide another set of observations, but the team still needs to decide which findings are valid. GitHub documents both missed problems and false positives in Copilot code review, and notes that generated fixes may be inaccurate or insecure. Require findings to identify the affected behaviour and explain how the problem can occur.
Ask automated review to investigate specific risks, such as inconsistent authorisation checks, changed error paths, unsafe deserialisation, missing transaction handling, or compatibility regressions. A long list of stylistic suggestions should not obscure a short list of defects.
Measure the work created by the reviewer as well as the defects it finds. Record accepted findings, rejected findings, duplicate findings, and changes made in response. This helps distinguish useful detection from comment volume.
Check the whole change at the relevant boundaries
Choose verification based on the effect of the change. An interface adjustment may need a consumer check; a schema migration may need compatibility and rollback exercises. A visual change needs inspection in the rendered interface. Running an unrelated unit test suite should not be described as validation of all these behaviours.
Inspect changes to the checks themselves. A removed assertion or weakened configuration can make a build pass while reducing the assurance it provides. Ask whether each test change follows from an intentional requirement change.
Security review should include the agent’s instructions, repository context, tools, extensions, and external services. OWASP’s 2025 Top 10 lists Prompt Injection as LLM01 and Excessive Agency as LLM06. The project is a risk and mitigation framework, not a software delivery standard. Its Excessive Agency material specifically connects risk to the systems and actions an LLM application can reach. See the OWASP GenAI Security Project’s 2025 Top 10, 2025 edition.
Ask whether untrusted issue text, documentation, test fixtures, source comments, or dependency metadata can influence actions outside the intended task. Limit write access, network reach, credential availability, and tool selection accordingly.
Read productivity claims in their original setting
A study’s task, participants, tools, date, and definition of success determine what its result can tell you. METR’s randomised controlled trial used data from February to June 2025, involved 16 experienced open-source developers completing 246 tasks, and primarily tested Cursor Pro with Claude 3.5 and 3.7 Sonnet. It reported that allowing AI increased completion time by 19% in that setting. See the METR study, published 2025.
The follow-up, published 24 February 2026, says its later experiment was an unreliable signal because developers who did not want to work without AI were less likely to participate or submit certain tasks. Concurrent use of multiple agents also made time measurement unreliable. METR concluded that the later data was only weak evidence about the size of any increase in speed. See METR’s experiment-design update.
Later capability evidence does not resolve this productivity question. METR’s May 8, 2026 analysis distinguishes uplift on old tasks, uplift on new tasks, and uplift in value. Agents may make previously unattractive work feasible, changing the task mix rather than simply reducing the time needed for the same work. See Task Substitution and Uplift.
Keep the dates, tested tools, and limitations attached to every number. Treat benchmark capability and self-reported speed as hypotheses to test on your own corpus.
Follow a change through the delivery process
Choose a small set of outcomes before the trial begins. Record elapsed time from a task becoming ready to acceptance, time spent in review, size of the change, rework, rollback or incident activity, and whether the change required follow-up fixes. Keep the boundaries consistent so an apparent improvement is not simply a change in what gets counted.
DORA’s 2025 State of AI-assisted Software Development report describes AI as an amplifier of existing organisational strengths and weaknesses. Its associated measurement guidance recommends retaining established frameworks while adding measures such as AI suggestion acceptance, model quality, trust, and review time where relevant. The report is survey and observational evidence, not a causal estimate of what an agent will do in a particular team. See DORA’s 2025 research and its measurement guidance, 2025.
Use delivery throughput and instability measures alongside local process measures. A faster implementation stage can coexist with larger changes, slower review, more rework, or weaker operational outcomes. Pair every speed measure with a quality or stability measure, and inspect batch size as an explicit control.
What we are watching
We are watching independent evidence on sustained agent work in established repositories, especially tasks involving ambiguous requirements, cross-service changes, migrations, and operational constraints. We are also watching whether longer-horizon benchmarks predict useful production work, or mainly measure narrow task classes such as software reproduction.
Further priorities are measurement designs that remain credible when developers select AI-suitable tasks, use multiple agents concurrently, or shift towards work that was previously uneconomic. Security work on prompt injection, excessive agency, MCP and extension trust, and sandbox escape paths remains directly relevant to coding-agent adoption.
For your next trial, define a bounded task, prepare a reproducible and isolated environment, give the agent only the permissions it needs, choose checks that can expose shared misunderstandings, and measure the work through review and delivery. Expand the scope only when evidence from your own team supports it.