The Methods Section Omits the Hard Part

A published methods section tells you what worked. It does not tell you what failed, why the protocol changed, or which parameter decisions took three months to validate. That knowledge is the hard part. And it lives nowhere a new researcher can find it.

There is an unspoken convention in scientific publishing: the methods section describes a process that was never quite as clean as it appears on the page.

The protocol you publish is the final version, written after the experiments succeeded. It omits the six failed convergence runs that preceded the working ENCUT setting. It omits the instrument calibration quirk that every researcher in the group knows about but no one wrote down. It omits the reason you switched from PBE to PBE+D3 for this material class specifically — the three papers that informed that decision, the four test calculations that confirmed it.

This is not a failure of scientific honesty. It is a structural feature of scientific communication. Papers are optimized for peer review, not for the next graduate student joining your group in September.

The methods section tells you what worked. The lab's institutional memory tells you how you figured that out. Those are different documents, and only one of them gets published.

The Published Method and the Practiced Method

Consider a typical VASP calculation in a computational materials lab. The published methods section will list: exchange-correlation functional (PBE), plane-wave cutoff (520 eV), k-point sampling (8×8×8 Monkhorst-Pack), PAW pseudopotentials. Peer reviewers nod. Editors accept.

What the methods section does not contain:

None of this is in any paper. All of it is essential. And it belongs to whoever is finishing their PhD this spring.

Why This Is Worse for Computational Labs

Experimental labs have some natural documentation forcing functions. A failed synthesis attempt produces a lab notebook entry. An instrument that breaks creates a service record. Protocol changes get captured in SOPs, at least for regulated environments.

Computational labs have no equivalent. SLURM does not log the reasoning behind the calculation. Git logs the code change, not why the change was made. A Jupyter notebook captures what was computed, not what was tried first or why the current approach was chosen over alternatives.

The knowledge that makes a computational research group productive — the parameter decisions, the method choices, the failure history, the domain-specific quirks — accumulates entirely in human memory. It transfers by proximity: the new student who sits next to the experienced postdoc, who asks the right questions before the right person graduates.

That transfer mechanism fails at scale. It fails when the postdoc leaves for industry in August and the new student arrives in September. It fails when the lab grows past six people. It fails when the question isn't asked because the person who would know the answer has already left.

The Replication Problem Is a Knowledge Problem

There is a large literature on the replication crisis in science. Most of it focuses on statistical issues: p-hacking, underpowered studies, researcher degrees of freedom. These are real problems. But there is a second replication failure mode that receives less attention.

When a lab fails to replicate its own results after key personnel change, or when an outside group fails to replicate a published result despite following the published methods carefully — the problem is often not statistics. It is that the published methods do not actually describe the method that was used.

Not because of deception. Because the full method is too granular, too contextual, and too tacit to fit in a methods section written for a general audience. The published method is a compression artifact. The real method is a living practice inside a specific lab, maintained by specific people, that degrades when those people leave.

Computational reproducibility is not just about sharing code. It is about capturing the reasoning layer that explains why the code is the way it is.

What Gets Left Behind

When a PhD student defends their dissertation, the lab typically receives:

• What they deposited: A thesis. Published papers. Code repositories (sometimes). A brief knowledge transfer meeting, if they remember to schedule it.
• What they take with them: Every parameter decision they made and why. Every failed approach. Every domain-specific heuristic they developed. Every piece of tribal knowledge they acquired from the postdoc before them.

The next student to work on the same project will reconstruct some of this through trial and error. Some they will ask about and receive partial answers by email. Some they will never figure out, and the lab will quietly move in a slightly wrong direction for two years before someone notices.

This is not exceptional. It is the ordinary state of knowledge management in most research groups. The methods section was never designed to prevent it.

The Documentation Instinct Gets This Wrong

The natural response to this problem is a documentation mandate: require students to write things down. Keep a group wiki. Maintain lab notebooks in Notion. Write README files for every codebase.

These instincts are correct about the problem and wrong about the solution. Documentation as a practice fails in research environments for a consistent set of reasons: the work changes faster than the documentation, the incentive to document is low relative to the incentive to do science, and documentation written for a general audience captures the what but not the why.

A README that says "use ENCUT=520" is incomplete. The README that explains why — including the failed alternatives, the specific material classes where this breaks down, and the two papers that informed the choice — is the document that actually prevents knowledge loss. Writing that document requires memory of the exploration process, not just the conclusion.

The problem is not that researchers fail to document. It is that by the time documentation seems necessary, most of the context needed to write it well has already left the building.

What Capturing the Methods-as-Practiced Actually Requires

The gap between published methods and practiced methods is not closable by a wiki or a notebook convention. Closing it requires capturing knowledge at the moment it is created — in the workflow, not after the fact.

When a researcher adjusts ENCUT and reruns a convergence test, the reasoning behind that adjustment needs to be captured then, not reconstructed in a documentation sprint six months later. When a VASP job fails and the fix is an NPAR setting that another researcher discovered last year, that fix and its history need to be findable by anyone in the group, not buried in a Slack thread.

This is what ResearchOS is designed to do. Not replace the methods section — but build the layer beneath it that the methods section was never designed to capture. The lab's accumulated reasoning, parameter history, failure archive, and domain-specific knowledge, indexed and queryable by the next researcher who needs it.

The methods section will always describe what worked. The gap it leaves — what was tried, what failed, why the current approach was chosen — is the knowledge that makes a lab productive. That knowledge deserves its own infrastructure.