Managing Research Projects

A practical system for getting research finished

John M. Drake

Odum School of Ecology & Center for the Ecology of Infectious Diseases

CEID Professional Development Workshop

Center for the Ecology of Infectious Diseases, Athens, GA

October 9, 2026

Every lab has a graveyard of projects that have been 80% done for five years.

In 2018 Andrew Tredennick, then a postdoc in my lab, wrote a protocol for a project with Eamon O’Dea, Pej Rohani, and me
The project: do early-warning signals of critical transitions work for real measles transmission? — weekly case reports, 1995–2005, from four cities in Niger (Agadez, Maradi, Niamey, Zinder)
The protocol fixed the question, the data, the analysis, the anticipated figures, and a checklist; then it was kept current as the work moved, with a changelog recording every change of plan
That project got finished: Tredennick et al. (2022), J R Soc Interface, six authors on the paper, adding Matt Ferrari and Andrew Park
Every lab — including mine — also keeps a graveyard of projects 80% done for years
Today is about the difference between the two
Conclusion: A written plan at the start, kept current in writing, is what separates finished projects from the graveyard. [2 min — ends 0:02]

Projects fail from entropy, not stupidity

Undefined endpoints — nobody said what “done” is
Mission creep — the question moved after the data arrived
Entropy — files, decisions, and intentions decay

Everything today is a cheap, mechanical defense against all three failure modes.

Roadmap

What a project is (and isn’t)
Choosing the project
The protocol — why written plans change outcomes
Structure: repositories, folders, reproducible code
Git and GitHub, the concept
Driving to done
You draft a protocol + discussion

What a project is

What are you working on?

A project is the work aimed at one product

Project = all the documents, data, and code aimed at one product
Product = something you can deliver: a paper, a chapter, a package, a data archive
One project = one product = one repository

If you can’t write a plan for it, it isn’t a project

A dissertation is not a project; each chapter is
Peer reviews, proposals, and presentations are not projects
An NSF or NIH grant is not a project either — it’s an umbrella funding mechanism that pays for several
A project is over when the product is delivered — the second paper from the same data is a new project, not scope growth

Choosing the project

The highest-leverage decision you will make

The stakes: only about 57% of PhD students finish within ten years

~57% ten-year completion (~63% in the life sciences), students entering 1992–95 at ~30 universities
Project choice is the highest-leverage, least-taught decision in a research career
Time spent choosing — weeks, even months — is not procrastination

Source: Council of Graduate Schools, PhD Completion Project (2008). Alon (2009): “Do not commit to a problem before 3 months have elapsed.”

The source: the Council of Graduate Schools’ PhD Completion Project, a multi-year study of program records at about 30 mostly research universities in the US and Canada; the 10-year figure follows the cohorts that entered 1992–93 through 1994–95 (about 12,000 students in ~300 programs) — 56.6% had finished by year ten
Life sciences, our neighborhood, did better at ~63%; humanities worst at ~49%
The completion number is about programs, not project scope — be honest about that
But it sets the stakes: the modal failure of a research career is non-completion, and the cheapest insurance is choosing well
Uri Alon runs a systems-biology lab at the Weizmann Institute; his 2009 Molecular Cell essay “How to choose a good scientific problem” is the classic on this; his lab rule, verbatim, for new students and postdocs: three months of reading, discussing, and planning before committing
Conclusion: The choice of problem is the highest-leverage decision you make, so spend real time on it. [2 min — ends 0:12]

Plot every candidate project on two axes

“If you do not work on an important problem, it’s unlikely you’ll do important work.”

— Richard Hamming, “You and Your Research” (1986)

Candidate projects plotted by feasibility and interest (gain in knowledge); adapted from the same two axes in Fig. 1 of Alon (2009), Mol Cell 35:726.

Who Hamming was: Richard Hamming (1915–1998), mathematician; worked on the Manhattan Project, then thirty years at Bell Labs, where he invented the error-correcting Hamming codes; Turing Award 1968
“You and Your Research” was a talk to about 200 staff at Bell Communications Research (Bellcore), March 7, 1986 — his answer to why some of his Bell Labs colleagues did Nobel-level work and most didn’t; the transcript is free online and worth assigning
Hamming sets the vertical axis: interest is non-negotiable — an unimportant problem done easily is still unimportant work
Personal history: I drew this around 2011 thinking I’d invented it; Alon published the same two axes in 2009 — independent invention, so these principles are discoverable, not folklore
Alon’s version: x-axis hard → easy, y-axis small → large gain in knowledge, with candidate problems along a Pareto front; the quadrant labels and the arrow here are mine
Define the axes: feasibility is data + skills + time; interest is strictly gain in knowledge, not fashionability
Alon’s own caution: “problems that are easy on paper are often hard in reality, and problems that are hard on paper are nearly impossible in reality”
Walking the arrow: it’s the whole argument — the same knowledge gain, reached by an easier route
Conclusion: Judge every candidate on feasibility and knowledge gain together, never on ambition alone. [3 min — ends 0:15]

Where I part ways with Alon: everyone works easy/high-gain

Alon: first problems are easy and modest; postdocs need feasible and high-gain; new PIs may take on a hard grand challenge — if it splits into tractable projects
My advice: difficulty is not a virtue — it is a cost that discounts everything downstream
For almost every hard problem there is an easier project that teaches the same lesson — find the easier route before you commit

Alon stages the diagram by career: a beginning student’s first problem should be easy so feedback comes fast; the second can move up the interest axis; postdocs need top-right projects because time is short; beginning PIs may choose a grand challenge that divides into many smaller good projects
So the disagreement is narrower than it sounds: I take Alon’s postdoc rule and apply it to everyone, students and PIs included
Say it plainly and invite disagreement at the discussion stop
The idea: if one project answers the same question with less difficulty, the hard version has nothing left to offer
Where easier versions come from: a different study system, an archival dataset instead of a new deployment, a model instead of an infeasible experiment, a sharp special case instead of the general problem
Why people miss them: difficulty is conspicuous and gets mistaken for value; people anchor on the first formulation of the question — and it’s the formulation that’s hard, not the question
The in-house example: “do early-warning signals work for real diseases?” The hard way = build real-time surveillance and wait years for elimination and re-emergence to happen
Tredennick’s route: fit transmission models to archival measles data from four Nigerien cities, then run the experiment in simulation — re-emergence by depleting population immunity, elimination by ramping vaccination from 70% to 100%
Honest about the trade: the paper tests EWS in empirically grounded simulations, not live on real surveillance data, and says so — most of the knowledge at a fraction of the cost
Honest caveat: if a genuine search turns up no easier version, the hard problem may deserve you (Hamming: a reasonable attack)
Conclusion: Choose the most feasible project that still delivers the knowledge you’re after — difficulty earns no credit. [3 min — ends 0:18]

Most published science is conservative — for a reason

Across ~6.5 million biomedical abstracts, 86% of the chemical relationships reported were already known; 14% were new
Papers introducing a new chemical were cited more — 12.9 citations on average versus 8.4 — but a project that fails produces no paper at all, and by the authors’ calculation the extra citations don’t make up for that risk
Highly novel papers had ~57% higher odds of a top-1% hit — and fatter tails at both ends
In a field experiment, expert reviewers gave the most novel proposals markedly lower scores

Sources: Foster, Rzhetsky & Evans (2015) Am Sociol Rev 80:875; Wang, Veugelers & Stephan (2017) Res Policy 46:1416; Boudreau et al. (2016) Manag Sci 62:2765.

Foster, Rzhetsky & Evans — sociologists of science (Foster now at UCLA, Evans at Chicago): turned 6.46M MEDLINE abstracts (1934–2008) into a network whose links are chemical pairs mentioned together; each new statement either repeats a known link (85.8%) or adds a new one (14.2%)
Unit warning: those percentages count relationship statements, not papers
Citations by paper strategy: 8.4 for papers that only repeat known links, 11.0 for papers adding at least one new link, 12.9 for papers introducing a new chemical
Their conclusion: they can’t observe failure rates directly, so they calculate the break-even — unless innovative projects fail only ~25–35% more often than conventional ones, which they think unlikely, the citation premium doesn’t cover the risk of never publishing
Exception: papers by winners of 137 major biomedical and chemistry prizes use the risky strategies more often — but that’s survivorship; the busts are invisible
Wang, Veugelers & Stephan — economists (KU Leuven; Paula Stephan at Georgia State): every Web of Science article from 2001, ~660,000 papers; “novel” = cites a pair of journals never cited together before, weighted by how distant the journals are; 89% of papers make no new pairing
Highly novel papers (top 1% of novelty in their field): ~57% higher odds of a top-1% citation hit over 15 years (the earlier working paper said ~40%); both tails fatten — about +18% odds of landing in the top 10% and +15% of the bottom 10%
And the payoff is slow: novel papers look worse for the first three years, pull ahead only from year four, sit in lower-impact journals than comparable papers, and draw their extra citations from other fields, not their home field
Boudreau et al. (Harvard Business School and Harvard Medical School): 150 short research proposals in a medical-school seed-grant competition on endocrine disease, 142 expert evaluators, each randomly assigned 15 proposals
What was randomized was who reviewed what; novelty itself was measured (share of never-before-paired MeSH keywords), so the novelty finding is a careful regression, not an experiment
Shape: scores rose slightly with modest novelty, then fell for the most novel fifth — the penalty is at the frontier
Their randomized finding: reviewers closest to a proposal’s topic scored it lower — experts are harsher judges of their own field
The through-line: risky choice is good for science and bad, on average, for you — within a thesis or tenure clock
Conclusion: On any timeline that matters to you, the ambitious-risky project is the worse bet — the field rewards risk, your career doesn’t. [3 min — ends 0:21]

The highest-impact papers are conventional work with one unusual combination

17.9 million papers, scored by how usual or unusual each pair of cited journals is
Papers that were mostly conventional but included one rare combination reached the top 5% of citations 9.1% of the time — versus 5.8% for purely conventional papers and 5.3% for purely novel ones
Only 7% of papers achieve that combination
Teams of three or more included an unusual combination in half their papers; solo authors in a third

Source: Uzzi, Mukherjee, Stringer & Jones (2013) Science 342:468.

Uzzi et al. — Brian Uzzi and Ben Jones at Northwestern’s Kellogg School: all 17.9M research articles in Web of Science; for every pair of journals in a paper’s reference list, compare how often that pair is co-cited against a randomized citation network
Each paper gets two scores: its median pairing (how conventional is the body of the paper) and its most unusual pairings (does it contain at least one rare combination)
The hit-rate figure uses papers from 1990–2000; a “hit” is the top 5% of citations after eight years, so the baseline is 5%
Four cells: conventional + one rare = 9.1%; conventional only 5.8%; rare only 5.3%; neither 2.1%
Reading: impact comes from one unusual connection embedded in otherwise established ground
Teams: one, two, and three-plus authors included a rare combination 36%, 40%, and 50% of the time — teams get the novelty in without losing conventionality
My gloss, not their finding: in project terms, put the novelty in the question and keep the methods you already command — their measure is reference lists, so it can’t tell question from method
Conclusion: Put the novelty in the question and keep the methods familiar — that’s my reading of where the hits come from. [2 min — ends 0:23]

Donald Stokes and Pasteur’s Quadrant

Donald E. Stokes (1927–1997) — political scientist; co-author of The American Voter (1960); dean of Princeton’s Woodrow Wilson School, 1974–1992
Pasteur’s Quadrant (Brookings, 1997) was published shortly after his death
His target: the old linear model — basic research upstream, applied downstream, purity and usefulness as opposite ends of one axis
Stokes split that into two independent questions — and the quadrant falls out

Point the project with Pasteur

Stokes (1997): two axes, not one. Use-inspired basic research — Pasteur’s quadrant — is where understanding and consequence advance together.

A different question from the first diagram: not “can I do it” but “where does it point”
Bohr (pure basic: atomic structure), Edison (pure applied: the light bulb, no interest in the underlying physics), Pasteur — whose work on spoiled wine and beer, silkworm disease, anthrax, and rabies was driven by practical problems and helped found microbiology on the way
The fourth corner: Stokes’s example is Peterson’s bird field guides — systematic description, neither theory nor application
Disease ecology lives in Pasteur’s quadrant
Use-inspiration is no citation sacrifice: Gao et al. (2024) compared 3,322 AI papers later cited in patents with 6,587 never cited in patents (1999–2013); the patent-cited ones also drew more scientific citations — an association, and the causation could run either way, but it’s no penalty
The link to the first diagram: use-inspiration is often what makes a problem easy/high-gain — the application supplies the data, the urgency, and the sharp question
Conclusion: Aim for use-inspired basic research — it is where understanding and consequence reinforce each other, not a compromise between them. [2 min — ends 0:26.5]

Discussion

Where does your current project sit on the feasibility–interest diagram?

Could an easier project teach the same lesson?

“Here to Help” — Randall Munroe, xkcd.com (CC BY-NC 2.5)

The protocol

Write the plan before the data exist

After trial registration arrived, positive results fell from 57% to 8%

55 large NHLBI trials of drugs and supplements, all with cardiovascular outcomes
Published before 2000: 17 of 30 positive on their primary outcome
Published 2000 or later, with outcomes registered in advance on ClinicalTrials.gov: 2 of 25 positive

Source: Kaplan & Irvin (2015) PLoS ONE 10:e0132382.

Kaplan & Irvin: Robert Kaplan, a health-outcomes researcher and former NIH official, with Veronica Irvin; they took every large NHLBI-funded trial (over $500k a year in direct costs) of a drug or dietary supplement, 1970–2012, and coded whether the primary outcome showed benefit
Why 2000: ClinicalTrials.gov launched that year, and every later trial in the set registered its primary outcome before publication; journals began requiring registration in 2005 and US law in 2007
What didn’t explain it: the share of placebo-controlled trials was flat (60% vs 64%), and industry co-sponsorship didn’t predict results
The authors are careful: an association, not proof — background cardiovascular care improved over the period, and NHLBI took a much more hands-on role in the later trials
What points to registration: 12 of the 25 registered trials still reported benefits on secondary outcomes — the effects hadn’t vanished, the freedom to promote them to “primary” had
The likely mechanism: researchers had to write the primary outcome down in advance
Conclusion: When trials had to name their outcome in advance, positive results all but disappeared — the strongest natural experiment we have on writing the plan down. [2 min — ends 0:32.5]

Four innocuous choices, left open, turn 5% false positives into 61%

Two outcome variables instead of one; peeking before adding observations; an optional covariate; dropping a condition
Each inflates error modestly; combined: a 60.7% false-positive rate at nominal p < .05
No fraud required — it’s enough that you would have analyzed different data differently (the “garden of forking paths”)

Sources: Simmons, Nelson & Simonsohn (2011) Psych Sci 22:1359; Gelman & Loken (2014) Am Sci 102:460.

Simmons, Nelson & Simonsohn — three social psychologists (Wharton, Berkeley); the paper coined “researcher degrees of freedom” and set off psychology’s replication reckoning
The setup: show that ordinary, defensible analysis choices — not fraud — can “prove” something impossible
The experiment: students heard either the Beatles’ “When I’m Sixty-Four” or a control instrumental (“Kalimba”), then reported their birth date; controlling for father’s age, the Beatles group came out younger — adjusted ages 20.1 versus 21.5 years, p = .04 — and a song cannot change your date of birth
How they did it: more than one outcome available, testing after every ~10 participants, an optional covariate, a third song they could drop — each defensible, each chosen after seeing the data; without the covariate the difference was not significant (p = .33)
The simulations, 15,000 runs with 20 per group: two correlated outcomes 9.5%; one peek then add 10 per group 7.7%; optional gender covariate 11.7%; drop one of three conditions 12.6% — each about 8–13%; all four together 60.7%
Gelman & Loken — Andrew Gelman, Columbia statistician, with Eric Loken: even one honest analysis walks the garden of forking paths, because the analysis you ran was contingent on the data you saw
The bridge: a plan that names the outcome, the sample size, the covariates, and the conditions in advance closes every one of those freedoms
Conclusion: An analysis with open choices can confirm anything, even the impossible — the protocol closes the choices before the data can exploit them. [3 min — ends 0:35.5]

This is us: 174 ecology teams, same data, opposite conclusions

Two unpublished datasets (blue tit sibling competition; Eucalyptus recruitment) analyzed independently by 174 teams
Effects ranged from significantly negative to significantly positive
Peer-rated analysis quality did not predict who landed where
In a survey of ecologists and evolutionary biologists, 64% admitted cherry-picking and 51% HARKing — Hypothesizing After the Results are Known: presenting an unexpected finding as if it had been predicted from the start

Sources: Gould et al. (2025) BMC Biology 23:35; Fraser et al. (2018) PLoS ONE 13:e0200303.

Pre-empt the “that’s psychology’s problem” objection: this is ecology and evolutionary biology, our datasets, our methods, published as a Registered Report
Gould et al. (lead author Elliot Gould, University of Melbourne): 174 teams, 246 analysts, two previously unpublished datasets — does sibling number affect blue tit nestling growth? does grass cover affect Eucalyptus seedling recruitment?
Blue tit: the average effect was clearly negative, but estimates ran continuously from large negative to zero, and a few were significant in the opposite direction; Eucalyptus: averaged near zero, with about a third significant — some positive, some negative
Peer reviewers rated each analysis; those ratings didn’t predict how far a team strayed from the average — competence doesn’t neutralize the spread
Their remedy is not a single better analysis: run robustness or multiverse analyses across the defensible choices, and treat any single analysis with skepticism
Where pre-specification fits: it stops you choosing the analysis after you’ve seen the result; it doesn’t make your plan the right one — robustness checks tell you whether the choice mattered
HARKing (term coined by the social psychologist Norbert Kerr, 1998): it’s invisible in the published record because the write-up looks like a confirmed prediction; only a dated protocol separates prediction from postdiction
Fraser et al. (University of Melbourne): anonymous survey of 807 ecologists and evolutionary biologists; “at least once” rates — 64% cherry-picked (didn’t report non-significant outcomes), 51% HARKed (ecologists alone 49%), 42% collected more data after checking significance
Not fraud, just what happens when no written plan constrains the write-up
Conclusion: Analytic flexibility is already in our field, and competence doesn’t neutralize it — pre-specify so you can’t choose after the fact, and check whether your choice mattered. [3 min — ends 0:38.5]

The Drake Lab protocol: a contract with your future self

A real protocol: Tredennick’s 2018 measles early-warning study — background, research questions, and study design, set down at the outset and kept current as the work moved.

Here’s the instrument: the first page of Tredennick’s protocol — title, originator with contact, authors, background, research questions
Its goal is to prevent mission creep: it fixes, in advance, the decisions that otherwise get relitigated in month eight — what question, what data, what analysis, what counts as done
What you’re seeing is the living document mid-project — seven pages by then, with results and a changelog
The question sharpened along the way: the protocol asks whether EWS track the effective reproduction number in the real case series; the paper tests whether EWS anticipate re-emergence and elimination in simulations from the fitted models — that evolution is what a changelog is for
I credit these pages with that project getting finished
Conclusion: The protocol is a contract with your future self, and its job is to kill mission creep. [2 min — ends 0:40.5]

Seven parts, one page is enough to start

Title and researchers — an originator with contact info
Background and rationale — 2–3 succinct paragraphs
Study design — steps + explicit research questions (for models: design it like an experiment — Drake 2025)
Data sources — with permissions needed
Analysis — including the anticipated figures, sketched before the data exist (for models: treatments, levels, and responses — Drake 2025)
Checklist — tasks with dates and owners
Changelog — the plan may change, but only in writing, with the reason

For computational projects, the study design is an experimental design

Treatments — what you vary (parameters, scenarios, interventions)
Levels — the values each treatment takes, chosen in advance
Responses — the outputs you will measure, named before the first run
Replication — how many stochastic realizations, and the seed policy
Control — the baseline scenario everything is compared against

After Drake (2025), “Modelling Like an Experimentalist,” Ecology Letters 28:e70251.

My 2025 Ecology Letters paper argues that modelling is experimentation: parameter regimes or data manipulations are treatments, replicated runs yield summaries, and comparisons across conditions reveal main effects and interactions
Treatments, levels, responses, and replication come from the paper; the seed policy and the explicit control scenario are additions for this checklist
The paper’s closing line: “If you cannot write down your treatments and responses, you are not modelling — you are doodling”
Unplanned simulation is the garden of forking paths with a random seed: runs are free, so without named treatments and responses you can always find a “result”
Treat a simulation study like a bench experiment — name the treatments, levels, responses, replication, and control before the first run
The Tredennick protocol is the worked case: the model, the fitting steps, the EWS set, and a test of the analysis on simulated data, all specified in advance — the paper itself treats re-emergence and elimination as treatments, with a gradient of susceptible depletion and a vaccination ramp, scored over 500 replicate simulations
Conclusion: A simulation study is an experiment — design its treatments and responses before the first run, not after. [2 min — ends 0:44.5]

Discussion

If your current project had a changelog, what would its first entry say?

“The General Problem” — Randall Munroe, xkcd.com (CC BY-NC 2.5)

Structure

Repositories, folders, and code that reruns

One project = one repository, named so a stranger can find it

Create the repository the day the protocol exists
Naming: surname-description (individual) or drakelab-description (team)
README.md is the front door: project name, originator, collaborators, the protocol — and a map of how the repository is organized
Private project, public product: the repo is the full internal history; the product (paper + archive) is the curated public face

A standard layout means never asking “where does this go?”

project/
├── README.md        protocol, or pointer to it
├── data/            raw, read-only, never edited
├── scripts/         01-clean.R, 02-fit.R, 03-figures.R
├── output/          derived data + figures, regenerable
└── documents/       manuscript, notes, correspondence

Two rules do most of the work: raw data are sacred; everything derived is disposable.

After Noble (2009) PLoS Comput Biol 5:e1000424 and Wilson et al. (2017) PLoS Comput Biol 13:e1005510.

Bill Noble, a computational biologist at the University of Washington, wrote “A quick guide to organizing computational biology projects” for his own students
Noble’s principle, verbatim: “Someone unfamiliar with your project should be able to look at your computer files and understand in detail what you did and why” — from the layout plus the README, the notes, and the scripts, not the folder names alone
His layout is data/, results/, doc/, src/, bin/ — ours renames them for a lab of ecologists, same logic
Greg Wilson founded Software Carpentry; Wilson et al. call these “good enough” practices deliberately: you don’t need to be a software engineer; a handful of habits — save the raw data, make it read-only, script every step — captures most of the benefit
Numbered scripts are the pipeline documentation — the ordering is the story of the analysis
Conclusion: One standard layout, two rules — raw data are sacred, derived output is disposable. [2.5 min — ends 0:53]

Undocumented projects decay

Information entropy: detail is lost from the moment of publication — specifics first, then generalities, with step losses at retirement, accidents, and death
Odds a dataset still exists fall 17% per year after publication
Even under strong archiving policies, 56% of ecology datasets were incomplete and 64% archived in ways that partly or wholly prevented reuse

Sources: Michener et al. (1997) Ecol Appl 7:330; Vines et al. (2014) Curr Biol 24:94; Roche et al. (2015) PLoS Biol 13:e1002295.

Michener et al. (Bill Michener and colleagues) proposed a metadata standard for ecology — a forerunner of the Ecological Metadata Language that DataONE and LTER archives use today
Their Figure 1 is the decay curve: information content peaks at publication; the specific details go first, then the general context; steps down at retirement or career change; an accident or obsolete storage can wipe it out at any time; and the investigator’s death ends it
The metadata you write today is the only antidote
Vines et al. (Tim Vines, then managing editor of Molecular Ecology): asked the authors of 516 papers (1991–2011) on plant and animal morphology for their data; where authors said what had happened to it, the odds it still existed fell 17% per year, and the odds of reaching any author by email fell 7% per year
The losses were mundane: stolen laptops, a parent’s attic, data stranded on Zip and floppy disks with no drive to read them
Roche et al. (Dominique Roche and colleagues, Australian National University): audited 100 Dryad datasets from 2012–2013 in seven journals with strong archiving policies; 56% were missing data, and 64% were archived in a way that partly or entirely blocked reuse
Roche’s lesson: depositing files isn’t archiving — completeness and metadata are the job
Conclusion: Undocumented work decays on a measurable clock, and the metadata you write today is the only thing that stops it. [2 min — ends 0:55]

The test: if `output/` was lost tonight, could one command rebuild it?

When published R code was re-run in a clean environment: 74% of scripts failed — library and working-directory errors among the most common
In ecology journals with code-sharing policies, 79% of articles shared data but only 27% shared code
Everything by script; relative paths; record seeds and package versions

Sources: Trisovic et al. (2022) Sci Data 9:60; Culina et al. (2020) PLoS Biol 18:e3000763; Sandve et al. (2013) PLoS Comput Biol 9:e1003285; Noble (2009).

Trisovic et al. (Ana Trisovic, Harvard’s Dataverse team): automatically re-ran 9,078 R files from 2,109 replication packages deposited on Harvard Dataverse, 2010–2020, mostly social science
Of the files that ran to an answer (about half hit the time limit), 74% stopped with an error; automated cleanup — removing setwd() calls, installing missing libraries — fixed many of them
The failures were trivial and preventable: missing libraries and hard-coded working directories
Culina et al. (Antica Culina, Netherlands Institute of Ecology): sampled 346 articles (2015–2019) from 14 ecology journals that require or encourage code sharing; 79% shared at least some data, 27% shared any code — and the authors say the true rates are lower, because these were the journals with policies
Ecology’s binding constraint is code, not data
The third bullet: Sandve et al.’s “Ten simple rules for reproducible computational research” supply scripting, seeds, and exact versions; relative paths are from Noble
Self-interest framing: the reviewer asking for a revised Figure 2 in eighteen months is asking future you, and future you remembers nothing — scripts are the memory
Conclusion: If one command can’t rebuild your results, you don’t have results yet — you have memories. [2.5 min — ends 0:57.5]

Version control is a lab notebook you cannot lose

Commit per logical change; push at least once a day.

Commits are dated snapshots of the whole project; branches let ideas fail safely; GitHub holds the shared copy. Mechanics — branching, merging, pull requests — get their own hands-on workshop: October 20, 2:55 PM.

Concept, not mechanics
A commit is a dated, signed snapshot of the whole project with a one-line note — a notebook entry you can rewind to; small commits, honest messages
Cadence: one commit per coherent change (not per file, not per day); push at least daily, so the shared copy is never more than a day behind your laptop — an unpushed commit that lives only on your machine is exactly the single point of failure the repo exists to prevent
A branch is a parallel line for trying an idea without endangering the main line — it replaces analysis_final_v3_REALLY_FINAL.R
Karthik Ram (rOpenSci) made the case in 2013 that git can hold the entire project — data (if small), code, figures, notes, manuscript; Perez-Riverol et al.’s ten simple rules (2016) show how GitHub turns it into collaboration infrastructure, with releases archived to Zenodo for a citable DOI
Every lab project is a repo under the DrakeLab organization from the day its protocol exists
Point people to the hands-on mechanics session: October 20 at 2:55 PM — today is concept only
Conclusion: Version control is a lab notebook you cannot lose — commit small, push daily. [3 min — ends 1:00.5]

Driving to done

The checklist is the engine; dependencies are where projects stall

The Tredennick protocol’s checklist, mid-project: struck-through tasks, surviving blockers, and a changelog recording why the plan changed.

Read the page: the cleaned dataset and the model code are struck through; cluster access at UGA and MIDAS — the dependencies — are struck through too; “Write the paper!” is the last line
Every task has a date, an owner, and a strikethrough waiting
Order the checklist so blockers hit early — data permissions, cluster access, coauthor availability surface in week one, not month eight
At every review (a check-in, a lab meeting, a one-on-one — whatever your cadence), ask three questions: what shipped, what’s blocked, what’s next
The tool you track tasks in doesn’t matter — GitHub Issues, a project board, Remember the Milk, a paper list, a whiteboard — what matters is that you actually hold the review on a regular cadence; a fancy tool with no review is worthless, and a scrap of paper reviewed every week keeps a project alive
Conclusion: The engine isn’t the tool — it’s the recurring review that answers shipped, blocked, next. [2.5 min — ends 1:03]

Done means delivered — then you stop

A project is over when the product is delivered: paper out, code and data archived (e.g., Dryad) with the repository linking to them
New ideas from old projects become spinoffs with new protocols and new repos
The portfolio view: one fast/safe, one core, one stretch — chapters as separate projects makes this natural

Exercise

Start your own protocol — right now

Fifteen minutes: draft the skeleton for one real project

Name the product — one sentence: what do you deliver?
Background — two sentences
Research question — and what would an answer look like? (a p-value? a pattern in a graph? a table of numbers?)
Three checklist items — each with a date
One dependency — the thing most likely to block you

Then two minutes: tell your neighbor your product in one sentence.

Discussion

What did you discover drafting it? Where do you disagree with the system?

“Standards” — Randall Munroe, xkcd.com (CC BY-NC 2.5)

The starter kit

Protocol template + worked example: github.com/DrakeLab/Wiki → lab-docs → project-protocols
Folder skeleton: README + data + scripts + output + documents
One repo per project, created the day the protocol exists
Definition of done: the product is delivered, the archive is public, you stop

You don’t need more discipline or better software. You need a plan you can print, a repo with five folders, and a definition of done. Write the protocol this week.

Backup

Does better documentation of analytic choices fix the many-analysts problem?

161 researchers, 73 teams, one dataset (ISSP) and one hypothesis — does immigration erode support for social policy? — 1,261 models submitted
Of 89 team-level conclusions: ~61% reject, ~26% support, ~13% “not testable”; estimates ran from large negative to large positive
After coding 107 analytic decisions plus expertise and prior beliefs, 95.2% of the variance in results remained unexplained
Expertise didn’t shrink it, and no team changed its model after seeing the others’ — the divergence is idiosyncratic, not bias

Source: Breznau et al. (2022) PNAS 119:e2203150119.

Breznau et al. (Nate Breznau, University of Bremen): 73 teams (161 social scientists) given the same International Social Survey Programme data and the same hypothesis
Design: teams declared their intended models in advance but were free to change them; a randomly chosen half took part in online deliberation before analysis, everyone deliberated afterward, and every participant ranked other teams’ model descriptions
Volume: one team judged the hypothesis untestable; the other 72 submitted 1,261 models (average ~17.5 per team, range 1–124)
Conclusions: 89 team-level verdicts because 16 teams reached separate conclusions for the two immigration measures — 54 reject, 23 support, 12 not testable (the paper prints 28.5% for “support”; the counts give ~26%)
Punchline: 107 coded decisions explained 2.6% of the variance and researcher characteristics at most 1.2% — 95.2% unexplained, the paper’s “hidden universe of uncertainty”
What it rules out: not bias or incompetence — higher methodological expertise didn’t reduce the spread, and no team voluntarily changed its model after seeing others
Versus the other many-analysts studies: Silberzahn et al. (2018) gave 29 teams one soccer dataset — do referees give more red cards to dark-skinned players? — and got odds ratios from 0.89 to 2.93, 20 teams significant and 9 not; Gould (174 teams, ecology) shows the same in our field; Breznau shows it can’t be explained away even after every identifiable choice is coded
What follows: pre-declaring a plan didn’t make these teams agree — it protects against choosing after the fact, not against choosing differently; the authors’ advice is humility about any single analysis, full documentation, and many-analyst or multiverse checks
Conclusion: No — documenting choices doesn’t explain the divergence, so pre-specify to stay honest and test robustness to know whether your choice mattered.

What do honest base rates look like?

Standard psychology literature: 96% of first hypotheses “confirmed” (n=152 papers)
Registered Reports: 44% (n=71)
Across 113 Registered Reports in biomedicine and psychology, ~61% of 296 hypotheses were not supported

Sources: Scheel, Schijen & Lakens (2021) AMPPS 4(2); Allen & Mehler (2019) PLoS Biol 17:e3000246.

For “does pre-specification just kill productivity?” — no, it reveals the true base rate the standard literature hides
What a Registered Report is: the journal peer-reviews and accepts the introduction and methods before the data are collected, so publication can’t depend on the result
Scheel, Schijen & Lakens (Eindhoven): the first hypothesis in a random sample of 152 standard psychology papers versus all 71 psychology Registered Reports published by 2018; they attribute the gap to less publication bias and/or less false-positive inflation
Allen & Mehler (two early-career researchers writing on open-science practice): tallied 296 hypotheses in 113 Registered Reports; half were replication attempts, which failed more often (66% unsupported versus 55% for novel hypotheses), so the comparison isn’t perfectly like-for-like
The 5–20% comparison figure is Allen & Mehler’s citation of prior work, not their own measurement
Conclusion: Pre-specification doesn’t lower the hit rate — it exposes the honest one.

How much do effects shrink on replication?

When an experiment is repeated, the measured effect is almost always smaller than the original report — the gap is the shrinkage
Reproducibility Project: Cancer Biology — for positive effects, median replication effect 85% smaller than original; 46% of effects replicated by a majority of five criteria
The project planned 193 experiments and completed 50 — missing data, under-described methods, and unhelpful authors blocked the rest
Nature survey of 1,576 researchers: >70% failed to reproduce another’s experiment; >50% failed to reproduce their own

Sources: Errington et al. (2021) eLife 10:e71601 and 10:e67995; Baker (2016) Nature 533:452.

The Reproducibility Project: Cancer Biology ran 2013–2021, the Center for Open Science with Science Exchange; it tried to repeat key experiments from 53 high-profile cancer papers published 2010–2012
50 experiments from 23 papers were completed, covering 158 effects; for positive effects the median replication effect was 85% smaller than the median original, and 92% of replications were smaller
46% = 51 of 112 effects that passed more of the five success criteria than they failed (40% of positive effects, 80% of null effects)
The 193→50 number is the sleeper, from the companion paper: none of the 193 experiments was described well enough to repeat without asking the authors; the data needed to compute effect sizes was public for just 4 of 193; a third of authors were unhelpful or never replied; and two-thirds of protocols needed modifications once work began, many of which couldn’t be carried out
Baker (Monya Baker, Nature reporter): an online questionnaire of 1,576 researchers who chose to respond — self-selected and self-reported, so a temperature reading, not a rate
Project management is reproducibility
Conclusion: Effects shrink by a lot on replication, and poor documentation stops replication before it starts.

Why doesn’t the system just reward good methods?

Model: when institutions reward publication volume, low-effort methods out-reproduce careful ones — no misconduct required
Mean statistical power to detect small effects in the social and behavioral sciences: 0.24, unchanged 1960–2011

Source: Smaldino & McElreath (2016) R Soc Open Sci 3:160384.

Smaldino & McElreath (Paul Smaldino, then at UC Merced; Richard McElreath, Max Planck Institute for Evolutionary Anthropology), “The natural selection of bad science”
The model: labs compete; those that publish more produce more “offspring” labs that copy their methods; selection for output drives effort down and false discoveries up, with no one cheating
The data: 44 published reviews of statistical power in social and behavioral science journals, 1960–2011; mean power to detect a small effect (d = 0.2) was 0.24, with no trend despite fifty years of warnings
A 2023 correction fixed a coding error in one part of the simulations; the authors and an independent reimplementation say the conclusions stand
For cynics who say the system won’t reward this discipline: the system’s defaults are exactly why individual labs need explicit countermeasures
Conclusion: The incentives select for speed over rigor, so the discipline has to come from you and your lab, not the system.

Doesn’t “work on important problems” contradict easy/high-gain?

“If you do not work on an important problem, it’s unlikely you’ll do important work.”
But: “It’s not the consequence that makes a problem important, it is that you have a reasonable attack.”
Great scientists keep 10–20 important problems waiting for an attack to appear

Source: Hamming (1986), “You and Your Research,” Bell Communications Research Colloquium.

References

References (1 of 2)

Allen C & Mehler DMA (2019) PLoS Biology 17:e3000246. doi:10.1371/journal.pbio.3000246

Alon U (2009) Molecular Cell 35:726. doi:10.1016/j.molcel.2009.09.013

Baker M (2016) Nature 533:452. doi:10.1038/533452a

Boudreau KJ et al. (2016) Management Science 62:2765. doi:10.1287/mnsc.2015.2285

Breznau N et al. (2022) PNAS 119:e2203150119. doi:10.1073/pnas.2203150119

Council of Graduate Schools (2008) PhD Completion and Attrition. Washington, DC

Culina A et al. (2020) PLoS Biology 18:e3000763. doi:10.1371/journal.pbio.3000763

Drake JM (2025) “Modelling Like an Experimentalist.” Ecology Letters 28:e70251. doi:10.1111/ele.70251

Errington TM et al. (2021) eLife 10:e71601. doi:10.7554/eLife.71601

Errington TM et al. (2021) eLife 10:e67995. doi:10.7554/eLife.67995

Foster JG, Rzhetsky A & Evans JA (2015) Am Sociol Rev 80:875. doi:10.1177/0003122415601618

Fraser H et al. (2018) PLoS ONE 13:e0200303. doi:10.1371/journal.pone.0200303

Gao X et al. (2024) Scientometrics 129:909. doi:10.1007/s11192-023-04925-w

Gelman A & Loken E (2014) American Scientist 102:460. doi:10.1511/2014.111.460

Gould E et al. (2025) BMC Biology 23:35. doi:10.1186/s12915-024-02101-x

Hamming RW (1986) “You and Your Research,” Bell Communications Research Colloquium

Kaplan RM & Irvin VL (2015) PLoS ONE 10:e0132382. doi:10.1371/journal.pone.0132382

Kerr NL (1998) Pers Soc Psychol Rev 2:196. doi:10.1207/s15327957pspr0203_4

References (2 of 2)

Michener WK et al. (1997) Ecological Applications 7:330. doi:10.1890/1051-0761(1997)007[0330:NMFTES]2.0.CO;2

Noble WS (2009) PLoS Comput Biol 5:e1000424. doi:10.1371/journal.pcbi.1000424

Perez-Riverol Y et al. (2016) PLoS Comput Biol 12:e1004947. doi:10.1371/journal.pcbi.1004947

Ram K (2013) Source Code Biol Med 8:7. doi:10.1186/1751-0473-8-7

Roche DG et al. (2015) PLoS Biology 13:e1002295. doi:10.1371/journal.pbio.1002295

Sandve GK et al. (2013) PLoS Comput Biol 9:e1003285. doi:10.1371/journal.pcbi.1003285

Scheel AM, Schijen MRMJ & Lakens D (2021) AMPPS 4(2). doi:10.1177/25152459211007467

Silberzahn R et al. (2018) AMPPS 1:337. doi:10.1177/2515245917747646

Simmons JP, Nelson LD & Simonsohn U (2011) Psych Science 22:1359. doi:10.1177/0956797611417632

Smaldino PE & McElreath R (2016) R Soc Open Sci 3:160384. doi:10.1098/rsos.160384

Stokes DE (1997) Pasteur’s Quadrant. Brookings Institution Press

Tredennick AT et al. (2022) J R Soc Interface 19:20220123. doi:10.1098/rsif.2022.0123

Trisovic A et al. (2022) Scientific Data 9:60. doi:10.1038/s41597-022-01143-6

Uzzi B et al. (2013) Science 342:468. doi:10.1126/science.1240474

Vines TH et al. (2014) Current Biology 24:94. doi:10.1016/j.cub.2013.11.014

Wang J, Veugelers R & Stephan P (2017) Research Policy 46:1416. doi:10.1016/j.respol.2017.06.006

Wilson G et al. (2017) PLoS Comput Biol 13:e1005510. doi:10.1371/journal.pcbi.1005510

Managing Research Projects

Projects fail from entropy, not stupidity

Everything today is a cheap, mechanical defense against all three failure modes.

Roadmap

What a project is

A project is the work aimed at one product

If you can’t write a plan for it, it isn’t a project

Choosing the project

The stakes: only about 57% of PhD students finish within ten years

Plot every candidate project on two axes

Where I part ways with Alon: everyone works easy/high-gain

Most published science is conservative — for a reason

The highest-impact papers are conventional work with one unusual combination

Donald Stokes and Pasteur’s Quadrant

Point the project with Pasteur

Discussion

Where does your current project sit on the feasibility–interest diagram?

Could an easier project teach the same lesson?

The protocol

After trial registration arrived, positive results fell from 57% to 8%

Four innocuous choices, left open, turn 5% false positives into 61%

This is us: 174 ecology teams, same data, opposite conclusions

The Drake Lab protocol: a contract with your future self

Seven parts, one page is enough to start

For computational projects, the study design is an experimental design

Discussion

If your current project had a changelog, what would its first entry say?

Structure

One project = one repository, named so a stranger can find it

A standard layout means never asking “where does this go?”

Two rules do most of the work: raw data are sacred; everything derived is disposable.

Undocumented projects decay

The test: if output/ was lost tonight, could one command rebuild it?

Version control is a lab notebook you cannot lose

Commit per logical change; push at least once a day.

Driving to done

The checklist is the engine; dependencies are where projects stall

Done means delivered — then you stop

Exercise

Fifteen minutes: draft the skeleton for one real project

Then two minutes: tell your neighbor your product in one sentence.

Discussion

What did you discover drafting it? Where do you disagree with the system?

The starter kit

You don’t need more discipline or better software. You need a plan you can print, a repo with five folders, and a definition of done. Write the protocol this week.

Backup

Does better documentation of analytic choices fix the many-analysts problem?

What do honest base rates look like?

How much do effects shrink on replication?

Why doesn’t the system just reward good methods?

Doesn’t “work on important problems” contradict easy/high-gain?

References

References (1 of 2)

References (2 of 2)

The test: if `output/` was lost tonight, could one command rebuild it?