Open Science

What is it? Why does it matter?

Rick Gilmore

Psychology & Child Study Center

Support

  • National Institutes of Health, R01-HD094830
  • National Science Foundation, OAS-2032713
  • John S. Templeton Foundation
  • Penn State Child Study Center

Agenda

  • A cautionary tale
  • What is open science
  • Why it matters
  • Let’s get to work…

A cautionary tale


(NYU Health Sciences Library, 2013)

What is open science?

Wikipedia

Sharing data

(Carroll, Herczog, Hudson, Russell, & Stall, 2021)

Sharing materials

  • Protocols
  • Surveys & displays, reagents, etc.
  • Data analysis
    • Pre-registered plans
    • Realized plans
    • Code
    • Code books/data dictionaries

Version control

https://github.com/penn-state-open-science/bootcamp-2023/commits/main

Open Access…

  • Blitz talk on PSU Open Access policy (up next)

Why does it matter?

What is science, really?

  • a stock of accumulated knowledge (facts & findings)

  • a set of characteristic methods

  • a set of cultural values (Merton, 1973, p. 268)

Richard Feynmann

…the idea that we all hope you have learned in studying science in school…It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty…

(Feynman, 1974)

Richard Feynmann

The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists…

(Feynman, 1974)

Richard Feynmann

…a specific, extra type of integrity that is not lying, but bending over backwards to show how you’re maybe wrong, that you ought to do when acting as a scientist. And this is our responsibility as scientists, certainly to other scientists, and I think to laymen.

(Feynman, 1974)

Richard Feynmann

…If you’ve made up your mind to test a theory, or you want to explain some idea, you should always decide to publish it whichever way it comes out. If we only publish results of a certain kind, we can make the argument look good. We must publish both kinds of result.

(Feynman, 1974)

Richard Feynmann

…if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment…

(Feynman, 1974)

Robert Merton

Hogwash!

Norms and counter-norms

Norm Counternorm
Faith in the moral virtue of rationality (Barber, 1952). Faith in the moral virtue of rationality and nonrationality (cf., Tart, 1972).
Emotional neutrality as an instrumental condition for the achievement of rationality (Barber, 1952). Emotional commitment as an instrumental condition for the achievement of rationality.

(Mitroff, 1974, p. 592)

Norm Counternorm
Universalism: The acceptance or rejection of claims entering the list of science is not to depend on the personal or social attributes of their protagonist; his race, nationality, religion, class and personal qualities are as such irrelevant. Particularism: “The acceptance or rejection of claims entering the list of science is to a large extent a function of who makes the claim” (Boguslaw, 1968:59)

(Mitroff, 1974, p. 592)

Norm Counternorm
Communism: “Property rights are reduced to the absolute minimum of credit for priority of discovery” (Barber, 1952:130). “Secrecy is the antithesis’ of this norm; full and open communication [of scientific results] its enactment” (Merton, 1949:611). Solitariness (or, “Miserism” [Boguslaw, 1968:59]): Property rights are expanded to include protective control over the disposition of one’s discoveries; secrecy thus becomes a necessary moral act.

(Mitroff, 1974, p. 592)

Norm Counternorm
Disinterestedness: “Scientists are expected by their peers to achieve the self-interest they have in work–satisfaction and in prestige through serving the [scientific] community interest directly” (Barber, 1952:132). Interestedness: Scientists are expected by their close colleagues to achieve the self-interest they have in work-satisfaction and in prestige through serving their special communities of interest, e.g., their invisible college (Boguslaw, 1968:59; Mitroff, 1974b)

(Mitroff, 1974, p. 592)

Science is a human activity

  • Humans are…
    • flawed
    • often illogical
    • emotional AND intellectual
    • biased
    • have blind spots
  • By what means can we mitigate or overcome these?

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives…We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

(Munafò et al., 2017)

(Munafò et al., 2017)

Claim

  • Better/stronger/less-biased methods…

\(\rightarrow\) better/stronger/less-biased inferences

\(\rightarrow\) more consistent (& realistic) expression of scientific values

Methods reproducibility

Methods reproducibility refers to the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.

(Goodman, Fanelli, & Ioannidis, 2016)

  • Does a typical journal article satisfy this?
  • Does project X from your lab group satisfy this?
  • Reproducible to your team \(\rightarrow\) reproducible to others

HARK1 the herald…

  • Are you sure you remember clearly what you believed before you collected study data?
  • Are you sure you remember what analyses you planned to do to test your predictions?
  • Did you write it down?

Why open science @ Penn State

(Gilmore, Hillary, Lazar, & Wham, 2023)

(Gilmore et al., 2023)

But my data are restricted…

(Gilmore et al., 2023)

Data management poses challenges

(Gilmore et al., 2023)

(Gilmore et al., 2023)

Why open science: The bigger picture

(Baker, 2016)

(Gilmore et al., 2023)

(Baker, 2016)

(Baker, 2016)

(Baker, 2016)

(Baker, 2016)

(Saul, 2023)

(Oransky & Marcus, 2023)

https://ourworldindata.org

https://ourworldindata.org

Rigorous, robust, reproducible science…

  • Has been essential to the improvement of human health and well-being
  • Will be increasingly important to these improvements in the 21st century and beyond
  • Can be strengthened
  • And must be

Open science practices can…

  • Help you
    • Manifest your most idealistic aspirations
    • Strike a realistic balance between self-serving and other-serving motivations
    • Avoid “fooling yourself”
  • Improve reproducibility
    • For you, your colleagues/collaborators, and others
  • Enhance credibility
  • Save time & money
    • Yours, colleagues/collaborators, taxpayers
  • Accelerate discovery
  • Improve scholarly rigor & reach

Let’s get to work…

Bootcamp

  • Today
    • Open Science @ Penn State
    • Tim Errington, “Replication of preclinical cancer biology: Challenges or opportunities?”
    • Alaina Pearce, “Good enough practices for data and project management”

May all our crises be good ones…

(Korbmacher et al., 2023)

Your thoughts?

Resources

About

This talk was produced using Quarto, using the RStudio Integrated Development Environment (IDE), version 2023.6.1.524.

The source files are in R and R Markdown, then rendered to HTML using the revealJS framework. The HTML slides are hosted in a GitHub repo and served by GitHub pages: https://penn-state-open-science.github.io/bootcamp-2023-open-sci-what-why/

References

Open Science: What & Why?

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452. https://doi.org/10.1038/533452a
Carroll, S. R., Herczog, E., Hudson, M., Russell, K., & Stall, S. (2021). Operationalizing the CARE and FAIR principles for indigenous data futures. Scientific Data, 8(1), 108. https://doi.org/10.1038/s41597-021-00892-0
Feynman, R. P. (1974). Cargo cult science. https://calteches.library.caltech.edu/51/2/CargoCult.htm. Retrieved from https://calteches.library.caltech.edu/51/2/CargoCult.htm
Gilmore, R. O., Hillary, F., Lazar, N., & Wham, B. (2023). Penn state open science survey. https://penn-state-open-science.github.io/survey-fall-2022/index.html. Retrieved from https://penn-state-open-science.github.io/survey-fall-2022/index.html
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341ps12–341ps12. https://doi.org/10.1126/scitranslmed.aaf5027
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., … Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1(1), 1–13. https://doi.org/10.1038/s44271-023-00003-2
Merton, R. W. (1973). The normative structure of science. In R. K. Merton & N. W. Storer (Eds.), The sociology of science: Theoretical and empirical investigations (pp. 267–278). The University of Chicago Press.
Mitroff, I. I. (1974). Norms and counter-norms in a select group of the apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Sert, N. P. du, … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021
NYU Health Sciences Library. (2013, November). Data sharing and management snafu in 3 short acts (higher quality). Youtube. Retrieved from https://www.youtube.com/watch?v=66oNv_DJuPc
Oransky, I., & Marcus, A. (2023). Science corrects itself, right? A scandal at stanford says it doesn’t. Scientific American. Retrieved from https://www.scientificamerican.com/article/science-corrects-itself-right-a-scandal-at-stanford-says-it-doesnt/
Saul, S. (2023). Stanford president will resign after report found flaws in his research. The New York Times. Retrieved from https://www.nytimes.com/2023/07/19/us/stanford-president-resigns-tessier-lavigne.html