Context: Invited talk presented during the Workshop on Memory and Skill, Duke University, Durham, NC, April 2016.
Title: Not knowing what we know: A call for a theory-neutral database for empirical results in psychology.
Abstract: Theorists have proposed various memory distinctions based on duration, capacity, awareness, or the nature of the retrieved content. We argue that debates about whether such systems are functionally and structurally distinct cannot be resolved by empirical evidence and that they have no scientific value, because they do not advance our understanding about the processes involved in various mnemonic tasks. Instead, we argue for a framework based on investigating task-dependent processes. In this framework all mnemonic acts recruit different combinations of processes depending on the nature of the task. However, one major impediment to task-dependent analysis of cognitive processes is the lack of organization of empirical results in psychology. We argue that there are fundamental problems with the way we store and organize empirical knowledge that prevent empirical integration and theoretical advancement. We call for a theory-neutral database for empirical results in psychology that systematically maps task parameters and behavioral outcomes for every published study in the field. Task parameters would include descriptors for the nature of the stimuli, the details of the paradigm and the procedure, etc., based on a systematic hierarchical nomenclature. This would not only allow us to integrate currently available empirical data, but can also drive empirical research by identifying gaps in our knowledge. It might further shift the focus away from hypothesis testing to precise parameter estimation. Such a database can be integrated with the publishing process, where researchers would have to submit task-parameters and summary data, as well as raw data with a standardized format. This would allow for automatic meta-analyses, and large-scale parameter estimation. This empirical integration could lead to advancements in theoretical understanding by stimulating cross-talk between paradigms, standardizing task descriptors, identifying gaps in our current knowledge and providing a systematic set of results for building and testing computational models and cognitive architectures.
Presentation slides: PDF
- Theoretical advancement is hindered by the lack of organization of empirical results
- Too many published empirical studies to integrate verbally
- Models and architectures are useful, but do not solve the problem
- We need a theory-neutral database for empirical results that systematically maps task parameters and behavioral outcomes for every published study in the field
- Structured syntax-based description of task parameters that allows easy comparison, literature reviews, data extraction and automated data-analysis
- Organization and integration of currently available knowledge
- Automatic meta-analysis and large-scale parameter estimation
- Identifying gaps in our knowledge
- Model testing and comparison