There’s a heated exchange in political science over a new Data Access and Replication Transparency (DA-RT) initiative.
Twenty-seven political science journals have signed onto a data share policy. Opposition to this policy has organized a signature campaign aimed at delaying implementation of DA-RT that now includes over 1,000 signatures.
For a list of the many blog posts on the topic can be found here.
My contribution is to give a quick overview of data access and replication practices from management. Below I will provide an overview of the data replication policies of these journals.
If you don’t get to the actual description I can give a quick summary that will make nobody happy.
If we divided the journals between mostly quantitative and qualitative journals, I think we would find that all of the journals with mandatory data access and replication policies largely focus on quantitative analysis.
But many of the journals with quantitative orientations, including a top statistics journal, don’t have a mandatory replication policy. The three big financial journals attempted a replication policy and it failed due to a major backlash from the editorial board.
Replication policies aren’t easy for any journal. But there seems to some models out there that can help guide us. But in this summary I don’t see any obvious model from a journal that focuses on qualitative work.
If you are still with me, I can provide some details.
Update: Here are replication studies in political science and economics.
Business School Publishing
Business schools are eclectic mixes of faculty with background in different disciplines. My department (International Business) has Ph.D.s in economics, management, political science, psychology, and sociology. This leads to complications in how to weight different types of research contributions (books, cases, journal article, policy pieces, etc). I recently posted that very few political science journals “count” in many business schools.
The narrowest list of the top business journal is the Financial Times 45 (FT 45). This is the gold standard of top business journals and I don’t know of a single department that wouldn’t count these journals as top contributions.
I went through the FT 45 journals in search of formal replication and data access policies. Of the 45 journals the vast majority of them have no stated data access or replication policy. At the end of this post I include the full list of FT 45 journals and their policies. If I missed something, please let me know.
Journals with Mandatory Data Access Policy
Many of the journals in the FT 45 with formal policies are economics and psychology journals. The American Economic Review, for example, has a clear mandatory data availability policy and has a repository for randomized control trails. Econometrical and the Journal of Political Economy have similar policies.
The Journal of Applied Psychology, the Journal of Consumer Research and the Journal of Consumer Psychology, and Marketing Science have mandatory data access policies. The Journal of Consumer Psychology also requires IRB verification for human subjects research.
This is the full list of FT 45 journals with mandatory data access and replication policies. That’s it.
Journals that Encourage Data Access
There are a handful of journals that urge authors to provide their data or at least provide extensive documentation of their data and analysis.
For example, American Accounting Review has the following statement:
The AAA’s Executive Committee policy (originally adopted in 1989, and amended in 2009) is that the objective of the Association-wide journals (The Accounting Review, Accounting Horizons, Issues in Accounting Education) is to provide the widest possible dissemination of knowledge based on systematic scholarly inquiries into accounting as a field of professional research and educational activity. To fulfill this objective, authors are encouraged to make their data available for use by others in extending or replicating results reported in their articles. Please see the policy on Data Integrity adopted in March of 2015 for more information.
The Journal of Accounting Research has one of the more complicated policies that fall short of full replication. The full description of their policy is can be found on their website. One interesting aspect of this policy include verification by the journal of access to the proprietary. If you have top secret data you need to prove that you accessed this data. Second:
Prior to final acceptance of the paper, the computer program used to convert the raw data into the dataset used in the analysis plus a brief description that enables other researchers to use this program. Instead of the program, researchers can provide a detailed step-by-step description that enables other researchers to arrive at the same dataset used in the analysis. The purpose of this requirement is to facilitate replication and to help other researchers understand in detail how the sample was formed, including the treatment of outliers, Winsorization, truncation, etc. This programming is in most circumstances not proprietary. However, we recognize that some parts of the data generation process may indeed be proprietary or otherwise cannot be made publicly available. In such cases, the authors should inform the editors upon submission, so that the editors can consider an exemption from this requirement.
Given the extensive use of proprietary data in Accounting, my opinion is that this is close to a mandatory replication policy. Authors using proprietary data such as Compustat must prove they have access to this database and provide clear instructions on how others with access could extract the data.
The Journal of the American Statistical Association is a clear example of a journal that recommends data sharing but doesn’t make it mandatory. This one was honestly a surprise to me.
Some publishers have general statements that apply to all of their journals. This one is from from Elseiver:
Research data is the foundation on which scientific, technical and medical knowledge is built, but there are challenges in making it accessible and shareable. In line with the STM Brussels Declaration, Elsevier envisions a future in which data can be easily and effectively stored, shared, discovered and used, in support of researchers and for the advancement of science and health. However, there are challenges in making research data accessible and shareable. We have developed our research data policy to address these challenges and will continue to actively support further
Despite the lack of technology, the publisher does allow you to publish your data in their “Data-in-Brief” feature for a fee of $200. Replication isn’t cheap.
The Journal of Applied Psychology (American Psychological Association) has the following policy:
In addition, APA Ethical Principles specify that “after research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release” (Standard 8.14). APA expects authors to adhere to these standards. Specifically, APA expects authors to have their data available throughout the editorial review process and for at least 5 years after the date of publication.
Journals with No Data Access or Replication Policies
In total, 35 of the FT 45 either had no data access policy or explicitly stated that data sharing isn’t mandatory.
In my review of management journal websites I couldn’t find any explicit journals statements with data access or replication policies. For example, the three Academy of Management (AOM) journals make no mention of data access or replication.
The Strategic Management Journal has no formal replication policy, although they do have a statement about P-hacking:
Data Snooping and P-hacking
SMJ strongly disapproves of data snooping and p-hacking practices in empirical research. Authors of submitted papers should not search databases for statistically significant coefficients with the intention of subsequently formulating hypotheses that fit the significant coefficients. Authors also should not adapt experimental designs with the primary intention of producing statistically significant results. In addition, authors of submitted papers should address the material significance (magnitude) of the results, in addition to statistical significance.
Finance journals also lacks formal policies. According to a previous editor, the Journal of Finance proposed a replication policy (along with the Journal of Financial Economics and the Review of Financial Studies). This caused a major backlash and was shelved.
Two economics journals are also silent on data access.
Summary of Journal Practices
- In-house replication of articles: None of the FT45 provide an in-house replication of journals. In contrast, numerous political science journals such as the American Journal of Political Science performs a pure replication.
- Mandatory data sharing: Seven journals have mandatory data sharing policies. These journals are economics journals, psychology journals, or marketing journals. The American Economic Review is a good model for this type of data sharing.
- Sharing code for propriety data: The Journal of Accounting Research provides details on expectations for the extraction of proprietary data.
- Encouraging Data Sharing: Some journals such as the Journal of the American Statistical Association provide editorial statements encouraging
- No Data Sharing Policies: Essentially all of management has no policy.
What do make of this evidence?
This isn’t my job. It is up to you to draw your own conclusions.
Here is a quick summary of the FT 45.
FT 45 Data Access and Replication Policies
Academy of Management Journal (Academy of Management)
No stated data access or replication policy
Academy of Management Perspectives (Academy of Management)
No stated data access or replication policy
Academy of Management Review (Academy of Management)
No stated data access or replication policy
Accounting, Organisations and Society (Elsevier)
No stated data access or replication policy
The Accounting Review (American Accounting Association)
No mandatory data access or replication policy but detailed statement from AAA encourage data sharing and have detailed guidelines.
Administrative Science Quarterly (Cornell University)
No stated data access or replication policy
American Economic Review (American Economic Association)
Mandatory data availability policy.
California Management Review (UC Berkeley)
No stated data access or replication policy
Contemporary Accounting Research (Wiley)
No stated data access or replication policy
Requires copies of other papers using this same data.
Econometrica (Econometric Society, Wiley)
Mandatory replication policy
Entrepreneurship Theory and Practice (Baylor University, Wiley)
No stated data access or replication policy
Harvard Business Review (Harvard Business School Publishing)
No stated data access or replication policy
Human Resource Management (Wiley)
No stated data access or replication policy
Information Systems Research (Informs)
No stated data access or replication policy
Ethical guidelines only documenting process:
Journal of Accounting and Economics (Elsevier)
No stated data access or replication policy
Journal of Accounting Research (University of Chicago, Wiley)
No blanket mandatory data sharing policy. But there confirmation of proprietary data and requires details on data collection.
Journal of Applied Psychology (American Psychological Association)
Formal data access policy
Journal of Business Ethics (Kluwer Academic)
No stated data access or replication policy
Journal of Business Venturing (Elsevier)
No stated data access or replication policy
Journal of Consumer Psychology (Elsevier)
Mandatory data access policy based on APA
Journal of Consumer Research (University of Chicago)
Data statement paragraph and mandatory data sharing.
Journal of Finance (Wiley)
No stated data access or replication policy
A letter from the previous editor on the backlash shelving the policy
Journal of Financial and Quantitative Analysis (Cambridge University Press)
No stated data access or replication policy
Journal of Financial Economics (Elsevier)
No stated data access or replication policy
Journal of International Business Studies (Academy of International Business)
No stated data access or replication policy
Journal of Management Studies (Wiley)
No stated data access or replication policy
Journal of Marketing (American Marketing Association)
Data access not required
Journal of Marketing Research (American Marketing Association)
Data access not required
Journal of Operations Management (Elsevier)
No stated data access or replication policy
Journal of Political Economy (University of Chicago)
Mandatory replication policy
Journal of the American Statistical Association (American Statistical Association)
Optional Data Policy
Management Science (Informs)
No stated data access or replication policy
Marketing Science (Informs)
Mandatory replication policy
MIS Quarterly (Management Information Systems Research Centre, University of Minnesota)
No stated data access or replication policy
Operations Research (Informs)
No stated data access or replication policy
Organization Science (Informs)
No stated data access or replication policy
Organization Studies (SAGE)
No stated data access or replication policy
Organizational Behaviour and Human Decision Processes (Elsevier)
No stated data access or replication policy
Production and Operations Management (Wiley)
No stated data access or replication policy
Quarterly Journal of Economics (MIT)
No stated data access or replication policy
Rand Journal of Economics (The Rand Corporation, Wiley)
No stated data access or replication policy
Review of Accounting Studies (Springer)
No stated data access or replication policy
Review of Financial Studies (Oxford University Press)
No stated data access or replication policy
Sloan Management Review (MIT)
No stated data access or replication policy
Strategic Management Journal (Wiley)
No stated data access or replication policy