For twelve years I taught political science at Washington University in St. Louis. This is my second and last year of teaching International Business at George Washington University before going back to political science. I am joining the Government Department at the University of Texas in Fall 2016.

A few of my friends have asked me about the differences in teaching at a professional school and teaching in the arts and sciences. This is a difficult comparison but my general message is that business school teaching is pretty tough. But there are rewards.

Teaching Loads

When I started my first job at Washington University in 2002 we had a 2-2 teaching load (four courses per year). After a study of the teaching loads at peer political science programs WashU shifted to a 2-1 teaching load.

In business schools, some faculty teach 2-2 (or higher teaching loads) but most “research active” faculty teach a three course load. Unlike most political science departments, faculty sometimes teach all three courses in a single semester and this can be the same class repeated three times. Faculty can teach three sections of “Introduction to International Business” or “Global Strategy” in a semester and not teach the following semester.

Sounds like a good deal? Keep reading.

Variety of Classes

In my previous job in political science, courses were either semester long undergrad lectures or semester long Ph.D. seminars. At Washington University most of us taught one large undergrad lecture, one smaller upper division undergrad class, and one either an undergrad seminar or Ph.D. class.

In business schools, there are a much larger variety of potential classes types, but the variety of topics you can teach is smaller is more limited. Does that make sense?  Let me explain.

Some business schools don’t have any undergraduate majors, while others (like GW) have a large undergraduate population. Ph.D. programs in business schools tend to be very small and many departments only offer one Ph.D. seminar across the whole department. For most business school faculty, you are teaching MBA students and most of these classes are the “core” classes. For me this would be teaching “global strategy” or “international management”.

But business schools have plenty of master’s classes that range from the regular daytime MBA, evening MBA, online MBA, accelerated MBA, executive MBA as well as numerous specialized masters programs. These programs come in different forms, ranging from a regular 15 week course, 7 week courses, to anything else you can imagine.  But these are different forms of similar core courses.

I found selecting my courses to be especially daunting given this variety of potential courses, students, and schedules. In my two years at GW I taught one 7 week MBA course, an large introduction to globalization (200+ students), an advanced undergrad course (50 students) and a Ph.D. course. If I stuck around there was a one week DC focused course, consulting abroad, as well as some weekend courses I would consider.

Course Choice

Unlike many social science departments, professional schools have much more rigid curricula and thus provide a much smaller number of courses. Similar to law schools that require every student to take courses such as Contracts and Civil Procedure, business schools have a large number of required courses that require staffing. This means that many faculty have very few choices on which courses they will offer, and in many cases faculty are hired  to teach a specific course. We currently have a position at GW that is essentially to help teach one of our core MBA classes (Global Perspectives).

Thus while there are many different course formats, most of these are variants of core courses and not electives.

On-Load vs Off-Load

What I found especially striking in the business school was the constant need for additional courses. Faculty were expected to fulfill their teaching obligations (three courses a year for research active faculty) and often had the opportunity to teach additional courses “off load” for additional pay. For many business schools these additional courses pay an additional 1/12 of your base salary for a 3 credit course.

I was under the impression that many business school faculty engage in consulting for additional income. After joining the business school it is clear that additional teaching, including more specialized certificate programs and executive education is that way that faculty can earn an extra income and provide a service to the school.  But this comes at a cost to your research time.

Evaluation and Teaching Quality

Before moving to my new job in a business school I asked a few other faculty that made a similar transitions about their experience. Despite what sounded like a comparable teaching experience, most of the faculty reported that they spent more time prepping courses and more overall energy in teaching at a business school.

I came from a department with great teachers (political science at WashU has some fantastic teachers who are even better researchers) and I generally had very strong course evaluations. I thought this tough transition wouldn’t apply to me.

Two years later I find that I am still spending considerably more time and energy teaching than I did in my previous job. Why?

First, there is considerably more pressure to be a “good teacher” in a business school. Like I said, my previous job had lots of great teachers, but in my current job we review other syllabi, talk about course evaluations, and have lots of opportunities to receive additional training as a teacher. Some of the teaching opportunities, such as executive education, are only open to some of the best teachers.

Much of this is based on teaching evaluations. I am very mixed on using this criteria for evaluating teaching given the many biases in course evaluations and the weak correlation between ratings and learning. But my point is factual. Evaluations matter.

Second, and more importantly, I find that teaching in a business school is less closely engaged with my research than in political science. When teaching political science courses I assign academic research on the topic, discuss research design, data quality, and the link between theory and evidence.

We do some of this in my management courses, but there is more focus on practical decisions by managers and more focus on cases. I read dozens of cases before I assign a single case in my classes.  It is fun to learn new materials and it is exciting teaching, but it has very little synergy with the research part of my job.

This isn’t meant as a complaint.  Teaching is a larger part of your job in a business school and the goal conflict between teaching and research is much starker than my own experience in political science.

Thinking about a job in a business school?

Lots of students and colleagues have asked me about jobs in business schools. Do I like it? Would your recommend it to others?

This is a complicated question. It’s a very different world that consists of different criteria for research quality, a very different service and administrative role in the job, and substantially different teaching.

The focus of this blog post is only on teaching.  For non-academics this may not make sense.  The public perception of professor is solely as a teacher.  But for the academics reading this blog we all know the very brutal job market and high tenure standards make research an extremely important part of our jobs. And service to your department, university, and professional become more important the longer you are in the job.

In my opinion, teaching in a business school comprises a larger percentage of your job (as does school service) than a position in political science. This leaves less time for research (or sleep).

But the teaching definitely has its rewards and good teaching leads to additional professional opportunities. I was lucky at GW to teach undergrads, MBAs, and Ph.D. students. I really love teaching undergrad students, especially when much of the teaching is focused on a liberal arts education that has a learning for the sake of learning component. Ph.D. classes are also extremely engaging, where there is such a close relationship between teaching and research.

I also found that my limited exposure to MBA teaching was very positive as well. I love living in the world of ideas. But many of us academics have taken to blogging or writing op-ed pieces as a way to start engaging in policy debates. MBA teaching engages you in a different form of policy debate. You have the opportunity to help educate managers in their decisions. Some of these managers are in the public sector (especially in DC) but many of them are in private firms.  If you want to make a real world impact, the MBA classroom could be for you.

But this exciting opportunity comes at a cost. I found that my job in a business school tilted my time and energy much farther away from research than my years in political science. We all have different ideals on how we spend our professional time.  I guess I am voting with my feet.

There’s a heated exchange in political science over a new Data Access and Replication Transparency (DA-RT) initiative.

Twenty-seven political science journals have signed onto a data share policy. Opposition to this policy has organized a signature campaign aimed at delaying implementation of DA-RT that now includes over 1,000 signatures.

For a list of the many blog posts on the topic can be found here.

My contribution is to give a quick overview of data access and replication practices from management. Below I will provide an overview of the data replication policies of these journals.

If you don’t get to the actual description I can give a quick summary that will make nobody happy.

If we divided the journals between mostly quantitative and qualitative journals, I think we would find that all of the journals with mandatory data access and replication policies largely focus on quantitative analysis.

But many of the journals with quantitative orientations, including a top statistics journal, don’t have a mandatory replication policy. The three big financial journals attempted a replication policy and it failed due to a major backlash from the editorial board.

Replication policies aren’t easy for any journal. But there seems to some models out there that can help guide us. But in this summary I don’t see any obvious model from a journal that focuses on qualitative work.

If you are still with me, I can provide some details.

Update: Here are replication studies in political science and economics.

Business School Publishing

Business schools are eclectic mixes of faculty with background in different disciplines. My department (International Business) has Ph.D.s in economics, management, political science, psychology, and sociology. This leads to complications in how to weight different types of research contributions (books, cases, journal article, policy pieces, etc). I recently posted that very few political science journals “count” in many business schools.

The narrowest list of the top business journal is the Financial Times 45 (FT 45). This is the gold standard of top business journals and I don’t know of a single department that wouldn’t count these journals as top contributions.

I went through the FT 45 journals in search of formal replication and data access policies. Of the 45 journals the vast majority of them have no stated data access or replication policy. At the end of this post I include the full list of FT 45 journals and their policies. If I missed something, please let me know.

Journals with Mandatory Data Access Policy

Many of the journals in the FT 45 with formal policies are economics and psychology journals. The American Economic Review, for example, has a clear mandatory data availability policy and has a repository for randomized control trails. Econometrical and the Journal of Political Economy have similar policies.

The Journal of Applied Psychology, the Journal of Consumer Research and the Journal of Consumer Psychology, and Marketing Science have mandatory data access policies. The Journal of Consumer Psychology also requires IRB verification for human subjects research.

This is the full list of FT 45 journals with mandatory data access and replication policies.  That’s it.

Journals that Encourage Data Access

There are a handful of journals that urge authors to provide their data or at least provide extensive documentation of their data and analysis.

For example, American Accounting Review has the following statement:

The AAA’s Executive Committee policy (originally adopted in 1989, and amended in 2009) is that the objective of the Association-wide journals (The Accounting Review, Accounting Horizons, Issues in Accounting Education) is to provide the widest possible dissemination of knowledge based on systematic scholarly inquiries into accounting as a field of professional research and educational activity. To fulfill this objective, authors are encouraged to make their data available for use by others in extending or replicating results reported in their articles. Please see the policy on Data Integrity adopted in March of 2015 for more information.

The Journal of Accounting Research has one of the more complicated policies that fall short of full replication. The full description of their policy is can be found on their website.  One interesting aspect of this policy include verification by the journal of access to the proprietary.  If you have top secret data you need to prove that you accessed this data. Second:

Prior to final acceptance of the paper, the computer program used to convert the raw data into the dataset used in the analysis plus a brief description that enables other researchers to use this program. Instead of the program, researchers can provide a detailed step-by-step description that enables other researchers to arrive at the same dataset used in the analysis. The purpose of this requirement is to facilitate replication and to help other researchers understand in detail how the sample was formed, including the treatment of outliers, Winsorization, truncation, etc. This programming is in most circumstances not proprietary. However, we recognize that some parts of the data generation process may indeed be proprietary or otherwise cannot be made publicly available. In such cases, the authors should inform the editors upon submission, so that the editors can consider an exemption from this requirement.

Given the extensive use of proprietary data in Accounting, my opinion is that this is close to a mandatory replication policy. Authors using proprietary data such as Compustat must prove they have access to this database and provide clear instructions on how others with access could extract the data.

The Journal of the American Statistical Association is a clear example of a journal that recommends data sharing but doesn’t make it mandatory.  This one was honestly a surprise to me.

Some publishers have general statements that apply to all of their journals.  This one is from from Elseiver:

Research data is the foundation on which scientific, technical and medical knowledge is built, but there are challenges in making it accessible and shareable. In line with the STM Brussels Declaration, Elsevier envisions a future in which data can be easily and effectively stored, shared, discovered and used, in support of researchers and for the advancement of science and health. However, there are challenges in making research data accessible and shareable. We have developed our research data policy to address these challenges and will continue to actively support further

Despite the lack of technology, the publisher does allow you to publish your data in their “Data-in-Brief” feature for a fee of $200.  Replication isn’t cheap.

The Journal of Applied Psychology (American Psychological Association) has the following policy:

In addition, APA Ethical Principles specify that “after research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release” (Standard 8.14). APA expects authors to adhere to these standards. Specifically, APA expects authors to have their data available throughout the editorial review process and for at least 5 years after the date of publication.

Journals with No Data Access or Replication Policies

In total, 35 of the FT 45 either had no data access policy or explicitly stated that data sharing isn’t mandatory.

In my review of management journal websites I couldn’t find any explicit journals statements with data access or replication policies. For example, the three Academy of Management (AOM) journals make no mention of data access or replication.

The Strategic Management Journal has no formal replication policy, although they do have a statement about P-hacking:

Data Snooping and P-hacking
SMJ strongly disapproves of data snooping and p-hacking practices in empirical research. Authors of submitted papers should not search databases for statistically significant coefficients with the intention of subsequently formulating hypotheses that fit the significant coefficients. Authors also should not adapt experimental designs with the primary intention of producing statistically significant results. In addition, authors of submitted papers should address the material significance (magnitude) of the results, in addition to statistical significance.

Finance journals also lacks formal policies. According to a previous editor, the Journal of Finance proposed a replication policy (along with the Journal of Financial Economics and the Review of Financial Studies). This caused a major backlash and was shelved.

Two economics journals are also silent on data access.

Summary of Journal Practices

• In-house replication of articles: None of the FT45 provide an in-house replication of journals. In contrast, numerous political science journals such as the American Journal of Political Science performs a pure replication.
• Mandatory data sharing: Seven journals have mandatory data sharing policies. These journals are economics journals, psychology journals, or marketing journals. The American Economic Review is a good model for this type of data sharing.
• Sharing code for propriety data: The Journal of Accounting Research provides details on expectations for the extraction of proprietary data.
• Encouraging Data Sharing: Some journals such as the Journal of the American Statistical Association provide editorial statements encouraging
• No Data Sharing Policies: Essentially all of management has no policy. 

What do make of this evidence?

This isn’t my job. It is up to you to draw your own conclusions.

Here is a quick summary of the FT 45.

FT 45 Data Access and Replication Policies

Academy of Management Journal (Academy of Management)

No stated data access or replication policy

Academy of Management Perspectives (Academy of Management)

No stated data access or replication policy

Academy of Management Review (Academy of Management)

No stated data access or replication policy

Accounting, Organisations and Society (Elsevier)

No stated data access or replication policy

The Accounting Review (American Accounting Association)

No mandatory data access or replication policy but detailed statement from AAA encourage data sharing and have detailed guidelines.

Administrative Science Quarterly (Cornell University)

No stated data access or replication policy

American Economic Review (American Economic Association)

Mandatory data availability policy.

California Management Review (UC Berkeley)

No stated data access or replication policy

Contemporary Accounting Research (Wiley)

No stated data access or replication policy

Requires copies of other papers using this same data.

Econometrica (Econometric Society, Wiley)

Mandatory replication policy

Entrepreneurship Theory and Practice (Baylor University, Wiley)

No stated data access or replication policy

Harvard Business Review (Harvard Business School Publishing)

No stated data access or replication policy

Human Resource Management (Wiley)

No stated data access or replication policy

Information Systems Research (Informs)

No stated data access or replication policy

Ethical guidelines only documenting process:

Journal of Accounting and Economics (Elsevier)

No stated data access or replication policy

Journal of Accounting Research (University of Chicago, Wiley)

No blanket mandatory data sharing policy. But there confirmation of proprietary data and requires details on data collection.

Journal of Applied Psychology (American Psychological Association)

Formal data access policy

Journal of Business Ethics (Kluwer Academic)

No stated data access or replication policy

Journal of Business Venturing (Elsevier)

No stated data access or replication policy

Journal of Consumer Psychology (Elsevier)

Mandatory data access policy based on APA

 

Journal of Consumer Research (University of Chicago)

Data statement paragraph and mandatory data sharing.

 

Journal of Finance (Wiley)

No stated data access or replication policy

A letter from the previous editor on the backlash shelving the policy

Journal of Financial and Quantitative Analysis (Cambridge University Press)

No stated data access or replication policy

Journal of Financial Economics (Elsevier)

No stated data access or replication policy

Journal of International Business Studies (Academy of International Business)

No stated data access or replication policy

Journal of Management Studies (Wiley)

No stated data access or replication policy

Journal of Marketing (American Marketing Association)

Data access not required

Journal of Marketing Research (American Marketing Association)

Data access not required

 

Journal of Operations Management (Elsevier)

No stated data access or replication policy

Journal of Political Economy (University of Chicago)

Mandatory replication policy

 

Journal of the American Statistical Association (American Statistical Association)

Optional Data Policy

Management Science (Informs)

No stated data access or replication policy

Marketing Science (Informs)

Mandatory replication policy

MIS Quarterly (Management Information Systems Research Centre, University of Minnesota)

No stated data access or replication policy

Operations Research (Informs)

No stated data access or replication policy

Organization Science (Informs)

No stated data access or replication policy

Organization Studies (SAGE)

No stated data access or replication policy

Organizational Behaviour and Human Decision Processes (Elsevier)

No stated data access or replication policy

Production and Operations Management (Wiley)

No stated data access or replication policy

Quarterly Journal of Economics (MIT)

No stated data access or replication policy

Rand Journal of Economics (The Rand Corporation, Wiley)

No stated data access or replication policy

Review of Accounting Studies (Springer)

No stated data access or replication policy

Review of Financial Studies (Oxford University Press)

No stated data access or replication policy

Sloan Management Review (MIT)

No stated data access or replication policy

Strategic Management Journal (Wiley)

No stated data access or replication policy

Right now the internet is blowing up with more news on the retracted LaCour and Green study.

But this post is mostly about a conference I attended today on replication in the social sciences hosted by the International Institute for Impact Evaluation (3ie). The conference followed Chatham House rules, essentially asking us to not attribute any of the comments to individuals.

The conference was mostly focused on development economics. 3ie has an ambitious replication initiative focused on a handful of influential papers. What I found really interesting is the tone of the discussion in this audience versus most of my interactions with political scientists. Numerous commentators make direct or indirect claims that the editors of the (econ) journals were the problem. They either had no replication policy or didn’t enforce their replication standards. One overview of the status of replication in economics is here.

As a political scientist, I felt like I was going back in time. I haven’t checked every journal, but I was under the impression that just about every major political science journal has a replication policy in place.

Now I checked. There is a damning study claiming only 18 of 180 political science journals have replication policies. The good news is that the American Political Science Review, American Journal of Political Science, and Journal of Politics all have replication policies and that impact factor is one of the best predictors of a replication policy.  But a few excellent journals do not.

Many journals now require archived data as a condition for publication and a few journals like Political Science Research and Methods and the American Journal of Political Science require a technical replication before publishing. My prediction is that within the next five year most of the major political science journals will require replication as a condition for publication.

I don’t have too much more to say about this, other than there seems to be major cultural differences in views on replication across fields that affects data sharing.

A number of presenters discussed the different types (and terms) of replication. Some of this seemed like inside baseball until we began discussing Michael Clemens paper on defining replication. I think the most important idea here is that the term “replication” carries a lot of baggage for the original researcher. Individuals (and journals) should encourage “replication” but be very careful in labeling a study as “failing to replicate”. For example, if a scholar made up the data, that is clearly a failure to replicate. But what if a study of land titling in country X doesn’t yield the same results as an identical land titling study as country Y. Labeling the original study as a failure to replicate suggests wrongdoing by the original author.

For those of us that have followed some of the nasty exchanges between original authors and authors of replication studies (some of these 3ie studies) it is easy to see how the threat of a label of “failed replication” could lead to defensiveness.  Many of the discussants highlighted the importance of engaging the original authors in a dialogue.  Our goal shouldn’t be to catch mistakes by authors. It is to correct them to help us learn more about the world.

One of the most thought provoking presentation was by Brian Nosek on the work being done through the Open Science Framework. Brian presented a number of replication studies.  This includes:

• The Reproducibility Project: Using a single year (2008) and three top psychology journals, 270 authors attempted to replicate 100 studies. There is a lot of information here, but the quick overview is that a very large percentage of the studies that attempted to collected new data and conduct the same analysis had insignificant results and smaller effects sizes. Given Clemens point on the definition of replication, let’s not call these failed replications. At least one person in the room labeled this publication bias. Either way, it is problematic.
• The Many Labs Project assigned 27 teams to attempt to answer the same substantive questions with the same data, but they had the discretion on the coding, method, covariates and any other specification decisions. There was huge variation in the answers that different findings.
• Nosek presented some evidence on which types of studies failed to replicate. He went through this very quickly and I can’t find any supporting materials online. But what was especially interesting is that an elite survey of experts in the field didn’t do an especially great job in predicting which studies replicated, but a prediction market did. The prediction markets under-predicted failed replications, but overall did a pretty decent job.  Basically, given the right (monetary incentives), experts could sniff out the studies that weren’t going to replicate.

Within all of the presentations there was an acknowledgement of incentive problems from original authors (to have a finding and not be proven wrong) and replication authors (to find something new/wrong to report). But I didn’t hear any clear solutions.

Finally, there was very little discussion of the LaCour and Green controversy. One commentator noted that this is a case of  replication success, not a failure. By making the data available science/Science was able to correct itself.

I’m not sure I am as optimistic. But it was nice to see that our discussions weren’t completely derailed by this crazy situation.

A few more sources.

Great resources from Gary King on replication here

Another study that attempts to define replication here

I forgot about Andrew Gelman’s Garden of Forking Paths

I am finishing my first academic year as faculty in a business school after twelve years in a political science department. As the only political scientists in our school I often have to explain the different political science journals.

Business schools are an exciting mix of faculty with Ph.D.s in management, economics, sociology, history, statistics, psychology, and political science. On top of these differences in training, faculty are housed in departments as different as marketing, finance, accounting, strategy, international business, decision science, and information systems.

These differences often require benchmark “lists” of accepted journals for everything from internal promotion, allocating teaching loads (reduced teaching for “research active” faculty) to the ability to advise Ph.D. students.

There are three broadly used lists. The FT 45, the UT-Dallas list, and the most comprehensive Association of Business Schools (ABS). The 2015 ABS guide just came out.  ABS ranks over 1,400 journals from 4-1 with 4 being the top ranking. Ok, they also give a * to some 4 journals, giving them. Grade inflation.

Just to give you some flavor for this, ABS ranks many economics journals. 23 are ranked 4 and 68 are ranking three.

Psychology also has their own set of rankings . Other fields, like sociology are ranked within the catch-all “Social Sciences” lists.

Without editorializing, here is the full list of all of the political science journals I could find.

Political Science Journals Ranked “4”

 

None

Political Science Journals Ranked “3”

Quarterly Journal of Political Science (added in 2014)

Electoral Studies (added in 2014)

Public Choice

New Political Economy

Public Opinion Quarterly (added in 2014)

Review of International Political Economy

West European Politics (added in 2014)

Political Science Journals Ranked “2”

Economics and Politics

Political Studies

Political Science Journals Ranked “1”

None

In a previous blog post I proposed a Public Policy Survey of Political Scientists. I proposed using the potential reviewer pool from an academic journal to identify a small group of experts to ask about a pressing public policy issue.

Michael Colaresi, journal co-editor at International Interactions, has agreed to use their potential review pool to identify experts on a given topic. As I noted in a previous post, there are costs and benefits of focusing on a journal review pool, as opposed to a broader membership. This post presents our first trial run.

We asked a total of 50 academics, all experts in this area, three questions about U.S. policy towards ISIS. We received 30 responses. Here we present the raw results without any commentary. Michael put together some nice graphics on the three questions and the correlations across questions.

All three questions were statements and respondents were presented with five options.

Question 1: The current airstrikes on ISIS will roll back their advances in the region (N=30).

Question 2: I support the President’s decision to strike ISIS.

Question 3: If the airstrikes against ISIS continues beyond two years in their current intensity, the majority of the public will not support continued airstrikes against ISIS.

Comments from the Respondents

For each question we gave respondents the option to provide additional background information or justifications for their answers. I am not going to provide individual comments here, but there were a number of common themes.

First, a number of respondents indicated that the airstrikes on ISIS aren’t sufficient to roll back ISIS, although they can slow or stop ISIS advances. A number of respondents also indicated that additional military force, including ground troops, would be necessary to roll back ISIS.

Second, while the majority of respondents agreed with the President’s airstrikes against ISIS, a few respondents indicated that this policy will likely fall short of the ambitious goals of stopping ISIS.

Third, respondents were not only mixed on the third question, the comments indicated an even larger divide on public support for bombing. But the most common response was that public support was largely contingent the U.S. avoiding any casualties.

Interpretation and Post-Game

The goal of this survey is to present expert opinions on a pressing policy issue. I have my own interpretation of the implications of this survey, and have some thoughts on the value (and limitations) of this type of expert survey. But this is for the next blog post.

We will certainly run another one of these surveys using the International Interactions potential review pool again next month. Please send me an email if you have thoughts about questions.

Can I add an Appendix to a Blog Post?

In case you are wondering about the correlations between the three questions:

Yesterday I proposed piloting a public policy survey of political science researchers.

This blog post had about 400 hits, which is sadly a very good day for my new blog. Either this is a good idea or a dumb idea. Don’t answer that.

I’ve received some excellent feedback on the idea. A few things that popped up:

First, the William & Mary Teaching and Research in International Relations (TRIP) snap poll actually does something very similar to what I proposed. This survey panel includes almost 3,000 teachers and researchers in international relations. While this survey is pretty broad, they ask people to identify their main fields of interest.

Second, quite a few people mentioned The Good Judgement Project.  This project aims to use crowdsourcing to predict world events.  Check out their leadership team.  This is no joke.

I like both of these projects quite a bit. So why should I provide yet another public policy survey?

My goal is to harness the expertise of a group of researchers active on a particular topic. To be honest, I am not sure how much this will differ from TRIP survey or Good Judgement Project results. Yes, Tetlock, I have read your work.

But why not try?

The worst case scenario is that I look silly. This happens all of the time. The best case scenario is that this is a great idea and someone or some group that is better trained than me actually does the right.

Maybe there is no upside.  Similar to all of my NSF grant proposals.

Here is the plan:

• I have constructed a draft Qualtrics survey here: http://tinyurl.com/lv4rzba
• International Interactions did a quick check of their reviewer pool on the topic and we can easily find 50 researchers to survey.
• I refined the survey and we send it out.
• We collect the responses and post them.
• Nobody reads this post or the results.

Feel free to take the survey for kicks.  To be clear, this survey isn’t fielded yet.  Your answers are just for my amusement.

I have been quietly pitching an idea of establishing a survey political scientists modeled after the University of Chicago’s Booth School IGM Panel. In the IGM survey, they ask an established panel of top economists one or two survey questions on topics like the impact of the minimum wage on unemployment, the economic returns to infrastructure spending, and the predicted economic impact of Scottish independence. See here for their latest survey:

My idea is slightly different. Rather than surveying the same group of scholars, why not ask different experts based on the topic? This allows us to harness the power of specialized knowledge and to include a much wider set of individuals to weigh in on topics.

The problem with this sort of proposal is that I am sure it will come under fire on how I select questions and how I establish a panel.

Rather than debate this, why not just give it a try?

So here is my proposal. First, I am going to select a questions, actually statements, with some feedback via this website or email.

Here is my first cut. How much do you agree or disagree with the following statements?

Statement 1: Effectiveness of airstrikes on ISIS

The current airstrikes on ISIS will roll back their advances in the region.

1. Strongly agree
2. Agree
3. Neither agree nor disagree
4. Disagree
5. Strongly Disagree
6. No opinion

Statement 2: Support for Strikes

I support the President’s decision to strike ISIS.

1. Strongly agree
2. Agree
3. Neither Agree nor disagree
4. Disagree
5. Strongly Disagree
6. No opinion

Statement 3: If the airstrikes against ISIS continues beyond two years, the public will not support continued action against ISIS.

1. Strongly agree
2. Agree
3. Neither agree nor disagree
4. Disagree
5. Strongly Disagree
6. No Opinion

Second, the editors at the journal International Interactions, have agreed to help me identify scholars on each topic, similarly to how they would find reviewers for research articles. Together, we will then ask potential panelists whether they would be willing to be a part of this project and provide answers.

There are obviously other models, but I’d like to at least give this one a try. Send me an email if you have suggestions.

Update:  I’ve already received some feedback on question working.  Minor changes incorporated.

This summer I moved from my position as an associate professor in political science to an associate professor in International Business at George Washington University.

During the summer I attended my two disciplines two flagship conferences.  The Academy of Management (AOM) conference and the American Political Science Association (APSA) conference.

For my political science readers, APSA is all too familiar.  Every Labor Day (yes, on Labor Day) we converge to a city and give very short paper presentations to tiny audiences.  We mostly tell ourselves that the conference is more about seeing colleagues, setting a hard deadline for a paper draft, and for professional development and networking.  This last part is very important to the many graduate students on the academic job market that kicks off as early as September 1.  So we go through a lot of effort and expense for a conference that is mostly valuable for the informal networkings.  And it is on Labor Day.  For now.

Many of the motivations for attending AOM are the same and the structure of panels (4 papers to a panel leaving each presenter 15 minutes if you’re lucky to present months of work) is mostly the same.  But there are some significant differences.

While APSA kicks off the academic job market, very few schools actually use APSA for interviews.  One reason for this is that APSA is just a little too early and most schools haven’t even started collecting applications yet.  But my take is not using our flagship conference to help to make better hiring decisions is a wasted opportunity.  We should probably move to after Labor Day.

But what I find even more odd is that all of this infrastructure could be harnessed for better research and networking experiences.  Here are two things that APSA could learn from AOM.

1.  A huge percentage of the panels, most of the panels in the first two days, are formally devoted to professional development.  I attended a day long event for junior faculty/new to management scholars.  I learned as much in this one day about my new profession as I probably learned in my first two years fumbling around with my career.  The highlights included a panel with journal editors and a breakout session on grant funding.

APSA has these professional development opportunities but the scale is quite different.

2.  AOM has a much more serious review process for paper and panel proposals.  Many of the management conferences require mini papers (something like 10 pages) and have external reviews of these proposals.  Unlike section heads in APSA who are required to pick and choose proposals based on a page or so abstract, AOM papers are at least partially vetted and authors are given feedback from the peer reviewers.  For my AOM panel we responded to peer reviews and made comments on the other presenters projects.

I think the AOM model institutionalizes better research and professional development.  The obvious major drawback from the AOM model is time.

I’m not sure if the AOM system is better, but it is at least worth considering.  We should also move it from Labor Day.

There is a lot that management could learn from political science.  More to follow.