ResearchMethods/Introduction/EvalResQues

Is this a good questions? Evaluating research questions

There are lots of questions researchers can ask, and not enough resources (time, money, access to participants...) to address them all. This means that researchers must evaluate the research questions they generate so that they can choose which ones to pursue. In this section, we consider two criteria for evaluating research questions: the interestingness of the question and the feasibility of answering it.

Is it interesting?

Do military conflicts encourage people's migration? Are men more likely to get involved in protests than men? How do councils distribute funds for traffic infrastructure? Do people prefer to vote electronically or in person?

Although it would be a fairly simple matter to design a study and collect data to answer these questions, you probably would not want to because they are not interesting. We are not talking here about whether a research question is interesting to us personally but whether it is interesting to people more generally and, especially, to the scientific community.

What makes a research question interesting in this sense? Here we look at three factors that affect the interestingness of a research question: the answer is in doubt, the answer fills a gap in the research literature, and the answer has important practical implications.

• Doubt. If the answer is obvious, the question is not interesting. A research question is interesting to the extent that its answer is in doubt. Obviously, questions that have been answered by scientific research are no longer interesting as the subject of new empirical research. In addition, even if a question has not been answered by scientific research, that does not necessarily make it interesting. There has to be some reasonable chance that the answer to the question will be something that we did not already know. But how can you assess this before actually collecting data? One approach is to try to think of reasons to expect different answers to the question—especially ones that seem to conflict with common sense. If you can think of reasons to expect at least two different answers, then the question may be interesting. If you can think of reasons to expect only one answer, then it probably is not. The question of whether women are less likely to get involved in protest than men is interesting because there are reasons to expect both answers. The existence of the stereotype itself suggests the answer could be no, but the fact that women tend to be more aware of social issues suggests the answer could be no. The question of whether people tend to migrate out of areas of military conflict is not interesting because there is absolutely no reason to think that the answer could be anything other than a resounding yes.
• Filling a gap. If at least part that the question has not already been answered by scientific research, then the answer will fill a gap in the research literature. On the other hand, it also means that the question may seem obvious for people who are familiar with the research literature. For example, the answer on how councils allocate money for infrastructure may be obvious to anyone who is familiar with the field of local governance.
• Importance. The final factor to consider when deciding whether a research question is interesting is whether its answer has important practical implications. Again, the question of whether human figure drawings help children recall information about war experiences they went through has important implications for how children are interviewed war-crime cases. The question of whether prefer to vote electronically is important to countries where voting is not obligatory, but in a country like Australia,it may not be interesting.

Is it feasible?

A second important criterion for evaluating research questions is the feasibility of successfully answering them. There are many factors that affect feasibility, including time, money, equipment and materials, technical knowledge and skill, and access to research participants. Clearly, researchers need to take these factors into account so that they do not waste time and effort pursuing research that they cannot complete successfully. Some international studies may be difficult to carry out. The participants may reside in another country, or you may not speak the same language. Some studies have include longitudinal designs, in which participants are tracked over many years. Other studies may be complex, non-experimental, and involve several variables and complicated statistical analyses. That kind of research is usually performed by teams of highly trained researchers whose work is often supported in part by government and private grants. Also, research does not have to be complicated or difficult to produce interesting and important results. Looking through a sample of professional journals will also reveal studies that are relatively simple and easy to carry out—perhaps involving a convenience sample of university students or reading through archives.

• Re-use. A final point on feasibility is that it is generally good practice to use methods that have already been used successfully by other researchers for similar research. For example, if you want to identify people’s attitudes towards refugees, it would be a good idea to use one of the many approaches that have been used successfully by other researchers (e.g., a survey which was tested for reliability and validity). This is good for two reasons:
  • If it has been tried before, it is probably feasible.
  • Building on past methods provides greater continuity with previous research. This makes it easier to compare your results with those of other researchers and to understand the implications of their research for yours, and vice versa.

"It's been done"

What if you find that it has been studied scientifically? Should you give it up? Not necessarily!

• If the question has been studied scientifically, and the research published, it means that it is of interest to the scientific community.
• The question can almost certainly be refined so that its answer will still contribute something new to the research literature.

That's why it is worth refining the question, as described above:

• Examine the mechanism behind descriptive answers'
• Examine in a different context: cultural, national, political
• Examine additional effects of the phenomenon

Evaluate research questions

1. Post your research question to the course discussion forum.
2. Find someone else's research question, and evaluate it.
3. Remember! Constructive criticism means suggesting an improvement, not pointing out a weakness.

OPTIONAL - EXTENSION PRACTICE

1. Practice: Evaluate each of the research questions you generated in question 2 in terms of its interestingness based on the criteria discussed in this section.
2. Practice: Find an issue of a journal that publishes empirical research articles. Pick two studies, and rate each one in terms of how feasible it would be for you to replicate it with the resources available to you right now. Use the following rating scale: (1) You could replicate it essentially as reported. (2) You could replicate it with some simplifications. (3) You could not replicate it. Explain each rating.

Credits

Paul Price, (2013, Updated version), Section 2.2 Generating good research questions