One of the most important things in a research project is a good research question. It is also one of the most difficult things. In the past twenty years, I have learned a few things about research questions that do not work well. Here is a list of fifteen things you should avoid, plus solutions, and a checklist.

  1. Identifying a topic, but not asking a question

If you write a sentence like ‘This research deals with aspects of Corporate Social Responsibility’ or ‘In this paper, I explore Corporate Social Responsibility from the perspective of theories on communication’ you do not ask a question. It is better to have a specific question than to have a broad topic (Eidlin, 2011). The problem with not asking a question is that you do not force yourself to say what you are really trying to research and create too much room for diversions and side tracks. A lack of focus is not merely a problem for yourself, but also for readers like your supervisor.

The solution to this problem is to think in terms of questions, and also write in terms of questions. It may seem obvious, but a research question is literally a sentence that ends with a question mark (“?”).

A good research question starts with words like ‘How…’ or ‘To what extent…’ or ‘Why…’. For descriptive research questions, start with ‘How…’ or ‘To what extent…’. For explanatory questions, use ‘Why…’. When you know that you are interested in a certain topic, try to formulate a question about it that starts with these words. This strategy does not always work. You may conclude that you need to think and read more before you are ready to formulate your research question. For instance, if you try to rephrase the examples above in terms of questions you will see that you do not get very far. Saying that you deal with ‘aspects of CSR’ is a way of saying you are not sure what your question is. Which aspects are important? A better example would be: ‘How do corporations decide how much to invest in a CSR strategy?’ or ‘Why do some corporations invest more in CSR than others?’ or even better: ‘How can differences between corporations in CSR strategies and investments be explained?’

A good research question is a question, not a topic.

2. Asking a definitional question

The question ‘What is altruism?’ is fascinating from many points of view and has puzzled numerous philosophers and researchers throughout history. It is a very complicated and challenging question. In fact, it is a question that exceeds the scope of your project. Also, it is a question that you cannot answer with new empirical research. The solution to this problem is not to ask a definitional question in your text. This is not to say that you should not ask definitional questions, or that they are meaningless. Of course you need to know what you mean by the concepts you use. Also you need to make sure that the reader knows the meaning of the concepts you use. Imagine you are a zoo keeper describing a rare species that you happen to have but that is hiding from visitors. You want your visitors to recognize the animal when it exposes itself. If you use a concept that is not common knowledge, explain its meaning the first time you use it. Just take the most common definition from the literature. If there is controversy in the literature about the definition of concepts that are important for your research, you can acknowledge alternative definitions in a footnote. But do not bother the reader with discussions about words when they do not have consequences for your hypotheses.

A good research question is testable.

3. Asking a meta question

The question ‘How has altruism been explained in previous research?’ is an example of a meta question. The meta question may be useful for you to answer when you are trying to get an overview of the literature. It could even be the perfect question to pose when you are writing a literature review. However, it is not a useful question for your empirical research.

The solution to this problem is to find the substantial questions that are hidden in the meta question. In this case an interesting (but very broad and difficult) question is ‘How can altruism be explained?’ You should ask a substantial question instead of a meta question.

A good research question is substantive.

4. Asking a yes or no question

An example of a ‘yes or no’ question is: ‘Does empathy influence altruistic behavior?’ This is an explanatory ‘Does X influence Y’ question. In the social sciences, it is hard to find either/or phenomena, like a light switch or an on/off button. Some things are either black or white, but most things are in shades of grey. Yes/no questions are crude and yield less informative answers than more refined questions. It is better to ask the more refined questions right away. You want your research to be as informative as possible. After all, the goal of science is to learn, not to be right.

The solution to the problem of asking a ‘yes/no’ question is to rephrase it in terms of a ‘to what extent’ question or in terms of a conditional question. ‘Yes/no’ questions are encompassed in ‘To what extent’ questions. So instead of the question ‘Does empathy influence altruistic behavior?’ it is better to ask: ‘To what extent is altruistic behavior motivated by empathy?’ If the answer to your initial yes/no question is positive, and altruistic behavior is indeed motivated by empathy, the follow-up questions are going to be: “How strong is the influence?”, “How does the influence work?”, “For whom is the influence stronger?” and “In which circumstances is the influence weaker?”.

Also, in almost all ‘Does X influence Y?’ questions there will be evidence from previous research that already shows some relationship. Typically, the research is less clear on how the relationship can be explained, how the relationship varies from one person to another, or how the strength of the relationship may depend on characteristics of the situation. Your study is more relevant if it asks a more refined follow-up question to which the answer is not yet known, such as ‘In which circumstances is altruistic behavior more strongly motivated by empathic concern?’

A good research question is informative.

5. Asking a black and white question

‘Is altruistic behavior determined by altruism or egoism?’ This type of question is different from the yes/no question because it suggests two positive alternatives. However, it suffers from the same problem because it assumes that the world is black and white: the motivation is either altruism, or egoism. In reality, people will be motivated by both altruism and egoism.

The solution to the black and white question is the same as the solution to the yes/no question: ask a ‘To what extent?’ question instead. The answer could be: ‘mostly by egoism’, ‘almost entirely by egoism’, or even in terms of a percentage: ‘for 95% by egoism’. In theory, the answer could be ‘entirely by egoism’. Only in that unlikely case, it would have been meaningful to ask a yes/no question.

A good research question is about shades of grey.

6. Including incorrect assumptions

The question “Why are women more generous than men?” assumes that women are more generous than men. Though in some contexts this assumption may be true, in others it may be incorrect (for a review, see Wiepking & Bekkers, 2012). Findings reported in the literature may not hold in other contexts. Check the assumptions in the question you ask.

A good research question is based on valid assumptions.

7. Omitting time and place

By including the time and place of your research, you situate the findings in a specific context. Omitting time and place suggests you will give a timeless answer to a question that is valid irrespective of the context of your study. It is very hard to uncover universal truths about the essentials of human nature by conducting an experiment among one hundred of your fellow students at your university or even by analyzing survey data about one hundred thousand citizens across the world. The solution is obvious: specify the time and the place for the phenomena that you are exploring and  trying to explain. Note that this solution may go against the advice that questions should be as informative as possible.

A good research question identifies time and place.

8. Leaving the comparison implicit

When you ask a question that involves a comparison, make it explicit. The fact that annual levels of giving to charity by high incomes in the Netherlands are less than 0.4% of their income (Bekkers, De Wit & Wiepking, 2017, p. 65) may lead you to ask: “Why do high income people give so little to charity?” When you ask a question like this, make the comparison explicit. A better formulation is: “Why do high income people give a lower proportion of income to charity than low income people?” This question is even more intriguing when you know that people in the lowest decile of the income distribution give about 1.2% of their income per year.

A question like: “Why do Protestants give so much to charity?” contains an implicit comparison between Protestants and other groups. Again, as per #6 above, when you make comparisons, check them. Though indeed Protestants tend to give more than Catholics or the non-religious, comparisons with other religious groups such as Muslims or Jews tend to show these groups give even more (Bekkers & Wiepking, 2011). A better research question would be “How can current differences in the US between religious groups in charitable giving be explained?”

In this particular example, you could doubt whether the amounts are ‘so much’ compared to the religious norm of tithing. Very few religious people actually give a tenth of their income (James & Jones, 2011). In this case, the more interesting research question would be: “How can current differences between religious groups in the US in the adherence to norms on charitable giving be explained?”

Generally speaking, questions including the word “so” include comparisons that you need to specify, and assumptions that you need to check.

When you ask questions about differences, make sure that they are meaningful. The question about the differences in generosity of citizens across the income distribution is about a meaningful difference: the generosity of the 10% lowest incomes is three times that of the 10% highest earning households in the top decile. Similarly, differences between religious groups are sizeable. The amount donated by Protestants in the Netherlands in 2015 was more than six times the amount donated by the non-religious. There is no rule of thumb for what constitutes a meaningful difference. Even a difference of just 1% can be meaningful when it has meaningful consequences.

A good research question is a about meaningful differences.

9. Asking a question that is too broad

‘To what extent is volunteering motivated by altruism?’. If you ask a very broad question such as this one, you cannot expect to finish your work by the deadline. There will simply be too many aspects that you should discuss to answer the question. In the course of dealing with these aspects, you run the risk of getting side tracked, encountering dead ends, and muddy paths and swamps. Remember to not only plan ahead but also check your schedule.

Questions that are too broad not only take too much time, but are also misleading to your readers. In all likelihood your research is not going to produce an answer to the question to what extent volunteering is motivated by altruism. You may be able to answer that question for a specific form of volunteering, among a specific group of people. Your question should give the reader an impression of the kind of research you are going to do to answer it.

The solution to this problem is to specify the phenomena that you are studying a bit further. You should specify time and place. You can specify aspects of the phenomenon you are interested in, e.g. by identifying a specific type of volunteering, or by making a comparison between different types, e.g., volunteering for religious or non-religious organizations. Also you can specify relations between aspects of the phenomenon you are interested in and other variables, e.g. by identifying conditions in which altruism may occur.

A good research question is narrow enough to be answered.

10. Asking a question that is too narrow

An example of a very narrow research question is: “How can a sports club that obliges members to serve as volunteers on Sundays deal with the problem of ‘no shows’?” This is a specific policy question that may have a high level of societal relevance to the volunteer manager and the president of the club, because they cannot run it without volunteers on Sundays. But the scientific relevance of the question may low, previous research will be difficult to find or non-existent, and the empirical research you can do to answer the question will be limited in scope.

The solution to this problem is to broaden the question. In this particular case, you could ask: “What is the relationship between mandatory service requirements and the willingness to volunteer among members of sports clubs?” A follow-up question would then be “Which conditions increase the willingness to volunteer for sports clubs with mandatory service requirements?”

A good research question is broad enough to be interesting.

11. Asking a question about a state rather than a change

“How is ethnic diversity related to civic engagement?” is a question about a state: it asks about the association between two variables at a given point in time. This kind of question is not ideal because theories in the social sciences can be tested more forcefully if their implications are formulated in terms of how things change. Try to reformulate your predictions about states at a given point in time into predictions about changes over time.

For instance, instead of the descriptive question about the relationship between ethnic diversity and civic engagement you could formulate the explanatory question: “How does ethnic diversity affect civic engagement?” This question will lead you to ask: “How does civic engagement change in communities that become more (or less) diverse?” By answering this question you will learn more than by answering the question whether civic engagement is higher in more diverse communities. The former research question leads you to look for change over time; the latter leads you to look for differences between communities at some point in time. In the latter case, lower levels of civic engagement in more diverse communities could reflect an influence of diversity on engagement as well as the reverse. In the former case you can look at the timing of events. If you know that first diversity increased and then the level of engagement declined, you may still not be able to conclude that the increase in diversity lowered the level of engagement, but at least you know that the change in engagement did not change the level of diversity.

A good research question is about changes rather than states.

12. Asking “Can” Questions about the World of Possibilities

“Can people overcome adversity in life?” is an example of a “can” question. It is not a good question, because it is too easy to answer it. As soon as you find one positive case, the answer is ‘yes’ and you’re done. The same goes for “Can X influence Y” questions. The answer in almost all cases is going to be ‘yes’. But that is not a very informative answer. We learn next to nothing from a demonstration that something is possible, unless everyone thinks it is not. We want to know whether X always affects Y, or whether there are situations in which the influence is absent, or stronger than in others. Is the influence similar for people with different characteristics, in different social groups and countries? How does the influence come about? We learn more when you report about the conditions in which it did not work. If what you are looking for is knowledge about moderators or mediators, it is better to ask about them right away.

“How can people make sense of adversity in life?” is an intriguing question, but not one that you will be able to answer with your research. Again, the trouble is in the word ‘can’, but in this case almost anything is possible. Some people make sense of adversity by gardening. Others by talking to friends. Still others turn to God. In all likelihood you would not be satisfied with any of these answers. Not all answers are equally useful or correct, but there is no way of knowing from the way your question is formulated. Because of the word ‘can’ it is not clear what kind of answer you are looking for.

Research questions about possibilities are difficult to answer when they lack a clear sense of direction. If you do have a few possibilities in mind, mention them in your question. In this particular case, it would also be helpful to specify a criterion. People may make sense of adversity in many ways, but you are probably interested in the origins of these different ways or their consequences. If you really do not have a clue what you are looking for, you should spend more time reading previous research and thinking about theories that may apply.

A good research question is about reality rather than possibility.

13. Asking a meaningless question

“Why are owners of yellow cars more likely to be female than male?” is a rather meaningless question. The presumption may be correct, but the answer is not consequential, neither for theory, nor for practice – unless the yellow reduces accidents. Make sure your question answers a meaningful question. This means not only that the question is about an important issue or burning question – we will get to arguments about scientific and societal relevance in section 2.3 and 2.4 below. Also you should make clear to the reader that the answer to your question is consequential. What would be the consequences for theory or practice of potential answers to your research question? What would be an informative answer that tells you whether a certain theory is right or wrong?

A good research question is one with consequential answers.

14. Asking a question that you do not answer

Perhaps it is obvious, but it is worth stating explicitly: make sure that your research in fact answers the question that you state in the beginning. If you do not answer the question you raise, readers will be disappointed. The implication is that you should adapt your research question to the research design of your study, at three different stages in the empirical cycle.

Design. First, adapt your research question to your research design when you are designing the study. When you work with existing data, make sure that the research question can be answered with the data at hand. When you have the opportunity to collect new data, make sure that you actually measure the concepts in the research question.

When the data you have do not include randomized treatments or randomly allocated events in natural experiments, be careful to ask explanatory questions. Make sure you use a design that allows for causal inference. See sections 4.1 and 4.2 below. 

You can phrase your research question somewhat broader than the measurements allow, but avoid asking a very broad question when your data and measures provide tangential evidence. Adapt the research question to the possibilities that the type of data you will use. Specify the context when the data pose limits to the generalizability of the results – keeping issue #6 above in mind.

The rule also works the other way around: when your data and methods do not allow you to answer the research question, you can also choose to continue searching for more suitable data and methods that do allow you to answer the question.

Analysis. Once you are in the fieldwork or data analysis stage, you may discover that what you thought was possible is in fact impossible. The dataset does not cover the countries or time points that you were interested in, it does not include good operationalizations of the concepts in your question, or it does not have the longitudinal structure for the variables you are most interested in. Also the assumptions for causal inference may not be satisfied. In these cases reformulating your research question to a question that your analysis does allow you to answer is often the best option.

Writing. Once you are done with your empirical work, check what you promised in the description of the design of your study. Do not mislead your readers or disappoint them by asking a different question than you are able to answer. To some extent, this is a matter of planning: reformulate your research question again and again, until it is capturing your research perfectly. If your research changes along the way, reformulate your original question.

A good research question is answered by your research.

15. Asking a question that others have answered already

Finally, you want to make sure that the question you are asking has not yet been answered already. Once you have started to gather previous research, you may be discouraged by how much research has already been done. This does not mean that your research is useless. Instead, you will often discover that most previous research is useless. Whatever you think the bar is for what constitutes a good answer to your question, raise it a bit further. Very few studies have a waterproof design and analyze complete and accurate data to produce definitive evidence on your question and conclude by saying that future research is not necessary.

A good research question is not yet answered by previous research.