Introduction from “Game Research Methods: An Overview”

Petri Lankoski & Staffan Björk

This volume is about methods in game research. In game research, wide variety of methods and research approaches are used. In many cases, researchers apply the method set from another discipline to study games or play because game research as discipline is not yet established as its own discipline and the researchers have been schooled in that other discipline. Although this may, in many cases, produce valuable research, we believe that game research qualifies as a research field in its own right. As such, it would benefit game researchers to have collections of relevant research methods described and developed specifically for this type of research. Two direct benefits of this would be to illustrate the variety of methods that are possible to apply in game research and to mitigate some of the problems; each new researchers has to reinvent how methods from other fields can or need to be adjusted to work for game research.

In our own work, we have been struggling to find suitable literature for courses focusing on game research methods. We have however noticed that researchers have been developing such methods through publications in journals and at conferences. Based on this, we decided to invite game researchers to share their knowledge about these research methods; the book you are now reading is the result of the responses we received to our call for methods. We aim to provide introduction to various research methods in context of games that would be usable to students in various levels. In addition, we hope that the collection is useful to supervisors who are not familiar with the games.

Research, according to Oxford Dictionary, is “[t]he systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions” (Research, n.d.). A method is a specific procedure for gathering data and analyzing data. Data can be gathered and analyzed in many different ways. Data can be gathered, for example, using questionnaires, interviews, and psychophysiological measurements. Quantitative analysis typically involves building hypothesis prior data gathering and testing those hypotheses using statistical methods to examine relations between measured variables. This means that the typical result is the rejection of a hypothesis or presentation of data that support it. In contrast, qualitative analysis of data involves generating categories based on data, abstracting the data based on these categories and building explanations of studied phenomena based on abstractions. Through this, qualitative research aims to provide rich descriptions of phenomena focusing on meaning and understanding the phenomena. Qualitative and quantitative data and analysis can be integrated in mixed methods approaches to gain richer understanding of a phenomenon.

The objectivity of scientific work and theories has been criticized. Hume’s (2006) argument that causation cannot be directly observed and that it is not possible to show things will behave the same way in the future proposed challenge to science in general. Popper (2002) agrees and claims that theories cannot be proven to be true or, in other words, validated but only falsified. Popper also maintains that a theory is not scientific if the theory does not produce predictions that can be used to falsify the theory. Popper’s account is not without problems because explanatory theories are considered valid scientific theories. Niiniluoto (1993), among others, argues that, for example, the evolutions theory is a good scientific theory that does not really have predictive power.

According to Dewey (2005), a theory is good if one can use it to predict or describe a phenomenon in ways that are useful. However, this does not mean that theories and models should not strive to describe data accurately or provide good predictions how things behave. This means that research is an iterative process where theories are rejected or refined based on new findings.

Yet another challenge to the objective knowledge is that observations are theory-laden (Kuhn, 2012). Observations depend on skills and, in some cases, from biological qualities:

• Visual perception depends on skill and biological factors: seeing certain kinds of dirt in the kitchen tells to some about critter visitors, whereas others see just dirt.
• People living in high altitude areas or near equator have often reduced sensitivity to see blueness (Pettersson, 1982). Hence, these people see colors differently.
• Determining what kind of atmosphere distant planets have. Because the gases of the atmosphere cannot be directly observed, light curves are used to determine the consistent of the atmosphere. Here, the observation is filled with the presupposition coming from the theory of refraction and observation is not independent from the theory.

 

The above discussion is intended to illustrate some reasons why scientific knowledge can fallible and approximate. However, science is still seen as the most reliable way to form knowledge about the world and observable phenomena. Therefore, it is crucial to be able to discuss about the quality of the results research produces. In all kinds of research, the question of validity and reliability is important. Procedures for ensuring validity of results should be part of every phase in research. These terms have different definitions in quantitative and qualitative.

Quantitative reliability is used to denote that the approach measures the same thing in the same way every time it is used. Validity refers to different related things: construct validity refers to that one is measuring X when one intends to measure X (e.g., if measuring zygomaticus major muscle is a good way to evaluate that a person is happy) and external validity or generalizability denotes if the results apply to different samples within the same population. In addition, quantitative reliability, validity, generalizability are discussed in more detail in Landers and Bauer (chapter 10) and in Lieberoth, Wellnizt and Aargaard (chapter 11).

Validity in qualitative research can be described as a factual accuracy between the data and the researcher’s account. Moreover, validity requires that the inferences made are grounded to data (cf.,Maxwell, 1992). Reliability can be described as consistency of approach over time and over different researchers. This means, for example, that different researchers should end up with similar codings or categories form the data (Creswell 2014). Saturation is a related concept. Saturation means that there is a point in qualitative study when collecting and analyzing data do not reveal new insights to the question in scrutiny (Guest, Bunce and Johnson, 2006). Validation and validity are discussed more in Lankoski and Björk (chapter 3) and Cote and Raz (chapter 7).

The issues of validity and reliability typically become settled in research fields as they mature and various approaches have been explored. However, some phenomena can be approached from several different approaches which all are appropriate and which inform one other. This creates interdisciplinary or multidisciplinary fields where many different views of how research should be conducted coexist. Game research is both a multidisciplinary research field and a new one. It is multidisciplinary because one can conduct research on the games itself and one can study the people who play games: on the actual activity of playing games, one can, in addition, choose many different theoretical and methodological grounding for each of approaches. It is due to this fact that we issued a general call for methods for this book; we wished to present the variety of methods possible and help researchers new to game research navigate which methods will be most appropriate for them.

Next, we provide a short introduction to research design. After that we give an overview on addressing quality of a research. Last, we outline the chapters in this collection.

Recommended reading

• Ladyman, J., 2002. Understanding philosophy of science. London: Routledge.

Research design

Research design is a systematic plan for an investigation. Research consists of multiple phases. The research first needs to have an idea about the topic of the research. After having the topic, reviewing research literature about the topic helps to determine if the topic is worth to pursue (e.g., what questions are already covered and how the topics have been studied), building understanding of the research topic, and helping formulating one’s research question and framing the problem. In quantitative (and mixed methods) approaches literature review has an additional role where the literature review along with the research question is used when formulating hypothesis for the study.

The research method should be selected based on the research question or topic. What is or what are the good method(s) to approach the area? However, it may also be prudent to be pragmatic and consider what skills various methods require. If there are no researchers in the team that have a prior experience to the method, extra care and time are needed to ensure that that the one knows the limitations and the possibilities of the method in adequate level.

Typical steps before research begins are as follows:

• Identifying the topic of research.
• Review of literature that relates to the topic area.
• Specify research question or hypothesis.
• Specifying what kinds of data are needed to study the question (the unit of analysis and the unit of observation; see below).
• Specifying what kinds of methods are needed to analyze the data.

 

Not all research designs use all these steps before research begins. For example, a grounded theory approach considers literature as a data for the research among the other types of data. As another example, quantitative explorative designs do not develop hypothesis because the point of the design is to build an initial understanding of a phenomenon.

A very important aspect of research design is the unit of analysis. The unit of analysis is the target of the study or more specifically what is the basic data type being studied. Looking at other fields of research, units of analysis can, in many cases, be described as related to the dimension of scale. Physics studies the very miniscule such as atoms while astronomy the large scale objects such as stars and galaxies, with chemistry and geosciences lay in between. Many of the disciplines related to games instead focus on more abstract types of data. The unit of analysis can, for example, be the phenomenological experience of a player, social interactions in cooperative play of a game, formal features of a game, or formal features in a group of games.

The unit of analysis decided upon which level data are collected. For example, collecting data by observing people playing a multiplayer game may be an approach level observation when wanting to investigate social interactions. However, all that is observed may not be part of the unit of analysis. In aforementioned example, the unit of analysis could be the social interaction while playing, and other data gathered might be ignored in this analysis when it is not relevant for the focus of the analysis.

The unit of analysis is closely connected to research questions and methods and often depends on these. Observations might be a more suitable method to gather information about social interactions than interviews. Formal analysis is adequate to understand whether the games in a same genre have some common systemic features or genre can be studied also in using method in linguistic analysis. Adequate method here depends on one’s unit of analysis.

Selecting a method is a part of designing ones research process. In more general, the systematic design of research includes planning

• sampling (i.e., what is a target population and how reach informants from that population) and recruiting informants
• how are the data gathered in detail (e.g., interview questions or themes, questionnaire questions, or instrumentation)
• how are the data analyzed
• how are the data stored after study and how it is anonymize, or how the data will be destroyed.

 

To test that the research design works, a pilot study can be used. In pilot study, the data gathering is tested with a small amount of participants, and the results are analyzed according to the design. The pilot is intended for checking that the design works as intended. After the pilot study, the researchers conduct the actual study, analyze the results (or qualitative studies; gathering and analyzing are iterative process) and report the result. Formats of reporting the results of research vary. Hence, it is recommended to read the earlier (bachelors, masters or doctoral) theses, conference papers, or journal articles and study how studies are reported in that context.

Recommended reading

• Creswell, J.W., 2014. Research design: Qualitative, quantitative, and mixed method approaches. 4th ed. Los Angeles: Sage.

Ethical considerations

When conducting a research on people there are always ethical issues to be considered. Care should be taken to set up a study in a way that it does not harm participants. If there are risks, the participants should be aware of that risk before deciding whether to participate in the study. In these kinds of cases, the potential benefits of the study should outweigh the risks. Conducting the study might need approval of ethical committee.

When a study is planned to involve children, a special care should be taken. The guardian consent of the children is needed and the guardian should have details about the study so that they can make educated decision whether their children participation. All games are not suitable to children, and it might not be even legal in some areas to let the children play mature or adult only games. For example, asking children to play Manhunt (2003) to study their reactions to violent content is ethically very problematic because it is generally held view that that kind of content is not suitable for children. It is also illegal to let children play mature or adult only game in some countries. One might also require a permit for the study from the ethical committee of ones university.

Even if ones study design does not involve blind experiments (1), anonymizing the data and results is a good practice in most the cases. This is extremely important when conducting studies about sensitive issues such as sexuality to anonymize participants when reporting results. In addition to anonymization the names, the context descriptions of the study should not give away or lead to the participants.

How to deal with ethical issues in interviews are discussed in chapter 7–9.

Ethics apply research throughout the process. In more general level, honesty and standard for the work are an important to keep in mind. Honesty means that the result should reported accurately. Standards of the work means that research procedures follow the established standards of one’s discipline. In addition, giving credits to all who were involved in the research is considered good practice.

Recommended literature

• Shamoo, A.E., 2009. Responsible conduct of research. 2nd ed. Oxford: Oxford University Press.

What is in this book

This volume is divided in six parts. Outside of those six parts, we have Olsson’s (chapter 2) a look at on thesis writing and how to understand or structure argument chain in an essay.

Parts I–IV focus on different types of research design. The first two parts focus on qualitative, descriptive research designs. The first part focuses on methods that aim to study games, and the second part focuses on designs for studying players or play. The aim of descriptive designs is to provide rich descriptions about phenomenon. However, the data gathering approaches are different when one is studying games compared with play or players. The third part takes a look at quantitative designs where the aim of the design is to provide quantifiable information about the unit of analysis and relations between the studied variables or examines causal connections. The fourth part looks at mixed methods designs. In mixed methods designs, qualitative and quantitative methods are used in conjunction to study the same phenomenon from multiple perspectives. Part V looks at the role of game development in research.

This collection does not cover all possible methods that can be used to study games, play, or players. We do not cover, for example, methods in platform studies, philosophical design, or historical design. Platform studies focus on the possibilities, limitations and influence of the gaming platforms; for example, see Montroft and Bogost (2009). Philosophical designs are relevant, for example, if one aims to develop definitions or is interested in ontological questions. Historical designs are intended to collect evidence from the past. Although games propose some challenges in these areas, the rich methodologies in these areas apply well to games. A particular area which is not explored in this book is that of research in education; a generic overview of this can be found in Bishop-Clark and Dietz-Uhler (2012), which can then be focused toward games.

Below, we provide overview of each part in this collection.

I Qualitative approaches for studying games

The first part looks at qualitative approaches for studying games. Games are seen as data and the research build understanding on games and how they work, provide experiences or information to its players. All chapters in this part consider games as systems. The data collection method in these approaches is playing a game or games that is under scrutiny. The methods also make assumptions about players or abstract them in some ways; for example, in Lankoski and Björk (chapter 3), the player is seen only in terms of what actions they can perform. These kinds of methods can arguably be seen as fundamental to much game research. Because in one degree or another, these often are needed to be able to use any of the other approaches; it can, for example, be difficult to understand play or player behavior in a game if one does not know what constitutes the gameplay.

Lankoski and Björk (chapter 3) provide a method for analyzing and describing the core components, or primitives that regulate the gameplay of a game. Zagal and Mateas (chapter 4) present a formal analysis approach that focuses on describing time in games using the concept of temporal frames. Last, Sköld, Adams, Harviainen, and Huvila (chapter 5) describe methodology for analyzing games as information systems.

II Qualitative approaches for studying play and players

The second part, qualitative approaches for studying play and players, provides methods that focus to actual play or player experiences. The game(s) played provides context but is not the main interest of study when using these methods.

Brown (6) introduces ethnomethodology where players are studied in their natural environments by, for example, observing play making field notes. The natural environment discussed in the chapter is online game worlds and forums. The chapter discusses especially challenges that the researchers encounter when studying intimate situations such as erotic role-play.

The rest of chapters in this part deal with different interview methods. Cote and Raz (chapter 7) covers in-depth interviews, how to plan and conduct interviews, and how to analyze interview data using thematic analysis. The analysis approach is useful for all kinds of qualitative data. Eklund (chapter 8) focuses on group interviews, and Pitkänen (chapter 7) stimulated recall interview approach.

III Quantitative approaches

The third part provides introduction to quantitative approaches in game research.

Landers and Bauer (chapter 10) review the fundamentals of statistical methods in game research. Lieberoth, Wellnitz and Aagaard (chapter 11) discuss limitations challenges of quantitative and how to critically interpret the results of the quantitative studies. Järvelä, Kivikangas, Ekman and Ravaja (chapter 12) provide an introduction to psychophysiology and a practical guide using games as stimulus for psychophysiological studies. Schott and Marczak (chapter 13) discuss various algorithmic solutions to isolate specific audiovisual aspects of a videogame for quantitative analysis. Wallner and Kriglsteinin (chapter 14) provide different methods to visualize gameplay data. Last, Svahn and Wahlund (chapter 15) introduces structural equation modeling as a method to study impact of gameplay.

 

IV Mixed methods approaches

The fourth part focuses on mixed methods approaches. In mixed methods research, qualitative approaches are used to build detailed understanding of a phenomenon, and quantitative approaches are used to evaluate magnitudes or references. The approaches are combined to draw strengths of each method.

Lieberoth & Roepstorff (chapter 16) provide introduction to mixed methods research approach, whereas Olsson (chapter 17) focuses on the repertory grid technique. Hook discusses grounded theory approach (chapter 18). Both repertory grid technique and grounded theory can be executed as qualitative or mixed methods design.

V Game development for research

The fifth and last part discusses game development for research.

Mohseni, Pietschmann and Liebold (chapter 19) discuss how to use modding in research. Their approach focuses on modding for experimental quantitative research. On the other hand, Waern and Back (chapter 20) looks at game design research where the design itself and design process are the topic of study. Their methodology is qualitative.

Acknowledgments

We are grateful to Marie Denward, Markku Hannula, Mikko Meriläinen, and Sofia Lundmark for giving feedback on chapter drafts.

References

Anon, n.d. Research. In: Oxford dictionaries. Available at: <http://www.oxforddictionaries.com/definition/english/research&gt;.

Bishop-Clark, C. and Dietz-Uhler, B., 2012. Engaging in the scholarship of teaching and learning: a guide to the process, and how to develop a project from start to finish. 1st ed. Sterling: Stylus Publishing, LLC.

Creswell, J.W., 2014. Research design: qualitative, quantitative, and mixed method approaches. 4th ed. Los Angeles: Sage.

Dewey, J., 2005. The quest for certainty and human nature. Kessinger Publishing.

Guest, G., Bunce, A. and Johnson, L., 2006. How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), pp.59–82.

Hume, D., 2006. An enquiry concerning human understanding. Project Guttenberg. Available at: http://www.gutenberg.org/ebooks/9662.

Kuhn, T.S., 2012. The structure of scientific revolutions. 4th ed. Chicago: The University of Chicago Press.

Ladyman, J., 2002. Understanding philosophy of science. London: Routledge.

Maxwell, J.A., 1992. Understanding and validity in qualitative research. Harvard educational review, 62(3), pp.279–301.

Montfort, N., 2009. Racing the beam: the Atari Video computer system. Cambridge, Mass: MIT Press.

Niiniluoto, I., 1993. The aim and structure of applied research. Erkenntnis, 38(1), pp.1–21.

Pettersson, R., 1982. Cultural differences in the perception of image and color in pictures. ECTJ, 30(1), pp.43–53.

Popper, K.R., 2002. The logic of scientific discovery. London: Routledge.

Rockstart North, 2003. Manhunt [game]. Rockstar.

Shamoo, A.E., 2009. Responsible conduct of research. 2nd ed. Oxford: Oxford University Press.

Footnotes

1. In blind experiments the researchers are not aware of the subjects identity or conditions (brands, placebo–treatment) to minimize the effects of the preconceptions of the researchers in the study.