Petri Lankoski and Mirjam P. Eladhari
Paper presented at Teaching Games: Pedagogical Approaches – DiGRA 2019 Pre-Conference Workshop(TGPA:DiGRA2019) August 6, 2019, Kyoto, Japan.
Abstract
This paper presents a case study of a Bachelor level game research methods course (15ECTS). The course covers observations, interviews, and introduction to statistical analysis. The course set-up follows constructive alignment design where the aim is that the learning goals, learning tasks, and evaluation are aligned. During the course, students first learn re-search design and later design their research based on a set of examples and conduct data gathering and analysis. The evaluation of the pedagogical approach used is based on students’ learning diaries where the focus is the methods and applying methods. Qualitative evaluation indicates that students can better describe their research designs and analyses.
Keywords
pedagogy, constructive alignment, research methods, game design research
Introduction
Game design students in university programs are in most cases taught iterative processes of design including stages of playtesting (Game Education SIG 2008). They aim to learn and are taught, how to create well-designed games. However, when they in the last parts of their Bachelor’s programme are to write academically for their examination projects, they are met with a new set of challenges: to conduct work that meets scientific formal requirements. In order to do so, they need to work in a structured manner and use sound methods for exploring their research questions, and finally be able to build an argument that they communicate in academic text, for example, their thesis or report.
At Södertörn University while teaching game development students, we saw that students had issues in planning research and applying game research methods in analysis in the Bachelor’s essay course. Even though students had taken a course that covered research methods, it seemed that the methods appeared rather abstract to many of our students. During the supervision, it was hard to instruct the specific stages of analysis as it seemed that analysis was a black box. In 2016, we redesigned the courses teaching research method, attempting at opening the black box by designing assignments for students where they incrementally learn to go tasks that are small in their scope to larger ones. They start with small tasks of data gathering and analysis and go on to replication studies to finally be more able to conduct their own research design in their bachelors’ projects.
This paper discusses the teaching approach we have used to teach game design research and presents a case study of a course thought to students preparing for their bachelor’s projects. Here, we describe the parts of the course that concerns game research, theory and methods, focusing on teaching research methods and leave other parts of the course out. We have tough the game research course we discuss here for three years. During these three years there were 27-34 students participating each year. Before that we had similarly focused research course (with 20-24 students) from 2013-2015 but using a different pedagogical approach
The game course we are discussing is part of three year educational programme in games, and is given in the beginning of the third year. Students participating have prior knowledge of various game analysis methods (e.g, formal analysis of gameplay (Lankoski and Björk 2015) or formal analysis of art (Carroll 1999) and stimulated recall interviews (Pitkänen 2015).
We redesigned the course using the idea of constructive alignment (Biggs and Tang 2011). With constructive alignment, Biggs and Tang mean that the learning task, teaching tasks, and evaluation are aligned. Alignment, as an example, means that a goal, such as to apply a method, is connected to a learning task, and uses the method in the practical task. The following evaluation focuses on the process of using the method—in contrast to the end result of the task (C.f. Biggs and Tang 2011).
In the course, students are completing a series of assignments, gaining hands-on experience of both gathering and analyzing data from players. They also get to practice how to design research.
Below we present the course in detail and offer some insights into the outcomes. The insights are based on supervision sessions during Bachelors essay course starting after this research course.
Case Study
The students are taught game design research over ten weeks on full time (15 ECTS). In the first half of this period, they learn game research methods through a series of workshops. In the second half, they conduct a longer study, replicating previous research of their choice, or altering the research design of a previous study. This set-up gives students an opportunity to practice designing research studies. Before the redesign, the first part of the course was based on the seminars where various research papers were discussed. Discussions included reflections on the method. The second part of the course was focused on a single method that student could choose. The method was taught during supervision sessions while student conducted a study. Below we describe the redesigned course in detail.
Part 1: Learning Game Design Research Methods
The assignments for learning and practicing methods consists of data gathering and analysis exercises. In data analysis assignment our students get the task to collect data of four participants’ play and play experience. The data gathering happens in our game lab (Figure 1) and consists of observations, surveys, and semi-structured interviews.
Figure 1. An example of recording a play session: a game stream and a two-camera set-up is used.
In forms of materials, students are given:
- game prototype (in the first course instance, we used three AAA games instead of the prototype)
- a script for a data gathering session along with a message that can be used to recruit participants
- a ready-made questionnaire
- a semi-structured interview guide.
The game prototype is a horror game containing three cases: a level without a monster, a level with a monster, and a level with a monster and forewarning about the monster.
The first assignment is focused on data gathering. Students are given a pre-formulated research question:
“How does design influence the horror experience?”
The question is investigated with the following foci:
- How is the the player experience influenced depending on whether a 1st person or a 3rd person point of view is used in the intro cut-scene?
- How is use of forewarning influencing playing experience?
Students record play sessions in our game lab. The lab is equipped with Elgato Game Capture hardware which allows for simultaneous recording of the game screen while two cameras record play (cf. Figure 1)
To prepare for the data gathering session, the students are instructed to read about conducting in-depth interviews (Cote and Raz 2015) before the sessions. The students are also instructed to write an assignment diary where they describe their data gathering and compare it against the given literature.
After the data gathering phase each student have a set of data they have collected themselves:
- Recorded play sessions, with 3 image streams and two audio streams
- Recorded interviews
- Questionnaire data.
The data listed above is used in three seminars focusing on data analysis. The themes for the seminars are:
- Thematic analysis of interview data
- Coding of observation data
- Quantitative analysis questionnaire data.
At the beginning of each seminar, we give a short introduction to the method. After each seminar students are asked to document their work and reflect on their learning on the seminar topics.
In preparation for the thematic analysis seminar, the students are instructed to transcribe the interviews they recorded and read method literature of thematic analysis: our own handouts along with Cote and Raz (2015) and Braun and Clarke (2006). During the seminar, students analyze their data in groups using inductive thematic analysis using iterative analysis approach where each iteration is developing analysis towards themes. We provide some practical tools for helping organizing data: mind map, thematic maps, and post-it categorization. While we ask students to describe the resulting themes briefly and provide a thematic map about their results, the focus for the learning diary is in the analysis process. We instruct students to reflect and compare their analysis process to the ones in the course literature.
During the second seminar, on the topic of the coding the observation data, we look at analysis using emergent codes. This seminar is structured similarly to the thematic analysis seminar. We ask students to read methods hands out (our own material) and conduct analysis using iterative process: finding an initial coding schema and then use that to code schema to analyze the data (and expand the schema if needed). The students are instructed to, when they write about this in their learning diary, focus on the analysis process and reflect upon it in a similar fashion to the previous seminar.
The seminar on quantitative analysis of questionnaire data, however, focus on interpretation of analysis results, rather than conducting quantitative analysis. As mentioned, the game prototype contains three cases: a level without monster, a level with a monster, and a level with a monster and forewarning about the monster. The questionnaire measures emotional experience. During the seminar the students (in groups) get
- A pre-formulated hypothesis stating that the forewarning is the most scary design.
- Box plots and violin plots of fear scores by conditions.
- ANOVA analysis of fear scores between groups and pair-wise t-tests.
- In order to introduce correlation, we also examine the relation between fear and number of game-overs. For this, we show scatter plots of these variables and Pearson correlation analysis results.
The tasks worked on during the seminar is providing interpretations and descriptions on what we can infer from the given figures and analyses in terms of given hypotheses. Here our goal is not to teach quantitative analysis methods per se, but rather general statistical literacy. The learning diary entry for this seminar focuses on the meaning of introduced concepts as well as the test results and their interpretation.
Above-described design is intended to follow constructive alignment (cf. Biggs and Tang 2011). All assignments and seminars include the learning activities (data gathering data analysis, analysis result interpretation) that are tied to the learning goals, and apply methods to data gathering or data analysis. In addition, the learning diary focuses on the data gathering and data analysis processes. This is intended to align learning tasks and learning goals.
Part 2: Applying Game Design Research Methods
When the second half of the course starts, student have fresh memories of conducting these structured assignments. The task they are given is to conduct a small study following a model of a previously published study. We give a list (Harmon-Jones et al. 2016; Azadvar
and Canossa 2018; Busselle and Bilandzic 2009; Lankoski 2016; Hart and Staveland; Wied et al. 1997; Downs and Smith 2010; Bazzini et al. 2010) of alternatives where the method used has been observations, interviews, and stimulated recall (A method we introduced in an earlier course). The potential example studies also include quantitative studies for those who would like an extra challenge, e.g. learning more about conducting quantitative research.
The students should build their own study based on the selected study as an example of research design. The students can change the research question to something similar and in the case of quantitative study they can also change measurement instrument.
The students write a report based on their study. The report is to focus on method, results, and discussion. The discussion is intended to include a deliberation on theoretical implications on the results as well as reflections about validity and reliability of the results.
In this second part of the course we have the aim that students should analyze the research design of the chosen study and build their own research design based on that analysis. The goal, understanding research design, and the learning task are aligned. In reporting of the study, students are not conducting a literature review. The reason for this is to help students to keep the focus on the learning task in applying research design in practical research task and reporting it.(1) The reason, again, focus on results and discussion on reporting is aligning the learning goal and learning task.
- Our previous courses include conducting a literature review. In addition, the Bachelor’s Thesis course is structured so that the students are required to conduct literature review first and motivate their research question on that literature review.
Reflections
The course structure described above has been iterated upon for three consecutive years. In the first iteration we required a more direct replication of the example study from the student. We also instructed the students to find example studies themselves. However, if they chose a study that had obvious methodological flaws, we asked them to choose another one. This approach turned out to be both difficult an unnecessarily restricting for the student’s learning process. Therefore we changed the approach in the latest iteration, providing students with a list of studies to choose from, and allowing for alterations of the research design.
Based on our observations during a five-year period teaching our Bachelor’s Thesis course, we see that the approach during the last three years appears to have improved students’ proficiency in research design and in application of research methods. These are the observations that we interpret as indicators of the improvement:
- In supervision, there is less need to discuss on general aspects of data gathering and supervision is focusing more on specific issues of research design.
- During supervision of data analysis, students talk about specific steps and details of data analysis using relevant concepts.
- Students enter with more considered and detailed research designs than earlier.
- The data analyses we see in supervision are more systematic than earlier.
We must stress that the changes we see in student performance might have multiple different reasons other than the learning task design and constructive alignment: we as teachers might have gotten better at teaching the subject or the students are more skilled.
Conclusions
Above we have presented a game (design) research course focusing on research methods. The course is built on the principles of \textit{constructive alignment} by Biggs and Tang (2011) where the learning goals, learning tasks, and evaluation are aligned. Our evaluations of students skill-sets during Bachelor’s Thesis course indicate that the students have a better command on research designs and how to apply research methods in practical work than before. While we cannot claim that the constructive alignment course design is the reason for seeing the positive development, we have observed that our research methods teaching has improved since we started to use the approach.
Bibliography
Azadvar, Ahmad, and Alessandro Canossa. 2018. “UPEQ: ubisoft perceived experience questionnaire: a self-determination evaluation tool for video games” [in en]. In Proceedings of the 13th International Conference on the Foundations of Digital Games – FDG ’18, 1–7. Malmö, Sweden: ACM Press. ISBN: 978-1-4503-6571-0. http://dl.acm.org/citation.cfm?doid=3235765.3235780.
Bazzini, Doris, Lisa Curtin, Serena Joslin, Shilpa Regan, and Denise Martz. 2010. “Do Animated Disney Characters Portray and Promote the Beauty-Goodness Stereotype?: DISNEY FILMS AND BEAUTY STEREOTYPES” [in en]. Journal of Applied Social Psychology 40, no. 10 (October): 2687–2709. ISSN: 00219029. http://doi.wiley.com/10.1111/j.1559-1816.2010.00676.x.
Biggs, J., and C. Tang. 2011. Teaching for Quality Learning at University. SRHE and Open University Press imprint. McGraw-Hill Education. ISBN: 978-0-335-24275-7.
Braun, Virginia, and Victoria Clarke. 2006. “Using thematic analysis in psychology”. Qualitative Research in Psychology 3, no. 2 (January): 77–101. ISSN: 1478-0887, 1478-0895. http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa.
Busselle, Rick, and Helena Bilandzic. 2009. “Measuring Narrative Engagement”. Media Psychology 12, no. 4 (November): 321–347. ISSN: 1521-3269, 1532-785X. http://www.tandfonline.com/doi/abs/10.1080/15213260903287259.
Carroll, Noël. 1999. Philosophy of art: a contemporary introduction. Routledge contemporary introductions to philosophy. London ; New York: Routledge. ISBN: 0415159636.
Cote, Amanda, and Julia G. Raz. 2015. “In-depth interviews for game research.” In Game Research Methods: An Overview, edited by Petri Lankoski and Staffan Björk, 93–116. Pittsburgh, PA, USA: ETC Press. ISBN: 978-1-312-88473-1.
Downs, Edward, and Stacy L. Smith. 2010. “Keeping Abreast of Hypersexuality: A Video Game Character Content Analysis.” Sex Roles 62, no. 11 (June): 721–733. ISSN: 1573.1 https://doi.org/10.1007/s11199-009-9637-1.
Game Education SIG, -. 2008. IGDA Curriculum Framework: The Study of Games and Game Development. Technical report. https://cdn.ymaws.com/www.igda.org/resource/collection/0DBC56DC- B7CB- 4140- BF3A- 22A9E92EC63A/igda_curriculum_framework_2008.pdf.
Harmon-Jones, Cindy, Brock Bastian, and Eddie Harmon-Jones. 2016. “The Discrete Emotions Questionnaire: A New Tool for Measuring State Self-Reported Emotions”, edited by André Aleman. PLOS ONE 11, no. 8 (August): e0159915. ISSN: 1932-6203. http://dx.plos.org/10.1371/journal.pone.0159915.
Hart, Sandra G., and Staveland. NASA Task Load Index. http://humansystems.arc.nasa.gov/groups/tlx/downloads/TLXScale.pdf.
Lankoski, Petri. 2016. “Embodiment in character-based videogames”. In Proceedings of the 20th International Academic Mindtrek Conference on – AcademicMindtrek’16, 358–365. Tampere, Finland: ACM Press. ISBN: 978-1-4503-4367-1. http://dl.acm.org/citation.cfm?doid=2994310.2994320.
Lankoski, Petri, and Staffan Björk. 2015. “Formal analysis of gameplay.” In Game research methods: an overview, edited by Petri Lankoski and Staffan Björk. Pittsburgh, PA: ETC Press. ISBN: 978-1-312-88473-1.
Pitkänen, Jori. 2015. “Studying thoughts. Stimulated recall as a game research method.” In Game Research Methods, edited by Petri Lankoski and Staffan Björk, 117–. ETC Press.
Wied, Minet de, Kathleen Hoffman, and David R. Roskos-Ewoldsen. 1997. “Forewarning of Graphic Portrayal of Violence and the Experience of Suspenseful Drama”. Cognition & Emotion 11, no. 4 (August): 481–494. ISSN: 0269-9931, 1464-0600. http://www.tandfonline.com/doi/abs/10.1080/026999397379890.
Petri Lankoski 