Research on Gamification in Education

Table of Links

Abstract and 1 Introduction

2. Background and Related Work and 2.1. Gamification

2.2. Game Design Elements and 2.3. Gamification Effects

2.4. Related Work on Gamification Negative Effects

3. Systematic Mapping and 3.1. The Research Questions

3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria

3.4. Applying the Search Strategy

3.5. Data Extraction

4. Systematic Mapping Results

5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements

5.1. Context and Participant Characterization

5.2. Focus Group Design

5.3. The Developers’ Perception on The Negative Effects

5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects

5.5. Participant Feedback

6. Limitations

7. Concluding Remarks

7.1. Future Research Directions

Acknowledgements and References

Acknowledgements

This study was financed in part by the Coordination for the Improvement of Higher Education Personnel (CAPES - Coordenacao de Aperfeicoamento de Pessoal de Nıvel Superior), Brazil, and by the National Council for Scientific and Technological Development (CNPq - Conselho Nacional de Desenvolvimento Cient´ıfico e Tecnol´ogico), Brazil - Finance Code 001.

We would like to thank the focus group participants for the valuable discussion that enriched this paper.

References

[1] Duolingo, About us: Mission., accessible in https://en.duolingo.com/info (2020).

[2] S. Deterding, From game design elements to gamefulness: Defining gamification., in: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, ACM, 2011, pp. 9–15.

[3] A. Rapp, F. Hopfgartner, J. Hamari, C. Linehan, F. Cena, Strengthening gamification studies: Current trends and future opportunities of gamification research., International Journal of Human-Computer Studies 127 (2019) 1–6, can be found at https://doi.org/10.1016/j.ijhcs.2018.11.007.

[4] O. Research, Gamification market by deployment (on premises, cloud based, and hybrid), enterprise size (small and medium enterprises, and large enterprises), end user, and geography (north america, europe, apac, and row)-global forecast up to 2025, can be found at https://www.orbisresearch.com/reports/index/gamification-market-bydeployment-on-premises-cloud-based-and-hybrid-enterprise-size-smalland-medium-enterprises-and-large-enterprises-end-user-and-geographynorth-america-europe-apac-and-row-global-forecast-up-to-2025 (2019).

[5] L. Piras, et al., Gamification solutions for software acceptance: A comparative study of requirements engineering and organizational behavior techniques, in: 11th International Conference on Research Challenges in Information Science (RCIS), 2017.

[6] Volkswagen, The fun theory 1 – piano staircase initiative, accessible in https://www.youtube.com/watch?v=SByymar3bds (2009).

[7] Volkswagen, The fun theory 2 – an initiative of volkswagen: The world’s deepest bin., accessible in https://www.youtube.com/watch?v=qRgWttqFKu8 (2009).

[8] Volkswagen, The fun theory 3 – an initiative of volkswagen: Bottle bank arcade machine, accessible in https://www.youtube.com/watch?v=zCt MzsnIUk (2009).

[9] B. Morschheuser, K. Werder, J. Hamari, J. Abe, How to gamify? a method for designing gamification., Proceedings Of The 50th Hawaii International Conference On System Sciences (2017) 1298–1307.

[10] D. M. Fernandez, S. Wagner, M. e. a. Kalinowski, Naming the pain in requirements engineering, Empirical Software Engineering 22 (5) (2017) 2298–2338.

[11] B. Boehm, V. Basili, Software defect reduction top 10 list., in: Foundations of Empirical Software Engineering, Springer Berlin Heidelberg, 2001, p. 426–431.

[12] T. e. a. Menzies, Are delayed issues harder to resolve? revisiting cost- tofix of defects throughout the lifecycle., Empirical Software Engineering 22 (4) (2016) 1903–1935.

[13] U. Nations, Sustainable development goals, available in https://www.un.org/sustainabledevelopment/education (n. d.).

[14] C. Almeida, M. Kalinowski, B. Feijo, A systematic mapping of negative effects of gamification in education/learning systems, in: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2021, pp. 17–24. doi:10.1109/SEAA53835.2021.00011.

[15] E. e. a. Mourao, On the performance of hybrid search strategies for systematic literature reviews in software engineering, Information and Software Technology 123 (2020) 106294.

[16] M. Nelson, Soviet and american precursors to the gamification of work, in: Proceeding Of The 16th International Academic Mindtrek Conference On - Mindtrek ’12, ACM Press, 2012, pp. 23–26.

[17] Y. Chou, Actionable Gamification: Beyond Points, Badges and Leaderboards, CreateSpace Independent Publishing Platform, 2015.

[18] K. Seaborn, D. Fels, Gamification in theory and action: A survey, International Journal Of Human-computer Studies 74 (2015) 14–31.

[19] B. Brathwaite, I. Schreiber, Challenges for Game Designers, Course Technology, 2009.

[20] G. Zichermann, C. Cunningham, Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media, 2011.

[21] K. Werbach, D. Hunter, For the win: How game thinking can revolutionize your business, 1st Edition, Wharton Digital Press, 2012.

[22] M. Denden, A. Tlili, F. Essalmi, M. Jemni, N.-S. Chen, D. Burgos, Effects of gender and personality differences on students’ perception of game design elements in educational gamification, International Journal of Human-Computer Studies 154 (2021) 102674.

[23] A. C. T. Klock, I. Gasparini, M. S. Pimenta, 5w2h framework: a guide to design, develop and evaluate the user-centered gamification, in: Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, 2016, pp. 1–10.

[24] A. Mora, D. Riera, C. Gonzalez, J. Arnedo-Moreno, Gamification: a systematic review of design frameworks, Journal of Computing in Higher Education 29 (3) (2017) 516–548.

[25] D. Gibson, N. Ostashewski, K. Flintoff, S. Grant, E. Knight, Digital badges in education, Education and Information Technologies 20 (2) (2015) 403–410.

[26] K. Werbach, D. Hunter, The gamification toolkit: dynamics, mechanics, and components for the win, University of Pennsylvania Press, 2015. [27] A. Toda, P. Valle, S. Isotani, The dark side of gamification: An overview of negative effects of gamification in education., HEFA 2017, Higher Education for All. From Challenges to Novel Technology-Enhanced Solutions (2018) 143–156.

[28] J. W. Rice, The gamification of learning and instruction: Game-based methods and strategies for training and education, International Journal of Gaming and Computer-Mediated Simulations 4 (4) (2012).

[29] P. Fotaris, T. Mastoras, R. Leinfellner, Y. Rosunally, Climbing up the leaderboard: An empirical study of applying gamification techniques to a computer programming class., Electronic Journal of e-learning 14 (2) (2016) 94–110.

[30] I. Bogost, Gamification is bullshit, accessible at http://bogost.com/writing/blog/gamification is bullshit/ (2011).

[31] L. Putz, F. Hofbauer, H. Treiblmaier, Can gamification help to improve education? findings from a longitudinal study, Computers in Human Behavior 110 (106392) (2020).

[32] A. Klock, I. Gasparini, A. Ogawa, M. Pimenta, Does gamification matter? a systematic mapping about the evaluation of gamification in educational environments, in: Proceedings of SAC 2018: Symposium on Applied Computing, 2006 - 2012, 2018.

[33] Z. Zaimudin, S. Chu, M. Shujahat, C. Perera, The impact of gamification on learning and instruction: A systematic review of empirical evidence, Educational Research Review 30 (2020).

[34] D. Johnson, et al., Gamification for health and wellbeing: A systematic review of the literature., Internet Interventions 6 (2016) 89–106.

[35] J. Hamari, J. Koivisto, H. Sarsa, Does gamification work? — a literature review of empirical studies on gamification, in: 47th Hawaii International Conference on System Sciences, 2013.

[36] J. Koivisto, J. Hamari, The rise of the motivational information systems: A review of gamification research, International Journal of Information Management 45 (191) (2018).

[37] V. Gabrielle, The dark side of gamifying work, accessible at https://www.fastcompany.com/90260703/the-dark-side-of-gamifyingwork.

[38] W. Hammedi, T. Leclercq, I. Poncin, L. Alkire, Uncovering the dark side of gamification at work: Impacts on engagement and wellbeing, Journal of Business Research 122 (2015) 256–269.

[39] A. Algashami, Gamification risks in collaborative information systems: Identification and management method, Ph.D. thesis, Department of Computing, Bournemouth University (2019).

[40] S. Hyrynsalmi, J. Smed, K. Kimppa, The dark side of gamification: How we should stop worrying and study also the negative impacts of bringing game design elements to everywhere, in: GamiFIN Conference 2017, 2017.

[41] S. Diefenbach, A. Mussig, Counterproductive effects of gamification: An analysis on the example of the gamified task manager habitica., International Journal Of Human-computer Studies 127 (2019) 190–210.

[42] M. Peixoto, C. Silva, A gamification requirements catalog for educational software: Results from a systematic literature review and a survey with experts, in: SAC ’17: Proceedings of the Symposium on Applied Computing, 2017, pp. 1108–1113.

[43] J. Majuri, J. Koivisto, J. Hamari, Gamification of education and learning: A review of empirical literature., in: GamiFIN Conference 2018, 2018.

[44] M. Alhammad, A. Moreno, Gamification in software engineering education: A systematic mapping, The Journal of Systems and Software 141 (2018) 131–150.

[45] E. Mendes, C. Wohlin, K. Felizardo, M. Kalinowski, When to update systematic literature reviews in software engineering, Journal of Systems and Software 167 (2020) 1–24.

[46] B. Kitchenham, S. Charters, Guidelines For Performing Systematic Literature Reviews In Software Engineering., Keele University and Hurham University Joint Report, 2007.

[47] K. Petersen, S. Vakkalanka, L. Kuzniarz, Guidelines for conducting systematic mapping studies in software engineering: An update, Information and Software Technology 64 (2015) 1–18.

[48] L. Roever, Pico: Model for clinical questions, Evidence Based Medicine and Practice 3 (2018) 2.

[49] A. Harzing, Publish or perish, available in https://harzing.com/resources/publish-or-perish. [50] D. A. Qatar Computing Research Institute, Rayyan, available in https://rayyan.qcri.org/welcome (2020).

[51] C. Wohlin, E. Mendes, K. R. Felizardo, M. Kalinowski, Guidelines for the search strategy to update systematic literature reviews in software engineering, Information and Software Technology 127 (2020) 106366. doi:https://doi.org/10.1016/j.infsof.2020.106366. URL https://www.sciencedirect.com/science/article/pii/ S095058491930223X

[52] J. Saldan˜a, The coding manual for qualitative researchers, 4th Edition, SAGE Publications Limited, 2021.

[53] S. Ejsing-Duun, H. Karoff, Gamification of a higher education course: What’s the fun in that?, in: Proceedings of the 8th European Conference on Game Based Learning - ECGBL 2014, 2014, pp. 92–98.

[54] S. Nicholson, Exploring gamification techniques for classroom management, in: Games+Learning+Society 9.0, 2013, acessible at http://scottnicholson.com/pubs/gamificationtechniquesclassroom.pdf.

[55] L. Haaranen, P. Ihantola, L. Hakulinen, A. Korhonen, How (not) to introduce badges to online exercises, in: Proceedings of the 45th ACM Technical Symposium on Computer Science, 2014, pp. 392–400. doi: 10.1145/2538862.2538921.

[56] B. J. Morris, C. Dragovich, R. Todaro, S. Balci, E. Dalton, Comparing badges and learning goals in low-and high-stakes learning contexts, Journal of Computing in Higher Education (2019) 573–603Available at https://doi.org/10.1007/s12528-019-09228-9.

[57] L. Z. Pedro, A. M. Z. Lopes, B. G. Prates, J. Vassileva, S. Isotani, Does gamification work for boys and girls? an exploratory study with a virtual learning environment, in: Prof. of the ACM Symposium on Applied Computing, 2015.

[58] M. Ortiz-Rojas, K. Chiluiza, M. Valcke, Gamification in computer programming: Effectson learning, engagement, self efficacy and intrinsic motivation, in: 11th European Conference on Games Based Learning, 2017, available at https://www.researchgate.net/publication/320372920 Gamification in Comp uter Programming Effects on learning engagement selfefficacy and intrinsic motivation.

[59] N. Zaric, V. Lukarov, U. Schroeder, A fundamental study for gamification design: Exploring learning tendencies’ effects, International Journal of Serious Games 7 (4) (2020) 3–25.

[60] A. Marinho, W. Oliveira, I. Bittencourt, D. Dermeval, Does gamification improve flow experience in classroom? an analysis of gamer types in collaborative and competitive settings, Brazilian Journal of Computers in Education 27 (2) (2019) 40–68. doi:10.5753/rbie.2019.27.02.40.

[61] C. Pilkington, A playful approach to fostering motivation in a distance education computer programming course: Behaviour change and student perceptions, International Review of Research in Open and Distributed Learning 3 (19), accessible at https://doi.org/10.19173/irrodl.v19i3.3664 (2018).

[62] R. Roy, S. Deterding, B. Zaman, Collecting pokemon or receiving rewards? how people functionalise badges in gamified online learning environments in the wild., International Journal of Human-Computer Studies 127 (2019) 62–80.

[63] L. Hakulinen, T. Auvinen, A. Korhonen, Empirical study on the effect of achievement badges in trakla2 online learning environment, in: 2013 Learning and Teaching in Computing and Engineering, 2013, accessible at https://doi.org/10.1109/LaTiCE.2013.34.

[64] M. Ohn, K.-M. Ohn, An evaluation study on gamified online learning experiences and its acceptance among medical students, Tzu Chi Medical Journal 32 (2) (2019) 211–215. doi:10.4103/tcmj.tcmj\_5\_19.

[65] A. Campos, E. Batista, A. Signoretti, R. Gardiman, C. Madeira, Gamifying activities in a higher education course, in: 13th European Conference on Games Based Learning, 2015.

[66] S. A. Kocadere, S. Caglar, The design and implementation of a gamified assessment, Journal of E-Learning and Knowledge Society 11 (3) (2015) 85–99, accessible at http://www.doi.org/10.20368/1971-8829/1070.

[67] L. de Marcos, A. Dominguez, J. Saenz-de Navarrete, C. Pages, An empirical study comparing gamification and social networking on e-learning, Computers & Education 75 (2014) 82–91.

[68] V. Naik, V. Kamat, Adaptive and gamified learning environment (agle), in: IEEE Seventh International Conference on Technology for Education (T4E), 2015. doi:10.1109/T4E.2015.23.

[69] R. McDaniel, R. Lindgren, J. Friskics, Using badges for shaping interactions in online learning environments, in: Software Engineering 2014, 2012, pp. 1 – 4. doi:10.1109/IPCC.2012.6408619.

[70] K. Berkling, C. Thomas, Gamification of a software engineering course and a detailed analysis of the factors that lead to it’s failure, 2013, pp. 1 – 4. doi:10.1109/ICL.2013.6644642.

[71] F. Tiefenbacher, Evaluation of gamification elements in a vr application for higher education, in: Systems, Software and Services Process Improvement, 2020, acessible at http://doi.org/10.1007/978-3-030-56441-4.

[72] G. Barata, S. Gama, J. Jorge, D. Gon¸calves, Improving participation and learning with gamification, in: 1st International Conference on Gamification, Vol. 14(6), 2013, pp. 55–72. doi:10.1145/2583008.2583010.

[73] W. Ghaban, R. Hendley, Understanding the effect of gamification on learners with different personalities, in: Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), 2019, pp. 392–400. doi:10.5220/0007730703920400.

[74] K. Davis, S. Singh, Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators, Computers & Education 88 (2015) 72 – 83. doi:http://dx.doi.org/10.1016/j. compedu.2015.04.011.

[75] K. Davis, H. Sridharan, L. Koepke, S. Singh, R. Boiko, Learning and engagement in a gamified course: Investigating the effects of student characteristics. (2018). doi:10.1111/jcal.12254.

[76] M. D. Hanus, J. Fox, Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance, Computers & Education 80 (2015) 152–161, http://dx.doi.org/10.1016/j.compedu.2014.08.019.

[77] L. Facey-Shaw, M. Specht, P. Rosmalen, J. Bartley-Bryan;, Do badges affect intrinsic motivation in introductory programming students?, Simulation & Gaming 51 (1) (2020). doi:10.1177/1046878119884996.

[78] S. Mahmud, H. Husnin, T. Soh, Teaching presence in online gamified education for sustainability learning, Sustainability 12 (9) (2020) 3801. doi:10.3390/su12093801.

[79] R. Roy, B. Zaman, Unravelling the ambivalent motivational power of gamification: A basic psychological needs perspective, International Journal of Human-Computer Studies 127 (2019) 38–50, accessible at https://doi.org/10.1016/j.ijhcs.2018.04.009.

[80] A. Hern´andez-Fern´andez, N. Olmedo-Torre, M. Pena, Is classroom gamification opposed to performance?, Sustainability 12 (9958), accessible at http://www.doi.org/10.3390/su12239958 (2020).

[81] A. Domınguez, J. Saenz-de Navarrete, L. de Marcos, L. Fernandez-Sanz, C. Pages, J.-J. Martınez-Herraiz, Gamifying learning experiences: practical implications and outcomes, Comput. Educ. 63 (2013) 380–392, acessible at http://dx.doi.org/10.1016/j.compedu.2012.12.020.

[82] D. Babichenko, L. Grieve, E. Bilodeau, D. Koval, Implementation and assessment of three gamification strategies across multiple higher education disciplines, in: 13th European Conference on Games Based Learning, 2019, accessible at https://www.doi.org/10.34190/GBL.19.016.

[83] R. Martin, A. Klein, Improved student independence through competitive tinkering, in: Frontiers in Education (FIE) Conference, 2017.

[84] M. Featherstone, J. Habgood, Unicraft: exploring the impact of asynchronous multiplayer game elements in gamification, Int. J. Human Computer Stud. (2018).

[85] M. Minge, D. Cymek, Investigating the potential of gamification to improve seniors’ experience and use of technology, Information 11 (2020) 1– 16, accessible at https://doi.org/10.3390/info11050249.

[86] O. Azevedo, F. de Morais, P. Jaques, Exploring gamification to prevent gaming the system and help refusal in tutoring systems, in: EC-TEL 2018, 2018, accessible at https://doi.org/10.1007/978-3-319-98572-5 18.

[87] C. Brom, T. Starkova, E. Bromova, F. Dechterenko, Gamifying a simulation: Do a game goal, choice, points, and praise enhance learning?, Journal of Educational Computing Research 57 (6) (2018) 1575–1613. doi:https://doi.org/10.1177/0735633118797330.

[88] L. Singer, K. Schneider, It was a bit of a race: Gamification of version control, in: 2nd International Workshop on Games and Software Engineering, 2012, acessible at http://www.doi.org/10.1109/GAS.2012.6225927.

[89] G. Barata, S. Gama, J. Jorge, D. Gon¸calves, So fun it hurts: Gamifying an engineering course, in: Foundations of Augmented Cognition, Springer, 2013, pp. 639–648.

[90] M. Iruela, R. Neira, What perception do students have about the gamification elements?, accessible in http://dx.doi.org/10.1109/ACCESS.2020.3011222 (2020).

[91] R. Bajko, J. Hodson, K. Seaborn, P. Livingstone, D. Fels, Edugamifying media studies: Student engagement, enjoyment, and interest in two multimedia and social media undergraduate classrooms, in: Information System Education Journal, Vol. 14(6), 2016, pp. 55–72.

[92] C. R. Prause, M. Jarke, Gamification for enforcing coding conventions, in: 10th Joint Meeting on Foundations of Software Engineering, 2015. doi:10.1145/2786805.2786806.

[93] A. Buisman, M. Eekelen, Gamification in educational software development, in: CSERC ’14: Proceedings of the Computer Science Education Research Conference, 2014.

[94] D. Codish, G. Ravid, Gender moderation in gamification: Does one size fit all?, in: Proceedings of the 50th Hawaii International Conference on System Sciences, 2017.

[95] D. Codish, G. Ravid, Personality based gamification: How different personalities perceive gamification, accessible at https://www.researchgate.net/publication/286518104 Personality based gamification How different personalities percive gamification (2014).

[96] E. Lavoue, B. Monterrat, M. Desmarais, S. George, Adaptive gamification for learning environments, IEEE Transactions on Learning Technologies, Institute of Electrical and Electronics Engineers (2019) 16 – 28.

[97] Z. Fitz-Walter, D. Johnson, P. Wyeth, D. Tjondronegoro, B. Scott-Parker, Driven to drive? investigating the effect of gamification on learner driver behavior, perceived motivation and user experience, Computers in Human Behavior 71 (2017) 586–595.

[98] Y. Attalia, M. Arieli-Attali, Gamification in assessment: Do points affect test performance?, Computers & Education 83 (2015) 57–63, accessible at https://doi.org/10.1016/j.compedu.2014.12.012.

[99] C. Chau, Y. Tsui, C. Cheng, Gamification for internet gaming disorder prevention: Evaluation of a wise it-use (wit) program for hong kong primary students, Frontiers in Psychology 10 (2019) 2468. doi: 10.3389/fpsyg.2019.02468.

[100] E. Gehringer, B. Peddycord, The impact of a forfeit-or-prize gamified teaching on e-learners’ learning performance, Computers Education 126 (2018) 143–152.

[101] J. Albuquerque, I. Bittencourt, J. Coelho, A. Silva, Does gender stereotype threat in gamified educational environments cause anxiety? an experimental study, Computers Education 115 (2017) 161–170.

[102] P. Papadopoulos, T. Lagkas, S. Demetriadis, How revealing rankings affects student attitude and performance in a peer review learning environment, in: CSEDU 2015: Computer Supported Education, 2015. doi:10.1007/978-3-319-29585-5\_13.

[103] P. Buckley, E. Doyle, Gamification and student motivation, Journal of Educational Computing Research 24 (6) (2014) 1162–1175. doi:10.1177/ 0735633118797330.

[104] T. e. a. Kredo, The effect of gamification on psychological and behavioral outcomes: Implications for cruise tourism destinations, Sustainability 11 (2009) 3002. doi:10.3390/su11113002.

[105] C. Wohlin, P. Runeson, M. H¨ost, M. Ohlsson, B. Regnell, A. Wesslen, Experimentation in software engineering, Springer Science & Business Media, 2012.

[106] V. R. Basili, H. D. Rombach, The tame project: Towards improvement-oriented software environments, IEEE Transactions on software engineering 14 (6) (1988) 758–773.

[107] J. Kontio, J. Bragge, L. Lehtola, The focus group method as an empirical tool in software engineering, in: Guide to advanced empirical software engineering, Springer, 2008, pp. 93–116.

[108] A. Martakis, M. Daneva, Handling requirements dependencies in agile projects: A focus group with agile software development practitioners, in: IEEE 7th International Conference on Research Challenges in Information Science (RCIS), IEEE, 2013, pp. 1–11.

[109] M. Kim, D. Notkin, Discovering and representing systematic code changes, in: 2009 IEEE 31st International Conference on Software Engineering, IEEE, 2009, pp. 309–319.

[110] E. de Souza Nascimento, I. Ahmed, E. Oliveira, M. P. Palheta, I. Steinmacher, T. Conte, Understanding development process of machine learning systems: Challenges and solutions, in: 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2019, pp. 1–6.

[111] A. Uchoa, E. Fernandes, B. Fonseca, R. de Mello, C. Barbosa, G. Nunes, A. Garcia, L. Teixeira, On gamifying an existing healthcare system: Method, conceptual model and evaluation, in: 2019 IEEE/ACM 1st International Workshop on Software Engineering for Healthcare (SEH), IEEE, 2019, pp. 9–16.

[112] E. Fernandes, A. Uchˆoa, L. Sousa, A. Oliveira, R. d. Mello, L. P. Barroca, D. Carvalho, A. Garcia, B. Fonseca, L. Teixeira, Vazazika: A software platform for surveillance and control of mosquito-borne diseases, in: 16th International Conference on Information Technology-New Generations (ITNG 2019), Springer, 2019, pp. 617–620.

[113] F. D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS quarterly (1989) 319–340.

[114] G. Norman, Likert scales, levels of measurement and the “laws” of statistics, Advances in health sciences education 15 (5) (2010) 625–632.

[115] J. Teixeira da Silva, Negative results: negative perceptions limit their potential for increasing reproducibility, Journal of Negative Results in BioMedicine 14 (12) (2015). doi:https://doi.org/10.1186/ s12952-015-0033-9.

[116] N. Matosin, E. Frank, M. Engel, J. Lum, K. Newell, Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture, Disease Models & Mechanisms 7 (2014) 171–173. doi:10.1242/dmm.015123.

[117] K. L. Matthews, M. Baird, G. Duchesne, Using online meeting software to facilitate geographically dispersed focus groups for health workforce research, Qualitative Health Research 28 (10) (2018) 1621–1628.

[118] J. Menary, S. Stetkiewicz, A. Nair, P. Jorasch, A. K. Nanda, A. Guichaoua, M. Rufino, A. R. Fischer, J. A. Davies, Going virtual: Adapting in-person interactive focus groups to the online environment, Emerald Open Research 3 (2021) 6.