Assessing Creative Thinking in PISA 2021: Role of Engagement in Problem Solving, Schemes and Mind Maps of Creative Thinking

Download Assessing Creative Thinking in PISA 2021: Role of Engagement in Problem Solving and more Schemes and Mind Maps Creative Thinking in PDF only on Docsity!

• Framework for the Assessment of Creative Thinking in PISA 2021: Third Draft
  • The case for assessing creative thinking
    • Why assess creative thinking?
    • What is the role of education in creative thinking?
  • Evidence-centred design as a general framework for the PISA 2021 assessment
  • Defining the assessment domain
    • What is creative thinking?
    • Domain generality versus domain specificity
    • Domains of creative engagement
    • Confluence approaches of creativity
  • Understanding and assessing creative thinking in the classroom
    • Individual enablers of creative thinking
    • Social enablers of creative thinking
    • Creative engagement
  • Implications for the design of the PISA 2021 creative thinking assessment
    • Focus and objectives of the PISA 2021 assessment of creative thinking.......................................
    • Domains of creative thinking included in PISA
    • Competency model of creative thinking
  • Distribution of tasks, response format and scoring methods in the cognitive test
    • Distribution of tasks
    • Response types
    • Scoring of the tasks
  • Example units and scoring methods in the cognitive test
    • Unit model in written expression
    • Unit model in visual expression
    • Unit model in social problem solving
    • Unit model in scientific problem solving
  • Design considerations and opportunities for additional indicators based on process data.
    • Accounting for domain and task-specific knowledge
    • Accounting for engagement with the task (task motivation)..........................................................
    • Design features to encourage students’ exploratory skills and trial and error................................
  • Test development and validation of the cognitive test
    • Ensuring appropriate coverage of the construct and cross-cultural validity
    • Validation and cross-cultural comparability of the assessment material
  • Scaling and reporting proficiency in the cognitive test......................................................................
  • Defining content for the PISA background questionnaires
    • Curiosity and exploration
    • Creative self-efficacy
    • Beliefs about creativity...................................................................................................................
    • Creative activities in the classroom and school..............................................................................
    • Social environment
• References
• Table 1. Possible ways to measure creative thinking facets across domains Tables
• Figure 1 Enablers and manifestations of creative thinking in the classroom Figures
• Figure 2 Proposed focus domains for the assessment
• Figure 3 Competency model for the PISA test of creative thinking.....................................................
• Figure 4 Examples of tasks in a written expression unit
• Figure 5 Examples of tasks in a visual expression unit
• Figure 6 Example of tasks in a social problem solving unit
• Figure 7 Example of template for a scientific problem unit

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT)

Framework for the Assessment of Creative Thinking in PISA 2021:

Third Draft

The case for assessing creative thinking

Why assess creative thinking?

1. Creative insights and advances have driven forward human culture across the world in diverse areas (Hennessey and Amabile, 2010[1]): in the sciences, technology, philosophy, the arts and humanities. Creative thinking is thus more than simply coming up with random ideas. It is a tangible competence, grounded in knowledge and practice, that supports individuals in achieving better outcomes, oftentimes in constrained and challenging environments. Organisations and societies around the world increasingly depend on innovation and knowledge creation to address emerging challenges (OECD, 2010[2]), giving urgency to innovation and creative thinking as collective enterprises.
2. While it is true that creative thinking drives the types of innovation that have a society-wide impact, it is also a more universal and democratic phenomenon than one might first believe. That is to say that every individual, to a greater or smaller degree, has the potential to think creatively (OECD, 2017[3]). Furthermore, there is a general consensus among psychologists and educators alike that creative thinking, understood as engagement in the thinking processes associated with creative work, can improve a host of other individual abilities, including metacognitive capacities, inter- and intra-personal and problem-solving skills, as well as promoting identity development, academic achievement, future career success and social engagement (Beghetto, 2010[4]; Plucker, Beghetto and Dow, 2004[5]; Smith and Smith, 2010[6]; Torrance, 1959[7]; National Advisory Committee on Creative and Cultural Education, 1999[8]; Spencer and Lucas, 2018[9]; Long and Plucker, (^2015) [10]; Barbot, Lubart and Besançon, 2016[11]; Barbot and Heuser, 2017[12]; Gajda, Karwowski and Beghetto, 2017[13]) (Higgins et al., 2005[14]).
3. Developing an international assessment of creative thinking can encourage positive changes in education policies and pedagogies. The PISA 2021 creative thinking assessment will provide policymakers with valid, reliable and actionable measurement tools that will help them to make evidence-based decisions. The results will also encourage a wider societal debate on both the importance and methods of supporting this crucial competence through education. This work in PISA is connected to another OECD project that aims at supporting new pedagogies that can foster creative thinking. For the past years, the OECD’s Centre for Educational Research and Innovation (CERI) has been leading an eleven- country study on ways of teaching and assessing creative and critical thinking with encouraging early results^1. (^1) Since 2015 CERI has led an exploration of the teaching and assessment of Creative Thinking in 11 countries

• Brazil, France, Hungary, India, Netherlands, Russia, Slovak Republic, Spain, Thailand, United Kingdom (Wales), and United States. Taking at its starting point the work piloted by Lucas, Claxton and Spencer (2013[124]) in England, it has prototyped a new teacher-friendly conceptual framework to think about creative and critical thinking in the classrooms in primary and secondary education. It has developed OECD rubrics on creative and critical thinking meant to support teachers to develop or improve pedagogical activities that nurture the creative and critical thinking skills of their students. An international network of

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT)

Evidence-centred design as a general framework for the PISA 2021 assessment

8. Evidence-centred design (ECD) (Mislevy, Steinberg and Almond, (^2003) [22]) provides a conceptual framework for developing innovative and coherent assessments that are built on evidence-based arguments, connecting what students do, write or create on a computer platform, with multidimensional competences (Shute, Hansen and Almond, (^2008) [23]; Kim, Almond and Shute, 2016[24]). ECD starts with the basic premise that assessment is a process of reasoning from evidence to evaluate specific claims about students’ capabilities. In essence, students’ responses to the assessment items and tasks provide the evidence for this reasoning process, and psychometric analyses establish the sufficiency of the evidence for evaluating each claim. Using ECD as an organising framework for the PISA 2021 creative thinking assessment can help to address a series of important test design questions, namely: which creative thinking constructs or processes does each task within the assessment reveal? Do the proposed scoring methods effectively recognise and interpret the evidence generated by students’ responses and interactions with the assessment platform? How is all of the evidence that is generated by students’ choices synthesised across multiple tasks? Is all of the evidence for a particular construct comparable when different students attempt different tasks?
9. ECD provides a strong foundation for the development of a valid assessment of the complex and multidimensional construct of creative thinking. It requires documented, explicit linkages among the test purpose, the claims made about the test takers and the evidence supporting the claims. Adopting the ECD process for the PISA 2021 creative thinking assessment requires the following sequence of steps:
   1. Domain definition: reviewing the relevant literature and engaging with experts to define the domain of creative thinking in an educational context. This foundational work clarifies the creative thinking competences that policy makers and educators wish to promote, and the types of creative expressions that 15-year-old students can achieve and that can be most meaningfully and feasibly assessed in PISA.
   2. Construct definition: describing the precise construct the PISA test will assess and specifying the claims that can be made about the relevant attributes of test takers on the basis of the assessment. In ECD terminology, this step is generally referred to as defining the Competency or Student Model (Shute et al., 2016[25]).
   3. Evidence identification: describing the evidence that needs to be generated in the test to support the subsequent claims made about test-takers (i.e. the behaviours or performances that demonstrate the skills being assessed, for example what students might select, write or produce, and which constitute evidence for the claims). In ECD, this is referred to as defining the Evidence Model. This step includes providing rules for scoring the tasks and for aggregating scores across tasks that extract the evidence required to support the claims (including process data stored in log files).
   4. Task design: identifying, conceptualising and prototyping a set of tasks that provide the desired evidence within the constraints of the PISA assessment. This stage corresponds to the Task Model step in ECD terminology.
   5. Test development: assembling the tasks into test formats that support all of the stated assessment claims with sufficient evidence. This corresponds to the Assembly Model step in ECD terminology.

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT)

6. Cross-cultural validation: ensuring that all assessment instruments provide reliable and comparable evidence across countries and cultural groups. This step is generally not discussed in ECD approaches, but is clearly important in the context of PISA.
7. Analysis and reporting: illustrating appropriate, meaningful and easy-to-communicate representations of the assessment results.

10. Validation and pilot studies can increase the iterative nature of this design cycle: for example, the analysis of validation data can inform choices regarding evidence identification and task design.
11. The structure of this framework document follows this sequence of evidence- centred design steps. First, creative thinking is outlined, both in general and specifically in an educational context. Then, the elements of the construct and the methods of evidence identification and collection are explicitly set forth. Finally, the framework discusses issues related to validation and reporting.

Defining the assessment domain

What is creative thinking?

12. PISA employs a definition of creative thinking that is relevant to 15-year-old students around the world. Creative thinking in PISA 2021 is defined as the competence to engage productively in the generation, evaluation and improvement of ideas, that can result in original and effective solutions, advances in knowledge and impactful expressions of imagination.
13. This definition of creative thinking is aligned with the one proposed by the Creative Thinking Strategic Advisory Expert Group (OECD, 2017[3])^2. It highlights the fact that students in all contexts and across all levels of education need to learn how to engage productively in the practice of generating ideas, how to reflect upon ideas by valuing both their relevance and novelty, and how to iterate upon ideas until they reach a satisfactory outcome. It has also been informed by the guidance of interdisciplinary experts and a comprehensive review of the literature on creativity.
14. While creative thinking is still an emerging construct, the broader yet intrinsically related construct of creativity has a strong research tradition. Plucker, Beghetto and Dow (2004[5]) define creativity as “the interaction among aptitude, process, and environment by which an individual or group produces a perceptible product that is both novel and useful as defined within a social context”, reflecting its multidimensional and social nature.
15. Achieving creative outcomes requires the capacity to engage in creative thinking, but it can also demand a wider and more specialised set of attributes and skills, such as intelligence, domain knowledge or artistic talent. For example, the ‘Big C’ creativity that is associated with technology breakthroughs or art masterpieces demands that creative thinking be paired with significant talent, deep expertise and high levels of engagement in a particular area, as well as the recognition from society that the product has value. (^2) The Strategic Advisory Board defined creative thinking as ‘…the process by which we generate fresh ideas. It requires specific knowledge, skills and attitudes. It involves making connections across topics, concepts, disciplines and methodologies’. This definition builds on the five dimensional model by Lucas, Claxton and Spencer (2013), that identifies five creative thinking habits - being inquisitive, being imaginative, persevering, collaborating and being disciplined.

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT)

21. A comprehensive meta-analysis of empirical studies examining the domains of creativity supports the existence of a math/scientific domain that is consistently distinct from other domains of creativity (Julmi and Scherm, 2016[38]). The meta-analysis indicates that stable patterns are visible across studies, generally corresponding to “the factors ‘hands on’ creativity, empathy/communication and math/science identified by Kaufman and Baer (2004[29]).”

Confluence approaches of creativity

22. ‘Confluence approaches’, or ‘componential theories’, describe creative thinking and creativity as multi-dimensional phenomena (Lucas, 2016[39]). Amabile’s (1983[34]; (^2016) [40]) componential theory of creativity outlines four necessary components for any individual to produce creative work: domain-relevant skills, creativity-relevant processes, task motivation, and a conducive environment. The model specifies that creative production fundamentally requires some base resources or raw materials (i.e. domain-specific skills, including knowledge and technical skills), a set of processes or skills for combining these base resources in new ways (i.e. creativity-relevant processes, including appropriate cognitive styles such as breaking out of performance scripts and keeping response options open), and a driver in order to do so (i.e. task motivation). It also suggests that a number of environmental factors can serve as either inhibitors or facilitators of creative engagement. These four components include both relatively stable elements and elements that are more amenable to development and social influences.
23. Sternberg and Lubart’s (1991[41]; (^1995) [42]) ‘investment theory of creativity’ suggests that six distinct yet interrelated resources are necessary for creativity: intellectual skills (such as synthetic and analytical skills); domain-related knowledge; particular ‘thinking styles’ (such as a preference for thinking in new way); motivation; specific personality attributes; and an environment that is supportive and rewarding of creative ideas. Sternberg (2006[43]) later elaborated on the importance of the confluence of these resources, explaining that creative endeavours are far more complex than the simple sum of each respective component. Interactions between different components may lead to a variety of outcomes: for example, high levels in many components could multiplicatively enhance creative engagement; in contrast, there may be a minimum threshold for each component below which creative achievements are not possible, irrespective of the presence or the degree of other components.

Understanding and assessing creative thinking in the classroom

24. Confluence approaches of creativity emphasise the importance of various internal resources for successfully engaging in creative work, as well as the importance of the environment in which creative work takes place. They thus provide a useful schema for the PISA assessment of creative thinking. However, in order to better understand children’s creative thinking, it is necessary to contextualise these approaches in a way that is relevant to students in their everyday school life (Glaveanu et al., 2013[44]; Tanggaard, 2014[45]).
25. Figure 1 sets out some key points of observation of creative thinking in the classroom, as well as the relationships between the respective elements. This model builds upon the five-dimensional model of creative thinking proposed by the Creative Thinking Strategic Advisory Expert Group (OECD, 2017[3]).

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) Figure 1. Enablers and manifestations of creative thinking in the classroom

26. Schools can influence several dimensions of students’ internal resources (described henceforth as ‘individual enablers’) for engaging in creative thinking, including: cognitive skills; domain readiness (domain-specific knowledge and experience); openness to new ideas and experiences; willingness to work with others and build upon others’ ideas (collaboration); willingness to persist towards one’s goals in the face of difficulty and beliefs about one’s own ability to be creative (goal orientation and beliefs); and task motivation.
27. As for the features of students’ social environments that might incentivise or hinder creative thinking (described henceforth as ‘social enablers’), the classroom culture, the educational approach of schools and wider education systems, and the broader cultural environment all represent distinct social environments for students. They can all influence the extent to which students value and invest in their own creative abilities, and can provide incentives or obstacles for engaging in creative thinking.
28. Finally, schools are arenas in which students’ manifestations of creative thinking, either as individuals or as part of a group, can be observed and measured. Creative achievement and progress in the classroom can refer to forms of creative expression (i.e. communicating one’s internal world and imagination through writing, drawing, music or other arts), knowledge creation (i.e. generating knowledge that is new to the group and understanding in a collaborative enquiry process), or creative problem-solving (i.e. finding creative solutions to a variety of problems across domains).
29. These distinct enablers of creative thinking in the classroom are strongly interconnected. Social enablers are inherently shaped by cultural norms, which in turn affect how students’ individual enablers are developed and honed.

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) a barrier for creative thinking, by resulting in fixation and a reluctance to think beyond those established routines.

35. Schools naturally have an important role to play in developing children’s domain readiness (knowledge and experience) in a range of subject areas in which students can express their creative thinking.

Openness to experience and intellect

36. There is a vast literature dedicated to identifying the personality traits that characterise ‘creative people’. Empirical studies examining the personality and behaviour of creative individuals typically employ questionnaire instruments and operationalise creativity as a relatively enduring and stable personality trait (Hennessey and Amabile, 2010 [1]). These studies have shown that many creative people share a core set of tendencies, but particularly ‘openness’: both ‘openness to experience’ and ‘openness to intellect’ (although both variants are seen as comprising the larger ‘openness’ factor) (Amabile, (^2012) [59]; Batey and Furnham, 2006[60]; Feist, 1998[61]; Prabhu, Sutton and Sauser, 2008[62]; Sternberg and Lubart, 1991[41]; Sternberg and Lubart, 1995[42]).
37. Kaufman et al. (2009[31]) found that openness to experience was the only one of the ‘Big Five’^3 personality dimensions that was significantly and positively correlated with creative achievements across all domains. The study was then repeated with Chinese participants, who recorded similar results (with the exception of creativity in the maths/science domain) (Werner et al., (^2014) [63]). McCrae (1987[64]) also found that divergent thinking was consistently associated with openness to experience, but not with the other remaining dimensions of personality. Meta-analyses of studies on creativity and personality have confirmed that openness to experience appears to be a common trait in creative achievers across domains, whereas other personality traits appear to interact with creativity only insofar as they benefit individuals within specific domains of endeavour (for example, ‘conscientiousness’ seems to enhance scientific creativity but detract from performance in the arts) (Batey and Furnham, 2006[60]; Feist, 1998[61]).
38. More specifically, ‘openness to experience’ refers to an individual’s receptivity to novel ideas, imagination and fantasy (Berzonsky and Sullivan, 19 (^92) [65]). It has been suggested that its predictive value for creative achievements across domains is due to its “broad constellation of traits with cognitive (e.g. fantasy, imagination), affective (e.g. curiosity, intrinsic motivation) and behavioural manifestations (e.g. being adventurous, stepping outside of one’s comfort zone, actively trying new things), all of which are related to creativity” (Werner et al., 2014[63]). Several scholars have further emphasised the importance of a sense of curiosity for successfully producing creative work (Chávez-Eakle, (^2009) [66]; Feist, (^1998) [61]; Guastello, 2009[67]; Kashdan and Fincham, (^2002) [68]).
39. ‘Openness to intellect’ is a related yet distinct trait that has also been shown to predict creative achievement. This construct refers to cognitive engagement with abstract and semantic information, primarily through reasoning (DeYoung, 2014[69]). In contrast to openness to experience, openness to intellect seems particularly correlated with scientific creativity (Kaufman et al., 2016[70]). (^3) Also referred to as the Five Factor Model of personality traits: Openness to experience; Conscientiousness; Extraversion; Agreeableness; and Neuroticism (see McCrae and Costa (1987[125])).

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT)

Goal orientation and creative self-beliefs

40. Persistence, perseverance and creative self-efficacy are all attitudes that have been shown to influence creativity by providing individuals with both a strong sense of goal orientation, and the belief that they can go on to achieve those goals.
41. Persistence – the act of single-mindedly continuing to invest effort towards one’s goal in spite of difficulty – and perseverance – enduring and overcoming difficulty to achieve one’s goal – are essential for creativity. Cropley (1990[71]) characterised creative individuals by “their willingness to expend effort”, and Torrance (1988[72]) emphasised perseverance as one of the main traits of creative individuals. Amabile (1983[34]) argues that the ability to concentrate effort for long periods and to persevere in the face of frustration is an important component of creative capacity.
42. Creative self-efficacy refers to the beliefs that individuals have about their own ability to perform a task creatively (Beghetto and Karwowski, 2017[73]). Goal orientation and creative self-beliefs are closely linked: several researchers consider creative self-efficacy essential in determining whether an individual will sustain effort in the face of resistance (i.e. persist) and ultimately succeed (i.e. persevere) in performing tasks creatively (Bandura, 1997[74]). These beliefs can in turn be influenced by prior performance history, mood and the social environment in which a task is performed (Bandura, 1997[74]; Beghetto, 2006[75]).
43. Efforts to stimulate creative thinking in the classroom might therefore aim to strengthen students’ beliefs in their creative abilities and their proficiency in self-regulatory attitudes and behaviours (including persistence and perseverance) (Davis and Rimm, (^1985) [76]).

Collaborative engagement

44. Contemporary research is increasingly looking beyond creative thinking as a purely individual construct and towards creative thinking as a collective endeavour, for example by examining the actions of teams in generating new knowledge (Thompson and Choi, (^2005) [77]; Prather, 2010[78]; Grivas and Puccio, 2012[79]; Scardamalia, 2002[80]). This particular understanding of creative thinking posits that creative work is the result of the interaction between an individual and their environment, including other individuals within that environment. Creative thinking and engagement is thus structured as a continuous cycle of “doing” (actions directed at the environment) and “undergoing” (taking in reactions of the environment) (Glaveanu et al., (^2013) [44]). Through collaborative engagement, teams can provide new answers to complex problems that are beyond the capabilities of any one person (Warhuus et al., 2017[81]).
45. Research on collaborative creative thinking shows that team members engage in a complex intentional, opportunistic, improvisational and emergent process, setting goals and monitoring progress as different members of the team assume leadership based on their own strengths. Being able to engage in dialogic and improvisational processes in particular creates the conditions for new ideas to emerge (Montuori, 2003[82]; Tsoukas, 2009[83]). Through collaboration, action is fused with idea creation and improvement, the reparation of weaknesses in ideas, and the discovery of new ways around dead ends.
46. The capacity to engage in collaborative work is an important driver of knowledge creation, also in a classroom context. Schools can provide a rich environment in which students can explore and build upon others’ ideas in an iterative process, and thus collaboratively create new knowledge. Students need to learn how to get inspired by the

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) anyone can make yet few actually do because they find the social costs to be too high. Schools therefore play an important role in encouraging students’ creative thinking by increasing the rewards and decreasing the social costs associated with it in the classroom (Sternberg, 2006[43]). For example, it has been argued that the pressures of standardisation and accountability in educational testing systems have reduced the room afforded to students for creative thinking in their school work (DeCoker, 2000[93]). Some researchers have even claimed that increasingly narrow educational approaches and assessment methods are at the root of a ‘creaticide’ affecting today’s young people (Berliner, 2011[94]).

Classroom climate

52. Organisational research has demonstrated the effects of certain features of the working environment on the creativity of workers. Informal feedback, goal setting, positive challenges, teamwork, relative freedom in carrying out tasks, and appropriate recognition and encouragement to develop new ideas are all environmental enablers of creativity (Amabile, 2012[59]; Zhou and Su, 2010[95]). Conversely, harsh criticism of new ideas, emphasis on the status quo, low-risk attitudes among top management, and excessive time pressures are among the environmental factors that can inhibit creativity (Amabile, (^2012) [59]). It could be argued that the effects of similar environmental factors could also apply to creative thinking in the classroom.
53. With regards to schools specifically, Nickerson (2010[96]) provides a list of school practices that can stifle creative thinking: (1) perpetuating the idea that there is only one correct way to do a task and only one correct answer to a question; (2) cultivating attitudes of submission and fear of authority; (3) adhering to lesson plans at all costs; (4) promoting the belief that originality is a rare quality; (5) promoting beliefs in the compartmentalisation of knowledge; (6) discouraging curiosity and inquisitiveness; (7) and above all, never permitting learning and problem solving to be fun.
54. Teachers are more likely to focus on teaching creatively and developing learner creativity within school and policy environments that encourage innovation (and accept its associated risks) and that allow them to develop and express their own creativity. Teachers thus need to understand the importance of students’ idea diversity, risk taking, and working with peers in order to accomplish difficult tasks. These approaches are all supported by teachers’ beliefs that creative thinking competences are something that can be developed in the classroom, even if this development takes time.
55. Beghetto and Kaufman (2014[97]) propose that teachers should monitor implicit messages sent by the classroom environment as well as actively cultivating an environment that helps students learn how to take charge of their own creativity. For example, this could be achieved by encouraging higher levels of student agency in setting goals, monitoring progress, identifying promising ideas, and taking collective responsibility for contributing to productive, creative team work. Teachers should also help students to recognise how and when creative thinking is task appropriate.
56. Some educational researchers have explored different ways of teaching and learning that increase the likelihood of knowledge creation. The research shows that creative thinking can be successfully engendered through collaboration in knowledge-building communities, in other words, when schools operate as knowledge- creating organisations in which students are directly engaged in sustained, creative work with ideas (Scardamalia and Bereiter, 2006[98]; Scardamalia and Bereiter, 1999[99]). When knowledge creation becomes an intentional activity that is integral to classroom life – a

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) norm of engagement – students can contribute new ideas to their community and work towards continually improving those ideas (Scardamalia, 2002[80]).

57. Knowledge creation can also be promoted through ‘questions of wonderment’. Questions of wonderment describe the process of trying to understand the world and trigger students to put forth their ideas about different phenomena (Scardamalia and Bereiter, (^1992) [100]; Bereiter and Scardamalia, 2010[101]).

Creative engagement

58. The creativity of students’ products provides indicators of their capacity to think creatively, particularly in tasks where much of the creative thinking process is ‘invisible’. Students’ creative products can therefore be useful to determine whether their creative thinking process has been successful (Amabile, 1996[35]; Kaufman and Baer, 2012[102]).
59. Over the years, an impressive body of literature on the importance and analysis of creative products across a range of domains has emerged. According to accepted definitions within the literature, creative products are both novel and useful as defined within a particular social context. In the context of schools, creative engagement can take distinct ‘everyday’ forms: for example, through expressive activities of writing, drawing, music or other ‘arts’ subjects; the creation of new knowledge and understanding; or the generation of creative solutions to different types of open problems. These forms of creative engagement in the classroom are multi-disciplinary and extend beyond traditional subjects, such as art and science (Beghetto and Kaufman, 2010[54]; Sawyer, 2011[103]).

Creative expression

60. Creative expression consists of both verbal and non-verbal forms of creative engagement, in instances where individuals communicate their internal world and imagination to others. Verbal expression refers to the use of language, including both written and oral communication. Non-verbal expression includes not only drawing, painting, modelling and musical expression, but also expressive movement and performance, for example dance and drama.

Knowledge creation

61. Knowledge creation refers to the advancement of knowledge where the emphasis is placed on progress rather than achievement per se, for example by establishing improved conceptual ideas such as better explanations or theories. Knowledge creation is not only reserved for discoveries of historical importance, but can also occur at all levels of society and in all domains. Scardamalia and Bereiter (1999[99]) have elaborated parallels among the work of scientists, designers and young students in creating knowledge: for example, it can be helpful for all, regardless of domain, to reconstruct knowledge in order to interpret the findings of others and to make sense of existing theories.

Creative problem solving

62. Closely linked to knowledge creation is creative problem solving. Not all cases of problem solving require creative thinking: creative problem solving is a distinct class of problem solving characterised by novelty, unconventionality, persistence, and difficulty in problem formulation (Newell, Shaw and Simon, 1962[104]). Creative thinking becomes particularly necessary when students are challenged with problems outside of their realm

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) or problem-solving). The assessment domains (and related tasks) must also be reflective of the realistic manifestations of creative thinking that 15-year-olds can realise in this context.

69. A second constraint is the amount of available testing time. Under the current design of PISA assessments, students will take a one-hour creative thinking test. This means that the range of possible assessment domains must necessarily be limited, in order to ensure that a sufficient amount of data is collected in each domain. As PISA aims to provide comparable measures of performance at the country level, rather than at the individual level, it is possible to apply a rotated test design in which students take different combinations of tasks within domains (with some overlap). Nonetheless, ensuring the ability to produce reliable measures of country-level student performance by each domain requires that a sufficient amount of testing time be dedicated to the tasks within each domain, therefore limiting the number that can reasonably be covered in the assessment.
70. A third constraint is the necessity to implement the creative thinking test within the standard PISA testing platform. The PISA test is administered on standard desktop computers with no touch-screen capability and no internet connection. The platform currently supports a range of item types and response modes, including multiple choice, text entry, drag and drop, hot spots (clicking on areas within a text or image), a chat interface, and interactive charts and graphs. While it has been possible to include new functionalities to the platform during the development of this assessment, such as a drawing tool, both the choice of assessment domains and the design of the tasks had to take into due consideration the technical limitations of the platform.
71. Taking these main constraints into account, and building upon the literature that discusses the different domains of creativity, the PISA 2021 creative thinking assessment focuses on two broad thematic content areas: ‘creative expression’ and ‘knowledge creation and creative problem solving’. ‘Creative expression’ refers to instances where creative thinking is involved in communicating one’s inner world to others. This thematic content area is further divided into the domains of ‘written expression’ and ‘visual expression’. Originality, aesthetics, imagination, and affective intention and response largely characterise creative engagement in these domains. By contrast, creative engagement in ‘knowledge creation and creative problem-solving’ involves a more functional employment of creative thinking that is related to the investigation of open questions or problems (where there is no single solution). It is divided into the domains of ‘scientific problem solving’ and ‘social problem solving’. In these domains, creative engagement is a means to a ‘better end’, and it can thus be characterised by generating solutions that are original, innovative, effective and efficient.
72. The four assessment domains represent a reasonable coverage of the creative thinking activities in which 15-year-olds typically engage, and reflect the nature of real world and everyday creative thinking. While they clearly do not exhaust all possible manifestations of creative thinking in school, they do provide a sufficiently diverse coverage of the construct of creative thinking as well as adequately respect the various logistical and technological constraints of the PISA 2021 assessment.
73. Finally, given that differences in cultural preferences for certain forms of creative engagement exist, as do differences in what is valued in education and in how subjects are taught across the world, we can expect some degree of variation in student performance across domains. By having students work on more than one domain, it will be possible to gain insights on country-level strengths and weakness by domain of creative thinking. The data may also uncover the differences in the extent to which students are encouraged to

FRAMEWORK FOR THE ASSESSMENT OF CREATIVE THINKING IN PISA 2021 (THIRD DRAFT) search for their own solutions and ways to express their ideas, with important implications for how creative thinking in different domains should be taught in school. Figure 2. Proposed focus domains for the assessment

Written expression

74. Written work represents a natural means for creative expression both inside and outside of the school context, and creative writing is important for developing children's cognitive and communication skills (Tompkins, 1982[106]). Good creative writing requires logical consistency; creative writers ask the readers to understand and believe in their imagination, and this requires that they focus on details and continuity. For example, even stories that are based on fantasy, with monsters and space aliens, need to obey a certain set of rules of logic and to make sense within the universe the author has created.
75. Individuals engaged in creative writing reflect upon the craft and process of writing, define expectations for their work, and respond imaginatively to the text of others (Carter, (^2001) [107]). These processes can stimulate many new areas of intellectual and emotional development for students, deepening their understanding of themselves and of the world (Essex, 1996[108]). Moreover, creative writing does not only apply to works of fiction: engaging in non-fictional writing can also be creative, such as writing slogans and tag-lines, and these forms of creative written expression can help students to understand and master basic rules of effective communication they need for their life.
76. In the cognitive test, students will need to demonstrate a capacity to express their imagination in a written format, respecting the rules and conventions that make written communication understandable and appreciated for its originality by different audiences. Several test unit templates have been designed for the domain of written expression. Students are asked to: engage in open and imaginative writing (with constraints limiting the length of written text that human raters will need to evaluate); generate ideas for various written formats by considering different stimuli, such as cartoons without captions or fantasy illustrations; and make an original improvement to someone else’s written work (as provided in the task stimuli).

Visual expression

77. In the domain of visual expression, students explore, experiment and communicate ideas and their own experiences using a range of media, materials and processes (Irish