Collaborative Problem Solving: Definition, Frameworks, and Assessment, Study notes of Design

Download Collaborative Problem Solving: Definition, Frameworks, and Assessment and more Study notes Design in PDF only on Docsity!

Collaborative

Problem Solving:

Considerations for the

National Assessment

of Educational Progress

Collaborative

Problem Solving:

Considerations for the

National Assessment

of Educational Progress

Authored by:

Stephen M. Fiore

Art Graesser

Samuel Greiff

Patrick Griffin

Brian Gong

Patrick Kyllonen

Christine Massey

Harry O’Neil

Jim Pellegrino

Robert Rothman

Helen Soulé

Alina von Davier

April 2017

• Chapter 1: Executive Summary Table of Contents
  • Introduction
  • Defining Collaborative Problem Solving
  • Assessment Design.
  • Assessment Development
  • Scoring Collaborative Problem-Solving Assessments
  • Reporting Results
  • Implications for NAEP.
• Chapter 2: Introduction
  • The Need for Collaborative Problem-Solving Assessments
    • Collaborative Problem Solving: An Essential 21st Century Skill
    • Collaborative Problem Solving: A New Assessment Domain
  • Collaborative Problem-Solving Skills: An Interdisciplinary Experience
• Chapter 3: Defining Collaborative Problem Solving (CPS)
  • Introduction
  • PISA (2015) Collaborative Problem-Solving Framework
    • Collaboration and Problem-Solving Processes.
    • Collaborative Problem-Solving Contexts
  • Assessing and Teaching 21st Century Skills (ATC21S) Framework
    • Collaborative Problem-Solving Model
    • Cognitive Skills in Collaborative Problem Solving.
  • Comparison of PISA and ATC21S Frameworks.
  • Other Perspectives on Collaborative Problem Solving
    • Group Communication Theory
    • Macrocognition in Teams.
    • Models That Integrate Collaborative Learning and Problem Solving
    • Additional Frameworks and Gaps
  • Conclusion
• Chapter 4: Assessment Design
  • Introduction
  • Nature of the Collaborative Task
  • Human-to-Human vs. Human-to-Agent Collaborations and Team Size
  • Team Composition iii
    • Distributions of Ability and Knowledge
    • Roles
    • Member Characteristics
  • Evaluations.
    • Individual Outcomes.
    • Team Outcomes
    • Process Measures
    • Moderators
  • Conclusion
• Chapter 5: Assessment Development
  • PISA 2015 CPS Task Development
    • Task Characteristics
    • Development Process.
  • ATC21S 2015 CPS Task Development
    • Task Characteristics
  • Conclusion
• Chapter 6: Scoring Collaborative Assessments
  • Introduction
    • Task Types That Inform Scoring
    • Additional CPS Task Types
  • Data from CPS Tasks
  • Scoring a CPS Task
  • Analyses That Can Inform Scoring Approaches
  • A Statistical Representation of Collaboration
  • Conclusion
• Chapter 7: Reporting
  • Current NAEP Reporting
  • What to Report: Elements Reflecting Construct Definition.
    • Collaborative Problem Solving
    • Content
    • Conditions/Contexts
  • What to Report: Assessment Grades and Years. iv
  • Score Structure for Reporting
    • Scales and Scale Scores
    • Achievement Levels and Score Precision
  • Reporting Groups
  • Score Reporting – Interpretive Supports
  • Score Report Design
  • Conclusion
• Chapter 8: Implications for NAEP
  • NAEP’s Influence on Policy and Practice
  • Collaborative Problem Solving: New Types of Data.
  • Measuring a Cross-Content Framework
  • Collaboration and Group Scores
  • Process Variables
  • Conclusion
  • References

Chapter 1: Executive Summary 2

Defining Collaborative Problem Solving

The term “collaboration” has different meanings in different environments. In K-12, collaboration almost always means an individual task can be solved by anyone in the group, but collaboration is also an instructional strategy to enable learning more efficiently or effectively. In the world of work (industry, military), the term “collaborative” usually means a group task in which no one member of the group can solve the task alone.

Collaborative problem solving involves two different constructs—collaboration and problem solving. The assumption is that collaboration for a group task is essential because some problem-solving tasks are too complex for an individual to work through alone or the solution will be improved from the joint capacities of a team. People vary in the information, expertise, and experiences that they can bring to bear in order to jointly solve a particular problem. More specifically, collaborative problem solving requires that people share their resources and their strategies in order to reach a common goal via some sort of communication process. Whether in an individual or group task, the group can be either face-to-face or virtual. In both cases, some technology is often used to facilitate collaborative problem solving.

Generally, collaborative problem solving has two main areas: the collaborative (e.g., communication or social aspects) and the knowledge or cognitive aspects (e.g., domain-specific problem-solving strategies). These two areas are often referred to as “teamwork” and “taskwork.” The primary distinction between individual problem solving and collaborative problem solving is the social component in the context of a group task. This is composed of processes such as the need for communication, the exchange of ideas, and shared identification of the problem and its elements.

The PISA 2015 framework defines CPS as follows: Collaborative problem solving competency is the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution.

Competency is assessed by how well the individual interacts with agents during the course of problem solving. This includes achieving a shared understanding of the goals and activities, as well as efforts to pool resources and solve the problem.

Within the PISA framework, three competencies form the core of the collaboration dimension: establishing and maintaining shared understanding, taking appropriate action to solve the problem, and establishing and maintaining group organization. The framework also identifies four problem- solving processes: exploring and understanding, representing and formulating, planning and executing, and monitoring and reflecting.

Chapter 1: Executive Summary 3

ATC21S has also developed a framework for assessing collaborative problem solving. This framework, like that of PISA, identifies dimensions of collaboration: participation, perspective- taking, and social regulation. Problem-solving skills include task regulation skills and knowledge- building and learning skills. The main distinction between the two frameworks is that ATC21S has an integrated approach, in which the distinctions between collaboration and problem solving are melded. For the purposes of defining collaborative problem solving, research must determine which of these approaches produces the appropriate level of granularity and accuracy in assessment.

Other research on cognition has identified some additional competencies that could be measured in an assessment of collaborative problem solving. These include group communication, “macrocognition” in teams, collaborative learning, and cognitive readiness.

Assessment Design

In addition to the definition of collaborative problem solving, there are a number of issues associated with the design of an assessment of that competency. These include the following:

• What is the nature of the assessment task(s)? What content is to be assessed? Is the task ill- defined or well-defined?
• Who is collaborating? Are they human-to-human or human-to-agent interactions? How many members are on the teams?
• What is the composition of the team? Is it homogeneous or heterogeneous? Do the team members assume similar or different roles?
• What is being evaluated? Is it individual or team outcomes? Is it process variables?

Assessment Development

The experience of ATC21S and PISA suggests that assessment task development for collaborative problem solving differs little from standard test development. The major departure is in conceptualizing what constitutes a test item and how students will respond to it. The ATC21S assessment involves human-to-human interaction, while the PISA assessment involves human-to- agent interaction. The choice of the approach has strong implications for many of the other issues. For instance, what will the task environment look like? What will be the group composition? What collaboration medium should be used? How can the scoring be implemented?

The human-to-human assessment approach is embedded in a less standardized assessment environment and offers a high level of face validity. A student collaborates with other students, so the behavior of both is difficult to control. Also, the success of one student depends on the behavior of the other student, as well as the stimuli and reactions that he/she offers. This has implications for scoring. How can the open conversation and large variety of stimuli be identified and utilized for scoring?

Chapter 1: Executive Summary 5

Reporting Results

Reporting is the most publicly visible component of a NAEP assessment. What is reported from an assessment plays a major role in how the assessment is received both by the general public and by the more direct stakeholders, including teachers and school, district, and state personnel.

There are a number of issues NCES must address in determining how to report results from a collaborative problem-solving assessment, including

• What will be reported: collaborative skill, problem-solving skill, or both? Will it be individual responses or team responses? Will it be within a content area or separately?
• What is the score structure: one scale or separate scales for sub-skills? Will the scores be reported using performance levels? Will the scores be mapped to sample items?

NAEP and PISA provide excellent models for score reporting on national and international assessments. Regular NAEP assessments, such as mathematics and reading, provide good models for design, analysis, scoring, and reporting using probability (cluster) sampling, balanced incomplete block designs for administration, item-response theory for scoring, scaling, and linking enabling comparisons across grades and across time. Unique NAEP items, such as the science hands-on tasks and interactive computer tasks, and the new Technology and Engineering Literacy (TEL) tasks provide other models for reporting. PISA 2015, the score report that was released in December 2016, provides a reporting model for collaborative problem solving.

Implications for NAEP

NAEP’s importance and credibility has provided the assessment with a large influence on policy and practice at the state and local levels. Because of NAEP’s prominence, NAEP and the National Assessment Governing Board need to exert caution in developing and implementing a new type of assessment. An assessment of collaborative problem solving would introduce a number of new elements to NAEP that can pose challenges to interpretation. These include the use of multiple content areas (and possibly content-neutral problems), the use of student groups in test administrations, and the use of process variables. NCES and the Governing Board should be very clear about the information the assessment results provide, as well as their limitations.

Despite these challenges, an assessment of collaborative problem solving would represent a bold step that could strengthen NAEP’s role as an assessment leader. As a so-called “21st century skill,” collaborative problem solving is considered vital to students’ future in the workplace. Policymakers and practitioners need to know how well students can demonstrate that skill, and NAEP is in an ideal position to provide that information.

Chapter 2: Introduction 6

The Need for Collaborative Problem-Solving Assessments

There is an increased interest in collaboration and teamwork in the workforce, higher education, and K-12 education. The Assessment and Teaching of the 21st Century Skills (ATC21S) included collaboration among the most important skills necessary for a successful career. While there is widespread agreement in the field of education that collaboration is an important skill, there is less agreement on how to build an assessment to measure it.

In 2015 the Organization for Economic Cooperation and Development (OECD) published its draft framework, which included strong rationale for the inclusion of CPS, calling collaborative problem solving “a critical and necessary skill across educational settings and in the workforce.” This plan by the OECD to add CPS to its 2015 PISA test has important implications for U.S. assessments, including the National Assessment of Educational Progress (NAEP).

NAEP, which is the largest nationally representative assessment that is administered regularly over time, measures what America’s students know and can do in various subject areas. A broad range of audiences use the assessment results, including policymakers, educators, and the general public. Subject areas range from traditional curricular subjects, such as mathematics, to non-curricular topics, such as technology and engineering literacy. Each NAEP assessment is built around an organizing conceptual framework. Assessments must remain flexible to mirror changes in educational objectives and curricula; hence, the frameworks must be forward-looking and responsive, balancing current teaching practices with research findings. The most recent NAEP framework created was the Technology and Engineering Literacy (TEL) framework, which was introduced in 2014.

Because of the growing importance of collaborative problem-solving skills in the educational landscape, the National Center for Education Statistics (NCES) decided that NAEP should investigate state-of-the-art CPS research and assessment before deciding whether an assessment of CPS should be added to NAEP. Therefore, NCES assembled a broad array of individuals to develop this white paper, with the goal of fully conceptualizing the assessment of CPS skills as it currently exists,

2 Introduction

Chapter 2: Introduction 8

issued in 2000, sought “to help educators make high school courses more relevant to the needs of a modern workforce and to help employers ensure that their employees possess appropriate, up-to-date skills” (U.S. Department of Labor, 2000). Subsequent surveys such as Are they Ready to Work (2006) by The Conference Board, Inc., the Partnership for 21st Century Skills, Corporate Voices for Working Families, and the Society for Human Resource Management have added business’ perspective regarding the new sets of knowledge and skills needed for the 21st century U.S. workforce. Key research findings such as the work of Levy and Murnane (2013) in Trends in Routine and Non-routine Tasks in Occupation s , United States, 1960 to 2009 , have cited the dramatic increase in demand for non-routine interpersonal and analytic skills and the corresponding decrease in demand for routine manual and cognitive skills.

For more than two decades, the data has consistently identified collaboration, critical thinking, problem solving, communication, and creativity/innovation as critically important skill sets for the future. The 2010 American Management Association P21 Critical Skills Survey of 2000+ business leaders found that over 70% of leaders surveyed identified these skills as priorities when hiring and evaluating employees. In March 2015, the World Economic Forum released a report, New Vision for Education , based on analysis of research from nearly 100 countries, and again reiterated the ongoing skills gap and the importance of 21st century skills, including collaboration and problem solving: “These gaps are clear signs that too many students are not getting the education they need to prosper in the 21st century and countries are not finding enough of the skilled workers they need to compete” (2015).

By 2002, organizations such as the Partnership for 21st Century Skills (P21), a non-profit coalition of business, education, and community leaders, began to emerge to build awareness of the seriousness of the skills gap and advocate for educational solutions. P21, with the assistance of a broad coalition of experts from business, education and government, developed the Framework for 21st Century Learning, a comprehensive set of student outcomes that articulated necessary skills, content knowledge, and interdisciplinary themes needed for the future. Central to the framework are the 4 C’s—collaboration, critical thinking and problem solving, creativity and innovation, and communication—the very skills that business, education, and researchers have identified as critical for the future. The framework has been documented in numerous books and most models of 21st century skills include collaboration, or collaborative problem solving, as important skills. (e.g., Fadel & Trilling, 2012; Trilling & Fadel, 2009; Wagner, 2008, 2010).

The National Research Council’s report Education for Life and Work, Developing Transferable Knowledge and Skills in the 21st Century (July 2012) analyzed the models and research (or lack thereof) around “21st century skills” and “deeper learning.” The report articulated the importance of skills and proposed clarifying the terms by dividing them into three domains: intrapersonal, interpersonal, and cognitive. Under this framework, collaboration resides in the interpersonal domain and problem solving in the cognitive domain. Additionally, the report outlined educational strategies and significant research recommendations.

Chapter 2: Introduction 9

The importance of collaborative problem solving as an educational outcome and important skill for life and work has continued to increase since the turn of the 21st century. Business continues to evolve requiring more cross-functional teams that work across international and cultural borders and possess complex cognitive, collaborative, and critical thinking skills (American Management Association, 2010). As Dede (2009) has observed,

The nature of collaboration is shifting to a more sophisticated skillset. In addition to collaborating face-to-face with colleagues across a conference table, 21st century workers increasingly accomplish tasks through mediated interactions with peers halfway across the world whom they may never meet face-to-face. … Collaboration is worthy of inclusion as a 21st century skill because the importance of cooperative interpersonal capabilities is higher and the skills involved are more sophisticated than in the prior industrial era.

It is no secret that traditionally the U.S. K-12 education system has focused almost solely on content delivery. “Little time is spent on building capabilities in group interpretation, negotiation of shared meaning, and co-construction of problem resolutions” (Dede, 2009). Developing 21st century skills, especially the 4 Cs, has been left to students to learn on their own. The growing body of evidence, including student performance on national and international tests such as PISA, suggests that this is no longer acceptable. The incorporation of collaborative problem solving into the Common Core State Standards has brought additional focus to the importance of teaching and assessing it.

CollAboRATIvE PRoblEm SolvINg: A NEw ASSESSmENT DomAIN

Collaborative problem solving is a new assessment domain which is developing rapidly; however, there is much to be learned. The Assessment and Teaching of 21st Century Skills (ATC21S) CPS assessment has emerged out of a coalition of advisors and experts as part of a project sponsored by Microsoft, Intel, and Cisco, and part of a larger effort to define, develop pedagogies for, and assess 21st century skills (Griffin, 2014). Like PISA, tasks are designed to elicit collaborative problem-solving behaviors by having students work in pairs and communicate through on-screen chat messaging. Unlike PISA, students are collaborating with other students rather than with computer agents. There is a clear delineation between social skills and cognitive skills. The ATC21S model emphasizes the development of both collaborative skills and the cognitive outcomes that can result from student mastery of those collaborative skills.

As mentioned earlier, OECD’s addition of CPS to its 2015 PISA test has important implications for U.S. assessments including NAEP (OECD, 2013). Internationally, significant focus on these skills has emerged in the educational plans of many countries – for example, Singapore’s ICT Masterplan and Israel’s national plan for Adapting the Educational System to the 21st Century. PISA’s inclusion of collaborative problem solving in its assessment is also driven by the need to teach and assess the skills most in demand, not just in the U.S., but internationally as well (Brannick & Prince, 1997; Griffin, 2014; National Research Council, 2011; Rosen & Rimor, 2012).

Chapter 2: Introduction 11

contexts may be required (Dede, 2012). The main challenge of the next generation of assessments, and the next generation of psychometrics, is how to model this knowledge in a way that meets the technical standards of traditional assessments.

In order to address these issues, the team of contributors to this white paper is diverse. To ensure that our interdisciplinary contributors produce a relatively integrated view, the authors address the following key questions:

1. How can we define collaboration and team output? What are the features of a good team?
2. How can we define the contributions of an individual to team performance?
3. How do we design an assessment that captures the process of collaboration together with the outcome of collaboration?
4. What are the features of a good interactive CPS task? Should it be a simulation or a game-based task? Is human-to-agent collaboration a good approximation of the CPS skills needed for the human-to-human interaction?
5. What type of interdependencies exist in the data collected from interactive CPS tasks? What is the best way to capture and analyze them? Will interactive CPS tasks result in “Big Data,” and if so, what is the implication for large assessments like NAEP?
6. What kind of statistical methods can be used to evaluate team dynamics? What is the unit of observation and the scoring rubric (Nelson, 2007)? For example, should we consider the specific actions taken by each student (and captured in log files)? Or should interim but summative stages for scoring be considered for an entire team? Should we model individual ability in the area of the assessment or the collective ability of a team?
7. Are there methodologies in use in one domain (e.g., group dynamics) that could be applied to another (e.g., collaborative problem solving)?
8. How do we score a CPS task? Should we report a multidimensional score or a combination of an individual score and a team score? Should we consider automatic scoring engines for analyzing the discourse data?
9. How will we define fairness in a CPS task?

These questions pinpoint features of the CPS construct that make its measurement considerations significantly different from traditional cognitive or even non-cognitive tests. Each of these issues is discussed in the individual chapters.

The purpose of this white paper is to inform decision making around the potential inclusion of CPS in NAEP. It is not intended to make recommendations about how NAEP should approach CPS, but rather to outline the possibilities for consideration.

Chapter 3: Defining Collaborative Problem Solving (CPS) 12

Introduction

Collaborative problem solving (CPS) is composed of two main elements: the collaborative, sharing, or social aspects coupled with the knowledge or cognitive aspects. Thus, the primary distinction between individual problem solving and collaborative problem solving is the social component. This is comprised of processes such as the need for communication, the exchange of ideas, shared identification of the problem, negotiated agreements, and relationship management. It is important to distinguish CPS from other forms of collaboration. In collaborative problem solving there is a group goal that needs to be achieved, the solution requires problem solving, team members contribute to the solution, and there is some foundation for evaluating whether the group goal has been achieved. Moreover, the activities of the team members are interdependent, with various roles, so that a single person cannot solve the group goal alone. The collaborative activities therefore require communication, coordination, and cooperation. These characteristics of CPS differ from collaborative learning, where the emphasis is on the learning of subject matter by individual team members in a group context but does not require a group problem that has to be solved with interdependency among team members. Similarly, collaborative work and collaborative decision making involve interdependent joint activities, but do not require problem solving.

This chapter defines collaborative problem solving that involves multiple people working interdependently towards a common goal (Fiore, 2008; Fiore et al., 2010; Graesser, Foltz et al., in press; Griffin, 2014). We describe two major frameworks that have been developed to define collaborative problem solving for the purposes of assessment. The first is the Program for International Student Assessment (PISA) assessment (Graesser et al., in press; OECD, 2013), which assessed CPS in several dozen countries in 2015. The second is the Assessing and Teaching 21st Century Skills (ATC21S) project, headquartered at the University of Melbourne, that developed a conceptual framework for CPS in 2010 with associated assessment tasks (Griffin, McGraw, & Care, (Eds.) 2012; Hesse, Care, Buder, Sassenberg, & Griffin, 2015). We then discuss some approaches to CPS that include additional factors that might need to be considered in CPS assessment.

Defining Collaborative

3 Problem Solving (CPS)

Chapter 3: Defining Collaborative Problem Solving (CPS) 14

CollAboRATIoN AND PRoblEm-SolvINg PRoCESSES

The PISA framework organized CPS competencies along two dimensions in a matrix that is shown in Table 3.1. One dimension addresses collaboration (the columns with 3 components) and the other problem solving (the rows with 4 components). The resulting matrix has 12 cells with skills that integrate collaboration and problem-solving processes. The collaboration dimension has three competencies. First is the concept of “establishing and maintaining shared understanding.” Here, students need to identify shared knowledge (what each other knows about the problem), identify the perspectives of other agents in the collaboration, establish a shared vision of the problem states and activities, and monitor and maintain relevant shared knowledge throughout the problem- solving task. Concrete actions include responding to requests for information, sending important information to agents about whether tasks are completed, verifying what each other knows, negotiating agreements, and repairing communication breakdowns. Second is the idea of “taking appropriate action to solve the problem.” Here, students need to identify the type of CPS activities that are needed to solve the problem and to follow the appropriate steps to achieve a solution. Both physical actions and acts of communication are typically needed to solve the problem. Third is the idea of “establishing and maintaining team organization.” Students need to organize and monitor the team to solve the problem; consider the talents, resources, and assets of team members; understand the roles of the different agents; follow the relevant steps for assigned roles; and reflect on the success of the team organization.

The problem-solving dimension directly incorporates the same competencies that were adopted in the individual problem-solving assessments of the PISA 2012 framework (Funke, 2010; Greiff et al., 2013; OECD, 2010). Specifically, there are four problem-solving processes that are needed to successfully address the kinds of complex problems being faced in the 21st century. First is the idea of “exploring and understanding.” In this stage, students need to interpret initial information about the problem, which is uncovered during exploration and interactions with the problem. Second is the notion of “representing and formulating.” Here students must select, organize, and integrate information with prior knowledge. From a process standpoint, this may include the use of graphs, tables, symbols, or words. Third is the process of “planning and executing.” This includes identifying the goals of the problem, setting any sub-goals, developing a plan to reach the goal state, and executing the plan. Last is the phase of “monitoring and reflecting.” Here the student is expected to monitor steps in the plan to reach the goal state, mark progress, reflect on the quality of the solutions, and revise the plan when encountering obstacles.

Chapter 3: Defining Collaborative Problem Solving (CPS) 15

Table 3.1.

Matrix of collaborative problem solving for PISA 2015 (OECD, 2013)

(1) Establishing and maintaining shared understanding

(2) Taking appropriate action to solve the problem

(3) Establishing and maintaining team organization (A) Exploring and Understanding

(A1) Discovering perspectives and abilities of team members

(A2) Discovering the type of collaborative interaction to solve the problem, along with goals

(A3) Understanding roles to solve problem

(B) Representing and Formulating

(B1) Building a shared representation and negotiating the meaning of the problem (common ground)

(B2) Identifying and describing tasks to be completed

(B3) Describe roles and team organization (communication protocol/rules of engagement) (C) Planning and Executing

(C1) Communicating with team members about the actions to be/ being performed

(C2) Enacting plans (C3) Following rules of engagement (e.g., prompting other team members to perform their tasks) (D) Monitoring and Reflecting

(D1) Monitoring and repairing the shared understanding

(D2) Monitoring results of actions and evaluating success in solving the problem

(D3) Monitoring, providing feedback, and adapting the team organization and roles

The 12-cell matrix provides a set of definitions for guiding assessment of collaboration skills engaged during problem solving. Some skills are assessed by the actions that the student performs, such as making a decision, choosing an item on the screen, selecting values of parameters in a simulation, or preparing a requested report. Other skills require acts of communication, such as asking other group members questions, answering questions, disseminating information, issuing requests, and giving feedback to others. Assessment of CPS, then, requires tracking the actions and communications of the individual being tested as the individual experiences a series of events, actions, and conversational speech acts by the other agents in the group. These experiences include obstacles to the collaboration, such as an agent that makes errors, fails to complete tasks, or does not communicate important information. The assessment tracks how well the human handles the various challenges and has skills associated with all 12 cells in the matrix. Details on the design, development, and scoring of items are provided in Chapters 4, 5, and 6.

CollAboRATIvE PRoblEm-SolvINg CoNTExTS

The PISA CPS 2015 framework also describes characteristics of problem-solving contexts that need to be considered. The problem-solving tasks require interdependency and joint activity among agents so that one agent cannot solve the group goal independently. Different problem- solving scenarios are identified, such as hidden profile (team members start out with distinct and complimentary pieces of information that must be shared), consensus in making a decision, and