Download Evaluating Learning Impact: Measuring Training Effectiveness and more Exercises Nursing in PDF only on Docsity!
Impact questions and answers
Evaluating Learning Impact - gathers, organizes and analyzes information regarding the impact of learning solutions against key business drivers; presents the information in a that meaningful to organizations; uses learning metrics and analytics to inform organizational decision making. Validity - A degree to which an evaluation instrument measures what the talent development professional intends for it to measure. How does training evaluation benefit an organization? - Provides documented results of training programs, which can be used to prioritize training needs at the organizational level. Enables decision makers to shift financial and other resources from training that has a small effect on corporate goals to training that has greater influence. Reliability - The ability of the same measurement to produce consistent results over time. Formative Evaluation - Collects information to enhance the training program during different intervals of the program. The effectiveness of a training program is assessed while it is being developed.
Impact questions and answers
Summative Evaluation - Is conducted after the training program is completed to determine the effectiveness of the program. The effectiveness is assessed when it is already available. Neil Salkind (2005) - Explains the relationship between reliability and validity by using the analogy of a dartboard. The evaluation instrument hits the bull's eye - it is valid. The instrument hits the bull's eye every time - it's valid and reliable. The evaluation instrument consistently hits some other part of the board - it is reliable but not valid. Steps of the Measurement Process - 1. Identify evaluation goals. 2 Develop evaluation design and strategy. 3. Select and construct the measurement tools. 4. analyze data 5. report data Purpose of a training evaluation? - Improve the design of the learning experience, determine business impact, cost-benefit and ROI Advantages of using an evaluation - Helps focus the training, helps determine whether the program was cost effective. Validates performance gaps and learner need. Secures client support and builds client relationships. Determine if results
Impact questions and answers
Meta-evaluation - An evaluation of an evaluation in which an outside person reviews an evaluation for its quality (to minimize bias). Kirkpatrick's evaluation method order - 1. Reaction - measures the degree to which participants find the training favorable, engaging and relevant. 2. Learning - measures the degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment. 3. Behavior - apply what they learned. 4 Results. Program Evaluation - Wholey, Hatry and Newcomer (2004) describe as systematic assessment of program results and if possible the systematic assessment of how the program caused those results. It assess the effect of the training program on learning. Balanced Scorecard Approach - Customer Perspective - Did the solution, initiative or practice meet the customer's need or expectation? Balanced Scorecard Approach - Innovation and Learning Perspective - Did the users gain the needed skills or knowledge?
Impact questions and answers
Balanced Scorecard Approach - Internal Business Perspective - Did the solution, initiative or practice have an effect back on the job? Balanaced Scorecard Approach - Financial Perspective - Did the solution, initiative or practice have a financial pay off? Split-Half Check of Reliablity - A type of test in which one test is split into two shorter ones and administered to the same group. Tyler's Model - Focuses on organizing learning experiences in a logical order of experiences for the learner. Focuses on evaluating the curriculum and revising those aspects that do not prove to be effective. 1. what objectives should be achieved. 2. what learning activities will assist. 3. how will curriculum be organized. 4. how should learner be evaluated? Performance Based Training (PBT) - Key items to be evaluated: Learning transfer and course materials Mean - Sometimes called the average, this measure considers the quantitative value of each number. Sum of all numbers/The number of values that make up the sum.
Impact questions and answers
Ordinal Data - It enables ranking items in order based on quality. Example: which age group do you prefer to teach? 1. 5-7 2. 8-12 3. 13 to 15 Interval Data - It not only allows rank ordering the items measured, but also quantifying and comparing the sizes of differences between them. Example: a temperature of 20 degrees is higher than a temperature of 10 degrees and an increase from 20 to 40 degrees twice as much as an increase from 10 to 20 degrees. Ratio Data - It allows rank ordering, quantifying, and comparing. Additionally, ratio variables feature an identifiable absolute zero point. Example: measures of time; measures of spaceA Measuring Variance in Statistics - The average squared deviation of each number from the mean of it's data set: Variance (s2) = Standard deviation from the mean / number of observations - 1 (or n-1). It involves squaring deviations, so the variance doesn't have the same unit of measurement as the original observations. Statistical Significance - Significance tests consider both random and nonrandom errors. More significance tests mean more accurate results. The term significant means probably true. (Not caused by chance). It is important to check the
Impact questions and answers
sampling procedure to avoid bias. Too many significance test produce some falsely significant relationships. Percentiles - A common form of standard scores, may be used to indicate how a person's score compares with others on a scale of one to 100. Effect size - A way of quantifying the size of difference or relationship between two groups. Correlation - Measures the relationship between two variables. Causation - Requires a highly controlled experimental environment. -1.00 correlation coefficient value - Shows an inverse relationship between two variables. +1.00 correlation coefficient value - Shows a prediction for y given x
Impact questions and answers
Discrete Variable - A variable whose attributes are separate from those of another variable. Examples could include participant gender or organizational membership. Dichotomous Variable - A variable that falls into one of two possible classifications. Example: Gender male or female Most critical step of hypothesis testing? - Stating the hypothesis Skewness - Asymmetry in the distribution of sample data values. It shows values on one side of the distribution farther from the middle than values on the other side. Outliers - A data point that is further from the others in the data set. It is an unusually large or small value compared to the others. Normal Distribution - A particular way in which observations tend to gather around a certain value instead of being spread evenly across a range of values. (Bell Shaped Curve)
Impact questions and answers
Problems from misusing evaluation data are? - Data not being provided to the appropriate groups. Organizations not using the evaluation data at all. Data not being used to drive improvement. Data being used for the wrong reason. How is the measure of variance used in statistics? - Standard deviation from the mean / Number of observations -1 (or n-1) Confidence Intervals - The degree of confidence is linked with the width of the confidence interval. It allows the possibility of being wrong. The degree of confidence measures the probability of that expectation to be true. The confidence interval is the range where something is expected to be. Provides more information than point estimates. Provide confidence that something will be within a very wide range and vice versa. Confounding Variable - An unknown or uncontrolled variable that produces an effect in an experimental setting. An independent variable that the evaluator didn't somehow recognize or control. It becomes a variable that confounds the experiment. Continuous Variable - A variable whose attributes form a steady progression. (ie time, speed or distance)
Impact questions and answers
Scaled Question - Used to determine opinions or attitudes by measuring direction (positive or negative) and intensity (strongly positive or strongly negative) Multiple Answer Question - Does not have exclusive answers. Used to help respondents remember and to ensure that they consider all viable options. What should you do during the process of experimental design? - Clearly identify the specific questions that an experiment is intended to answer. Organize an experiment properly. Attempt to identify known or expected sources of variability in experimental units. Focus on designing experiments that improve the precision of the answers. Why is it important to validate business drivers before determining the appropriate measurement for an evaluation? - Business drivers are the internal and external factors that direct an organization's strategy, goals, business needs and performance needs. Dependent Variable - This variable should be the one that is measured.
Impact questions and answers
Extraneous Variable - This variable influences an experiment's outcome. Are undesirable variables that influence the relationship between the variables an evaluator is examining. They add error to the experiment. Quantitative data collection - Measures yield hard data. Can be stated in terms of frequency, percentage, proportion or time. Qualitative Data Collection - Measures yield soft data. More intangible anecdotal, personal and subjective. opinions, feelings etc. Population - A sample portion for which an evaluator is interested in collection data. Disadvantages of exams and tests as data collection methods - Test anxiety; impractical to include both knowledge and skill assessments; incomplete without interview. Doesn't reveal the reason a person performed a certain way. Test may not be constructed properly to justify how the results are used. Selection Bias - The error of distorting a statistical analysis by pre or post selecting the sample.
Impact questions and answers
Random Selection - The process of drawing the sample of people for a study from a population. Cause and Effect Analysis - Helps to identify possible causes for variation. Random Assignment - The process of allocating the sample to different groups or treatments in the study. What factors should be kept in mind to avoid bias in an interview? - A protocol for consistency in questioning. The sample is representative of the population. Participants understand the questions. One-way analysis of variance - This model allows for comparing several groups of observations, all of which are independent but possibly with a different mean for each group. Two-way Analysis of variance - This model studies the effects of two factors separately and together.
Impact questions and answers
Completely randomized design - This model assumes that treatments are allocated to experimental units completely at random. Completely randomized block design - This design, participants are put into groups-known as blocks- of the same size as the number of treatments. Force-Field analysis - Helps to identify forces that maintain status quo Hypothesis Testing - Helps to identify issues from a given sample. Factors that Influence Power in Statistical Test - The kind of statistical test being performed; sample size; the size of experimental effects; the level of error in experimental measurements 95 percent confidence interval - One time in 20 the findings were caused by chance Benefits of Forecasting - Predicts measurable values for a program's outcomes that have not yet been observed. It allows for the comparison between predicted values and actual data collected.
Impact questions and answers
Return on expectations (ROE) - A process that requires taking the day collected at each of Kirkpatrick's four levels and putting them into a logical chain of evidence that unifies the learning and business functions. Two levels of learning must be measured: Monetary value and the extent the learner produces performance results Talent Management analytics - The use of talent data to do such tasks as improve business performance, predict turnover, measure the business impact of leadership development programs, and determine the effectiveness of onboarding programs. cross-tab table - A table that shows a pictorial comparison of results of two or more questions and helps in analyzing cause and effect and complimentary relationships. Content Analysis - How data is coded, the meaning of the data and what alternative meaning should be considered, and whether or not reliable data patterns are emerging.
Impact questions and answers
Process Analysis - The tools or support materials used in completing the process, whether or not the use was consistent, the variables affecting the process, and the interdependencies that existed in the data. Quantitative Analysis - Includes identifying and explaining data discrepancies, determining what relationships exist in the data and how the data can be measured Two axis matrix - One set of variables is a row of descriptive terms (horizontal) and a second set of terms is (vertical) Three axis matrix - Cube shaped object used to express a set of abstract variables, such as judgements of quality, intervals of time, or types of things. Flowchart - Organizing and synthesizing information that contains input process output items, decisions points, directions of flow, step Event Networks - A system oriented synthesis model used to describe and make visible what should be rather than what is
