Download CPCE Research and Program Evaluation Test and more Exams Advanced Education in PDF only on Docsity!
CPCE: Research and Program
Evaluation Test
Research vs. Evidence Based Inquiry - ANS-Research: the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question. Evidence-based inquiry: the search for knowledge using empirical data which has been gathered systematically. Quantitative Research - ANS-Assumes social facts have a single objective reality. Tends to study samples or populations. Researchers try not to influence collection of data (instruments). Statistical methods comparing and contrasting groups occurs. Researchers examine for causes and relationships. Qualitative Research - ANS-Assumes multiple realities socially constructed by individuals and groups. Tends to study individual units - person, family, community — in naturalistic setting. Researchers may be primary instrument for collecting data (through observation). Researchers' impressions, judgments and feelings may be used. Goal is to describe the nature of things. Inductive vs. Deductive Research - ANS-Inductive: this research begins at the real world, practical level. It tends to be descriptive, correlational, or historical and leads to the building of theory. Deductive: this research springs from theory which is already established. This research tries to determine what the relationships are between elements of the theory and may be experimental in nature.
Quantitative (Non-Experimental Designs): Survey - ANS-This may occur through questionnaires, interviews, etc. and is used to measure attitudes, perceptions, etc. (For example: Public Opinion Poll). Often the response rate of survey research is low, below 50 percent. Unless you know that the characteristics of the non- respondents are similar to the characteristics of the respondents, you must be cautious in generalizing. Quantitative (Non-Experimental Designs): Descriptive - ANS-Descriptive: this research simply describes an existing state of events. Numbers may be used to characterize groups or individuals. Quantitative (Non-Experimental Designs): Comparative - ANS-This research method investigates whether there are differences between two or more groups. There is no manipulation of conditions experienced by each group. Quantitative (Non-Experimental Designs): Correlational - ANS-This research method uses the correlation coefficient to determine the degree of relationship between two or more variables or phenomena. Quantitative (Non-Experimental Designs): Ex Post Facto - ANS-Also called causal- comparative, this research design studies possible causal relationships among variables ex post facto (after the fact). You do not manipulate any variables; the focus is on what has already happened (after the fact). You may generate 'several' reasons (causes) for the relationships you discover. For example: Two employment agencies conduct job dubs. In one agency, the job clubs are member-initiated and run whereas in the other, employment counselors provide instruction and guidance. In examining the job placement rates of members for the past year, you find that the members of the professionally led groups had a higher placement rate. Typical statistics used in ex post facto research are the t -test and analysis of variance, Quantitative (Experimental Designs): True Experiment - ANS-This research is characterized by the use of experimental and control groups with random assignment to each. Experimental designs are used to determine cause-and-effect relationships. For experiments, there are design variations such as: a. Treatment and control group with posttest only. b. Treatment and control group with pretest and posttest. c. Two different treatment groups with control group and posttest, and many other combinations.
Pilot Study - ANS-A small-scale research effort often used to determine the feasibility of a large scale effort with emphasis on refining procedures and instrumentation. Longitudinal Research - ANS-Consists of collecting data from the same group of individuals over a period of time. This is also called a panel study. For example, this author studied the career development of school children beginning when they were second graders and re-interviewing them every two years until they were high school seniors. Cross-Sectional Research - ANS-Consists of collecting data from different groups at the same time and examining these differences. For example, one approach to studying career development is collecting data from each grade of students, for example from second to twelfth grade, at the same time. Within-Subjects vs. Between-Subjects - ANS-Within-subjects: examining what changes occur within the members of a group. Between subjects: examining what changes occur between two or more groups. Meta-Analysis - ANS-This is research comparing findings across studies, i.e., the results of many studies are examined simultaneously and one or more research questions answered. For example, a number of studies have examined what works in various counseling situations. Examining a number of these studies simultaneously can help determine what counseling techniques work for what kind of clients, with what kinds of problems, under what conditions. Internal Validity - ANS-Experiments are internally valid to the extent that extraneous variables have been controlled. In other words, to the extent that the treatment variable is the only one producing the observed changes, the experiment is internally valid. Threats to Internal Validity: Selection of Subjects - ANS-Differences in the results between two groups may not be due to the treatment variable experienced by one group because the composition of the two groups are different to begin with (probably not randomly selected). Threats to Internal Validity: Instrumentation - ANS-Differences in results between two or more groups may be due to instruments which are unreliable or because the instruments are changed during the study. Or, perhaps the observers recording data become fatigued or bored and they record behaviors differently over time.
Threats to Internal Validity: Maturation - ANS-Results may be due to maturational or other changes in the subjects and not due to the treatment being applied. This is especially important if research data is gathered over a long period of time. Threats to Internal Validity: Mortality or Attrition - ANS-Losing subjects during the study could lead to different results than if everyone had stayed. Subjects with the most or least amount of important characteristics to the study may be the ones dropping out. (Think of normal curve extremes.) Threats to Internal Validity: Experimenter Bias - ANS-The responses of the subjects may be influenced by the researcher. This may occur by treating some subjects differently, reinforcing different behaviors, as well as the presence of many other variables which deliberately or unintentionally influence subjects. Threats to Internal Validity: History - ANS-These may be the extraneous incidents (e.g., national or local tragedies or other occurrences) which influence subjects in a research project. These events might interact with the independent variable so the results which occur may be due to a combination of the event and the variable under study. Threats to Internal Validity: Statistical Regression - ANS-Sometimes subjects in a study are recruited because of extreme high or low scores (e.g., self-esteem, social skills) on the dependent variable being measured. Due to statistical regression, future measures would expect these individuals to score closer to the mean score even without any intervention. External Validity - ANS-An experiment is externally valid to the extent that the results may be generalized to people and situations beyond the study. There are several threats to the external validity of experiments and some of these are also threats to internal validity. Threats to External Validity: Selection of Subjects - ANS-If subjects are not randomly selected, the results may only apply to the subjects in the study. Furthermore, the results can only be generalized to people with similar characteristics. Can we generalize results of research with college students to other adults in general? Threats to External Validity: Ecological Validity - ANS-The research has ecological validity if the results can be generalized from one setting or circumstance to another. Sometimes the circumstances, conditions or physical surroundings of the research are so unique that the results cannot be generalized beyond that study. Threats to External Validity: Subject Reactions (Reactivity) - ANS-If any of the following are present, the results may not generalize beyond the study:
Stratified Sampling - ANS-This refers to selecting in such a way that major subgroups in the population will be sampled. These subgroups may be based on ethnicity, gender, age, etc. Proportional Stratified Sampling - ANS-This refers to randomly selecting the same proportion of individuals for the sample as they represent proportionally in the major subgroups in the population. For example: If one-half of a population is Hispanic and one-half is white, you would randomly select your sample to be one-half Hispanic and one-half white. Cluster Sampling - ANS-The unit is not an individual but naturally occurring groups of individuals such as classrooms, city blocks, etc. Clusters are randomly selected for study. Purposeful Sampling - ANS-In some studies, there may be no interest in generalizing findings so purposeful sampling may be carried out. Possibilities are comprehensive sampling where every case or event is selected, or there is extreme-case or typical- case selection. Samples of Convenience/Volunteer Samples - ANS-Non-random or non-probability samples. These samples cannot be counted on to yield a normal distribution of scores. However, they could yield very useful and important data. Sample Size - ANS-Sample size influences statistical hypothesis testing. Tables for determining appropriate sample size are available. A general rule is that 5 to 10 percent of the population is adequate. The suggested minimum sample sizes for different kinds of research are: Correlational: 30 Ex post facto and experimental: 15 Survey: 100 A table of random numbers lists computer-generated random numbers used to assign to individuals. Selecting from this table assures randomness. Descriptive vs. Inferential Statistical Analysis - ANS-Descriptive: sometimes called summary, these techniques are used to describe the data collected for a research sample or population, and include means, standard deviations, frequency counts, and percentages. Inferential: used to make inferences from the sample to the population. The goal is to determine the probability of some event occurring. Common statistical tools used for this are the i-test and analysis of variance.
Parametric vs. Nonparametric Statistical Analysis - ANS-Parametric: used when a sample is randomly drawn from a population and the data is normally distributed. You have para (two-sided) data that yields a bell-shaped curve. Also, you assume that the variance of the sample you are studying is homogeneous (similar) to the variance of the population from which your sample is drawn. The scores would give you a normal curve -- more or less. Examples of parametric statistics are: i-test and analysis of variance. Nonparametric: used when you cannot make any assumption about the shape of the curve or variance of the population scores (that is, they may not be normally distributed and variances may not be homogeneous). You want to make a statement about a school's academic performance but you only collected scores from the "gifted and talented" students. You will not have a normal distribution of academic scores for the school. Examples of nonparametric statistics are: Chi-square, Mann-Whitney U Test, and Wilcoxon Signed-Ranks Test, all to be reviewed later. Independent Variable - ANS-This is the variable you manipulate or vary to see what change occurs in the dependent variable. Sometimes you group or categorize the independent variable. For example, you categorize a group of individuals by gender, male and female. Or, you categorize high school students as freshmen, sophomores, juniors, and seniors. The independent variable precedes or is antecedent to the dependent variable. The independent variable is sometimes called: stimulus variable, predictor variable or experimental variable. Dependent Variable - ANS-This is the variable you are measuring or trying to change. The value of this variable depends upon the value of the independent variable you selected. The dependent variable is sometimes called: response variable, outcome variable or criterion variable. Research Questions and Hypotheses - ANS-Some investigators ask research questions to be answered. Some investigators formulate hypotheses (often in the null form) to test. The null hypothesis states there is no difference between the variables or groups measured. Ex: There are no significant differences in final academic grades (GPA) between boys and girls finishing the tenth grade. One alternative hypothesis (directional) states that one group's score will be significantly different from another group's score (one-tailed test). Ex: The GPA of girls will be higher than the GPA of boys finishing the tenth grade. Another alternative hypothesis (nondirectional) states that there will be differences between the groups but which group has higher or lower scores is not indicated (two- tailed test). Ex: The GPA of girls and boys finishing the tenth grade will be different.
b. master's degree — licensed social worker (LSW) c. Ph.D. degree - psychologist (Ph.D.) You are attempting to determine self-esteem differences in clients following counseling and are curious as to whether level of training of therapist might lead to different levels of self-esteem. Since you now have three groups (the levels of training) the /-test is no longer appropriate. You would use the one-way analysis of variance - when you have only one variable (level of training) at three or more levels. Completing the calculation for the analysis of variance yields an F value and this value must be compared to the values listed in an F Distribution Table to determine whether significant differences are present. Computer data analysis programs now provide this information automatically. ANOVA: Factorial - ANS-You would use the factorial analysis of variance (ANOVA) to simultaneously determine whether mean scores on two or more variables (factors) differ significantly from each other and whether the factors interact significantly with each other. In our example in Item 18, we are comparing self-esteem scores of clients who have been counseled by female and male counselors. Let's assume we want to compare self- esteem scores with a second counselor variable as well, kind of training. Our two independent variables then are (I) gender (two kinds -female and male) and (2) kind of training. Let's stay with the same three kinds of training we used in Item 22. The one dependent variable is the measure of self-esteem. This IS a 2 X 3 ANOVA design. By completing this calculation, one could deter- mine if there are any significant differences in client self-esteem based on therapist gender, or kind of therapist training, or whether there is a significant interaction between gender and training. This is called a factorial analysis of variance because the effects of two or more factors (independent variables) are being measured. ANOVA: Multivariate (MANOVA) - ANS-When your analysis involves more than one dependent variable, you cannot use Factorial ANOVA. The appropriate statistic is multivariate analysis of variance (MANOVA). If you were measuring something else about the clients in addition to self-esteem, that would be more than one dependent variable. In our continuing example (See Item 23 - ANOVA), we had two independent variables - gender and kind of training. Our single dependent variable was a measure of self- esteem. Let's assume that we also want to measure the locus of control (internal vs. external orientation) of clients when they conclude therapy. We have each client complete a locus of control scale in addition to the self-esteem inventory. Whenever you have two or more dependent variables, you cannot use ANOVA. You must use multivariate analysis of variance (MANOVA).
Analysis of Covariance (ANCOVA) - ANS-Analysis of covariance is used similarly to analysis of variance except that the influence of one or more independent variables on the dependent variable is controlled. This means that initial group differences are adjusted statistically on one or more variables that are related to the dependent variable. For example: In our continuing example (Item 18 and onward), we are looking at the impact of therapists' gender on clients' self-esteem. Let's assume that we do not have random assignment of clients to therapists. In fact, let's assume those most female clients are assigned to female counselors and most male clients are seen by male counselors. On the pretest of self-esteem, we find that women clients score an average of five points higher than male clients. In ANCOVA, we make use of such information which is obviously related to our dependent measure, self- esteem. If we were measuring the impact of therapist gender on clients' self-esteem, women therapists would have started out five points higher on client self-esteem. So we statistically adjust client self-esteem scores (for example, add points to the males pretest scores and subtract points from the female pretest scores) before making the comparison. In this case, the covariate is the systematic difference in self-esteem scores on the pretest (women clients score an average of five points higher than men clients). Consequently, we conduct an analysis of covariance - adjust the scores. Then we do the analysis of variance. Post Hoc or Multiple Comparisons Test - ANS-If your analysis of variance yields a significant F value, you still will not know which particular pair of mean scores is significantly different from each other. You must apply one of the post hoc (after the fact) tests to determine whether different particular group means or combinations of group means are significantly different. The most commonly used post hoc or multiple comparison tests are: Scheffe's (most conservative) Tukey's HSD (Honestly Significant Difference) Newman-Keuls Duncan's new multiple range test Your computer data analysis program will conduct these automatically. Nonparametric Tests - ANS-When you cannot assume that your distribution of scores is normally distributed (resembles a normal curve) or that the variance of your sample is similar to the variance of the population (homogeneity) you must use nonparametric statistics. Some examples are: Mann-Whitney U test: when you collect data from two samples that are independent from each other and the scores are not normally distributed.
following statement might represent the data; As individuals get more anxious about the counselor exam, their hostility increases (positive relationship). Factor Analysis - ANS-This is a statistical method using the correlation coefficient to determine whether a set of variables can be reduced to a smaller number of factors. For example: A factor analysis of all the items on a long inventory with 15 scales may uncover only four or five factors independent of each other underlying the scales. Thus many of the constructs of the 15 scales overlap with each other. Likert Scale - ANS-This is a widely used technique for measuring attitudes or opinions. It allows for several response choices, e.g., a five (or seven or nine) point scale. Biserial Correlation - ANS-An appropriate correlation coefficient to use when one variable yields continuous data and the other yields data that is dichotomous. Degrees of Freedom - ANS-The number of observations that are free to vary. Double-Blind Technique - ANS-This occurs when neither the researcher nor the subject knows who is getting the active substance or the placebo. Halo Effect - ANS-This is the tendency for the observer (researcher or data collector) to form an early impression of the person being observed and then letting this impression influence observations or ratings of that individual. Halo effect may be positive or negative. Heteroscedasticity - ANS-One end of a distribution of scores has more variability than the other end resulting in a fan-like appearance. Homoscedasticity - ANS-There is an equal distribution of scores throughout the range of scores, i.e., around a line of best fit. Inter-Rater Reliability - ANS-In qualitative research, the reliability calculated by correlating the responses of several raters. Observer Bias - ANS-This is the tendency of researchers to see, hear and remember what they want to. Placebo - ANS-Originally meant an inert drug (typically, a sugar pill); now it also means a control treatment that gives subjects the same amount and kind of attention as experimental group subjects get (this reduces Hawthorne and Rosenthal effects). Rank-Order Correlation (Spearman rho) - ANS-Used when the values of the variables are reported in rank form rather than continuous.
Counseling Program Evaluation - ANS-The emphasis on accountability in the human services field accelerated in the 1970s and continues today. The push for accountability comes primarily from funders including Health Maintenance Organizations, insurance companies, government funding sources at all levels, and others. This accountability in human services reflects what the business world adopted many years ago as the 'bottom line.' There is an acute need to demonstrate the efficacy of counseling in general and the effectiveness of specific theories, techniques, and approaches in particular. The emphasis on short-term therapy, often between six and twelve sessions, argues for research and evaluation to determine what works well for what kinds of problems with what clients under what circumstances. Counseling program evaluation requires that goals and measurable objectives be specified in advance. Without those, evaluation data has little relevance. The effectiveness of counseling techniques and processes often occurs on an individual client basis. Evaluation: Formative vs. Summative - ANS-Evaluation is the systematic collection of evidence of the worth of a program, process, or technique. a. Formative evaluation: This is ongoing, process evaluation to measure the effectiveness of a technique or part of a program. Formative evaluation tries to determine how well a new technique, process or treatment works. b. Summative evaluation: This is summary or product evaluation designed to measure the effectiveness of a program, usually conducted at the end of a cycle such as a school year, fiscal year, etc. Summative evaluation is conducted to see how well agency or program goals have been met. Usually a product (document) is generated so this is 'product' evaluation versus 'process' evaluation (formative). Ethical Issues in Research - ANS-Confidentiality: no one should have access to the data except for the researcher and research assistants if any. Release of research data to others is only ethical with the consent of the subject. Deception: deception may be justifiable if no risk to subjects is involved. Such research should be followed by debriefing of the subjects. Informed consent: subjects should be informed of the research they will be participating in and give their consent. Significance of research results should outweigh the potential benefits denied the control group. Research must be approved by a 'human subjects committee' or Institutional Review Board when conducted within institutions or agencies when federal funding is involved.
