Download CPCE Research and Program Evaluation.pdf and more Exams Nursing in PDF only on Docsity!
CPCE: Research and Program Evaluation
Research vs. Evidence Based Inquiry - Research: the systematic process of collecting and analyzing data for some purpose such as investigating a problem or answering a question. Evidence-based inquiry: the search for knowledge using empirical data which has been gathered systematically. Quantitative Research - Assumes social facts have a single objective reality. Tends to study samples or populations. Researchers try not to influence collection of data (instruments). Statistical methods comparing and contrasting groups occurs. Researchers examine for causes and relationships. Qualitative Research - Assumes multiple realities socially constructed by individuals and groups. Tends to study individual units - person, family, community — in naturalistic setting. Researchers may be primary instrument for collecting data (through observation). Researchers' impressions, judgments and feelings may be used. Goal is to describe the nature of things. Inductive vs. Deductive Research - Inductive: this research begins at the real world, practical level. It tends to be descriptive, correlational, or historical and leads to the building of theory. Deductive: this research springs from theory which is already established. This research tries to determine what the relationships are between elements of the theory and may be experimental in nature. Quantitative (Non-Experimental Designs): Survey - This may occur through questionnaires, interviews, etc. and is used to measure attitudes, perceptions, etc. (For example: Public Opinion Poll). Often the response rate of survey research is low, below 50 percent. Unless you know that the characteristics of the non- respondents are similar to the characteristics of the respondents, you must be cautious in generalizing. Quantitative (Non-Experimental Designs): Descriptive -
Descriptive: this research simply describes an existing state of events. Numbers may be used to characterize groups or individuals. Quantitative (Non-Experimental Designs): Comparative - This research method investigates whether there are differences between two or more groups. There is no manipulation of conditions experienced by each group. Quantitative (Non-Experimental Designs): Correlational - This research method uses the correlation coefficient to determine the degree of relationship between two or more variables or phenomena. Quantitative (Non-Experimental Designs): Ex Post Facto - Also called causal-comparative, this research design studies possible causal relationships among variables ex post facto (after the fact). You do not manipulate any variables; the focus is on what has already happened (after the fact). You may generate 'several' reasons (causes) for the relationships you discover. For example: Two employment agencies conduct job dubs. In one agency, the job clubs are member- initiated and run whereas in the other, employment counselors provide instruction and guidance. In examining the job placement rates of members for the past year, you find that the members of the professionally led groups had a higher placement rate. Typical statistics used in ex post facto research are the t - test and analysis of variance, Quantitative (Experimental Designs): True Experiment - This research is characterized by the use of experimental and control groups with random assignment to each. Experimental designs are used to determine cause-and-effect relationships. For experiments, there are design variations such as: a. Treatment and control group with posttest only. b. Treatment and control group with pretest and posttest. c. Two different treatment groups with control group and posttest, and many other combinations. For example: Sixty college freshmen are enrolled in an English class. Thirty are randomly assigned to a one-hour per week writing lab, the others comprise a control group. End-of-semester essay examination results are analyzed to see if the lab was associated with better writing skills. Quantitative (Experimental Designs): Quasi-Experiment - This research is similar to experimental research except that randomization of subjects to treatment and control groups is not possible. It may be that no control or comparison group is available. Results from such research will not be as unequivocal as results from a true experimental study. For example: An elementary school has two classrooms of fourth graders. Each classroom of fourth graders is taught arithmetic by a different method for the school year. In May, arithmetic achievement is compared for the two classrooms using scores on a national exam. There was no random assignment of students to the two classrooms and the 'arithmetic teaching methods' were not randomly assigned to the two classes.
Between subjects: examining what changes occur between two or more groups. Meta-Analysis - This is research comparing findings across studies, i.e., the results of many studies are examined simultaneously and one or more research questions answered. For example, a number of studies have examined what works in various counseling situations. Examining a number of these studies simultaneously can help determine what counseling techniques work for what kind of clients, with what kinds of problems, under what conditions. Internal Validity - Experiments are internally valid to the extent that extraneous variables have been controlled. In other words, to the extent that the treatment variable is the only one producing the observed changes, the experiment is internally valid. Threats to Internal Validity: Selection of Subjects - Differences in the results between two groups may not be due to the treatment variable experienced by one group because the composition of the two groups are different to begin with (probably not randomly selected). Threats to Internal Validity: Instrumentation - Differences in results between two or more groups may be due to instruments which are unreliable or because the instruments are changed during the study. Or, perhaps the observers recording data become fatigued or bored and they record behaviors differently over time. Threats to Internal Validity: Maturation - Results may be due to maturational or other changes in the subjects and not due to the treatment being applied. This is especially important if research data is gathered over a long period of time. Threats to Internal Validity: Mortality or Attrition - Losing subjects during the study could lead to different results than if everyone had stayed. Subjects with the most or least amount of important characteristics to the study may be the ones dropping out. (Think of normal curve extremes.) Threats to Internal Validity: Experimenter Bias - The responses of the subjects may be influenced by the researcher. This may occur by treating some subjects differently, reinforcing different behaviors, as well as the presence of many other variables which deliberately or unintentionally influence subjects. Threats to Internal Validity: History - These may be the extraneous incidents (e.g., national or local tragedies or other occurrences) which influence subjects in a research project. These events might interact with the independent variable so the results which occur may be due to a combination of the event and the variable under study. Threats to Internal Validity: Statistical Regression - Sometimes subjects in a study are recruited because of extreme high or low scores (e.g., self-esteem, social skills) on the dependent variable being measured. Due to statistical regression, future measures would expect these individuals to score closer to the mean score even without any intervention.
External Validity - An experiment is externally valid to the extent that the results may be generalized to people and situations beyond the study. There are several threats to the external validity of experiments and some of these are also threats to internal validity. Threats to External Validity: Selection of Subjects - If subjects are not randomly selected, the results may only apply to the subjects in the study. Furthermore, the results can only be generalized to people with similar characteristics. Can we generalize results of research with college students to other adults in general? Threats to External Validity: Ecological Validity - The research has ecological validity if the results can be generalized from one setting or circumstance to another. Sometimes the circumstances, conditions or physical surroundings of the research are so unique that the results cannot be generalized beyond that study. Threats to External Validity: Subject Reactions (Reactivity) - If any of the following are present, the results may not generalize beyond the study: i. Hawthorne effect - refers to the influence in performance which occurs when subjects receive attention or know they are participating in research. ii. Demand characteristics -- these are all the cues, information, knowledge, even rumors the subject has heard about the experiment which are likely to influence his or her performance. iii. Experimenter bias -- these are the changes in the subject's behavior brought about by the researcher's expectations, behaviors or attitudes. It may also be called the Rosenthal effect. He conducted research into this phenomenon and called it the Pygmalion effect which refers to the self- fulfilling expectation of doing well because it is expected. iv. Placebo - broadly, the placebo refers to any control treatment. The placebo should be identical to the experimental treatment except for the critical item being studied (new drug, parenting skills training, reaction to violence in films, etc). Even so, control subjects may be influenced by the placebo and react in unintended ways. Threats to External Validity: Novelty and Disruption Effect - The measured effect of the treatment on the subjects may be due to its novelty or the disruption it causes. Being selected for a research project may be exciting and energizing; as it continues it may begin to disrupt routine and one's typical schedule. When novelty and disruption wear off or stabilize, there may be no long-term effects of the treatment. Levels of Measurement - a. Nominal: the numbers represent the variable's qualities or categories. With nominal data, you would ordinarily use a nonparametric statistic such as chi-square. b. Ordinal: the numbers represent differences in some magnitude of the variable. You can order the data, from top to bottom, from high to low. For example: Scores on a Psychology 101 final exam can be ranked from highest to lowest. c. Interval: the intervals between the numbers on a scale contain the same amount of the variable throughout the scale. These intervals provide a constant unit of measurement. For example: On a standardized test, the distance (interval) between 11 and 12 is the same as the distance between 24 and 25. Other examples are Fahrenheit or centigrade temperature scales.
Inferential: used to make inferences from the sample to the population. The goal is to determine the probability of some event occurring. Common statistical tools used for this are the i-test and analysis of variance. Parametric vs. Nonparametric Statistical Analysis - Parametric: used when a sample is randomly drawn from a population and the data is normally distributed. You have para (two-sided) data that yields a bell-shaped curve. Also, you assume that the variance of the sample you are studying is homogeneous (similar) to the variance of the population from which your sample is drawn. The scores would give you a normal curve -- more or less. Examples of parametric statistics are: i-test and analysis of variance. Nonparametric: used when you cannot make any assumption about the shape of the curve or variance of the population scores (that is, they may not be normally distributed and variances may not be homogeneous). You want to make a statement about a school's academic performance but you only collected scores from the "gifted and talented" students. You will not have a normal distribution of academic scores for the school. Examples of nonparametric statistics are: Chi-square, Mann-Whitney U Test, and Wilcoxon Signed- Ranks Test, all to be reviewed later. Independent Variable - This is the variable you manipulate or vary to see what change occurs in the dependent variable. Sometimes you group or categorize the independent variable. For example, you categorize a group of individuals by gender, male and female. Or, you categorize high school students as freshmen, sophomores, juniors, and seniors. The independent variable precedes or is antecedent to the dependent variable. The independent variable is sometimes called: stimulus variable, predictor variable or experimental variable. Dependent Variable - This is the variable you are measuring or trying to change. The value of this variable depends upon the value of the independent variable you selected. The dependent variable is sometimes called: response variable, outcome variable or criterion variable. Research Questions and Hypotheses - Some investigators ask research questions to be answered. Some investigators formulate hypotheses (often in the null form) to test. The null hypothesis states there is no difference between the variables or groups measured. Ex: There are no significant differences in final academic grades (GPA) between boys and girls finishing the tenth grade. One alternative hypothesis (directional) states that one group's score will be significantly different from another group's score (one-tailed test). Ex: The GPA of girls will be higher than the GPA of boys finishing the tenth grade. Another alternative hypothesis (nondirectional) states that there will be differences between the groups but which group has higher or lower scores is not indicated (two-tailed test). Ex: The GPA of girls and boys finishing the tenth grade will be different. Significance Level - There is always a possibility that in regard to your research results and the null hypothesis you posed, you will make a mistake in rejecting the null hypothesis when you should not, or in accepting
the null hypothesis when you should reject it. The significance level you select will determine your likelihood of making either mistake. The significance level may be set by you before data analysis at .05, .01 or some other level such as .001. The level selected is your willingness to make an error, that is, rejecting the null hypothesis, when in fact there is not a significant difference between the groups. (Conventional significance levels used in research are .05, .01. and .001.) At .05, you are willing to accept the possibility of rejecting the null hypothesis in error five times out of one hundred times (if you performed the experiment 100 times). Type I (Alpha) Error - Type I or alpha error are the names we use in referring to the rejection of the null hypothesis (which states that there is no difference) when it is correct. In short, you make an error. Remember: at a significance level of .05, you will reject the null hypothesis five times out of 100 when in fact there is not a real difference. You can change the probability of Type I error by changing your significance level. Type II (Beta) Error - Type II error refers to the failure to reject the null hypothesis when there is, in fact, a difference. As the significance level goes down (e.g., .05 to .01), Type I error decreases but Type II error increases, i.e., the failure to reject the null hypothesis when you should. As one type of error goes up, the other goes down and vice versa. Remember: Alpha (Type I) error: Reject (null hypothesis) when shouldn't. Beta (Type II) error: Retain (null hypothesis) when shouldn't t-Test - The r-test is used to determine whether the mean scores of two groups are significantly different from each other. It can only be used when there are two groups (two mean scores).You would compare your obtained value of t (from the calculation using the formula) with the value of t presented in a Table of t Values to make that determination. Computer data analysis programs now provide this information automatically. It should be noted that even though statistical significance may be reached, especially with large groups, there may be no practical significance to the difference. ANOVA (One Way) - Let's assume that in our Item 18 example, you as the agency director, notice that you have therapists trained at three levels. These levels are: a. master's degree — licensed professional counselor (LPC) b. master's degree — licensed social worker (LSW) c. Ph.D. degree - psychologist (Ph.D.) You are attempting to determine self-esteem differences in clients following counseling and are curious as to whether level of training of therapist might lead to different levels of self-esteem.
esteem. If we were measuring the impact of therapist gender on clients' self-esteem, women therapists would have started out five points higher on client self-esteem. So we statistically adjust client self- esteem scores (for example, add points to the males pretest scores and subtract points from the female pretest scores) before making the comparison. In this case, the covariate is the systematic difference in self-esteem scores on the pretest (women clients score an average of five points higher than men clients). Consequently, we conduct an analysis of covariance - adjust the scores. Then we do the analysis of variance. Post Hoc or Multiple Comparisons Test - If your analysis of variance yields a significant F value, you still will not know which particular pair of mean scores is significantly different from each other. You must apply one of the post hoc (after the fact) tests to determine whether different particular group means or combinations of group means are significantly different. The most commonly used post hoc or multiple comparison tests are: Scheffe's (most conservative) Tukey's HSD (Honestly Significant Difference) Newman-Keuls Duncan's new multiple range test Your computer data analysis program will conduct these automatically. Nonparametric Tests - When you cannot assume that your distribution of scores is normally distributed (resembles a normal curve) or that the variance of your sample is similar to the variance of the population (homogeneity) you must use nonparametric statistics. Some examples are: Mann-Whitney U test: when you collect data from two samples that are independent from each other and the scores are not normally distributed. Wilcoxen signed-rank test: when you have scores for two samples and these scores are correlated (that is, you matched them or got two scores for each individual- repeated measures). However, the scores do not approximate a normal distribution. Kruskal-Wallis test: when you have more than two mean scores on a single variable. This is a nonparametric one-way analysis of variance. Chi-Square - Chi-square is another nonparametric test and is used when you have nominal data (groups or categories). This statistic is used to determine whether two distributions differ significantly. For example, you might be curious whether more boys than girls wear jeans to high school. The null hypothesis would be: There are no significant differences in the number of boys and girls who wear jeans to high school. You would use the Observed frequency of jean-wearing with the Expected frequency. This is called a contingency table. (4 squares: top yes/no, left boys/girls). Is the wearing of jeans contingent on the gender of the student? Solomon Four Group Design - This design examines the effect of any pretest used on the experimental treatment.
Group: Pretest / Treatment / Posttest A: Yes/Yes/Yes B: Yes/No/Yes C: No/Yes/Yes D: No/No/Yes This design allows you to determine whether the pretest by itself made a difference (Group B), whether the treatment by itself made a difference (Group C), whether a combination of pretest and treatment made a difference (Group A), or whether nothing made a difference (Group D). Multiple Regression - This is the use of the correlation coefficient to determine the strength of the relationship of predictor (independent) variables on a criterion (dependent) variable. Multiple regression adds together the predictive power of several independent variables (predictors). For example: Predictor variables such as high school GPA, class rank, and ACT scores may be used to predict the criterion (outcome) variable, which could be end-of-college freshman year GPA. Scatterplot - This is a graphic representation of the relationship between two variables for a group of individuals. Each 1 on the graph is an individual who receives a score on the X test (e.g., anxiety) and a score on the Y test (e.g., hostility). The following statement might represent the data; As individuals get more anxious about the counselor exam, their hostility increases (positive relationship). Factor Analysis - This is a statistical method using the correlation coefficient to determine whether a set of variables can be reduced to a smaller number of factors. For example: A factor analysis of all the items on a long inventory with 15 scales may uncover only four or five factors independent of each other underlying the scales. Thus many of the constructs of the 15 scales overlap with each other. Likert Scale - This is a widely used technique for measuring attitudes or opinions. It allows for several response choices, e.g., a five (or seven or nine) point scale. Biserial Correlation - An appropriate correlation coefficient to use when one variable yields continuous data and the other yields data that is dichotomous. Degrees of Freedom - The number of observations that are free to vary. Double-Blind Technique - This occurs when neither the researcher nor the subject knows who is getting the active substance or the placebo. Halo Effect -
evaluation is conducted to see how well agency or program goals have been met. Usually a product (document) is generated so this is 'product' evaluation versus 'process' evaluation (formative). Ethical Issues in Research - Confidentiality: no one should have access to the data except for the researcher and research assistants if any. Release of research data to others is only ethical with the consent of the subject. Deception: deception may be justifiable if no risk to subjects is involved. Such research should be followed by debriefing of the subjects. Informed consent: subjects should be informed of the research they will be participating in and give their consent. Significance of research results should outweigh the potential benefits denied the control group. Research must be approved by a 'human subjects committee' or Institutional Review Board when conducted within institutions or agencies when federal funding is involved. Writing Research - Most counseling and psychological journals require that the writing style presented by the Publication Manual ofthe American Psychological Association be used. Sexist language is to be avoided. Never submit a professional research manuscript to more than one journal for publication consideration at a time.
