In searching for completed program evaluations online, I found one that I thought would be particularly interesting. It is an evaluation of the Fulbright Scholar Program. I’d been aware of the program for some time, knowing the prestige with which the program and its participants are regarded. The program offers grants for students, scholars, teachers, professionals, scientists, and artists to study and work abroad. Many participants have had very successful careers. The ones I was aware of when finding the report are Julie Taymor (movie and stage director), and Renee Fleming (opera singer). Upon researching the program further, I found many other familiar names, including many Nobel and Pulitzer Prize winners.
The program evaluation can be found at: http://exchanges.state.gov/media/pdfs/ope/completed/execsummaries/usscholar_fulbright.pdf
The evaluation was completed by SRI International, a non-profit research institute located in California. They sent a survey to a stratified random sample of 1004 U.S. Fulbright Scholar alumni who participated in the program between 1976 and 1999. 80% of those contacted completed the survey.
The introduction section of the assessment report clearly states the goals of the Fulbright Program, and gives an overview of it. As I read the goals, I considered that their broadness might make measuring success in them quite difficult. For example, how easy is it to measure how well a program “increase[s] mutual understanding between the people of the United States and the people of other countries”? As well, the goals of the assessment were to ascertain the broader impacts of the program and assess and document the contribution of the experience of the program on its alumni. Upon reading the full report, I am convinced that the survey was very well designed and has successfully met its mandate.
The evaluation does not specifically state what questions were asked on the survey, but the responses are grouped in sections that clearly relate back to the goals listed at the beginning. In many cases, qualitative and quantitative measures are employed. The responses lead me to believe that some questions were multiple choice questions, and others were open questions where the participants would write their answers in sentences. Some sections contain subsections to show different ways the goals were met. For each goal, a variety of bullet-points relating directly to that goal are listed. In every case, the points stated appear to me to be quite comprehensive. In the case of each goal, I am convinced that the data gathered are accurate, and that the data listed in each section is relevant to seeing that that goal is met.
I have trouble finding too many weaknesses with this evaluation. I wonder if part of the impressiveness of this report stems from the fact that those who would have won scholarships are likely to be very bright, well-spoken, well-written individuals who can articulate their thoughts with ease. Although that could contribute, what impresses me the most is that the report is organized to address each of the goals of the evaluation and then shows a variety of different evidence that relates to each goal, either showing that it was met (which happened in most cases) or the degrees to which it was not. I would consider this to be a very good example of a program evaluation.
Hi Anne,
ReplyDeleteWhen I went to the link to your PE I found an excellent summary of the evaluation. I found it very impressive that 80% contacted completed the survey.
Would this be an example of Scriven's Summative outcome model?
I noticed that Laura mentioned the Impact model, I am not certain if it fits under one of the models in our guru ppt. ?
Glenys
Sounds like a well thought out program, too. I think it is always easier to evaluate a program that is well designed because the evaluation follows from the good design. It is almost built in.
ReplyDeleteI am wondering about the use of qualitative and quantitative data. I read somewhere that you generally can't mix methods because they are built on different paradigms. Are open-ended questions qualitative? I need to revies!!!
Good job, Anne.
Hi Laura!
ReplyDeleteMy interpretation is that open-ended questions that I believe were asked in this instance are qualitative. My impression is that when the author discusses the emerging themes, that it is based on open-ended questions.
As well, It seems to me that other questions were ones where the participants would check off boxes that apply. This data is presented in the article as, for example, "90 percent participated in social activities with students, faculty, or other professional
colleagues."
It's difficult to interpret exactly what the survey entailed without looking at it. A lot of my comments were based on my impressions or interpretations.
Glenys -
ReplyDeleteI was also impressed at the high rate of response, but perhaps it is because so many of the participants have done research themselves and realize the importance of gathering data.
I have great difficulty categorizing the study under one model as the models seem to overlap so much. I think that Scriven could apply.
Anne
ReplyDeleteA solid choice to base this assignment upon. I agree that great things are often done by great people and this evaluation in a fine example of that sentiment. I too am amazed at the 80% response rate. Agin that says something about the participants as well as the survey. This high rate would also give great validity to the results of the evaluation. What makes you chose Scriven as your model?
Jay
I had a very difficult time choosing a model, because they seem quite similar to me, and if you allow a little liberty in applying them, it seems that many of them could suit the review. I considered each of them individually. I chose Scriven, because the report seemed to focus on the goals and roles of the program and its participants. These were a recurring theme in the report. The Levine model didn't really fit because the report didn't take up positive and negative opposing viewpoints. The Rippey doesn't suit because it doesn't involve all of those impacted. Naturalistic applies somewhat because of the social interactions represented in the program. The Stake model had some aspects that would fit - such as the congruence between what is intended and observed, and the focus on the relationship between the variables. The CIPP model could be applied. The Provus didn't seem to fit as well because of the comparison of each step to what is should be. I felt that the focus on goals and roles was so prominent that the Scriven model was my best choice.
ReplyDelete