Survey Says!

Assignment 5: Survey Design

Purpose of Survey:
This survey targets facilitators and coordinators who work in a literacy organization. The purpose of the survey is to gather information and opinions regarding current program processes (how the programs are planned and supported) and program outcomes measurements (methods and efficiency of methods used).
The results could help guide future budgets, regarding resources and professional development (program evaluation techniques). Another potential benefit to this survey is the opportunity for staff to reflect on their own practice; reflection is a critical activity that is often set low on a long list of other priorities. This reflective exercise would also help set the stage for the upcoming evaluation plan that will be initiated over the next few months.
I shared Draft 1 of the survey with 5 staff: 2 coordinators and 1 staff member from a workplace literacy program, 1 coordinator from a family literacy program, and the executive director of the organization. I received feedback from 4 of 5. Based on their feedback and comments, I revised the survey, named Draft 2.

Comments on Survey Design:
Since the organization is small (10), I purposely left out demographics that I believe do not impact the nature of the data sought, and may very likely identify each ‘anonymous’ staff member. I tried to collect only pertinent information. (In my experience, often data is collected from program participants that is no longer relevant or perhaps never was. I have seen templates for surveys that were relevant for one program’s requirements that were duplicated in other programs, whose requirements did not match the data collected). I could have asked how long the staff had been with the organization, or how long they had been facilitating programs in other organizations, but I didn’t deem it relevant to my purpose. My purpose is to have staff share their current perspectives regarding program development, support and evaluation, which will hopefully involve reflection on their own expertise levels. Although not the main focus, I believe professional development should be based on current need and not years of experience. (I may be mistaken; I’m open to any challenges to this perspective)

Rationale for changes in Surveys:
1. Question 2 was added that involved a reworking of vocabulary: (2) How do you know you are making a difference? (What would convince others that your program is successful?) I believe this language helps to remove the tendency to look at the word ‘evaluation’ with fear and trembling and makes it clear and offers another way of looking at the idea of evaluation.
2. Question 3: ‘statistics’ was added, since I omitted this. In my view this is an output, and I’m not as interested in it as longer term outcome measurements. However, it is an evaluation technique and belongs here.
3. Question 4 was added to accommodate the request for more depth surrounding staff confidence levels: a) Please rate your confidence level regarding your expertise in evaluation (i.e.: terms, methods) b) Please explain the reasons for your rating. I think this gage of facilitator perception is an important factor when planning a participatory program evaluation.
4. Question 6: It was suggested to remove (if any). I included it in to allow for the option of not answering if not relevant. However, I removed it to reduce wordiness.
5. Question 9: ‘Websites’ was added as resource source. Oops- a major omission in draft 1.
6. Question 10 Draft 1 question 7 was changed to: a) Have you have felt adequately supported/helped by supervising staff? b) Have you felt adequately supported/helped by your colleagues? c) Please share specific examples of when you felt supported and/or how you responded in situations where you experienced a lack of support. This change was based on 2 differing responses:
a. The executive director had strong feelings around the term ‘superior’: “I choke when I see ‘superior’- is there a better way of putting that?” I changed the term to “supervising staff”. in question 10.
b. Based on feedback from another staff member, I removed the rating scale regarding employee support, since the sample size is very small (only 10 staff, including 3 supervisory staff) and honesty may not be accurately measured. Instead, I added a request for examples of both support and lack of support, along with strategies used to address them. This gives the question a more positive spin and elicits specific instances that could be used in future to support and guide.
7. One staff suggested using this survey as a planning tool as well as a data collection device; “Perhaps respondents could be asked to set a goal for their own/program (evaluation) improvement for the next six months/year.” I didn’t add it to the survey due to the extra time required by participants, but think it’s a great idea to present in another format, such as during the evaluation process. I struggled with whether it would be too much to ask in a survey...thoughts?

Staff Evaluation Survey Draft 1
Thanks for your feedback! This information will help guide future program/project planning. Please contact me at phone #:_________ or by email: ________with any additional suggestions or comments.
1. What do you see as the strengths of your program/project?

2. Regarding resource materials (such as: program-specific manuals &curricula, related reference books & magazines, & research documents), please rate your resources available to plan/develop your program on the scale below:

0 Low 1 2 3 4 5 6 7 8 9 High 10
3. How would you improve your resource materials?
____include more resource books (i.e.: research, theory, collections of activities/ideas)
____adapt/expand current curriculum/program topics/themes
____adapt/expand current curriculum/program handouts
____no improvements are needed
____other:___________________________________________________

4. Which methods of evaluation do you currently use to measure outcomes?
___post-program participant surveys ___interviews ___anecdotal records
___photos ___focus groups
___other:_______________________________________________________

5. Do you feel satisfied with the evaluation methods you currently use?

Yes, I’m satisfied___ Somewhat satisfied___ Not satisfied___

6. What (if any) methods of evaluation do you suggest to improve measurement of your program/project goals?
7. a) On the scale below, rate the level of support &/or guidance you have received from your superior(s):

0 Low 1 2 3 4 5 6 7 8 9 High 10
b) On the scale below, rate the level of support &/or guidance you have received from your colleagues:

0 Low 1 2 3 4 5 6 7 8 9 High 10
c) Please explain the reason(s) for your choices.
8. Are there any further suggestions that you can share to improve your program?

Thank you for taking the time to share your feedback!

Staff Evaluation Survey Draft 2
Thanks for your feedback! This information will help guide future program/project planning. Please contact me at phone #:_________ or by email: ________with any additional suggestions or comments.
1. What do you see as the strengths of your program/project?
2. How do you know you are making a difference? (What would convince others that your program is successful?)

3. Which methods of evaluation do you currently use to measure outcomes?
___post-program participant surveys ___interviews ___anecdotal records
___photos ___focus groups ___statistics
___other:_______________________________________________________

4. a)Please rate your confidence level regarding your expertise in evaluation (ie: terms, methods):

0 Low 1 2 3 4 5 6 7 8 9 High 10
b)Please explain the reasons for your rating:
5. Do you feel satisfied with the evaluation methods you currently use?
___Yes, I’m satisfied ___Somewhat satisfied ___Not satisfied
6. What methods of evaluation do you suggest to improve measurement of your program/project goals?
7. What percentage of time do you give towards evaluation for your project?
___under 5% ___5 – 10% ___11 – 25% ___26 – 40% ___above 41%
8. Regarding resource materials (such as: program-specific manuals &curricula, related reference material, & research/theory information), please rate the resources available to plan/develop your program on the scale below:

0 Low 1 2 3 4 5 6 7 8 9 High 10
9. How would you improve your resource materials?
____include more resource books (i.e.: research, theory, collections of activities/ideas)
____adapt/expand current curriculum/program topics/themes
____adapt/expand current curriculum/program handouts
____no improvements are needed
____websites
____other:___________________________________________________

10. a) Have you have felt adequately supported/helped by supervising staff?
___ regularly ___ rarely ___ occasionally

b) Have you felt adequately supported/helped by your colleagues?
___ regularly ___ rarely ___ occasionally

c) Please share specific examples of when you felt supported and/or how you responded in situations where you experienced a lack of support:

11. Are there any further suggestions that you can share to improve your program?
Thank you for taking the time to share your feedback!