Project: Feedback

Project planning for evaluation

In the previous three topics, we looked at applying a strength-based approach, group facilitation, and working with a group of young people as they participate in planning and carrying out a project, event, or activity.

Collecting feedback and evaluating a project after its completion are an important part of the process. Project Checkpoint 4 includes making a plan to collect data for later evaluation. As soon as possible after implementation, you will conduct a debrief session with the participants to collect feedback on their experiences, observations, and opinions. You will need to organise this data and submit it as part of your assessment.

This topic includes a Debrief Plan template and activity ideas for the session if you wish to use them. Read the assessment criteria carefully – make sure your feedback and debrief session plan will enable you to collect what you need to submit for assessment. Plan a debrief session that makes sense for your project and what you want to learn from the experience. You will also use this information again in the next Module to carry out an evaluation of the project.

Task: Facilitating effective planning

Read through Assessment 4.1, 4.3 focusing on Project Checkpoints 4 and 5.

• Identify what planning for evaluation you and the participants need to do.
• Think about how you can best facilitate the group to go through the process.

Why debrief and collect feedback data?

Planning and implementing the project is a lot of hard work and learning for you and the group. At the project’s conclusion there are two main things you want to know:

1. Was it a success? Did it achieve its objectives?
2. What did we learn? And can we transfer this learning into future projects or new areas?

You can only answer these questions well if you have good data to support your conclusions. Getting good data require two main conditions:

1. Careful planning before implementation on what information you will need, and the best way to collect the data.
2. Collection and collation (organising) of data immediately after implementation, or as soon as possible. 

What is the evaluation?

There are many types of evaluation used in different fields, although they share some core attributes. Here we focus on the purpose of evaluation for this project. The emphasis is on impact evaluation to measure effectiveness. Did this project do what it was intended to do?

To evaluate effectiveness and/or impact you first need criteria (standards, principles, statements) to measure against and objective data to accurately support decisions or judgements made in relation to the criteria. Objective data is based on facts and information from different sources, not simply personal feelings, or unsupported opinions. Your observations and professional judgements can be included in the evaluation data but is balanced out by feedback from other stakeholders and relevant facts about the situation.

Measurable criteria

The criteria you use to measure the project’s impact and effectiveness include the goals or objectives set at the beginning. Did the project do what you intended it to do? Did the project meet the goals decided on by the participants?

However, the project’s impact can be broader than simply meeting its goals. Also, consider if the right goals were set. It is possible to meet goals yet still feel dissatisfied with the outcome, or to not meet goals yet still feel satisfaction with how the project turned out. It is also possible to have unexpected, yet positive outcomes. And it is possible for unforeseen things to go wrong.

So, when you evaluate the project’s effectiveness think about questions like:

• Did it meet its stated goals?
• How did different stakeholders feel about the project – satisfied or dissatisfied?
• If the project met its goals are you and the participants still satisfied with how it went? Why do you feel this way?
• If the project did not meet all its goals, is this okay? Why or why not?
• Did the project play out as intended? Do you know why it did, or why it did not? If you could go back in time and make changes to the plan before implementation, what would you do?
• Did you have the resources, people, space, money, time, equipment, materials, and training that you allowed for in the plan?

The project evaluation can include a judgement about the quality of the goals and objectives, along with the planning process used to achieve those goals. However, you are not just evaluating the success of the project (ability to meet stated goals) but also its value as a learning experience:

• What did you learn from this project? About your practice? About yourself? About the young people and how to work with them?
• What did the participants learn? About themselves? About transferable knowledge and skills? About developing confidence and working with others?

Objective data

You determine the impact or success of the project through the relevant data you collect from various sources. For an evaluation, you cannot simply rely on your own subjective observations and opinions. You must collect data on the participants’ perspectives and experiences, and possibility other stakeholders too.

We generally think of data as qualitative or quantitative (depending on the method used to collect it). Both types are valuable sources that give different types of information. For this project, you will most likely use qualitative data as it is one project with one group of participants.

Examples of quantitative data includes statistics and survey results. Quantitative methods collect standardised data in a systematic way. Everyone is asked the same questions or provides the same type of data points. The range of allowable variation is small and standardised. For example, a question in a survey with only four possible options to choose from. Standardised data makes it possible to compare data groups and generalise more widely. It is also easier to collect responses from more people faster. However, while this data can show ‘what’ a situation is it does not necessarily show why or how or convey nuance and complexity.

Examples of qualitative data include interviews and focus group discussions. Qualitative methods collect data that has a wide range of possible responses. Even if everyone is asked the same open-ended question, each person can answer in their own way. Qualitative data is usually about ‘why’ and ‘how’. It can explore effects and unexpected outcomes. But collecting qualitative data takes longer and uses a much smaller sample size. It can be difficult to generalise using qualitative data.

The two methods can be used together. For example, qualitative methods can produce stories, anecdotes, narratives, case studies, and real-world examples that help to contextualise and give voice to quantitative data results.

Later in this topic, we explore some data collection tools, such as surveys, interviews, focus group discussions (such as participant debrief sessions), documentation, and gathering other evidence like photos or videos.

Plan for evaluation

Plan to evaluate the project so that you capture the right information. An evaluation must be based on more than your own subjective opinions and perspective. You want data from the participants and possibly other stakeholders to support any evaluations you make.

• What data do you need to collect?
• Who do you need to collect it from?
• How will you collect this data?

The answers to these questions will depend on the nature of your project and the participants.

Task: Making plans for collecting data

Read through the section Plan to collect data for evaluation in Project Checkpoint 4 in Assessment 4.1, 4.3. For each bullet point, think about who could give you that information and how you might go about collecting it out from them. Remember, your own observations and notes are also a useful source of information, so long as you do not rely solely on your own perspective. The evaluation also needs to consider the perspectives of the participants and other relevant stakeholders.

Which methods of data collection are best?

There is no one correct answer to this question. How you collect data depends on what you want to know and the nature of the situation or project you are evaluating. It can also depend on who you are getting your data from.

Select the methods that will best serve your purposes. The only data collection you must do for assessment purposes is the debrief session following implementation of the project. However, if you can also collect data from other sources your evaluation will be more reliable and more useful.

Privacy and transparency

When you collect data for evaluation you must protect privacy – do not use names or identifying descriptions. However, you may use demographic descriptions; for example, 18-year-old European female school leaver, or 16-year-old male Chinese new migrant.

Be transparent with those you collect data from. They should be told if a conversation or written work is being recorded, what it will be used for, and who will have access to it. You should also explain how you will protect their privacy and that they will remain anonymous.

Survey

A survey lets you get a similar range of data from a larger group of people in a shorter amount of time. You can hand out a written survey, do it online, or face-to-face. A survey provides standardised information that you can compare.

You need to pay careful attention to how you design a survey. Keep it as short as you can and try to write simple questions that most people can clearly understand. Professional survey companies often test questions several times to make sure their questions are not ambiguous (interpreted in different ways by different people) or difficult to answer. Before you use a survey, ask one or two people to try it out and give you feedback – is the survey clear and simple for them to read and answer?

In a survey, you can ask open-ended questions, closed questions, scaled questions or multi-choice questions.

• Open questions in a survey should be kept to a minimum. Open questions invite longer answers and a wide variety of responses because it is up to the survey respondents to decide how to answer the questions. Responses to open questions are more difficult to compare and correlate from a larger group, but can give unexpected, yet valuable insights. Open questions in a survey demand more effort and time to analyse, so use with care.
• Closed questions in a survey are useful because people can answer them quickly and you can analyse a lot of responses quickly. Responses are straight forward to collate and compare. Most closed questions invite yes/no or true/false type answers, and they don’t usually ask about why or how.
• Scaled questions where respondents give a rating from 1 to 5 are useful because they can indicate attitudes of a larger group. However, they do not explore why people have those attitudes or explain the nuances.
• Multi-choice questions give a set range of options for the people answering the survey to choose from. They have similar advantages to closed questions but provide more information.

It is also possible to combine different types of questions, for example, multi-choice questions and open questions. With a multi-choice question, you can provide list of expected response options, and then an open question to gather other information that you (as the survey designer) may not have thought of. You may also wish to combine a closed question with a scaled question and additional open questions. Compare the two examples below:

1. What did you enjoy about our performance? Choose all options you agree with.
   • Dancing
   • Singing
   • Solo singing
   • Costumes
   • Music selection
   • Other
2. What did you enjoy most about our performance?
3. What is one thing that you would have made it more enjoyable for you?

The types of questions you use in a survey depend on what you want to know, how many people you want to survey, and how much time they have. Also, people are usually happier to answer questions that have a limited range of options and are quick to answer.

For this project, use surveys if you want feedback from a lot of people. For example, the group runs a food stall at a community event, and you want feedback from customers. You could ask a sample of customers to answer 4 or 5 questions about the food and service. These could be listed on a single page for them to write on, or someone in the group asks them the questions and copies down their answers. Depending on the project, it would be useful for participants to have at least some of their stakeholders answer a survey. They can use the results to help inform their debrief.

Or, you can design and include a survey to help you evaluate the project from the participants’ perspective. Before you write the survey, refer back to the stated goals for the project. The feedback you get should help inform you about the success of the project and if it was able to meet its goals. Ask the participants to complete a survey form so you can compare their responses on specific questions.

Explore further

If you want to quickly create an online survey, you can use a site like SurveyMonkey, which also has a lot of good advice and templates for writing questions.

• Here is an online article from SurveyMonkey with tips for writing questions: Writing good survey questions¹ You can use these tips for both online and off line surveys.
• If you need to see examples of how to design and word questions, you might find this article useful: Surveys 101: A simple guide to asking effective questions.² It also includes advice on how to design questions that work well for surveys.

Interviews

A good way to get more in-depth feedback is to interview participants and other stakeholders. Interviews can be short or long. They can be structured with a specific set of questions to ask, or statements to respond to. Structured interviews with a list of questions for each person allow you to do some comparison of information. Interviews can also be unstructured, with the topics to discuss left up to the person being interviewed.

The advantage of an interview is that you can:

• Ask more precise questions and ask follow-up questions to learn about one person’s experience and perspective in greater detail.
• Learn more complex information about a situation, including aspects you were unaware of.
• Avoid a literacy barrier, which can be an issue with a written or online survey.
• Learn a lot from body language and other non-verbal data during an in-person interview.

However, interviews have a few pitfalls to be aware of:

• Interviews take a lot of time, so you usually only talk with, at most, a handful of people.
• In-person communication may include personal or controversial content – such as issues associated with personal dynamics and interactions.
• If you allow participants to do interviews, do so with care. They may unintentionally ask insensitive questions or not know what to do if someone becomes rude, defensive, or upset.
• Be conscious of interviewer bias. Unless the whole conversation is recorded and transcribed, the interviewer adds a layer of interpretation to what the other person says.

You can always use a mix of survey and interviews. For example, ask all the participants to complete a survey about their experience of the project, and then do a more in-depth structured or semi-structured interview with two participants from the group.

An interview should feel like a conversation. Ask open-ended questions so the other person is encouraged to talk and explain their experiences and opinions in greater detail (for example “Tell me about …” rather than “Did you enjoy …”). Try to avoid yes/no questions. If you require a yes/no answer, make sure to include follow-up questions such as “Why or why not?”.

Even though it is a good idea to go into the interview with some structure – a list of questions or points of topics you want to cover – try to use a conversational, back-and-forth style when conducting the interview. This will build greater rapport with the person you are interviewing and help them to feel more relaxed and expressive.

Explore further

If you would like to learn more about how to conduct interviews, you may find this article on qualitative interview techniques³ useful. While it is a little long and assumes a higher standard than you need for the project, it does provide very useful advice on how to conduct an interview, how to prepare an interview guide beforehand, and how to use the guide during the interview.

Focus group discussion

This is useful way to collect data on a group project. The debrief session you conduct with the participants for the assessment is a type of focus group discussion; it enables you to explore the group’s perspective of the project. You could do a focus group with everyone or break up into smaller groups.

A focus group discussion provides complex information on the experiences, feelings, values, beliefs, and preferences of the participants. With a focus group you are more likely to hear the group’s consensus or what most people in the group see as the ‘right’ answer. Usually, if you want to go deeper into alternative perspectives within a group, you need to talk one-to-one, especially about sensitive topics. Of course, some individuals are more comfortable sharing alternative thoughts and experiences in a group setting than others. This may also depend on the level of trust you have with the group and how long you have all known each other.

Keys to a successful group discussion:

• Create a relaxed and welcoming environment and set expectations for the discussion up front.
• Have a plan to deal with any conflict and tension that may arise, especially if it is a topic people are likely to feel passionate, sensitive, or defensive about.
• Frame the conversation as a safe space – everyone has a legitimate experience and/or opinion; we are each an expert on our own experiences and emotions – I think …, I feel …, and so on.
• Have a plan to draw out quiet or shy participants or provide an alternative way for them to contribute.

Hopefully, by the time of the debrief the participants have gotten to know each other well and have built up trust and good rapport with each other and with you, so the debrief will be an open and insightful conversation, and everyone will feel able to contribute. However, it is still best to be prepared for other outcomes.

Documentation

Some data you use for evaluation of the project may come from documentation, such as your own notes or journal, participants’ written content, or the organisation’s paperwork and documents such as policy statements, health and safety procedures, reports, and other paperwork requirements.

What is the debrief?

As soon as possible, after project implementation, set aside time to debrief with participants. During this debrief session, participants reflect on and share experiences and insights. You will capture this data and use it to supplement your own perspective and help inform your evaluation of the project.

The purpose of the debrief is to:

• celebrate success and achievement
• reflect on the process and identify potential improvements
• identify what was learned from the experience
• capture the participants' perspectives on the project.

The debrief session should feel like a conversation broken up into sections and activities. Its purpose is for participants to explore how they did for themselves and determine what participation in the project achieved and what they learned from the experience. You can share your positive appreciation for their efforts and identify any specific things they did or learned that you were impressed by but do so at the end of the session. During the session, try to avoid telling participants how they did.

Debrief session structure

• Plan the session to keep it interesting and ensure you cover the topics you want explored.
• Prepare to capture the discussion. This could include taking photos of mind maps on the white board, or the paper sheets participants write down their key ideas on, for example. Or you may decide to take notes under different headings.

If a survey or interview(s) of stakeholders were undertaken during or immediately following implementation bring this information to the debrief session so participants can include it as part of their discussions.

Sample structure for debrief session

This is one possible structure for a debrief session. Please modify to suit your purposes or use your own approach:

1. Opening, such as a karakia, mihimihi, pepeha (or similar, as appropriate for the group).
2. Quick warm up to get everyone laughing and talking, and in the mood to share their ideas.
3. Brief explanation of the session: why you are running it, what they can contribute. Also, thank participants for their hard work and that you appreciate their time and effort.
4. What happened? Activity based around recall of the whole project process from the initial brainstorm to end of implementation.
5. So, how did it go? Activity based around the success of the project, referring to the goals of the project and what participants learned from this experience.
6. Now what? Activity about how to use what they learned going forward.
7. Closing and transition to celebration activity.

After the debrief

Have a ritual and/or reward to mark the project’s completion. For example, a shared lunch, games afternoon or go out for a treat. Use this as an opportunity for the young people to show their appreciation for each other. For example, randomly pair them up and ask them to share with the group what they appreciate about the other person’s contribution or what they learned from them; or participants write anonymous letters that you read out – each person gets a positive mention!

Another way to mark the project’s completion in a positive way, and if it is appropriate, hand out certificates. Ask a manager (or the person who approved the original proposal) to come and say a few words to the group and hand out the certificates. This provides external validation for their efforts (and yours too!). Do this in a way that is culturally appropriate for the group.

Conversations and activities

General tips:

• Use open questions to encourage deeper conversations and more opportunity for participants to contribute and explore their experiences, opinions, insights, and perspectives more fully.
• Have a range of different activities to keep session fun, engaging and the brain active. Micro breaks, changes in activity type, physically moving around all help to maintain participation and engagement.
• Arrange people and the space with care. Discussion in circles encourages more open conversations. Keep subgroups separate. Remove distractions. Make sure space is comfortable.
• Have specific tasks with defined outcomes to help maintain focus. With an activity have a set list of things to achieve through the discussion. Facilitate full group conversations to keep them on track or ask colleagues to help with subgroups. Capture the participants’ key points as part of the activity – worksheet to complete, poster to write on, notes on a whiteboard you can photograph.
• Use accountability to keep the session on track – subgroups report back to the main group or to another subgroup, or complete a section of a poster.
• Keep your own responses as neutral as possible. You want to capture the actual perspectives and ideas of the participants, not the ones they think you want to hear. Avoid showing approval for some contributions but not others. Try to respond to all contributions in the same way. Ask follow-up questions for more detail or clarification rather than general expressions that can signal your own opinions with subjective responses like ‘good answer’, ‘sweet’, ‘I like it!’ If you compliment, make it is about the process not the content of what they contribute as this has the potential to skew the conversation and thus the data you collect.

Remember, not everyone contributes in the same way. Some hold back or are too shy to speak up. To include their perspectives, talk with them one-on-one later, or use activities that allow participants to write down their ideas or share them with one other person rather than in front of the whole group. If one or two people are too dominant give them other jobs to distract them; for example, they hand out materials or monitor a timer. You can also include activities in which participants give non-verbal responses – such as drawing or movement to show their opinion.

The most important thing is to do what works best with your group.

Activity ideas

These are suggestions only. Use what makes sense to you.

What happened?

Break the project’s timeline up into logical sections; write each section’s name at the top of a large sheet of paper and pin these around the room in order. Give each participant a marker, so they can walk around and write or draw pictures of key things that happened at each stage. Each person contributes what they remember. They can talk about it and remind each other or tell stories. Everyone may have different memories of the events and together the group can fill in the gaps. Once it looks like most of the details are in place, as a group retell the story of the project.

You can add to this by handing out sheets of emoji stickers that the participants place at different points on the sheets to reflect how they were feeling at that stage of the project.

You can also underline in a different colour or use a highlight stick to indicate anything unexpected that happened – anything that was not part of the original implementation plan. The group can have a brief discussion about why something unexpected happened and how they dealt with it.

Remember to take photos of the final sheets as evidence to submit as part of your assessment and to capture data for your evaluation.

So, how did it go?

Now everyone has a clear story of what happened, start the debrief to determine the success of the project and what they learned as a result.

Reintroduce the original goals participants set for the project at the beginning of the planning process. These should be clearly stated in the planning documents. Divide the group up into subgroups and give each group one or more cards with a goal written on it. For each card (goal), they create a mind map or similar divided into three sections: What went well for achieving this goal? What needed improvement to better meet this goal? What were the challenges for this goal? As a subgroup they discuss and write key words and phrases on the mind map. Then, following the discussion the subgroup decides if the goal was met or not.

Each subgroup then presents their conclusion on the goal they analysed and explains their reasoning. If there is time, the wider group can also add their comments to the mind map. There are no right or wrong responses, rather the aim is to interpret the available evidence to come to a consensus or agree to disagree. Manage the group to disagree respectfully, ensure everyone has a chance to present their perspective, and know that it is okay to change your mind. Avoid giving your own opinion (at least until the end) even if asked.

If you have feedback from other stakeholders like survey results or comments from interviews, share these with the group. Does this change their thinking in any way?

The group comes to a consensus, if possible, on which goals were met and which fell short. Pin each goal card to the board in three groups – goals met, goals undecided and goals not met. Discuss and explore why. Move the cards around the board as the group discussion changes.

Then, ask: Where these the right goals for this project? How could we have improved them?

Put participants back into subgroups (or mix them up). Tell them to get into a time machine and go back to talk to their past selves at the very start of the project. But they can only give three pieces of advice about what to do the same and/or what to do differently. They write these on a sheet of paper to share and compare with the full group.

Remember to take photos, notes and/or examples of participants’ written work.

So, what did we learn?

This could be done as individual participants or working in pairs. Have worksheets prepared with sentence starters on them with space to write down their ideas. You want to find out some of what participants learned because of their involvement with the project. What you ask about depends on your programme and the overall work you are doing with the young people. You might prefer to do this as a survey and follow up with two or three longer individual or subgroup interviews.

Examples of sentences for participants to complete:

• One new thing I learned how to do on this project is ….
• One thing I am better at now after this project is ….
• I used to think (or believe) …… but after this project I changed my mind. Now, I think ……
• From 1 to 5, how much better are my [xxx] skills after doing this project. Give an example of your improvement.
• Something good I learned about myself while doing this project is ….
• Something good I learned about [others] while doing this project is …
• Someone I am thankful for on this project was … because …
• Someone I learned from on this project was... From them I learned, …
• The number one thing I got out of being on this project is …

Capture the range of responses from the participants. Some of them may surprise you. One way to do this is to select sentence stems and put them on individual cards. Participants select one or two cards each and have a few minutes to think. They then share with a partner or in a small group, so everyone gets a chance to talk or practice what they want to say. Then, invite participants to share with the wider group. Take notes on the common themes as they come up; write these as key words or phrases on the whiteboard for the following activity.

Now, what?

Each participant selects one thing they learned or valued most about participation with the project. They write a short description, draw a picture or comic, or do a short skit or role play showing how they could use this skill or change in themselves to improve another area of their life. They can share this with a partner or the whole group.

If you want to end on a positive note, and there is time, randomly assign each participant with the name of another participant. They write a short, anonymous letter thanking this person for what they brought to the project and the work they did.

Task: Using engaging activities for data collection

Please look though Section 2 – Evaluation tools, in the document A mini-evaluation pact for youth groups and organisations.⁴ Here you will find several excellent activity ideas that are a fun and interesting way to collect debrief data from participants that you can use to inform a later evaluation. Use or modify these activities as alternatives to the activity suggestions above. You may even find some of these suggestions like the post card are an interesting way to get feedback from other stakeholders.

Task: Resources for conducting feedback and debrief sessions

Please read through Evaluating Participation Work – The Toolkit.⁵ This resource has a lot of useful templates and activity ideas to collect feedback from participants and others. While some are intended for younger children most would be useful for working with young people or could be modified. The document also includes related useful templates like consent forms and questionnaires that are easier for those who struggle with reading or writing. You will also find useful activity instructions and worksheets that can be adapted for the debrief session. Select any that you think would be useful and that your group might enjoy. These worksheets will enable you to collect information from the participants to use for evaluation purposes.

Task: Design a debrief session

Look at the Debrief plan template below (you can download it from the assessment section at the end of this topic) and, if it is useful for your purposes, use or modify it to design a debrief session to facilitate as soon as possible after implementation. If possible, set aside a full morning or afternoon, but adapt this depending on your participants and the project. It is valuable for participants to take some time to think back over the project to reflect on their success and what they have gained through participation in the project.

Also, read over Checkpoint 4 and Checkpoint 5 for the project. As evidence to support evaluation you may use your own notes and observations, feedback from stakeholders including participants – survey, interviews, and the debrief with participants, which can include the debrief session itself and any follow up interviews or conversations.

Read through the following sections of Assessment 4.1, 4.3

• Project Checkpoints 4 and 5
• Task 2 – Holistic safety
• Task 3 – Project reflection
• Task 4 – Submission of project evidence

You should now know what you need to obtain data on the project. This includes how to obtain feedback data from participants and other stakeholders. It also includes a suggestion on how to facilitate a debrief session with the participants. The Debrief plan template is optional. You may modify it to fit with how you do things. Also, if your organisation has an established way of debriefing or collecting follow up data you may use this instead.

You will use data from your own notes and observations, the debrief session and other feedback to complete evaluation of the project in the next Module.

Debrief Plan template

You should now be ready to plan to collect feedback and prepare for the debrief session following the project’s implementation.