Participatory Monitoring and Evaluation

What is M&E?

When we invest human and financial resources in a violence and crime prevention project, we want to know that it was or is worth doing – or, if things do not result in some kind of desired progress, we want to know what we have to change, and how, so that the desired progress is more likely to be achieved. And we might be looking for best practices, which can be replicated in other places with a similar context.

If we receive external funds, those who provide financial support demand accountability. An institution with a mandate for contributing to community safety, for instance a municipality, needs to show what it has done to fulfil its mandate. Systems which aim at delivering such information are called monitoring and evaluation (M&E) systems.

M&E is part of the implementation of the project (except the ex-post evaluation) and needs to be considered in the planning of activities (activity plan) as well as in the planning of resources (budget).

• Monitoring is a systematic process carried out throughout the project. Depending on the type of data and the project, information might be gathered, eg., once every three or six months, or even monthly. Monitoring data can compare to flashlights on certain aspects of the project, at certain times or intervals.
• Evaluations are realised during, at the end or at a certain point after the end of a project. They are called midterm, final and ex-post evaluations. Depending on the evaluators, there are internal and external evaluations. Evaluations analyse the relevance, efficiency, effectiveness, impact and sustainability of a project.

M&E and behaviour change

We propose to work with impact chains and to describe progress in terms of the behaviour change of important actors in and outside of a community or region.

M&E deals with the relationship between an effort or activity and its results. When these results are directly related to what we have done or implemented in our project, then these are called outcomes (direct results).

If we follow the approach of “behaviour change”, then our outcomes are expressed as behaviour changes. When it is expected that they show only in the long term, because some processes take longer time and might result as consequences of the outcomes, we talk about impacts (indirect results).

In this regard, our M&E process will analyse which activity (or intervention) or group of activities lead to an observed behaviour change of important actors. And we analyse whether our expected impact chain works as we expected it to work.

This is easy when we work with simple activity-result relationships. We can positively attribute one outcome to certain activities, which have been implemented in order to achieve this expected outcome.

Defining Indicators

Indicators are essential elements of M&E. They describe changes envisioned for the future. Depending on the prioritised M&E approach, indicators are either:

• SMART – Specific, Measurable, Achievable, Relevant and Time-bound; or
• SPICED – Subjective, Participatory, Interpreted and Communicable, cross-checked and compared, Empowering, and Disaggregated.

There are three basic types of indicators:

1. Quantitative indicators are expressed through numbers, like: “30% reduction of number of hotspots in a certain area and time” or “50% reduction of alcohol selling shebeens in a certain area and time”. They describe something that is easily observed or measured.
2. Qualitative indicators describe something that is not easily observed or measured. For instance empowerment, social cohesion, changes in the value system of people or a feeling of (un-)safety. Quality  might also be expressed using numbers. For example, in bigger events with many people or in interviews, it is possible to ask people to rank between 0 and 10, 0 being the negative extreme for feeling safe and 10 being for feeling very safe.
3. Proxy indicators are helpful where direct measurement is not possible. They describe something that is more easily observed, eg. “people in the street after dark” when we want to describe the “feeling of safety”.

The approach, described in this toolkit, proposes two different kinds of indicators, which can be either quantitative, qualitative or proxy indicators:

1. Progress indicators (sometimes called milestones, in “outcome mapping”, or progress markers): they describe 3 steps of progress: what we “expect to see” as direct result from an activity, what we would “like to see” and what we would “love to see”.
   
   Indicators for the example above could be:
   
   “Expect to see” - For example: In each of five schools, two teachers (one male, one female) provide sports classes for adolescent boys and girls, combining it with exercises to manage conflicts in a non-violent way.
   “Like to see” - For example: In each of five schools, two teachers achieve a constant participation of XX girls and XX boys because their sports classes are attractive and interactive.
   “Love to see” - For example: In each of five schools, two teachers have found that their afternoon sports classes are where participating girls and boys bring in their interests, and find time and space for confidential exchange.
    
2. Impact indicators describe the impact of the prevention measures on the quality of life of the people in the project area.

Another important element of an M&E system is the baseline data. For each indicator a baseline value is needed, which describes the situation regarding this specific indicator at the beginning of a project. Against the baseline, changes are measured. Baseline data thus describes the situation which will be changed.

M&E of violence prevention

M&E in the field of violence and crime prevention is challenging because the issue is complex. We discuss this:

1. Social violence and crime prevention initiatives show some of its results only after a long time:
   A Greek proverb says:

   “A society grows great when old men (and women) plant trees whose shade they know they shall never sit in”.

   Violence within the family, lack of parental supervision, violence as part of daily life and rampant sexism in society are some examples of crucial risk factors in violent behaviour. It is important to address such risk factors if we want to achieve sustainable prevention results. Impacts of successful projects in these areas often show only after longer periods of time. Even the five year cycle of the IDP is too short to achieve fundamental societal changes. Still, it is possible to see first positive effects. We have to accept that phenomena in our society which are deep-rooted, interlinked, and developed over longer time, need a longer time to be changed.
   
   However, there are things that can be done immediately and that show results in the short or medium term. Therefore a good mixture of short, medium term and long term effects is quite important, because everybody needs to see that the efforts made, and work done, are not in vain.
2. The description of a behaviour change is not always easy: We cannot measure the subjective feeling of safety of people in the same way as we measure temperature, speed or the number of reported assaults. And we cannot exclude the very personal perception of the people who express their feeling of safety or unsafety. We can more easily describe a behaviour change in people’s actions in relation to their feeling of safety.
   
   
   Example

   “Frequently, old people are sitting together outside their houses to chat or relax”. Or: “After dawn, there are still many people, men and women alike, on the street walking or chatting. Children are playing outside”

3. Statistics do not provide objective information: This also applies to crime statistics, not only in South Africa, but worldwide. You might want to use official statistics on the situation of violence and crime in your area to complement your baseline data. It is advisable, though, to use these with caution. Numbers or percentages are often used to give the impression that they describe “the objective truth”. Yet they depend on a lot of factors, not least on the standpoint of those who collect and analyse the information.
   
   
   Example

   Rape is one of the most under-reported crimes in South Africa. This means statistics show far less cases than have actually occurred. According to a study by the Medical Research Council (MRC) carried out in Gauteng in 2010, only one in 25 rapes were reported to the police.

   There are many reasons: On the one hand police officers tend to record less cases of domestic violence, abuse and rape as this would negatively affect their overall performance rating. On the other hand, raped women and abused children

   • fear reporting, because many feel ashamed, guilty or humiliated,
   • fear that others might not believe or would accuse them of lying, feel pity or love towards the person abusing,
   • are threatened by the perpetrator, do not trust the police officers of the local police station,
   • fear that they might have to relive the trauma in court and during investigation,
   • are afraid to upset the stability of their family
   • or are afraid of being stigmatised and labelled as “damaged”.
4. The results of M&E can vary depending on the methods used: The results of M&E depend, for instance, on the depth of collection of information, the way it is collected, the selection and number of people interviewed and documents consulted. If M&E is done with women and men separately, the results are often different, because perceptions and priorities differ. The same happens when working with young and older people separately. This does not mean that one group produces the correct and the other one the wrong results. It rather means that we have to deal with various “realities”. The differences can occur even when defining indicators in a participatory way. In this case, a thorough discussion is necessary in order to arrive at results which are supported by all groups.
   
   Sometimes problems or errors can also occur in the documentation or processing of information.
5. The results of M&E can vary depending on the person assessing: For example, someone who is simultaneously responsible for the realisation of a crime prevention project might be biased and wants to show good results. Someone who does not speak the language of the people in the community where violence and crime is prevalent, and who might not fully understand the culture, will most probably make a different assessment to someone who is part of this culture.

Participatory Monitoring and Evaluation

There are various purposes and advantages of a participatory M&E approach, many of which are already mentioned here: “Participation in South Africa”.

In a participatory M&E process, community members become active planners, decision-makers and actors. Instead of 'recipients of services' they become 'agents of change'.

If people whose situation will be changed participate actively in the planning and assessment of a project - and if they participate in decision making - then they will have a high sense of ownership and motivation towards the project. This is because their specific interests and needs are taken into account.

Local people are experts on their local context. In a participatory process, community members are not reduced to "recipients of services". They are planners, decision-makers and actors. They actively participate in a learning and decision-making process in which they themselves are "agents of change".

Understood this way, participatory M&E is a process of capacity development and transformation. It provides high transparency for the community on its process and progress. It contributes to the “empowerment” of the local community:

• to actively start a process of change towards community safety,
• to systematically review this process,
• learn from own success and failure,
• assess the progress and adapt the planning, if necessary.

There is debate about participatory M&E. Some researchers strongly believe in the scientific approach, which they see as objective and believe is the only way to prove results. Others, however, state that no tools generate absolute, valid truth. 

They try to combine various perspectives in order to describe “reality” as best as possible. This includes different subjective perceptions of the people affected by the prevention measures. Participatory M&E and rigorous impact M&E do not exclude each other. It depends on M&E objectives whether to use one or other approach. Elements of both can be combined too.

The objectives of such a participatory M&E process are:

• A Process Approach and Community Empowerment: to carry out the M&E process as a process of common learning, with the participation of the all relevant stakeholders in the community, specifically ensuring the participation of young people, as they play an important role in violence and crime prevention;
• Community assessment of progress and impact of violence prevention measures, verifying M&E results among all relevant stakeholders;
• Transparency: to work with results of M&E in a way that they are accessible to all community members; that means results can be checked at any time by anyone, and they are presented in a comprehensive way.

Example: participatory M&E process

Based on the preceding information, this section describes an option for a participatory M&E process, along with some participatory tools to support such a process.

The Impact Chain

The impact chains, which were “constructed” in the planning process (then called actors’ chains), provide an important basis for the M&E process later on. They describe what we expect to happen after implementing the planned activities properly.

Two Levels of M&E

For each M&E level, we can use different tools and templates. The levels are complementary.

It is good to have a small M&E team throughout the process, with one person responsible for the co-ordination, documentation and filing of the results, e.g., a safety co-ordinator and selected people from the community.

1. Monitoring on activity and output level
   
   A well-developed planning logic is crucial, if we have to manage more complex violence prevention measures or projects. We need to define clearly why we plan certain activities, and which outputs we expect from implementation of the activities.
   
   On this level, we look for answers to the following questions:
   a. Have all activities been carried out as planned?
   b. How much did they cost compared to the budgeted amounts?
   c. Have the activities carried out produced the expected outputs (e.g., people trained, with skills, knowledge or awareness)? Have facilities, services or infrastructure been provided? If there have been deviations, what were the reasons?
   
   The main objective of monitoring on this level is to ensure the timely and proper implementation of activities leading to the achievement of the expected outputs. A proper use of resources and transparency is another objective.
2. Monitoring of Outcome and Impact of the implemented activities
   
   It is easier to monitor on the activity and output level. Many organisations focus only on this. But that leaves important questions about outcomes and impact unanswered:
   
   Outcome:
   1. Do people make use of the outputs provided (knowledge, skills, facilities, services, etc.)?
   2. And if they make use of the things provided, what real difference/s does this make to them? Are these positive and/or negative differences?
   
   Impact:
   1. Do the people (in the different groups in our community) feel or perceive changes in the quality of life?
   2. How do the different groups describe these changes? And specifically: how do young people – female and male – describe changes of their quality of life?
   3. What has contributed to the described changes?
   
   At this point, organisations should also be open to looking at negative effects, so that necessary adaptations in the planning can be considered. The “do-no-harm-approach” supports looking specifically at non-intended outcomes and negative impacts.

Using participatory M&E tools

Depending on the size of the community, we have to consider how to include the different stakeholders in the community and their specific viewpoints. We might work with representatives for the different stakeholders in meetings or workshops, so that facilitation is possible and with fewer participants.

There are various possibilities: we can organise workshops in different locations with smaller groups, that is, we repeat the same workshop with different groups, thus ensuring that many people can participate. Or the community can select representatives to participate in workshops. Special attention should be paid to the participation of marginalised groups and to ensure that the voices of young people, women and children are heard.

It is also important to ensure that all the different stakeholders in a community are considered separately, that is, we need the points of view, ideas or perceptions of women, men, young people, older people, and so on. In this process, it is important to foreground the perspective of marginalised groups in particular.

M&E is about collecting information, analysing, comparing and discussing the information, assessing progress and revising the planning, when necessary. We have different ways to do that:

Direct observation along with interviews can be carried out by community members themselves. Still, we need somebody with M&E experience:

• to provide guidance for the direct observation, e.g. what is it that we want to observe, what information do we need?,
• to develop templates to record the information or photos;
• to formulate the guiding questions of the interviews, and provide guidance on how to conduct the interviews.

We need somebody with facilitation skills and M&E experience (or a team with a good facilitator and an M&E specialist) for the facilitation of the workshops. For more information, read "What makes a good facilitator?".

Monitoring on activity and output level

Activity plans and budgets are basic instruments for this level of monitoring. Below you will find some examples (templates at the end of this chapter). Monitoring allows groups or participants to examine planned and realised activities; it also shows where additional activities need to be considered.

• Activities Monitoring – Example & Template
• Financial Monitoring – Example & Template
• Meeting Calendar – Example & Template
• Output Monitoring – Examples & Template

Tools

Below you will find some tools which are helpful when you start to work with people on the topics of crime and violence, as well as prevention. They help to get participants to tune in to the topics and the way of working.

Tool 1 - Behaviour Changes – New Ways of Doing Things, New Ways of Thinking ( 120 min.)

Objective: To generate information from self-assessment of the behaviour change of the selected boundary partners as well as participants’ ideas on necessary next steps or adaptation of planning. Download

Tool 2 - The M&E Web ( 60 min.)

Objective: To support the assessment of the progress of a group or organisation which has, for instance, participated in training workshops or events for the exchange of experiences, and/or which has received some other kind of support. Download

Tool 3 - SWOT Analysis of our Prevention Initiative ( 120 min.)

Objective: To enable stakeholders to reflect on the ongoing process. To assist participants to generate useful information from this community self-assessment of the process so far. To generate information that serves to define the next steps and possibly adaptations in planning. Download

Tool 4 - A Look at our Quality of Life ( 120 min.)

Objective: To enable he different stakeholders in the community to:

• identify changes, discuss trends and progress with regard to the formulated impact of the project.
• be aware of different existing perspectives.
• identify possible next steps to further strengthen the positive changes and counteract negative ones.
• elicit diverse views on the indicators.

Download

Tool 5 - Rich Picture – Mind Map ( 120 min.)

Objective: To enable participating stakeholders to describe and analyse changes with regard to indicators that were formulated in the planning phase. Download

Tool 6 - How did our strategies work (influence matrix)? ( 120 min.)

Objective: To ensure stakeholders closely examine the changes which they identified in their community and thoroughly analyse what might have led to these changes. Download

Literature & more guides

Anderson, Marie B. (2004): Experiences with Impact Assessment: Can we know what good we do? Berghof Research Centre, Berlin.

Earl, S., Carden F., Smutylo, T. (2001): Outcome Mapping. Building Learning and Reflection into Development Programs

GTZ (2007): The concepts of participatory impact monitoring.

Guijt, I.& Woodhill, J. (2002): Managing for Impact in Rural Development – A Guide for Project M&E, Rome, Italy: International Fund for Agricultural Development (IFAD).

Neubert, Susanne (2010): Method for Impact Assessment of Programmes and Projects (MAPP). In: Verfahren der Wirkungsanalyse – Ein Handbuch für die entwicklungspolitische Praxis – Arbeitskreis Entwicklungspolitik DeGEval – Deutsche Gesellschaft für Evaluation (Hrsg.) Freiburger Beiträge zur Entwicklung und Politik, Arnold Bergsträsser Institut, p. 88-96

World Bank (2002): Sleeping on our own Mats: An Introductory Guide to Community Based Monitoring and Evaluation.

A news service focusing on developments in the field of monitoring and evaluation methods relevant to development programmes with social development objectives. Managed by Rick Davies: http://mande.co.uk/

• Offers among others: Most Significant Change (MSC) – a detailed description of the full approach.

World Bank Website: Participation and Civil Engagement.

Participatory Methods Website of the Institute of Development Studies (with Robert Chambers as associated researcher)

NGO IDEAS – German Network of NGOs working on participatory M&E. The website offers among others a description of MAPP, Method for Impact Assessment of Programmes and Projects, a two-day M&E workshop with participatory tools, developed by Susanne Neubert, DIE, Bonn/Lusaka

OSF South Africa – Crime and Safety Project website with policy brief on “Monitoring and Evaluating Community Safety Projects Affordably”.