Summaries

Day 1

Opening Session

CIMMYT Director General Masa Iwanaga opened the Workshop, summing up its purpose that, despite the impressive volume of impact-assessment studies conducted by the CG as a whole (a figure of over 800 studies was quoted by David Raitzer), participants had been gathered "not to celebrate victory, but to try to address major issues, including change in institutional culture and change in institutional relationships." He was followed by members of the organizing committee. The main objectives of the workshop were presented as building capacity among the participants for effective impact assessment, and enabling the participants to share their experiences with fellow practitioners. The workshop was aiming to address difficult-to-approach issues in impact assessment, such as: institutional learning and change; assessing stakeholder empowerment; project impacts on women; costs and benefits of participatory research; impacts on the ability of rural communities to be innovative; and participatory methods in impact assessment. The issue of quality assurance while mainstreaming participatory research was raised, along with the challenge of institutionalization - linking impact assessment with knowledge management and institutional management. Meanwhile, Roberto La Rovere stressed the need for a paradigm shift for impact assessment away from pure economics. Participants were then split into small groups to discuss their expectations of the workshop, which were fed back to the plenary.

Session 1. Perceptions of impact assessment for institutional learning and change
Two papers made up the first session. The first focused on the role of ex-post impact assessment, as assessed via a questionnaire to a sample of CGIAR members and follow-up interviews. Despite the high demand for poverty-focused impact studies from donors, the results themselves are not the primary decision-making factor when it comes to allocating funds! These findings suggest needs for broader dissemination of impact reports, with careful integration of the findings into other fund-seeking activities. The role of external program and management reviews in donors' decision-making was mentioned, so it is important that impact-assessment reports be brought to the attention of review team members. The second paper looked at the establishment of the Inter-Center Initiative on Institutional Learning and Change (ILAC Initiative). As its name suggests, ILAC is interested in institutions, experiential learning, and change; it arose out of an understanding that former preconceptions of what makes a successful program are probably out-dated and that the field of impact assessment is moving on at a fast rate.

Discussion of this session highlighted such aspects as:
· The relationship between the evaluator and the project - should the evaluator be independent?
· Impact assessment should now move beyond accountability to learning, but the two should not be seen as contradictory.
· Is more and "better" impact assessment enough? Rather look at the political context.
· Importance of timing - ex-post impact assessment is good for accountability, but does not help the ongoing project.
· Who is interested in learning? Currently, management and scientists, not donors.
· The "unspoken issue" of power, even within a single institution.

Session 2. Moving from assessment to learning and change: Collaborative impact assessment
Prepared by Stephen Biggs

Summarizing Session 2 is really hard, as the Workshop as a whole kept on reinforcing issues that came up in that session. For example, we had a paper by Yogesh Bhatt on the new management framework being introduced into IWMI. For me, one of the really interesting features of this was that Yogesh himself came from a background of working with CARE International in India, and brought with him all the experience of working with a large international NGO in a complex international and local political context. In the last session of the workshop, when we were discussing in a subgroup issues of research ethics and how to support socially responsible people and institutions, Yogesh was able to suggest using the principles and experiences of "Do No Harm"; as CARE International had a wide experience of using these methods. 1For me, this was evidence of the broadening of the skills of CG Centers staff so that they were able to effectively address issues of poverty reduction and social exclusion, where generally changing power structure is involved and sometimes takes place in violent contexts.

Diversity across CG Centers, other organizations and networks: There were four papers in Session 2. Three from CG Centers and one from a large NGO, which was managing a global partnership. What was significant was the diversity in the papers and the different ways in which organizations and networks were working. The paper from CIP showed that this CG Center was taking seriously the MDGs and had reformulated its mandate to focus on these goals, rather than making them an "add on." The way this was being implemented with NGOs and government agencies meant there were all the time effective voices monitoring CIP's research for poverty and social inclusion relevance. ICRAF's work was very different, but in some ways similar. In the presentation, we heard about how ICRAF was assessing with others the experiences of 28 organizations that were working together in "scaling up" technology. Interestingly coming up with familiar experiences of how some powerful organizations were not wanting to work with others in sensible transparent ways for a whole range of parochial and personal reasons. What was refreshing about these developments in a CG Center was the way the issues are being documented and addressed effectively2 . On the old familiar theme of "it is difficult for social scientists and natural scientists to actually works together," it was also refreshing to hear from ICRAF's experiences throughout the workshop that "doing joint multidisciplinary work on poverty and social inclusions topics" was not a major problem. Much of their funding was project based and donors required poverty relevance, in practice as well as on paper. The close monitoring of the projects for poverty relevance was helping to ensure a poverty bias in the work. In IWMI, it is too early to tell how the new management and monitoring system will work. However, from the evidence so far, it would appear that the institution is committed to new rules and procedures. By having a new staff member to run the new system, who comes from another international NGO, it means that new experience has been introduced into a CG organization.

Managing the unexpected: All the papers reflected a healthy view that nothing was set in stone or was permanent. The monitoring systems really did seem to be set up to monitor, not only previously agreed plans, but also how to learn from and respond to unexpected outcomes, and unexpected changes in the wider political and economic context.

Importance of poverty reduction and social inclusion goals: Each of the four papers reflected (in different ways) an interest in poverty reduction and social inclusion, and effective institutional changes that are bringing this about. In the case of CIP, the MDGs now figure high in their mandate. In the case of IWMI, any research in the organization does not get approved unless social development goals are substantially addressed in the research proposals. In the ICRAF and Prolinnova cases, the research challenges and experiences of developing empirically based, cost-effective indicators were well addressed. Significantly, all papers did not have lists of what "should be done," but rather reported on experiences of how to do it.

Scaling up and relevant social science research: The issue of actual experiences of scaling up were especially addressed in the CIP and ICRAF papers. For me, what was refreshingly interesting was the emphasis on learning in each case of the great variety of ways useful technologies had or had not been "scaled up." Perhaps the conclusion from a global public good perspective of these research findings is: there is no one institutional model for "scaling up" that can be transferred between political, cultural, spatial and temporal settings. All that can be done is to strengthen and support local capability to be able to take decisions that are relevant to the local ecological, economic, political and cultural environment. This is certainly a different research finding from those that led to the promotion of such "one size fits all" institutional models as the T&V system, participatory research methods, some cooperative models.

Working outside of the CG box: The papers in this session certainly produced evidence that things have changed in the CG system. Sometimes in the past, CG Centers described partnerships in a way that showed how they did research and partners adapted and transferred innovations in local circumstances. Clearly this no longer holds in the CG system, as reflected by the papers in this session and many other workshop discussions. For example, the frequent references to the FAO-supported Community Integrated Pest Management (CIPM) and Farmer Field School (FFS) experiences, shows that many CG Centers and programs have learnt a lot from other actors; and intend to keep going in these new directions of learning from others.

Personal observation: I was part of the early days of CG activity in the 1970s. What has been so refreshing for me about this Workshop is that many new activities are happening on the ground, in which CG organizations are but one of several actors. The old unhelpful idea of who is doing "research" and who is doing "development" has broken down. In many situations, it appears that poverty reduction, social inclusion and other MDGs are now serious objectives. There is a great range of diverse institutional arrangements in place that are working, and lessons are being leant and acted upon.

Day 2
Prepared by Jemimah Njuki

Session 3. Methods and constraints for institutional learning and change

Three papers were presented under this session: the first on participatory monitoring and evaluation, the second on assessment of participatory elements in agricultural research for institutional learning and change; and the third on the challenges of out-scaling participatory methods in agricultural research. I find the theme of this session very timely and important. Organizations need to learn from experience, from what is working and what is not working, and to adapt their strategies and ways of working accordingly. One of the ways of doing this is by organizations making use of evaluation and impact-assessment results. For these to be useful as a learning tool, there needs to be a re-focus of evaluation and impact assessment from just their accountability functions to include learning and change. From this workshop, we have heard how participatory monitoring and evaluation (PM&E) can be used as a tool for institutional learning and change and, more importantly, for providing feedback mechanisms between different stakeholders, including communities. Some strategies for promoting learning and change from evaluations and impact assessments are:
· Inclusion of key decision-makers in designing evaluation questions and in reflecting on the evaluation results;
· Summarizing evaluations and impact-assessment reports into short and simple learning points;
· Recognizing and promoting informal methods of learning;
· Putting in place systems for tracking progress, analyzing and learning;
· Having internal mechanisms for "stopping and learning" from what people are doing-we are so busy doing things (writing proposals, implementing them, doing impact assessment and moving on to the next proposal) that we forget to take time to stop and learn from what we are doing;
· Exploring from within and outside the CGIAR System where learning systems and mechanisms have been successful.

Some of the key questions that beg an answer from these learning and participatory processes are: Who needs to participate? Should we aim for maximum participation or optimum, strategic participation? While these are difficult questions to answer, one thing is clear, it is no longer a matter of us "good participatory guys" vs. them "bad non-participatory guys." Different kinds of projects will require different strategies for the participation of other stakeholders, but what is clear is that it is crucial to involve the key beneficiaries (to ensure that our research is relevant) and the key decision-makers including donors (who can and will eventually influence change).

One of the challenges of participatory approaches and process is the scaling out. Opportunities and lessons for successful scaling out of participatory approaches exist. However, we need to ask ourselves some hard questions as participatory approaches practioners:

What are we scaling out? Is it technologies, approaches, principles? Those scaling out participatory approaches and principles need to take into account the local context. What works well in one situation may not work well in another due to differences in local, national and even international contexts. Because of this, there needs to be room for adaptation of these processes. In a lot of cases, we do not want to see anyone interfering with or changing our perfect location-tested processes in the scaling-out processes. We want them implemented in the same way. This is not going to be possible. We have to be flexible and allow for adaptations to these processes if we want them to work in other contexts.

What is required for scaling out? Several things are required to scale out participatory processes. One is the recognition by scientists that they alone may not achieve the wide-scale scaling out of these processes. We need to recognize the role of policy and engage policy-makers in the process, as well as partners who have the scope for wide-scale implementation of these processes, such as national research and extension programs. Of the ingredients that cannot be overvalued in the scaling out process is the need for capacity-building. Last but not least, we need to learn from successful scaling out of past participatory process and a good example of these is the scaling out of the IPM Farmer Field Schools in Asia and recently in Africa.

Session 4. Project impacts on stakeholder empowerment

We had another very interesting session on empowerment and the difficulties of measuring empowerment and more important questions of should we even be trying to measure it.

The general feeling on measuring empowerment can be summarized in the following quotation on evaluation and impact assessment:

    The first step is to measure what can easily be measured; this     is okay as far as it goes!!

    The second step is to disregard what cannot be easily measured     or give an arbitrary value; this is dangerous!!

    The third step is to presume that what cannot be easily     measured is not very important; this is blindness!!!

    The fourth step is to say that what cannot be easily measured     may not even exist; this is suicide!!!

How often we do this with empowerment!!!

From the presentations in this session, we saw different forms or types of empowerment, different definitions, and different stages of empowerment. We heard of project-based empowerment processes, but also of transformational-based empowerment. What comes out from these presentations is the need to look at empowerment both as a precondition to participatory process but also as a consequence of these processes, and the need see empowerment not just as an end in itself but also to focus on what the outcomes of empowerment are.

Session 5. Project impacts on women

The CGIAR is characterized by a few pockets of gender research in different Centers. Although it is important to look at what project impacts are on women, there is also a need to be pro-active in carrying out gender research. The three presentations in this session showed some positive and maybe some negative consequences of projects on women, but to ensure more positive impacts on women, we need to do more. Some of the practical suggestions for putting gender research on the agenda of the Centers that came out of the discussions included:
· Mainstreaming gender research and making gender visible in the strategic plans of the Centers including in the MTPs.
· Putting in place a quality management system for proposals that ensures that proposals being developed and funded under the Centers take gender into account.
· Building capacity for gender research in Centers: The discussions noted that although there may be willingness and commitment on the part of scientists to understand gender, to carry out gender research and to integrate gender considerations in their work, the capacity to do so may not be there. There needs to be a concerted effort by gender specialists in the Centers and by PRGA to build this capacity. This needs to be coupled with a mentoring process and there may be a lot to learn here from the Gender and Diversity Program.
· Networking among scientists involved in gender research in the Centers in order to share experiences and lessons.
· Need to broaden the scope of gender research: Some of the presentations under this session showed different ways of integrating and carrying out gender research. It is evident that just having a gender variable or including a gender variable in analysis is not enough. There needs to be more than this.

Day 3

Session 6. Benefits and costs of participatory research
The four papers presented in this session looked at various aspects of the costs and benefits of participatory research: one was a direct comparison of "conventional" and participatory research approaches; one looked solely at the cost-benefit analysis of crop breeding; and the third at integrating participatory components into "conventional" research projects. Finally, session-facilitator Roberto La Rovere looked at the elements typically assessed in determining the costs and benefits of participatory research. He also stressed the difficulties frequently encountered in such evaluations, such as putting an economic value on "international public goods" and social benefits (e.g. reduced health risks from reduced pesticide use). He suggested that such analysis were all-too-easy to manipulate simply by ignoring some costs or emphasizing some benefits. This led into small-group discussion of issues, checklist, methods and best practices for assessing costs and benefits (2 groups), cost-effectiveness of participatory research, and ease of manipulation vs. ethics.

Issues raised in discussion included:
· The role of manipulation in advocacy.
· The need to provide real costs.
· Counterpart costs.
· How to calculate cost-effectiveness vis-à-vis the use of (e.g.) indigenous knowledge that has taken years to develop and is usually a hidden cost (conversely, training costs for researchers in conventional science are equally hidden) - e.g. ethnographic approach to farmers' transaction costs: the risk issue is important to farmers; do we have a model to understand their viewpoint?
· It is easy to ignore some costs, e.g. negotiation, interaction, farmers' opportunity costs.
· What are the costs of not having participatory research, in terms of (e.g.) adoption, breeding assumptions? Need to monitor throughout.
· Costs-benefit is irrelevant if we understand the value of participatory research, but we still need to reduce costs, e.g. by capacity-building, cheaper forms of measurement.
· What would have happened in the CG without participatory research? What has been the influence of CG participatory research practitioners who have moved on to influence other institutions?

Session 7. Project impacts on rural innovation capacity
Prepared by Jonathan Hellin

The four papers in this session looked at the extent to which participatory research has contributed to innovation capacity. Susan Kaaria looked at farmer innovation as a result of participation in Local Agricultural Research Committees (CIALs). One of the key questions that arose is that while there is some evidence that one of the benefits of CIAL methodology is improved food security, it is not always clear if farmers who already have greater food security are attracted to the CIALs in the first place. Esbern Friis-Hansen's paper on farmer innovation in Uganda stressed that much of the innovation was due to experimental learning in Farmer Field Schools along with empowerment and innovation on the part of the extension staff involved in the project. Joshua Ramisch looked at the strengthening of field ecology for improved soil-fertility management in western Kenya. The focus of the research was very much on farmer learning.

Jonathan Hellin presented an overview paper that discussed the notion of farmer empowerment as a primary objective of participatory research and posed two questions for discussion:

1. Under what circumstances is it reasonable to expect participatory research projects to have a direct impact on farmer empowerment and rural innovation capacity?
2. Should farmer empowerment and stimulation of farmer innovation capacity be primary objectives of organizations engaged in participatory research?

The premise is that farmer participation in agricultural research is important and necessary in order to increase the efficiency and impact of agricultural research and technology development. Furthermore, farmer empowerment is valuable and desirable, and while it can result from participatory research, direct empowerment per se should perhaps not be the main objective of participatory research conducted by research organizations per se. Wider farmer empowerment leading to innovation can perhaps be best brought about by organizations such as NGOs that are more grass-roots based and tend to have a longer-term interaction with farmers than researchers do. Time did not permit the participants in the workshop to reach a consensus on this important issue.
__________________________________________________

1 In Nepal, where currently there is much violent social conflict, CARE International and many other development agencies have found the "Do No Harm" useful (Anderson, Mary B. 1999. Do No Harm: How aid can support peace - or war. Lynne Rienner, London, UK. Anderson, Mary B. (ed.) 2000. Options for Aid in Conflict: Lessons from field experience. The Collaborative for Development Action, Cambridge, MA, USA).

2 Such process documentation involving CG Centers is not new; nor is the reporting of what powerful actors do. For example see Lewis (Lewis, D.J. 1998. Partnership as process: Building an institutional ethnography of an inter-agency aquaculture project in Bangladesh. chapter 6, pp. 99-115 in Mosse, D., Farrington, J. and Rew, A. (ed.) (l998) Development as a Process: Concepts and methods for working with complexity. ODI and Routledge, London and New York). However, what is interesting here are the examples of how some actors are effectively addressing such power structure and political/cultural issues.