Working Groups & other discussion forums

Day 1

Session 1: Perceptions of ILAC

Q1. What practical actions can we take collectively and individually after the workshop to address the paradox of strong demand for Impact Assessment but limited evidence of direct influence on decisions?

*Involve donors in the IA process in critical stages of the decision-making process (taking them on board)
*Impact Assessments need to cover the complexity of farmers' decision-making and agricultural innovation processes in order to deliver the outcomes that correspond with donors' demand
*Provide short briefs (1 page) that donors/decision-makers can read given their limited time for absorbing information (note of caution: potential trade-off between providing condensed information and doing justice to complexity)
*Identify the key people that need to be informed and focus your 'information package' to them Þ strategic way of informing people (right timing, right targeting)
*Make decision-makers prepared for failures as a normal part of human actions; build confidence, ownership and trust with decision-makers (long-term process)
*Look into wider issues of credibility in our efforts to create impact (e.g. funding for projects/programs vs. funding for administration)
*Build alliances with other people in the organization that are committed to institutional change
*Be more realistic about the impact that research can have within the lifetime of a project; raised awareness might lead to project impact long after the project has been phased out (which raises the questions: when to measure impact? and are we looking too much into immediate impact)
*Raise awareness of donors about impact pathways (these might not always be linear)
*{Be strategic - joining an existing 'wave,' not necessarily in at start (if appropriate)}

Q2. How to blend formal and informal methods and different world-views with reference to large-scale impact assessment? (noting that this integration is easier at village scale with more focused networks)

Click here to see the graphic

· {A definition was provided - that of Oxfam}
· Importance of values depends on level of participation
· Local context-specific choice of methodologies
· CGIAR is good at involving partners
· CGIAR bad in analyzing social processes
· Honesty: explain failures and learn [from them]

Session 2: Collaborative impact assessment

Q3. How do you bring about institutional changes that give rise to more poverty reduction, environmental sustainability and social inclusion?

Overview:
· To be effective, need to understand and act on processes, political context and institutions
· Principles of political and cultural action are embedded in particular activities and their context. Example of Grameen Bank, where new employees are immediately sent to spend 6 months in the field and to be as critical as possible about what they observe. Many drop out, but those who stay really understand what is going on.

Action:
Donors: key to change! But fickle, attracted to success
Farmer field school (FFS) story:
· became proven success; all donors want to give to them, but the desire to fund FFSs is greater than the number of FFSs that are truly capable of implementing the program. As FFSs emerged that were not successful, there was a general backlash against FFSs
· In Nepal, existing network of FFSs wanted to defend their quality. Because there was enough power in farmer trainers and other supporters, there were representations to Ministry, which succeeded in blocking unqualified FFSs.

Centers:
· Different cultures probably means different approaches; an example is that the merging of CIAT and TSBF has been difficult because of very different cultures or styles
· Another example is that ICRAF is the only Center with the mandate of 'research for development'; all others steer away from associations with development per se
· Use self-assessment and learning from failures
· Learning from other institutions
· One Center has new DG whose philosophy is to 'let people run free and then bring back together'
· Decentralization common to many Centers and such Centers are more likely to have scientists aware of and responsible to local needs. This is often problematic vis-à-vis demands from the Centers for a unified position or corporate approach, or at least not including "extras."

Good practice:
· Using self-assessment as part of the culture where impact and failures are part of the learning
· Need positive models to give us insights on how to change institutions.

Leadership: "champion"
· cannot be trained, anointed
· Finding allies
· Dealing with power structure
· May be messy; 'blood on the ground'
· Requires determination, sustained effort.

Q4. How do we do IA and M&E that meet collaborative needs and include both science and development objectives?

Summary: Increase in number of different types of partners or stakeholders à multiple IA information needs à IA becomes more complex and there is a need to apply different methods à challenge be inclusive and ensure that IA is manageable.
Elements that can help:
· Involve multiple partners in the definition of the key questions of the IA and in drawing conclusions from the findings
· Separate tasks for IA and tasks for capacity-building.

Should we use the term collaborative here? Should we rather use the term partnerships? 'Collaborative' does not necessarily reflect the degree of commitment, dedication, responsibility sharing, resource-sharing as compared to 'partnerships.' On the other hand, collaboration may be short term or long term, it can be a partnership but does not have to be.

The challenge in the question to be addressed, it was agreed, is really dealing with different/multiple needs for IA information. A graphic illustration was produced:

Click here to see the graphic

To get an idea of the different types of partners/stakeholders and their different IA information needs, the group decided to list different partners/stakeholders and their IA information needs. There was talk about also listing the consequences of not meeting these needs.
Three groups of partners/stakeholders were identified related to a local-global continuum:
· Micro, e.g. community-level partners
· Meso, e.g. regional- or national-level partners
· Macro, e.g. international-level partners.

The listing of needs and consequences was not completed, as the discussion moved on to the issue of accountability:

Demands/needs for IA information

Immediate (demands)	Those who control research funding and jobs
Secondary (needs)	Those who need the information for their own research or development activities
Tertiary (needs)	Those who benefits from research and development activities

However, there was agreement that this is not the preferred way to think about accountability structure - How should the accountability structure for IA look and be like?

The group identified possible elements and discussed at length, but did not reach conclusion or agreement.

Elements that could be considered to influence accountability structure:
· MDGs ?
· Social and economic inclusion ?
· Equity ?
· Human rights ?

The group observed that these are all goals in themselves - but whose goals?

Day 2

Session 3: Methods and constraints for ILAC

Q5. Use of data (from PM&E processes) for learning purposes and project re-adjustment for enhancing impact and who needs to participate in these processes? How do we learn from people who have done this already?

· Having a community-level process that feeds into an institutional/R&D process
· In utilization-focused evaluation that involves decision-makers in reflecting on data
· Having a systematic process for tracking indicators
· 'Stopping' to reflect and learn - {balance creating space for reflection with promotion of good practice, avoiding micro-management (e.g. reflection time in CCER)}
· Ensuring formal PM&E processes run alongside informal processes
· Ensuring flexibility in the process "leaving room for serendipity"
· à donors […]
· {IPGRI's management team has been transformed in respect of allowing time for reflection, providing empirical evidence that a Center can learn and change - other Centers could learn from this experience, which needs documenting}
· {knowledge sharing in ICT-KM is looking at Center annual meetings, trying to transform them into learning fairs for interaction and reflection (CIFOR, IWMI cases)}.

Q6. How do we scale out PR processes while at the same time maintaining quality, rigor and relevance of these processes? How do we learn from people who have already done this?

· There are experiences of scaling out already happening: need to extract lessons, promote cross-case learning
· Process and principles can be replicated — how many times?
· Capacity-building should be a priority: identify key (and interested) components of the innovation system; practical learning, mentoring, etc.
· Institutional commitment for learning and change, and political support needed — engaging from the beginning
· Replication has risks: (a) technologies that may not fit; (b) tools without principles
· For replication of technologies generated through PR, a differentiation should be made between input-based and knowledge-intensive approaches that are tied to process
· Quality control? Institutional commitment — final test in the real world: accountability
· {Scaling out requires compromise for the masses}
· {Differentiate scaling-out processes from project-based processes - need policy involvement}
· {Need to engage at political level}

Session 4: Empowerment

Q7. What experiences have participants had in trying to assess empowerment processes? What differences have the concept of farmer empowerment made to how research and development organisations operate?

Outputs:
· There appears to be more than one discourse on empowerment:
-The 'new' discourse, focussed on projects and heavily influenced by the World Bank. This discourse includes a big concern with measurement, for the purposes of external assessment. In addition to World Bank efforts, other interesting work in this area has been done by DANIDA, CARE, INTRAC
-The 'old' discourse, focussed on social movements and influenced by Friere. The achievements are self-justifying and therefore no need for external assessment. Many examples from Latin America, including Campesino and Campesino
· The group discussion repeatedly came back to the experience of IPM Farmer Field Schools, which seems to bridge the old and new discourses (perhaps IPM is more 'new,' and FFS is more 'old'). This may explain why measurement of FFS impact has been controversial, and why success has sometimes been followed by examples of 'the empire strikes back'
· The 'new' discourse is problematic for a number of reasons:
-is empowerment zero-sum or relative?
- is empowerment transformational or reversible?
-who decides what is 'good' empowerment and what is troublemaking?
-can there be too much empowerment of farmers, with negative consequences for other sections of society?
-should empowerment be seen as a precondition or a consequence for technological development?
· Old or new, the group agreed that greater awareness of power would greatly help the work of agricultural scientists and extensionists. We all need to understand relationships and dynamics in the communities/societies where we work, and we need to appreciate our own power. There are good examples of 'transformational' learning among development professionals (India, Indonesia, South Africa) that help to achieve this better understanding of roles and relationships. The CG Centers need more of this.

Session 5: Gender

Q8. How have some people in research context been able to create gender conducive environment?

Good practices to learn from:
· Commitment/support from leadership (e.g. IWMI)
· All proposals are vetted for gender where relevant in the IWMI Quality Management System
· "Agents of Learning" Gender teams with representation from all departments (IIRR)
· ICRISAT Gender and Diversity (G&D) Associates (25 at all levels) also looking at Gender and Research, and could work from within.

Problems:
· Budget given for gender and then not used for gender
-Who monitors gender? Who understands it?
· Some/many? Scientists do not understand gender in research
· Vacuum of trained, mentored scientists for the future
· No follow-up to CG Social Science 2002 meeting, where gender was discussed
· Dependency on one gender specialist in a Center: overload, reduced reach
· Short tenure at some Centers makes it difficult to hold expertise.

Possible actions:
· Capacity-building on being a change agent (e.g. G&D course; ILAC training)
· Including attention to gender in performance evaluations of scientists where appropriate
· Mentoring/coaching of biological scientists at Centers
· Addressing the needs of Centers such as CIMMYT that would like to learn more
· Finding out whether in some Centers G&D teams could be advocates for gender research
· Identifying donors who will, who would:
-monitoring grants to Centers for incorporating gender in research
-establishing a challenge program
· Providing a bonus system for gender research; prizes
· Funding a challenge program for gender research
· Expanding connections, such as looking outside the CG box for experts who regularly work with donors, for assistance on making the case for gender issues
· Centers establishing dialog with senior officials in collaborating countries (e.g. IRRI in Pakistan)
· Technical meetings on gender and research.

Day 3

Session 6: Benefits and costs [Q9]

· Is easiness to manipulate [data] a real issue? {Should we not manipulate in our advocacy role?}
· Comprehensive about costs and benefits {e.g. counterpart costs}
· [Cost comparisons between PR and conventional research may be unfair comparision, as processes are very different]
· Costs of not having PR?
· How to calculate costs (issue of investment in capacity-building)
· Transaction costs on seed acquisition (ethnographic approach)
· Easy counterfactual is too artificial
· Political economy of benefit-cost analysis (mentioned paper) {ISNAR workshop 15 years ago covered much of what is being treated as 'new knowledge' today - rather it was not adopted then by practitioners}
· Easy to ignore some costs {e.g. negotiation, interaction, opportunity costs of farmers}
· Consequence of PR not being cost-effective {e.g. adoption, breeding assumptions}
· {How to calculate cost-effectiveness, given hidden costs (e.g. indigenous knowledge) that take years to accumulate (but compare cost of, e.g., PhD) - risk issue is more important to those imparting indigenous knowledge; do we need a model to understand their viewpoint?}
· {What are we really looking for? Improving the efficiency of research may go beyond costs and benefits to augmenting and replacing}
· {Cost-benefits irrelevant if one understands the value of PR, but do need to reduce costs of (e.g.) capacity-building, huge demands for measurement}
· {What would have happened in the CGIAR without PR? What was the influence of people in the CG who then moved on to influence others?}

Session 7: Rural innovation capacity

Q10: Methodology

Qualitative - quantitative: respect for qualitative
· What is impact assessment
control
attribution?
· Realistic counterfactual to PR
· Define 'conventional research'
· Point of PR is broad sharing {collaboration with other disciplines}
· Point is to answer the question
· Powerful tool for reflecting local interpretation
heterogeneity
· Question of quality / rigor relevant for both
· {There will always be some doubters}
· {Capacity issue — i.e. sloppy handling of qualitative data à degraded notion of value of qualitative data}.

IA — control, attribution:
-Context required for pro-poor technology development
-Control groups
-Context of control groups
-Rival explanations
-Counterfactual random or not?
-Minimal set of critical factors shared by group members
· Be clear about the difference when comparing
· The role of natural science 'technologies,' etc.

Q11. Primary objective of PR

· Produce international public goods (IPGs)
· Cannot draw line between research and development
· IPG must be rooted and understood in local contexts
· What is 'cutting-edge science'?
· Social science? Not separate out — public goods
· Orientation towards global public goods does not exclude sectors

Session 8: PR methods in IA

Q12. Research ethics

· Evaluation assessments have some statements on ethics (African eval. ass. - AFREA; American EA — AEA)
learn from these existing guidelines
>CG has ethics committee and guidelines (maybe not focusing on research processes)
>need guidelines focusing on research process.

Q13. Areas of focus

· Confidentiality
· Pictures of community— being used by R&D organization
· Gender aspects
· Guidelines developed should include social scientists
· Research not going through ethics
· In capacity-development processes where community are involved - PRA (how do communities benefit?)
· Intellectual property rights (respecting farmers' seed) — scientists copyrighting farmer property.

Q14. With PR

· Taking "no more vampire scientists"
= using farmers as a 'control' group, is it ethical (introducing untested programs maybe more unethical?) — a good document idea
partners can keep us ethical (as example)
· CB - research ethics boards with community seat on the board
· Ethics of visitors are important as well
· How we work with communities raising status of some can cause "risk" on some members - use locals to decide
· Compensating farmers for participating?
· Precautionary principle of "do no harm" at community è example from Francesca's research
· Social responsibility is scaling up — creating socially responsible innovation
· Learn from universities that have "ethical guidelines"
· How PR scientists present their findings— "not always" the truth
· Recognizing all those who contribute to the research
· Bring research results back to communities and local government
· "Learn from community library with all research publications"
· Ethical issues when dealing with institutional innovations and power structures - especially when dealing with scaling up.

Integrating perspectives

· photos as farmer communication
strengths and weaknesses
· more conceptual issues difficult to understand
· contribution of farmers — scientists
· research— extension
· IA — to confirm
— to get additional information

Better to be vaguely right than precisely wrong
· Context — what is in it for me?
· Integrating / representing different perspectives