Ranking and scoring exercises are common in participatory research work, for example in determining farmer preferences for crop varieties, tree species or other commodities. However, they rarely involve eliciting opinions of many individual farmers so as to establish the repeatability of study findings. This paper demonstrates ways in which ranking of evaluation criteria can be combined with evaluation scores to produce generalizable results concerning farmers' overall preference for a set of bean (Phaseolus vulgaris) varieties. The use of weighted indices is explored. Results were found to be robust across all indices. However, farmer evaluations for yields did not correlate well with actual yields measured on their own plots. Farmers often gave high evaluation scores for yield performance of new bean varieties, even when they yielded quite poorly in plots in their own fields. Reasons for this are discussed.