Funding for large research initiatives, such as those funded through the National Institutes of Health U mechanism, has increased since 2010; however, there is little published research on how to evaluate the success of such initiatives. Here, we describe the collaborative evaluation planning process undertaken by the Interactions Core of the Collaborating for the Advancement of Interdisciplinary Research in Benign Urology (CAIRIBU) research community, a clinical and translational research initiative funded by the National Institute of Diabetes and Digestive and Kidney Diseases. Evaluation is necessary to measure the impact of our work and to allow for continuous improvement efforts of CAIRIBU activities and initiatives. We developed and implemented an iterative seven-step process that engaged the Interactions Core, NIDDK program staff, and grantees at each step of the planning process. Challenges faced in planning and implementing the evaluation plan included the time burden on investigators to submit new data for evaluations, finite time and resources for evaluation work, and the development of infrastructure for the evaluation plan. We call on funding agencies to include more explicit requirements for evaluation participation from grantees, as well as dedicated funding to support the evaluation process, in future funding opportunity announcements for large research consortia.