We present an unconditionally energy stable and uniquely solvable finite difference scheme for the Cahn-Hilliard-Brinkman (CHB) system, which is comprised of a Cahn-Hilliard-type diffusion equation and a generalized Brinkman equation mod-eling fluid flow. The CHB system is a generalization of the Cahn-Hilliard-Stokes model and describes two phase very viscous flows in porous media. The scheme is based on a convex splitting of the discrete CH energy and is semi-implicit. The equations at the implicit time level are nonlinear, but we prove that they represent the gradient of a strictly convex functional and are therefore uniquely solvable, regardless of time step size. Owing to energy stability, we show that the scheme is stable in the time and space discrete and norms. We also present an efficient, practical nonlinear multigrid method . comprised of a standard FAS method for the Cahn-Hilliard part, and a method based on the Vanka smoothing strategy for the Brinkman part . for solving these equations. In particular, we provide evidence that the solver has nearly optimal complexity in typical situations. The solver is applied to simulate spinodal decomposition of a viscous fluid in a porous medium, as well as to the more general problems of buoyancy- and boundary-driven flows.