A $p$-local finite group consists of a finite $p$-group $S$, together with a pair of categories which encode ‘conjugacy’ relations among subgroups of $S$, and which are modelled on the fusion in a Sylow $p$-subgroup of a finite group. It contains enough information to define a classifying space which has many of the same properties as $p$-completed classifying spaces of finite groups. In this paper, we examine which subgroups control this structure. More precisely, we prove that the question of whether an abstract fusion system $\mathcal{F}$ over a finite $p$-group $S$ is saturated can be determined by just looking at smaller classes of subgroups of $S$. We also prove that the homotopy type of the classifying space of a given $p$-local finite group is independent of the family of subgroups used to define it, in the sense that it remains unchanged when that family ranges from the set of $\mathcal{F}$-centric $\mathcal{F}$-radical subgroups (at a minimum) to the set of $\mathcal{F}$-quasicentric subgroups (at a maximum). Finally, we look at constrained fusion systems, analogous to $p$-constrained finite groups, and prove that they in fact all arise from groups.