A set S of permutations is forcing if for any sequence $\{\Pi_i\}_{i \in \mathbb{N}}$ of permutations where the density $d(\pi,\Pi_i)$ converges to $\frac{1}{|\pi|!}$ for every permutation $\pi \in S$ , it holds that $\{\Pi_i\}_{i \in \mathbb{N}}$ is quasirandom. Graham asked whether there exists an integer k such that the set of all permutations of order k is forcing; this has been shown to be true for any $k\ge 4$ . In particular, the set of all 24 permutations of order 4 is forcing. We provide the first non-trivial lower bound on the size of a forcing set of permutations: every forcing set of permutations (with arbitrary orders) contains at least four permutations.