Let $U$ be a smooth affine curve over a number field $K$ with a compactification $X$ and let ${\mathbb {L}}$ be a rank $2$, geometrically irreducible lisse $\overline {{\mathbb {Q}}}_\ell$-sheaf on $U$ with cyclotomic determinant that extends to an integral model, has Frobenius traces all in some fixed number field $E\subset \overline {\mathbb {Q}}_{\ell }$, and has bad, infinite reduction at some closed point $x$ of $X\setminus U$. We show that ${\mathbb {L}}$ occurs as a summand of the cohomology of a family of abelian varieties over $U$. The argument follows the structure of the proof of a recent theorem of Snowden and Tsimerman, who show that when $E=\mathbb {Q}$, then ${\mathbb {L}}$ is isomorphic to the cohomology of an elliptic curve $E_U\rightarrow U$.

Given a non-isotrivial elliptic curve over an arithmetic surface, one obtains a lisse $\ell$-adic sheaf of rank two over the surface. This lisse sheaf has a number of straightforward properties: cyclotomic determinant, finite ramification, rational traces of Frobenius elements, and somewhere not potentially good reduction. We prove that any lisse sheaf of rank two possessing these properties comes from an elliptic curve.