We use methods of differential astrometry to construct a small field inertial reference frame stable at the micro-arcsecond level. Using Gaia measurements of field angles we look at the influence of the number of reference stars and the stars magnitude as well as astrometric systematics on the total error budget with the help of Gaia-like simulations around the Ecliptic Pole in a differential astrometric scenario. We find that the systematic errors are modeled and reliably estimated to the μas level even in fields with a modest number of 37 stars with G <13 mag over a 0.24 sq. degrees field of view for short timescales of the order of a day for a perfect instrument and with high-cadence observations. Accounting for large-scale calibrations by including the geometric instrument model over such short timescales requires fainter stars down to G=14 mag without diminishing the accuracy of the reference frame.