This paper considers the effect of an exponential variation in the background density field (as exists in compressible atmospheres) on the structure and dynamics of the quasi-geostrophic system, and compares the results with the corresponding Boussinesq limit in which background density variations are assumed small. The behaviour of the compressible system is understood via a closed-form analytic expression for the Green's function of the inversion operator relating potential vorticity and streamfunction. This expression makes explicit the anisotropy of the Green's function, inherited from the density profile, which has a slow, algebraic decay directly above the source and an exponential decay in all other directions. An immediate consequence for finite-volume vortices is a differential rotation of upper and lower levels that results in counterintuitive behaviour during the nonlinear evolution of ellipsoidal vortices, in which vortex destruction is confined to the lower vortex and wave activity is seen to propagate downwards. This is in contrast to the Boussinesq limit, which exhibits symmetric destruction of the upper and lower vortex, and in contrast to naive expectations based on a consideration of the mass distribution alone, which would lead to greater destruction of the upper vortex. Finally, the presence of a horizontal lower boundary introduces a strong barotropic component that is absent in the unbounded case (the presence of an upper boundary has almost no effect). The lower boundary also alters the differential rotation in the lower vortex with important consequences for the nonlinear evolution: for very small separation between the lower boundary and the vortex, the differential rotation is reversed leading to strong deformations of the middle vortex; for a critical separation, the vortex is stabilized by the reduction of the differential rotation, and remains coherent over remarkably long times.