Given a hyperbolic invariant set of a diffeomorphism on a surface, it is proved that, if the holonomies are sufficiently smooth, then the diffeomorphism on the hyperbolic invariant set is rigid in the sense that it is C$^{1+}$ conjugate to a hyperbolic affine model.