We study the magnetic field structure and evolution for 26 X-ray brightening events in coronal holes and quiet Sun regions, including bright points and jets. We found that all brightening events are associated with bipolar regions and caused by magnetic flux emergence followed by cancellation. The emission fluctuations seen in the X-ray bright points are associated with reoccurring magnetic cancellation in the footpoints. An X-ray jet presents similar magnetic behaviour in the footpoints but its magnetic flux cancellation rate is much higher than in the bright point. Comparing coronal holes and the quiet Sun, we do not find differences in their corresponding magnetic field behavior.