Beam instabilities that arise in the compression zone of a gyrotron oscillator can degrade the beam quality and hence adversely affect the operating characteristics of the device. This paper investigates a class of space-charge instabilities that are related to unstable Bernstein modes. These are investigated both by simulations with a 2½-dimensional fully electromagnetic particle-in-cell code and by solving the linear dispersion equation (obtained with kinetic theory). Use of the code makes it possible to study effects that cannot be taken into account in the linear dispersion relation, such as the effect of static self-fields. The simulation strongly suggests that the instability is convective. The dependence of growth rate and frequency spectrum on beam parameters is calculated both with simulation and the analytical method, and the results show good agreement. A detailed analysis of the electromagnetic fields calculated in the simulation shows that the unstable waves are not (as has frequently been assumed in the past) purely electrostatic.