The state of iron in coal gangue minerals is an important factor in determining the potential for value-added utilization of this solid waste; this is especially true for the coal gangue coming from the Pingshuo open-pit mine in China. The objective of the present study was to characterize the petrological, mineralogical, and chemical states of Fe in the coal gangue from the Carboniferous Taiyuan Formation. Methods used included polarizing microscopy, X-ray diffraction (XRD), scanning electron microscopy–energy-dispersive spectroscopy (SEM–EDS), X-ray fluorescence, micro-Fourier-transform infrared (micro-FTIR) spectroscopy, and Mössbauer spectroscopy. The coal gangues are mudstones, silty mudstones, and pelitic siltstones, which are composed primarily of kaolinite, quartz, feldspar, pyrite, illite, and magnesite. In coal gangue, the Fe was found to occur in ferruginous minerals, in crystal-lattice substitutions, or in a colloidal state. The ferruginous minerals in the coal gangue are pyrite and marcasite, and the pyrite morphologies are framboidal, euhedral octahedral crystals, subhedral granular crystals, and irregular crystals. The results of SEM–EDS and micro-FTIR confirmed that the lattice substitution of Fe in the coal gangue minerals occurred mainly in kaolinite, resulting in two types of kaolinite: iron-containing and iron-free kaolinite. The former may be transformed from volcanic biotite and the latter from volcanic feldspar. The Mössbauer spectra of kaolinite showed intense doublets with isomer shift and quadrupole splitting values consistent with tetrahedrally coordinated Fe3+ and ocahedrally coordinated Fe2+, suggesting the presence of two types of substitution sites: (1) Fe2+ replacing Al3+ in the octahedral sheet; and (2) Fe3+ replacing Si4+ in the tetrahedral sheet. This study has important theoretical significance for the high-value utilization of coal gangue.