We report the structure and physical properties of two quasi-two-dimensional triangular antiferromagnetic materials, Co0.66Al2Se3.53 and Ni0.61Al2Se3.55, which show highly magnetically frustrated characters. Powder X-ray diffractions demonstrate that Co0.66Al2Se3.53 and Ni0.61Al2Se3.55 possess identical space group of P-3m1 with lattice parameters a = 3.8089(1) Å, c = 12.676(1) Å and a = 3.7880(1) Å, c = 12.650(1) Å, respectively. Analyzing the susceptibility data of Co0.66Al2Se3.53 reveal a Curie Weiss temperature of −216 K, and a spin-freezing transition temperature of 4.5 K, giving a frustration index f = −θcw/Tf ≈ 48. Ni0.61Al2Se3.55 possesses an effective moment of 2.38 µB, a Curie–Weiss temperature of −62 K with no sign of spin-freezing transition down to 2 K. The AC susceptibility data of Co0.66Al2Se3.53 suggest a spin glass-like transition, but no intersite mixing between Co2+ and Al3+ was observed from the X-ray photoelectron spectroscopy measurements.

The low temperature properties of double exchange model in triangular lattice are investigated via truncated polynomial expansion method (TPEM), which reduces the computational complexity and enables parallel computation. We found that for the half-filling case a stable 120° spin configuration phase occurs owing to the frustration of triangular lattice and is further stabilized by antiferromagnetic (AF) su-perexchange interaction, while a transition between a stable ferromagnetic (FM) phase and a unique flux phase with small finite-size effect is induced by AF superexchange interaction for the quarter-filling case.