We report an on-going blank-field multi-wavelength deep and wide survey of the Subaru/XMM-Newton Deep Survey Field (SXDF). The SXDF has been the focus of a wide range of multi-wavelength observing programs spanning the X-ray to the radio. These observations cover a large enough area (the initial optical imaging covers $\sim$1.3 deg$^{2}$) and depth ($B=28.2$) that they are not affected by large-scale structures (which exist on tens of Mpc scales) and allow us to study the distribution and evolution of high-$z$ galaxies and AGNs, and thus constrain theories for their formation. Our early results include: i) an indication of the primeval Large Scale Structure (LSS) at z $\sim$ 5.7, ii) an indication for the down-sizing of galaxy formation at z $\sim$ 1, iii) identifications of passively evolving systems, and evidence for early formation and the passive evolution of present-day early-type galaxies, and iv) discovery of a large number of optically obscured QSOs. As for the next step forward - we express our hope to use the next generation optical/IR Extremely Large Telescopes (ELTs) to obtain larger and deeper spectroscopy samples of the high-$z$ objects.