Bone materials are characterized by an astonishing variability and diversity. Still, because of ‘architectural constraints’, their fundamental hierarchical organization or basic building plans remain largely unchanged during biological evolution. These building plans govern the mechanical interaction of the elementary components of bone (hydroxyapatite, collagen, water; with directly measurable tissue-independent elastic properties), which are here quantified through a multiscale homogenization scheme delivering effective elastic properties of bone materials: At a scale of 10 nm, long cylindrical collagen molecules, attached to each other at their ends by ∼1.5 nm long crosslinks and hosting intermolecular water inbetween, form a contiguous matrix called wet collagen. At a scale of several hundred nanometers, wet collagen and mineral crystal agglomerations interpenetrate each other, forming the mineralized fibril. At a scale of 5 microns, the solid bone matrix is represented as collagen fibril inclusions embedded in a foam of largely disordered (extrafibrillar) mineral crystals. Remarkably, needle and sphere type representations of disordered minerals deliver quasi-identical mechanical behavior of such extrafibrillar porous polycrystals. At a scale above the ultrastructure lacunae are embedded in extracellular bone matrix, forming the extravascular bone material. Model estimates predicted from tissue-specific composition data agree remarkably well with corresponding stiffness experiments across cortical and trabecular materials, which opens new possibilities in the exploitation of computer tomographic data for nano-to-macro mechanics of bone organs, especially in combination with currently investigated extensions towards damage and failure.