Morphological description based on features of the olive stone, such as its surface and shape, can help to determine an olive cultivar's identity. The description, however, is based on visual examination and is thus affected by the examiner's expertise. Although the eye has the capacity to discern texture and shape, the values that are assigned to score different levels or descriptor states, such as a highly scabrous to smooth surface or a circular to elliptic shape, are categorical values. Studies on scoring methodology have shown that the assignment to categories or classes is problematic. The purpose of the present work was to classify olive cultivars by computer-image analysis of olive stone characteristics using mathematical tools, such as fractal geometry and moments. Fractals were used to extract texture information, and moments were used to extract shape information. The results revealed an overall classification accuracy of more than 90% using a Mahalanobis distance. The fractals and moments calculated for stones from genetically identical trees of the same cultivar did not show any statistically significant differences. As environmentally independent and robust morphological descriptors, both fractals and moments showed potential for accurate and efficient classification of olive cultivars and eventual description of olive diversity.