The effects of the farming system (conventional-organic-abandoned) and soil management (native cover crop vs tillage) on vascular plant species were analyzed in sloping olive groves (>20%) in 20 different locations in Andalusia, SE Spain. The soil management techniques included Organic Tillage (OT), Organic Cover Crops (OC), Conventional Tillage (CT), Conventional Non-Tillage (CNT), Abandoned Cover Crops (AC) and Abandoned Woody (AW). Data for the vascular plant species were recorded through three line transects of 30 m with a bar perpendicularly touching every 1 m of the measuring tape. Environmental variables were also recorded at plot level to assess their influence. Dependent variables, such as species abundance, richness and diversity indexes were studied using univariate analysis (one-way ANOVA, Kruskal–Wallis test) while multivariate statistics (ANOSIM, SIMPER, DCA) were used for analyzing the data matrices. We found that the different combinations of farming system and soil management affect biological diversity in terms of individual abundance, plant cover, species richness and diversity, species and family composition. Life forms and species distribution patterns are also affected. The main environmental variables affecting the plant taxa were those related with soil and climate characteristics, slope, olive age and intensive land uses at landscape level, including the percentage of artificial surfaces. The lowest levels of biodiversity (e.g., species richness) were found in the tilled olive groves (CT = 8.1 sp. ± 2.2, OT = 10.0 sp. ± 5.4). Surprisingly, the organic tilled groves (OT) were very poor in species compared to those with native plant cover (OC = 27.9 sp. ± 3.0). The latter, however, showed similar species richness to the abandoned olive groves (AC = 21.2 sp. ± 3.7, AW = 27.2 sp. ± 3.0). Possible solutions for increasingly uncompetitive sloping olive groves include conversion to organic with native plant cover or abandonment for rewilding.