In organic farming the control of perennial weed species, in particular Cirsium arvense, can be a major concern for farmers, especially if there is no regulation through perennial forage production. To test whether the stubble cleaner (SC), an enhanced skim plow (PL), is as effective in the control of C. arvense and other weeds as conventional ploughing and perennial forage production, an organic field experiment was established. Three different tillage/crop rotation systems were compared: an SC system and a PL system, both in a cereal-based crop rotation and an additional PL system in a crop rotation that included a perennial alfalfa–grass mixture (PLALF). In the SC system, tillage was carried out solely with the SC, while in the PL and PLALF systems, ploughing was alternated with chiseling. In the fifth year, each main plot was divided into subplots, and seven different cover crop treatments were integrated into each of the three systems. The effects of the three systems and the cover crop treatments on weed cover and density, weed biomass, and weed diversity in the sixth and seventh year of the experiment are the subjects of this paper. The choice of cover crop species was of minor importance for weed control. The PLALF system was generally more effective in controlling C. arvense than the PL and SC systems. No significant differences between the PL and SC systems regarding the control of C. arvense could be identified in four of five assessments. The SC system had significantly higher total weed density than the PLALF and PL systems in both years. However, the differences in weed emergence between the PL and SC systems diminished until the assessment of weed cover and biomass in the main crops, when no significant differences between these two systems (2012) or no differences at all (2013) could be identified. Species richness was not significantly influenced by the tillage/crop rotation system in both years. Evenness and Shannon–Wiener index were significantly higher in the PLALF and PL systems than in the SC system on most assessment dates in 2012. In 2013 there was no clear trend regarding evenness and Shannon-Wiener index probably due to a hoeing operation.. In conclusion, for weed control, the choice of crop rotation was more important than the choice of tillage method, while for the diversity of the weed community, the choice of tillage method was more important than the crop rotation.