Text classification methods have been widely investigated as a way to detect content of low credibility: fake news, social media bots, propaganda, etc. Quite accurate models (likely based on deep neural networks) help in moderating public electronic platforms and often cause content creators to face rejection of their submissions or removal of already published texts. Having the incentive to evade further detection, content creators try to come up with a slightly modified version of the text (known as an attack with an adversarial example) that exploit the weaknesses of classifiers and result in a different output. Here we systematically test the robustness of common text classifiers against available attacking techniques and discover that, indeed, meaning-preserving changes in input text can mislead the models. The approaches we test focus on finding vulnerable spans in text and replacing individual characters or words, taking into account the similarity between the original and replacement content. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on four misinformation detection tasks in an evaluation framework designed to simulate real use cases of content moderation. The attacked tasks include (1) fact checking and detection of (2) hyperpartisan news, (3) propaganda, and (4) rumours. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions, e.g. attacks on GEMMA being up to 27% more successful than those on BERT. Finally, we manually analyse a subset adversarial examples and check what kinds of modifications are used in successful attacks.

Novel prediction methods should always be compared to a baseline to determine their performance. Without this frame of reference, the performance score of a model is basically meaningless. What does it mean when a model achieves an $F_1$ of 0.8 on a test set? A proper baseline is, therefore, required to evaluate the ‘goodness’ of a performance score. Comparing results with the latest state-of-the-art model is usually insightful. However, being state-of-the-art is dynamic, as newer models are continuously developed. Contrary to an advanced model, it is also possible to use a simple dummy classifier. However, the latter model could be beaten too easily, making the comparison less valuable. Furthermore, most existing baselines are stochastic and need to be computed repeatedly to get a reliable expected performance, which could be computationally expensive. We present a universal baseline method for all binary classification models, named the Dutch Draw (DD). This approach weighs simple classifiers and determines the best classifier to use as a baseline. Theoretically, we derive the DD baseline for many commonly used evaluation measures and show that in most situations it reduces to (almost) always predicting either zero or one. Summarizing, the DD baseline is general, as it is applicable to any binary classification problem; simple, as it can be quickly determined without training or parameter tuning; and informative, as insightful conclusions can be drawn from the results. The DD baseline serves two purposes. First, it is a robust and universal baseline that enables comparisons across research papers. Second, it provides a sanity check during the prediction model’s development process. When a model does not outperform the DD baseline, it is a major warning sign.

An international benchmark has been organized by EDF R&D during 2008–2010, gathering 11 partners. Its main objective is to quantify the confidence in numericalmodels. The built-up dynamical system considered is a pump actually in service in powerplants, considered in its work environment. Blind eigenfrequency numerical values relativeto the pump assembly fixed in concrete present a larger variability than the 5% roughvariability concerning the separate parts, essentially due to the modeling of the boundaryconditions and interfaces at macro level. For the nine first eigenmodes considered, experimental-numerical correlation shows a frequency error less than 15% for the freesub-structures and nearly 30% for the free five-component system. Though the first overallmodes are correctly identified, the frequency error is significantly larger for theclamped pump assembly, with however MAC coefficients higher than 0.9 for two modes; butthe frequency error can be reduced to less than 9% for the four first modes after updatingprocedure. A lesson to draw is that measurement information is needed to improve thequality of theoretical built-up structure. After this benchmark, a more ambitious researchprogram will follow as a FUI project, gathering the observation of numerical andexperimental variabilities, updating model improvement and numerical quantification of thetotal uncertainty.

A benchmark is organised to quantify the variability relative to structure dynamicscomputations. The chosen demonstrator is a pump in service in thermal central units, whichis an engineered system with not well-known parameters, considered in its workenvironment. The blind modal characterisation of the separate pump components shows a5%–12% variability on eigenfrequency values and a less than 15% frequency error incomparison with experimental values. The numerical-experimental MAC numbers reach 0.7 atthe maximum, even after updating. An example of modal results on the pump assembly fixedis presented, which shows a larger discrepancy with measurement values, essentially due tothe modelling of the interfaces and boundary condition, and to the possible simplificationof the main components F.E. models to reduce their size. Though a significant frequencyerror, the first overall modes are correctly identified. If this tendency can be confirmedfrom all the participants’ results, the conclusion to be drawn is that, if the predictivecapability of F.E. models to represent the dynamical behaviour of sub-structures issatisfactory, the one relative to structures that are built-up of several components doesnot allow their confident use. Additional information issued from measurements is neededto improve their accuracy.

With increasingly thin margins and new technologies, it is important that farm managers know their cost of field operations on a per unit basis (e.g., acre, ton, bale). Accurate per unit costs give confidence when constructing enterprise budgets and evaluating new technologies, such as no-till. Custom rates are often used as a proxy for per unit costs; however, this research, using entropy and jackknife estimation procedures, found that custom rates understate total ownership and operating costs by approximately 25% for an average Kansas farm. Estimates from these models are then used to benchmark actual costs against expected cost.