Transformer-based neural language models achieve state-of-the-art performance on various natural language processing tasks. However, an open question is the extent to which these models rely on word-order/syntactic or word co-occurrence/topic-based information when processing natural language. This work contributes to this debate by addressing the question of whether these models primarily use topic as a signal, by exploring the relationship between Transformer-based models’ (BERT and RoBERTa’s) performance on a range of probing tasks in English, from simple lexical tasks such as sentence length prediction to complex semantic tasks such as idiom token identification, and the sensitivity of these tasks to the topic information. To this end, we propose a novel probing method which we call topic-aware probing. Our initial results indicate that Transformer-based models encode both topic and non-topic information in their intermediate layers, but also that the facility of these models to distinguish idiomatic usage is primarily based on their ability to identify and encode topic. Furthermore, our analysis of these models’ performance on other standard probing tasks suggests that tasks that are relatively insensitive to the topic information are also tasks that are relatively difficult for these models.

This paper demonstrates workflows to incorporate text data into actuarial classification and regression tasks. The main focus is on methods employing transformer-based models. A dataset of car accident descriptions with an average length of 400 words, available in English and German, and a dataset with short property insurance claims descriptions, are used to demonstrate these techniques. The case studies tackle challenges related to a multilingual setting and long input sequences. They also show ways to interpret model output and to assess and improve model performance, by fine-tuning the models to the domain of application or to a specific prediction task. Finally, the paper provides practical approaches to handle classification tasks in situations with no or only few labelled data. The results achieved by using the language-understanding skills of off-the-shelf natural language processing (NLP) models with only minimal pre-processing and fine-tuning clearly demonstrate the power of transfer learning for practical applications.

Enterprise social network messaging sites are becoming increasingly popular for team communication in engineering and product design. These digital communication platforms capture detailed messages between members of the design team and are an appealing data set for researchers who seek to better understand communication in design. This exploratory study investigates whether we can use enterprise social network messages to model communication patterns throughout the product design process. We apply short text topic modelling (STTM) to a data set comprising 250,000 messages sent by 32 teams enrolled in a 3-month intensive product design course. Many researchers describe the engineering design process as a series of convergent and divergent thinking stages, such as the popular double diamond model, and we use this theory as a case study in this work. Quantitative and qualitative analysis of STTM results reveals several trends, such as it is indeed possible to see evidence of cyclical convergence and divergence of topics in team communication; within the convergence–divergence pattern, strong teams have fewer topics in their topic models than weaker teams; and teams show characteristics of product, project, course, and other themes within each topic. We provide evidence that the analysis of enterprise social networking messages, with advanced topic modelling techniques, can uncover insights into design processes and can identify the communication patterns of successful teams.

This paper aims to address an important yet under-studied issue – how does violence from the side of the protestors affect overseas support for a democratic movement? The importance of this question is twofold. First, while violence and radicalization are not exactly unfamiliar territories for scholars of contentious politics, they do not receive as much attention when their effects spill beyond the domestic arenas. Second, this study seeks to examine international solidarity with democratic movements at the civil society level, which differs substantially from the conventional elite-centric approach when it comes to the intersection between democratization and international relations. Against this backdrop, this paper considers the relationship between violent tactics employed by the protestors during the anti-extradition movement and the sentiment expressed by people elsewhere towards the protests. To this end, a total of 9,659,770 tweets were extracted using Twitter Application Programming Interface during the period of 1 June 2019–31 January 2020. Leveraging computational methods such as topic modelling and sentiment analysis, findings in this paper demonstrate that a majority of foreign Twitter users were supportive of the protestors while held relatively negative sentiments against the government as well as the police. In addition, this study reveals that, broadly speaking, violence might cost a democratic movement by its international support, but could also garner more attention at times. Despite its restricted scope, this paper hopefully will shed some useful light on the dynamics underlying international solidarity for a democratic movement abroad as well as the complex mechanisms of interactions between people who protest at home and those who observe from overseas.

European Union legislators, CJEU judges and EU law scholars have produced streams of texts which determine both what EU law is and how it is perceived. We explore what these distinct “voices” tell us about the EU’s legal and policy priorities using a mega corpus compiling more than 200,000 legislative acts, 55,000 court rulings and opinions, and 4,000 articles from a leading EU law journal. Applying an unsupervised machine learning technique known as probabilistic topic modelling, we find that economic integration remains the focus of EU law, but that scholars tend to emphasize rights issues more and ignore certain topics, such as farming regulations, almost entirely. The relationship among these partly interdependent, partly autonomous voices, we suggest, can be conceptualized in terms of co-evolution. Legislation influences issue attention on the CJEU, which, in turn, influences what law professors choose to write about.

Competition in the US Congress has been characterised along a single, left-right ideological dimension. We challenge this characterisation by showing that the content of legislation has far more predictive power than alternative measures, most notably legislators’ ideological positions derived from scaling roll call votes. Using a machine learning approach, we identify a topic model for final passage votes in the 111th through the 113th House of Representatives and conduct out-of-sample tests to evaluate the predictive power of bill topics relative to other measures. We find that bill topics and congressional committees are important for predicting roll call votes but that other variables, including member ideology, lack predictive power. These findings raise serious doubts about the claim that congressional politics can be boiled down to competition along a single left-right continuum and shed new light on the debate about levels of polarisation in Congress.

The Engineering Design field is growing fast and so is growing the number of sub-fields that are bringing value to researchers that are working in this context. From psychology to neurosciences, from mathematics to machine learning, everyday scholars and practitioners produce new knowledge of potential interest for designers.

This leads to complications in the researchers’ aims who want to quickly and easily find literature on a specific topic among a large number of scientific publications or want to effectively position a new research.

In the present paper, we address this problem by using state of the art text mining techniques on a large corpus of Engineering Design related documents. In particular, a topic modelling technique is applied to all the papers published in the ICED proceedings from 2003 to 2017 (3,129 documents) in order to find the main subtopics of Engineering Design. Finally, we analyzed the trends of these topics over time, to give a bird-eye view of how the Engineering Design field is evolving.

The results offer a clear and bottom-up picture of what Engineering design is and how the interest of researchers in different topics has changed over time.

Why do issues “fade” from the problem stream? This is an important but underresearched question, which this article examines by looking at the dynamic interaction between frames and frame sponsors. We develop a novel methodological approach that combines algorithmic coding (topic modelling) with hand-coding to track changes in the presence of frames and frame sponsors during periods of intense problematisation (“problem windows”) both within continuous contexts and diachronically across different contexts. We apply this approach empirically in a corpus of newspaper articles that pertain to the coal seam gas controversy in Australia – a divisive policy issue where frame conflicts are common. We find that elite actors have a particularly decisive impact on the problem stream in terms of both the evolution and duration of debate. Further, problem windows close in response to three different mechanisms: elite frame convergence; public statements (by government and industry); and elections.

Despite a near unanimous agreement that human trafficking is a morally reprehensible practice, there is confusion around what qualifies as human trafficking in the United States. Adopting a mixed-method strategy, we examine how human trafficking is defined by the public; how contemporary (mis)understanding of human trafficking developed; and the public opinion consequence of this (mis)understanding. The definition of human trafficking has evolved over time to become nearly synonymous with slavery; however, we demonstrate that media and anti-trafficking organisations have been focussing their attention on the sexual exploitation of foreign women. We show that general public opinion reflects this skewed attention; the average citizen equates human trafficking with the smuggling of women for sexual slavery. Using a survey experiment, we find that shining light on other facets of human trafficking – the fact that human trafficking is a security problem and a domestic issue – can increase public response to the issue.