2.1. Research using audit data

Ferraz and Finan (Reference Ferraz and Finan2008) are the first to use detailed audit data in linking these with the results of elections. They find that corrupt mayors exposed in an SAI audit are less likely to be reelected in Brazil. Audit data from Brazil are also used in Ferraz et al. (Reference Ferraz, Finan and Moreira2012) to study the effects of corruption on education. Multiple studies have utilized the rich audit data from Brazil to study the effects of corruption on accountability, reelection, government performance, and public sentiment (Melo et al., Reference Melo, Pereira and Figueiredo2009; Pereira et al., Reference Pereira, Melo and Figueiredo2009; Pavão, Reference Pavão2018; Boas et al., Reference Boas, Hidalgo and Melo2019). Brazil stands out as a frequent case study because their SAI randomly implement audits, and their reports are publicly accessible with readily available data. Outside of Brazil, Dunning et al. (Reference Dunning, Grossman, Humphreys, Hyde, McIntosh, Nellis, Adida, Arias, Bicalho and Boas2019b) find that across six countries the effects of audit information on elections vary by context (Dunning et al., Reference Dunning, Grossman, Humphreys, Hyde, McIntosh and Nellis2019a). In South Africa, Berliner and Wehner (Reference Berliner and Wehner2021) find that municipal audits can lead to an electoral response when these report poor performance.

Recent literature has also utilized audit data published by the national-level SAI in Mexico. Chong et al. (Reference Chong, De La O, Karlan and Wantchekon2015) use these audit reports to run an experiment exploring whether voters punish corrupt mayors. Arias et al. (Reference Arias, Larreguy, Marshall and Querubín2019) build on this and implement an experiment in Mexico to identify how reporting on audit discrepancies to voters affects their support of candidates. Larreguy et al. (Reference Larreguy, Marshall and Snyder2020) also use national-level audits to explore the effects of corruption on elections and find that information that is widely disseminated through radio and television can hurt the electoral returns of the accused party. In all three cases, the national-level SAI data are readily available on the ASFs online platform for reviewing specific cases. The national-level SAI in Mexico is limited to audits of intergovernmental transfers from the central government to SNGs, but given Mexico has over 2,500 SNGs it is impossible for this institution to audit each and every one of them, instead only auditing a sample of SNGs. In 2021, for example, the ASF of Mexico audited 249 municipalities of 2,471 (Auditoría Superior de la Federación, 2022), leaving more than 90% of local governments outside of that year’s analysis. As noted above, state SAIs are responsible for performing full audits of local governments and thus have significantly better geographic coverage. Each ASE is responsible for disseminating the results of their audits to the general public, but none publish a searchable database of their findings. Instead, their findings are in the annual publications of their audits, with the data and details stored within the text of each document. This information is neither pre-processed nor readily available for use in research which explains why the existing literature has largely ignored it as a source of data. To overcome this limitation, I use natural language processing to extract fiscal data from the text.

2.2. Text-as-data in political science

Recent applications of text-as-data approaches exploit the ever-increasing sources of online archived news for research in the social sciences. Grimmer and Stewart (Reference Grimmer and Stewart2013) utilize text-as-data to understand what actors are involved in achieving policy change and what topics are of interest to voters. Applications of machine learning in generating data are a growing approach in political science (Wilkerson and Casas, Reference Wilkerson and Casas2017). These tools are also being popularized in public administration literature which accelerates the data collection process (Hollibaugh, Reference Hollibaugh2019). In conflict research, text-as-data is also used to track drug trafficking organizations (Osorio and Beltran, Reference Osorio and Beltran2020) and the presence of insurgents (Osorio et al., Reference Osorio, Reyes, Beltran and Ahmadzai2020). In this paper, I leverage natural language processing tools to systematically extract information from audit reports.

3.1. Case study: ASE-Sinaloa

The case study for this paper are the audits produced by the subnational SAI in the state of Sinaloa, Mexico. The ASE-Sinaloa is an excellent pilot because their audit reports are similar and consistent throughout the time period studied which facilitates the systematic extraction of information. Their website publishes historical audit reports from 2002 through the present. Over each time period, the contents and structure of the reports vary given updates in audit procedures and requirements, but the vocabulary and items audited remain the same. For example, in 2002 the audit report simply consists of a recommendation of whether the municipalities finances were in order that excluded any detailed information about specific accounts. By 2010, the content of an audit report became much more detailed, specifying the amounts audited and the location of different funds. Starting in 2012 and through 2016, these audit reports began specifying the total value of pliegos de observaciones or fiscal discrepancies identified through an audit of a given municipality. Each document would summarize the discrepancies and publish their total monetary value. Pliegos do not explicitly imply corruption, but they do signal errors, omissions, or potential misappropriation that may merit a further investigation. The values of pliegos are not reported in any official budgetary databases compiled at any government level in Mexico, this information only exists in the audit report that identified them. If a researcher were interested in studying what percentage of municipal funds was tied to discrepancies, they would have to individually review each audit to determine that amount and then combine this information with official budgetary data.

3.2. Collecting audit documents

Each state in Mexico has its own separate ASE that inspects the finances of the state and municipal governments. All 32 ASEs publish their audits through their online portals, but only a few publish all their historical audits. I reviewed the website of each ASE and identified the temporal coverage of the audits they do publish. Table 1 breaks down the years covered and the number of state ASEs within each category of coverage. Of the 32, only 9 publish more than 10 years of audits while the majority had less than 5 years published on their website. I had initially placed an access to information request to obtain the audits from ASEs that did not publish their entire historical collection online through Mexico’s National Institute for Access to Information (INAI in Spanish), but these were temporarily put on hold as the world entered various lockdowns caused by COVID-19. Given I did not have complete coverage of all ASEs, I decided to focus on Sinaloa because their information was readily available but also because of my particular familiarity with the contents of their audits and the budgets of municipalities in this state.Footnote ³ The ASE-Sinaloa website publishes all of their audits in scanned PDF files that are simple to download. As the first case study, it offers an opportunity for identifying the procedures and content review required to extend extraction of fiscal data to audits produced in other states that also publish a collection of their historical audits.

Table 1. Number of ASEs that publish historical audits

The collection of audit documents was downloaded from the ASE-Sinaloa website through a web crawler using Scrapy in Python. The crawler searched for all PDF documents throughout its website and downloaded them, this process collected a total of 3,759 documents. Many of the documents contained legal information on the organization and for specific audits, others contained appendixes and supporting materials. I subset the actual audit reports published for each municipality on a biannual basis from 2008 through 2016. This reduces our collection to 321 total audit reports that cover Sinaloa’s 18 municipalities over a 9 year period. Unfortunately, each audit report is a scanned copy of the official document that contains signatures, stamps, and alterations to the document made during its progression through the state legislature as it was reviewed and approved. Contrary to traditional PDF documents where a user can search through the text, scanned images are much more difficult to process because the items on a page are saved as pictures rather than words. I overcome this limitation by systematically pre-processing each document using optical character recognition (OCR) software textract and tesseract in Python to convert each PDF into a machine-readable plain text file.

The documents are written in Spanish and the OCR software struggled to recognize various accented characters. For example, the character ó was frequently confused with a 6 which highlights some of the existing limitations with using OCR outside of the English language. To correct for these errors, I identified a list of commonly misspelled words across the documents and generated a dictionary of their correct spelling, and then I programmed a function that read through each document and replaced misspelled words, this helped standardize the errors produced by the OCR and reduced the amount of noise in the text.

The audit reports all follow the same structure, they use the same language, and all contain the content of interest. They also use the same boilerplate language to specify the legal standing of their audits that repeat a description of the procedures used in applying the audit, and of other organizational information that is redundant across all the documents. Figure 1 shows a sample boilerplate paragraph identified in the audit reports. This is their opening paragraph specifying that article 53 of the state’s Constitution establishes the legal obligation for the ASE to review the public accounts of the state government and of the municipalities.

Figure 1. Boilerplate paragraph example.

Buried within the boilerplate content and legal terminology is the actual result of the audit, specifically the value of discrepancies identified in the audit. Given the abundance of noise in these documents, it is necessary to filter out the non-relevant text from each document. In order to filter the irrelevant chunks of text, I split each document into paragraphs so that I can distinguish between sections of text that contain important information related to the results of the audit and those that simply contain descriptions of the legal requirements they followed, details on how and when the audit was applied, and other information not relevant to my pre-defined classification criteria. To each paragraph, I apply a cleaning function that eliminates white space, rogue characters, and other errors introduced through the OCR conversion. This produces over 28,000 paragraphs across the 321 documents, the high volume of paragraphs further highlights the challenge of reviewing each document.

3.3. Text classification

After having split each document into paragraphs, it is now possible to create a text classification algorithm that can read the contents of each paragraph and determine the likelihood of it being relevant or non-relevant. Reviewing all of the 28,000 paragraphs would be a time-intensive task that might produce inconsistent results based on the reviewer’s attention, fatigue, or disinterest with the topic as the number of paragraphs read increases. Instead, I use supervised machine learning to train a text classification model that can categorize the population of paragraphs based on the structure, contents, and similarities shared with a sample of paragraphs reviewed by a human. The supervised approach requires that the individual reviewer follows consistent classification criteria throughout the sample of articles read, and to review a sample of paragraphs that contain a wide range of potential paragraphs. This method allows a model to make inferences about all the documents based on the contents of a sample reviewed in detail and produces more consistent results.

From the 28,000 documents, I extracted a random sample of 1,300 paragraphs to use as the training data for the text classification model. I relied on the text annotation software Prodigy in Python that offers a friendly user interface where the reviewer can read the text and assign a label to each paragraph. In Prodigy, I read the individual paragraphs and labeled them as relevant or non-relevant based on the following criteria: if a paragraph mentioned a type of discrepancy, the approval of a specific line item, the monetary value of a discrepancy, or any other mentions of specific results of the audit, then these were classified as relevant, all else was classified as non-relevant since they are not of interest to the research objective of this paper. This differs from a string matching method where the researcher can search within the text for keywords or paragraphs that contain numbers. In such an approach, one would have to supply the classification algorithm with a robust and comprehensive set of rules to match on that will miss variations in spelling or formatting not covered in the rules. A text classification model uses machine learning to learn what variations of word combinations, variations in spelling, and particularities in paragraph formatting to determine the probability of a paragraph classified to either label. The advantage of it is that the model will learn what features are important without the user explicitly stating them, this minimal input is then used to generate out-of-sample predictions on paragraphs it has not seen. Given the OCR process introduced a lot of noise into the text, a machine learning text classifier adapts better to the variation in the spelling and structure of words rather than relying on a finite list of terms to match on.

The main limitation of machine learning is that it requires a large corpus and a diverse set of training data for it to learn the variety of features within each label. A typical text classification model would require thousands of observations for it to produce reliable results, but Prodigy contains a set of internal tools that accelerate the production of an accurate model. For this project, I used the “model-in-the-loop” feature that learns which paragraphs are most likely to be informative to a classification model. It focuses on examples that are the most ambiguous while avoiding the reclassification of items that it already understands. For example, all of the audits started with the same paragraph specifying the legal standing of the audit that contained no relevant information for the project, after the third time reading this paragraph the internal model understood that this content was very likely to be classified as not relevant and thus stopped suggesting it in the training data and default classifying all of the same examples as not relevant. It also understood that paragraphs that made any mention of monetary values were very likely to be classified as relevant and stopped suggesting them. Instead, the interface provided me with the paragraphs it was the least certain about, specifically the most ambiguous and fringe cases contained in the sample that it would learn the most from. This allowed me to focus on paragraphs that contained unique information or examples that were less common across the documents and reduced the classification time from weeks to a few hours. The result is 800 paragraphs labeled as non-relevant and 482 as relevant. The annotated paragraphs serve as training data for a logistic regression classification model that produces a combined F1 score of 0.91 for both labels.

I use the logistic regression model to generate predictions on all of the 28,000 paragraphs. The model reviews the contents of each paragraph and estimates the probable label for each based on the features it learned from the training data. The audits from the ASE-Sinaloa were very similar across time and this made the implementation of a classification model fairly simple. In general, the model performed well on the boilerplate content but struggled with paragraphs that contained overlapping information of boilerplate and relevant data. Fortunately, these were most often classified as false positives rather than false negatives which implies the classification over-identified relevant paragraphs rather than over-excluding them from the final output. The text classification model when implemented on the full population of documents allowed me to reduce the 28,000 paragraphs across 321 audits to 6,517 paragraphs that contained relevant information.

3.4. Information extraction

Although the text classification process filtered out thousands of paragraphs and over a million non-relevant words, the product is still a collection of 6,517 paragraphs that would be challenging to manually review. The audits may include information on specific line items of a municipalities spending, references to purchases procured, and a disaggregated explanation of each discrepancy identified. It may also contain information on items that were not purchased, excess in tax collections or fees, and other information that contains a monetary value. This information is important, but for the purposes of this paper, I focus on a single item that summarizes the findings of the audit to showcase the usefulness of these documents and to present a method for extracting information that can be extended to the rest of the data. My objective is to identify the total value of all discrepancies reported in the audit and extracting that information from the text.

To extract the monetary value of the sum of discrepancies, I implement an additional tool from natural language processing: named entity recognition (NER). The text classifier assigned labels to an entire paragraph, a NER model reviews each word and the contextual information surrounding it to classify entities such as locations, nouns, specific terms, and monetary values. The entities I am searching for are unique to this context and thus require a NER trained from scratch. I again rely on Prodigy to build a NER model that can identify the specific entities that I am searching for. The first are mentions of pliegos de observaciones which translates to a “folder of observations.” Pliegos are all of the discrepancies identified in the audit and are generally followed by the monetary value of their sum. This is the actual amount of monies tied to the discrepancies, together they are the specific information I am interested in extracting.

Using the Prodigy interface, I annotate a random sample of 150 paragraphs from the 6,517 classified as relevant to create the training data required by the NER model. Figure 2 presents an example of the annotation interface along with a paragraph I annotated with the labels of interest. The paragraph states that after comparing the approved budget with the actual amount spent, there is a variation that totals a specific amount. Simply searching for monetary values would return a long list of amounts without any contextual information, that is why I have trained the NER model to identify three specific items: monetary values, a mention of a discrepancy, or a mention of a budgetary item recommended for congressional approval. Prodigy specifies the label assigned to each annotated entity, in this example next to the highlighted amounts the label of “MONEY” appears, and next to the word variacion the label “DISCREPANCY” is tagged. Although this example is specific to the items I am particularly interested in, it could easily be extended to the identification of other key terms and values across the documents, it would require the user to annotate additional paragraphs that contain the information the user is searching for. This opens the possibility of identifying payments to companies or individuals, the procurement procedures followed, the amount of unspent sums, excess taxes collected, or other items of interest to researchers and policymakers. The user could elect to build a NER model from scratch or to simply update the model trained here with the additional observations, reducing the time and effort required to extract information significantly.

Figure 2. NER annotation in Prodigy.

The 150 annotated paragraphs are then used to train the NER model using Spacy in Python. The NER model returned a 0.93 F1 score which suggests it is performing well at classifying the relevant entities. I reviewed a subset of individual paragraphs and confirmed that the model was identifying words associated with a discrepancy and monetary values; therefore, I used this model on the 6,517 paragraphs. The NER model read each paragraph and identified the entities of interest based on the information it learned in the training step.

The NER model returns a list of the entities and their tag for each paragraph, specifically, it found mentions of pliegos followed by their monetary values and extracted that information. Figure 3 demonstrates how the NER model highlights and tags each entity it identified. In this paragraph, the document is summarizing that the total value of discrepancies found in the audit amounted to $2,886,965.33 MXN. From the list of entities identified by the NER model, I filter for mentions of pliegos followed by their monetary values to produce a dataset with 125 observations where each row represents the total value of discrepancies reported in the audit of a municipality for a particular year.

Figure 3. NER identification in paragraphs.

The use of natural language processing and machine learning facilitates the extraction of information contained deep within the contents of a scanned audit report. I converted 321 scanned documents into machine-readable text and then split these documents into over 28,000 paragraphs. I train a text classification model to differentiate between paragraphs that contain relevant information from those that do not, resulting in 6,517 potentially relevant paragraphs. I extract the total monetary value of all discrepancies reported in an audit using NER and the final product is a dataset with 125 observations that can be used for additional analysis. This pipeline and the models developed can be applied to other cases with minimal user input. For audits produced by other state ASEs, the user would need to update the classification model and the NER model on a random sample of paragraphs extracted from those documents so that the model would recognize their particular nuances. This process can be scaled to a much larger collection of audits, or it can be applied to more recent audits published by the ASE-Sinaloa to continue extracting data on discrepancies. This approach can be extended to countries that do not publish their audits in Spanish, the user can follow the pipeline of splitting documents into paragraphs, training a classifier, and then a NER model to identify the specific information they are in search of.