Developing Computer Vision and Machine Learning Strategies to Unlock Government-created Records

Developing Computer Vision and Machine Learning Strategies to Unlock Government-created Records | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Developing Computer Vision and Machine Learning Strategies to Unlock Government-created Records Greg Jansen, Richard Marciano This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5105914/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This work explores the development of AI and ML computer vision techniques to unlock digitized handwritten US Census records from the 1950s, which includes over 6.5 million images and was only recently made available to the public on April 1, 2022, following a 72-year access restriction period. The 1950 Census offers a unique window "into one of the most transformative periods in modern American history, revealing a country of roughly 151 million people who had just recently emerged from the hardships and uncertainties of World War II and the Great Depression." ( census.gov ). This computer vision and machine learning work is part of a larger case study based in Sacramento, California focusing on creating a so-far unseen window into the fate of the Japanese American community. Sacramento once housed the fourth largest Japantown community on the West Coast and saw its community forced out twice in a decade: in 1942 during WWII Japanese American Incarceration (the largest single forced relocation in US history), and in 1954 during urban renewal where Japantown residential and business districts were leveled. Our project uses AI-based computational treatments to help recover and memorialize the history of the erased Sacramento Japantown. Moreover, we contrast these findings with the processing of the 1940 Census (released in 2012), thus producing a novel "before-and-after" representation. We demonstrate a workflow for extracting demographic information using image segmentation, computer vision techniques, and deep learning for handwritten character recognition. These techniques are generalizable to other cities, states, and communities, and demonstrate AI-assisted strategies to unlock vital demographic information. The approach highlights the potential benefits of computational techniques on social justice issues. The workflow represents an AI-assisted filtering process for Census records, with a user interface for computationally driven page review. The goal is to automate the culling of pages to select a smaller subset of pages that can then be further targeted or crowdsourced. Computer Vision Machine Learning Artificial Intelligence 1950 U.S. Census records Sacramento WWII Japanese American incarceration Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 1. Background: Japanese American WWII Incarceration 1.1 Japanese American Incarceration In 1942 a network of 10 incarceration camps was created from California to Arkansas. Over 120,000 civilians of Japanese ancestry, two-thirds of whom were U.S. citizens, were deported into incarceration camps between 1942 and 1946. Major federal records associated with the War Relocation Authority (WRA), the agency established to handle the forced relocation and detention of Japanese Americans during World War II, include: (1) The “Japanese American Internee Data File, 1942 – 1946” , with camp intake records of evacuated Japanese Americans, also known as WRA Form 26. (2) The “Final Accountability Rosters of Evacuees at Relocation Centers, 1944-1946 , also known as FAR, with camp outtake records of evacuees at the time of their final release or transfer. and (3) The National Archives “Internal Security Case Reports, 1941 - 1947” , a very significant WRA records series, comprising narrative reports prepared by camp investigators, police officers, and directors of internal security, relating cases of alleged “disorderly conduct, rioting, seditious behavior,” etc. at each of the 10 camps, with detailed information on the names and addresses in the camps of the persons involved, the time and place where the alleged incident occurred, an account of what happened, and a statement of the action taken by the investigating officer. Over the last decade, since 2015, the authors of this paper have explored computational treatments of these types of records. Our initiative is called “Computational Thinking to Unlock the Japanese American WWII Camp Experience.” 1 Examples of this work include: (1) Terp Magazine Winter 2023 : Truth in Exile , 2 (2) SAA’s Archival Outlook July/August 2023 : Giving Data Back through Computational Scholarship , 3 with a Jupyter Notebook, 4 (3) NHK World Japan television documentary 2022 : Dear Mr. Collins – 80 Years Since the Japanese American Internment , 5 and (4) Mellon Foundation Sawyer Seminar blog 2019 : Data, Desert Islands, and Digital Dark Ages . 6 This work also includes a number of publications (Beltran et al. 2021), (Marciano et al. 2018), (Marciano et al. 2019), (Underwood et al. 2017). 1.2 Focus on Sacramento In Dec. 2021, the authors of this paper conducted a University of Maryland iSchool undergraduate seminar. 7 One of the final student projects was called: A Tale of the Dislocation of a Neighborhood: from Sacramento to the Camps . This was first project where neighborhood Census data from 1940 was combined with WWII Incarceration data to identify 13 Japanese American families and reveal the stories behind their deportations. 8 Figure 1 locates the 13 families that lived in the southwest corner of Sacramento in 1940 (per the 1940 Census). The map highlights a triangular 1950 Census enumeration district in orange, ED 70-150, and suggests a spatial methodology for using both Census surveys, to snapshot community members before and after Incarceration. This paper features a case study based in Sacramento that generalizes the Dec. 2021 study to all neighborhoods, focusing on creating a so-far unseen window into the fate of the Japanese American community. Sacramento once housed the fourth largest Japantown community on the West Coast and saw its community forced out twice in a decade: in 1942 during WWII Japanese American Incarceration (the largest single forced relocation in US history), and in 1954 during urban renewal where Japantown residential and business districts were leveled. Our project relies on computer vision and machine learning to help recover and memorialize the history of displaced residents. 2. Census and AI 2.1 On the 1950 Census Our approach to the extraction of information from the 1950s US Census focused on the handwritten population schedules, which recorded the details of individual households and people. (with up to 38 data items collected for all persons including name, address, age, race, gender, marital status, place of birth, and employment). 9 Since our research is entirely concerned with Japanese American households in Sacramento, we decided to create an augmented document review workflow in order to help us target the effort of manual transcription of all Japanese American households. Instead of trying to extract all of the recorded data, we isolate the Race column and, for now, only attempt to detect the presence or absence of the ‘W’ character (which stands for White). Figure 2 shows part of page 32 of a population schedule form for the 1950 Census enumeration district (ED 70-55) of the City of Sacramento, CA. ED 70-55 has a total of 39 pages. The first 5 lines show the handwritten information collected for 2 households: the first household is Japanese (Jap) and spans lines 1 through 3, and the second household is White (W) and spans lines 4 through 5 – as shown by the highlighted red boxes in the Race column. The first household is located at 519 Q St. (as shown in columns 2 and 3) and has 3 family members: (1) Nobujiro SAKO, head of household, age 76, born in Japan, (2) his wife, Shisa, age 57, born in Japan, and (3) his daughter, Nobuko, age 30, born in California, and never married. The second household is located at 1617 5 th St. and has 2 people: (1) Harry COLLIAS, head of household, age 27, born in California, and never married, and (2) Carl SETTLE, lodger, age 28, divorced, born in Missouri. Census population schedules are organized around enumeration districts (EDs). “An enumeration district, 10 as used by the Bureau of the Census, was an area that could be covered by a single enumerator (census taker) in one census period. Enumeration districts varied in size from several city blocks in densely populated urban areas to an entire county in sparsely populated rural areas”. Figure 3 shows part of the 1950 Sacramento ED map with its 193 areas. As part of our project, we georeferenced the 1950 ED Map for Sacramento, and vectorized each of the 193 boundaries, thus creating a GIS shapefile layer with 193 polygon features. In Figure 3, we highlight in yellow, the particular ED illustrated in Figure 2, ED 55 (technically ED 70-55 as ED numbers had two parts, where the 70 prefix designated the county and the 55 suffix was for the specific area within the county). ED 55 comprised 5 city blocks (14, 15, 16, 22, and 23). The first household shown in the first 3 lines of the ED 55 page 32 population schedule of Figure 2, is located on Q St. at the bottom, and the second household (lines 4 through 5) is within ED 55 on 5 th St in the middle. 2.2 On the Emerging Use of AI One of the earlier uses of computer vision and AI to automatically extract handwritten data from the population schedules was with the 1940 Census data. This work, sponsored by the National Archives and Records Administration (NARA) and carried out by researchers at the NCSA Supercomputing Center (Diesendruck et al. 2012) used a computer vision technique called Word Spotting to index names from nearly 4 million high-resolution scanned ED forms. Traditionally, Census data has been extracted through large-scale crowdsourcing initiatives such as those carried out by FamilySearch.org and Ancestry.com. It is only recently with the 1950 Census released in 2022, that the use of AI has emerged to speed up the extraction process. NARA has used an AI Optical Character Recognition (OCR) text extraction tool to extract the names from the digital images of the census schedules. This is not a trivial process due to the challenges of deciphering handwritten text (illegible handwriting, poor image quality, variations in census taker handwriting styles, etc.). Ancestry.com, in an effort to deliver a searchable index of the 1950 U.S. Census to customers faster than before, announced in 2022 that it would be using “proprietary AI handwriting recognition technology”. The strategy seems to be a combination of automated extraction followed by a traditional crowdsourcing approach to mitigate AI errors. The innovative approach we explore in this paper focuses on the more targeted goal of identifying ‘W’ entries in the Race column, which has led to the emergence of a two-part strategy: (1) the first phase involves the use of computer vision techniques to segment the form, using clues in the form layout to identify all the boxes within the race column (this is an algorithmic approach that was implemented with the OpenCV software package in Python code), (2) the second employs deep learning, using a set of neural networks to analyze images of individual race column boxes, a fairly classic handwritten character recognition (HCR) application of machine learning. For the machine learning program, we used the TensorFlow framework via the Keras (version 3) Python library. The rest of the paper describes our particular data extraction workflow. 3. Data Extraction Workflow In this section, we outline the two-step process of extracting demographic information from the census population schedules. All of the code that implements these steps is explained within Jupyter code notebooks that are publicly available for download and reuse from the CASES (Computational Archival Science Education System) website. 11 There are several code notebooks that walk the reader through the entire process step-by-step, including the download of census data and handwritten character training data. In addition, a recent August 2024 panel at the annual meeting of the Society of American Archivists (SAA) on “The Impact of AI on the Future of Archival Work,” featured a recording of this data extraction workflow by Greg Jansen. 12 The process starts with the high-resolution JPEG images of census population schedules, as photographed lying in the center of a dark mat. These are full-color image files, but the photographs that were digitized appear to be black and white. The census forms themselves are ragged on some edges, printed very faintly in places, and often photographed on top of a small sheaf of other pages. Whether a result of the scanning, photography, or the original printing, there is a faint horizontal banding visible in each image that is a consistent source of noise. Often faint lines are showing through from printing on the back of the page. These forms and the way the information was recorded were designed for human transcription into tabulating systems. Therefore, they often include handwritten notes that cover entire lines and run across many columns. So even if we can do a perfect job of detecting the demographic box and the code within it, we cannot account for these line notes and numerous other exceptional conditions. Our process reduces the need for human review of Census pages, but it cannot replace it yet due to these inherent challenges that result from the forms and recording techniques. The first phase of image segmentation takes in each page and, after a number of steps, returns a list of coordinates that represent the pixel boundaries of the demographic cells within that page. Using these coordinates, we extract the cell images from the page and perform a few extra computer vision steps that will clean them up for HCR. Following clean-up, the cell images are fed into two different machine learning models that are trained to look at an image and predict the likelihood that the image represents a ‘W’ character. Each prediction is a measure of the confidence the model has that the image can be classified as a ‘W’ character. Based on the confidence produced by two different models, we implement an efficient workflow for human review of any demographic cells that cannot be confidently classified. The workflow allows a person to review at a glance the uncertain demographic cells from up to six census pages at the same time. They may then check a box by any page that warrants later transcription, due to a demographic cell of interest. The review workflow omits any demographic cells that are confidently identified as a ‘W’ character. Now that we have a sense of the overall process, let us explore each step in turn and dive into how these techniques are applied to a tricky feature extraction problem. 3.1 Image Segmentation The image segmentation process was the most painstaking in the whole project. Each census form is a little different and the layouts vary slightly, as we will explore later. Computer vision is a practice that benefits from deep familiarity with diverse techniques and when to apply them to a given problem. We consulted several experts in order to formulate our approach to the census forms and even so, the project stalled at several points when we had to find a better method. Note that the description below will omit many intermediate steps too numerous to include in this overview. For the curious, we recommend consulting the Jupyter notebooks and the complete code. 3.1.1 Resize and Remove Background Mat The first order of business is to remove the border of the image completely, including the darker mat below the page, as well as the edge of the paper page itself or any of the edges of pages below the current page. The black area surrounding each page isn't adding any information that helps us process page contents and will in fact distract algorithms that we want to have focus on the black ink areas. This step relied on a CV technique of shape detection, specifically a morphological transform, to find the largest contiguous white shape in the image, i.e. the page itself. We then create a mask or stencil which includes all of the areas outside of the white page shape, this is a map of which pixels are beyond the page. In the last step, we overwrite all of the pixels included in the mask with the color white. 3.1.2 Threshold to Black and White Pixels In this step, we want to reduce the pixel information in our image from 256 levels of gray-scale information to a simple 2-level image of black and white pixels. In order to do this, we have to decide on a threshold, above which a pixel will be made white and below which a pixel will be made black. We used a CV algorithm, called Otsu’s threshold, to determine a useful threshold for each image individually. We then converted the image to black and white. Finally, we inverted each image, so that you get a page with white ink and a black background. This is done because CV algorithms are generally designed to work on shapes or ink drawn in ones (white) with zero values (black) used to represent empty space. 3.1.3 Thin or “Skeletonize” the Lines This next step is one of several in which we try to align the lines printed on the page with the rows and columns of the pixels in our image. This means bringing the form into alignment with the X and Y axis of the image. This will help immensely with future steps. Skeletonizing is a highly descriptive technical term for an image processing operation that takes a binary image like ours and removes ink from the narrower sides of shapes, thinning them until each shape is only one pixel wide. (Saha et al. 2016). It kind of looks like the bones within the shapes. If you did skeletonize the shape of a person, it would look kind of like bones except that the skull would be reduced to a vertical line, not a solid. Single-pixel width shapes are helpful when we want to distinguish handwritten curvy scribbles from the long straight lines that make up the paper census form. It is another way of removing extraneous information (thickness) from our image in order to perform the next step, which will be detecting long lines and then rotating our image so that those lines are mostly perfectly horizontal or vertical. The OpenCV code to skeletonize a binary (black and white) image is a single line, but we’ve also specified the use of Zhuang Suen’s algorithm to do so, following earlier census work at NCSA that provided the framework for much of the CV work. (Diesendruck et al. 2012) 3.1.4 Detect lines Long, straight lines within the page image with the Hough Lines Transform (HLT) algorithm, which in essence tallies all the lines to which each white pixel might contribute. This technique yields a list of lines that are described as the combination of an offset and an angle. In this case we are only concerned with horizontal lines, so we tell the HLT algorithm to constrain itself to a range of angles within ten degrees or so off the horizontal axis. This saves us some extraneous computation. 3.1.5 Rotate the Image to Horizontal Given all of the long, near-horizontal lines printed in the image, we need to find the proper angle to align them with our image axes, as well as we can. What we can do first is throw away any lines that are outliers, meaning their angles are very different from most of the other lines. In the image, these show up as diagonal. We group our lines by their approximate angle and then discard all but the most numerous group. Taking this representative group of close-to-horizontal angles, we get the average angle. The last step is to then rotate the whole image by this angle, bringing everything into alignment. Now the white pixels for a vertical line in the image will mostly fall into a single pixel column. The same will be true for horizontal lines and pixel rows. Often the angle of rotation is vanishingly small, but across the whole image, this can cause a line to stray into a neighboring pixel row or column. 3.1.6 Fit a Uniform Template to Each Normalized Image (slide and/or magnify) So, we now have an image where the printed form lines, thinned to one pixel wide, align with our image axes. The handwriting on the page is also thinned to a single pixel wide, but it will not align with the image axes generally, as the writing is scattered into many rows and columns. This means that when we add up all of the “ink” with a row, we can see a distinct spike whenever we get to a horizontal rule in the form. The same spikes are visible at each vertical line if you add up the values in the image columns. So, there is a pretty clear signal here for where our lines fall in terms of rows and columns. To restate our goal here, we are looking for the “Race” column in the form, which will be marked by two vertical lines. Each form row, representing a census person, is also marked off by form lines. How do we figure out which spikes represent the lines that we need for image segmentation? The answer is that we need a map of the form lines we are expecting to find. Then we will need to fit our map onto the observed territory in each individual image. In this case, our map is more like a template of expected form line offsets. We can consider each axis in turn, but we will take the vertical lines as an example. In general, the forms share the same template, except that the forms may shift slightly under the camera. Also, the camera distance or magnification can be slightly different. So, if we know where we find the vertical lines in one image, we can expect to find them again in every image after some shifting (translation) and magnification of their expected positions or offsets. Since we have twenty-four vertical form lines, we can use every line at once to confirm evidence of the correct magnification and translation. The brute force process is to simply run through a reasonable range of translations and magnifications, until we find one that puts the spikes, I.E. the most ink, at the expected locations of lines in our template. This is accomplished in two nested loops that compare the ink found at template offsets for each value of magnitude and translation. We use the same process to apply a template of horizontal form lines, but that includes a new twist to be described next. 3.1.7 The Hunt for the Horizontal Line Template At some point, deep in the thorny thicket of computer vision analysis, we found ourselves well and truly lost, with no fitting map of horizontal lines. Our horizontal template was seen to fit well on many of the pages, but on some others, it was fitted quite poorly, obviously out of place and shoddy. It was after some consternation that it finally dawned on us that the printing on some pages was not like the others. A notes field that is printed at the bottom of some forms was inexplicably printed at the top of others. As if to deny us any hope of orderly processing, this devilish Notes field was sometimes large and sometimes small. How then to fit a template to these variably horizontal lines? Do we need to try three or four different templates before finding the one that matches notes field placement and size? We seized upon some hope when we realized that we might try aligning a smaller template consisting of just those horizontal lines that we care about, I.E. the lines that are consistent and can guide us to the correct rows in the population schedule. Using a template that started with the lines in the column headers and ended with the twenty-five lines used to record individuals, we were able to generally get better results. The brute force process of magnification and translation had to explore a much larger range of translation but would usually find a good fit. However, when the fit was wrong, it was generally wrong by one or two printed rows. Since the table rows are equally spaced, this makes some sense, as many different offsets produce significant overlap between template rows and the image. We still needed a better way to home in on the correct fit to get consistent results and reduce the range of solutions being considered. While studiously ignoring the taunts of the ungovernable notes field, we spied a clue that just might work. If you squint at a page image, one line stands out from the rest. Approximately in the middle of the page, there is a heavier line that starts at the top of the table headers and continues on, down to the end of the individual record rows. Sometimes printed faintly or not at all in the lower part of the page, we see that this heavy line is generally printed well at the top of the table, and there, at the top of the heavy line, we have a landmark that we may use to as a starting place for aligning the template. The CV technique we can use to find the heavy line shape is another morphological transform, this time with some parameters that will restrict the shapes identified to be vertically long and wider than most lines. This restriction is imposed through the use of what is called a kernel, a pattern of pixels that must fit within all the identified shapes in one or more positions. We used a kernel that was fifty pixels high and just two pixels wide. Prior to running the morphological transform, we also included a step called “dilation” to specifically repair any small breaks in the heavy vertical line due to misprints, ensuring that we will find it more consistently. Of all the shapes that the morphological transform detects, we choose the one that is vertically the longest. In almost all cases this combination of steps will yield a shape whose upper limit can be used to initially align our small horizontal lines template, before proceeding to the magnification and translation fitting steps. In those cases where the heavy line is not found, we simply fall back to our older, less consistent approach, which is often adequate. 3.1.8 Slice Regions of Interest Using the Adjusted Template Now that the template of form line offsets has been fitted to the territory, the row and column spikes that represent lines in each image, we can use that adjusted template to pick out the particular areas that are of interest to us. In our case, we want to extract images of every cell in the race column. In the template, the Race column is between the 12 th and 13 th vertical lines. Within the smaller horizontal template, the individual record rows begin at the 3 rd line and continue through the 28 th with thirty roughly equal spaces between each row. Since most census recorders seemed to write demographic code on the line, rather than above it, we extended each cell down the page slightly, in order to capture the entire handwritten code. 3.2 Handwritten Character Recognition (HCR) The HCR process operates on individual demographic cell images. It attempts to predict, through two different kinds of machine learning models, whether or not a particular cell contains the ‘W’ character. The models are trained on a large collection of handwritten characters, designated as EMNIST, that were originally collected by the US National Institute of Standards and Technology (NIST) (Cohen et al. 2017). Some subsets of the original NIST data were actually based on handwritten characters extracted from census records. Each training image is 28 by 28 pixels with some gray levels. The NIST recommends that models based on EMNIST work best when the target characters, which must have the same image dimensions, have been centered by their bounding box, instead of their center of mass. If you look at the images above you can see that they all have margins of about the same size all around them. Unfortunately, we were unable to center our demographic cell images by the bounding box and were forced to use the center of mass technique, due to the interference of the horizontal lines on which the characters were written. Below you can observe a typical census demographic cell that was extracted by the previously described CV segmentation technique. In Figure 12, you may see the dashed-looking form line that remains below the ‘W’ character. The white box represents a 28 by 28-pixel area that has been drawn at the center of mass of this image, or as close to that as the margins allow. You can see that a bounding box around just the ‘W’ would yield a different result, especially where you can see the ‘W’ touching the right side of the white box. This bounding box discrepancy was not yet resolved in our project and may have impacted the results. We are working on some additional CV techniques designed to remove the form lines more completely. However, the issue of the form line was addressed in training, as explained below. 3.2.1 Adding “census form noise” to EMNIST Training Images Since we could not find a CV method to remove the form lines from our target images, we decided to do the next best thing and train our machine learning models on images that included similar noisy lines. For that, we would have to modify the EMNIST images before training the models. The TensorFlow framework has some affordances that support this kind of pre-training step on your training data, and we would end up using several. We generated randomized noisy lines and added these to the lower rows of the training images. Other pre-training operations include the translation of pixel values into normalized floating-point numbers (fractions between 0 and 1) and the re-labeling of character classes to a binary, either 1 to indicate a W or 0 indicating not W. Using these two possibilities together as one logical output, we ask the model to produce a single floating-point number between zero and one that indicates the confidence that the character in the image is a ‘W’. Finally, another concern about the training dataset is that the frequency of the ‘W’ class is about one in twenty-six, as the dataset is balanced with equal representation of all letters. This means that the model will learn that ‘W’s are somewhat rare, about as frequent as any other letter. However, in the census data the ‘W’ is by far the most frequent character that we see. We followed an approach in training of setting a calculated initial bias 13 to address this imbalance between the training data and the census data. 3.2.2 Feed-forward Neural Network Setup and Training We define the most straightforward neural network (NN) model first and customize it for our input parameters and the output we want. This model is defined as a sequence that takes in a 28 by 28 tensor array, representing image pixels. The input is first flattened into a 784-unit long tensor array of one dimension (28 times 28). Then each of these input features is connected to each of the units in the subsequent neural network layer. The dense NN layer stores weights associated with each of these connections and those weights are used to determine the value each unit will pass to the next layer in the model. These connection weights are what we adjust during training. An activation formula (ReLU) is supplied to this layer. The default activation formula is "linear", which means that unit output pretty directly reflects the input and connection weights. Our activation formula of ReLU means that unit output never exceeds a certain maximum value and that any unit output below a certain threshold is attenuated or lessened. This privileges the output of units that are strongly weighted and also receiving input. Each unit in the first dense layer is subsequently connected to the single unit in the last layer. The activation there is sigmoid , not ReLU. Sigmoid is an activation function that converts the output value to a number between zero and one, with one representing total model confidence that the image is a ‘W’ character. After we define the neural network layers and activations in the first step, we compile that model, adding an optimizer, a loss function, and a custom function for gathering metrics during training. The loss function compares each prediction to the known class in the training data. The model will be trained over many passes in order to minimize this loss measure. Our loss function is binary, which means that the model is judged on how well it predicts a single value based on the input it receives. After all of this setting up, the fitting or training of the model to the training data is shockingly straightforward. In a single line of code, we can ask for the model to be fit in 15 epochs , which means that the model will pass through the training data fifteen times, adjusting connection weights and improving its accuracy with each pass. We trained the model on a laptop with an Intel Core i7-12800H processor, without connecting TensorFlow to a separate GPU. Even so, the training on all 88,800 images was completed in under five minutes. 3.2.3 Convolutional Neural Network Setup and Training A convolutional neural network (CNN) is one that includes so-called convolved layers that are not densely connected. Instead, connections are made between groups of units, called kernels, that take input from overlapping visual regions in an image. This pattern of bundled neuron connections that divide the visual field is inspired by the organization of the animal visual cortex. While a deeper exploration of CNN organization is beyond the scope of this article, one may find many references on the subject. The model configuration in our TensorFlow code was based on a tutorial we found that was specific to HCR the task. 14 In the setup code above, you can observe the creation of the two additional “convolved” layers along with the specification of their kernel dimensions in pixels. Unlike the first deep learning model, the pixel inputs are not flattened at the beginning into a single dimension but kept in two dimensions such that they can be divided into the overlapping visual regions used by the CNN kernels. When all neurons are connected between two layers, the two-dimensional arrangement of the input image is irrelevant. 4. Results 4.1. Model predictions of a ‘W’ in census records Next, we ran both of our deep learning models on example pages from the 1950s US census and reviewed our results. Census recorders tend to develop a shorthand script for recording codes, as they work quickly to record their data. We saw that ‘W’ characters with cleanly separated line strokes were identified readily enough by both ML models. However, any more casual scribbles would throw off our carefully trained models. In our testing of several hundred census pages, we determined that more than two-thirds of ‘W’ codes could be reliably automatically recognized, eliminating the need for human review of those records. This includes a necessary margin of caution on our part since one could simply accept ‘W’s that were predicted with much lower model confidence levels. 4.2. Applying HCR to transcription workflow In order to apply this technology to filtering census records, we built a rudimentary user interface for page review into another Jupyter notebook. The interface presents six pages of filtered demographic cells for review on each screen. Any recognized ‘W’ codes are omitted, so the user only has to review a smaller subset of the demographic codes. If any page has codes that are interesting for further study or transcription, the user may check a box next to that page and “save the page for later”. A log file gathers these saved pages, which they can return to later. In Fig. 17 you can see a screenshot of the interface with four pages of demographic images shown. For each page around twenty ‘W’ characters have been filtered out, as noted in the page labels. In our interface tests, we found that a deliberate user could review one screen of six census pages in around 11 seconds. That means that one can review each census page for demographic codes in less than two seconds a significant time saving over manual review. One could plausibly expect to complete a review of the 1950s Sacramento metro area, a population of approximately 216,000 people, in four hours with this workflow, not including some necessary breaks to refresh your eyes. 5. Next Steps There are a number of follow-up questions that remain after the research we describe. For one thing, it is without a doubt possible to better tune the images that we send to the predictive models. This includes ongoing work on CV approaches to eliminating the form lines that remain, which seems feasible, given that we know their approximate location and their shape. Some kind of morphological transform can be developed that will dissolve these lines prior to the recognition task. Elimination of the form lines would also make it possible to apply a “bounding box” approach to centering the characters in the 28 by 28 input images with a margin, as recommended in the EMNIST data set notes, bringing our target images into better alignment with the training data. Beyond improving the image preparation steps, there are also newer deep learning models that have been developed, especially transformer models, which may provide a more state-of-the-art approach to the recognition task. In particular, we think that the OCR-free document understanding transformer, known as Donut, offers much to be explored. (Kim et al. 2022 ) Having outlined these opportunities, the first step is to clean up and publish the code notebooks that were used to perform this work so that others can fully explore our work. The code notebooks document in a narrative and specific way the iterative process by which the segmentation, recognition, and user interface tasks were developed, along with the exploration of the difficult puzzles we encountered along the way. 6. Conclusion It is rewarding, but intrinsically challenging, to attempt to apply recent AI technologies to archival materials that were printed or handwritten before the digital era, especially given that the information was originally designed for human transcription and tabulation. The field of machine learning is moving forward by leaps and bounds, yet adoption within archives is constrained by many factors, ranging from ethical concerns to the availability of expertise and resources. We chose in this study not to use general-purpose cloud offerings but rather to evaluate and adapt more traditional computer vision techniques to our specific challenges with historical materials. In many ways to deliver a project like this one feels like dropping a little paper boat into a rushing torrent. The variety of new machine learning models and their many applications are frankly astounding. We are very grateful to be able to explore just a few of these opportunities with regard to the 1950 US Census. Unfortunately, we cannot recommend that a technologist in the archives pursue such projects casually. The preparation of materials was a painstaking process that unfolded over the course of several weeks. It required consultation with outside computer scientists with whom we had unique access. In the meantime, our Japanese American household study proceeded, using more time-consuming manual review and transcription, led by Dr. Marciano and two 2023 LEADING fellows. This covered both the 1940 and 1950 US Census records for Sacramento and stretched over several weeks, in which we also crowdsourced the creation of two Japanese American household datasets. This allowed us to assess the time that would have been saved had the automated review workflow been used, thus, demonstrating its labor-saving potential. Due to the time investment required, these endeavors require a strong commitment to the outcomes and a realistic cost-benefit analysis. In our case, generalizing these techniques to the quite large dataset of every city in the United States in the 1950s, would provide a substantial benefit to related research. Another substantial benefit, we think, is being able to share the code, notebooks, and narrative that explain the development process from beginning to end. We hope that the project can help to further propel the community of practice for AI and archives, giving practitioners more guidance and contacts to consult when evaluating the application of machine learning to archival records. In the long run, we must remember that new automation, when applied with due care and consideration, opens up new opportunities for our own labor (Johnson & Acemoglu 2023 ). We must plan to approach new roles alongside our community of practice that engages innovative forms of computation to enhance the public good. Declarations Author Contribution All authors wrote and reviewed the manuscript. References Beltran, L., Ping O’Brien, E., Jansen, G., Marciano, R. (2021) “A Framework for Unlocking and Linking WWII Japanese American Incarceration Biographical Data” , 2021 IEEE International Conference on Big Data, Dec. 15, 2021, Orlando, FL. See: https://ai-collaboratory.net/wp-content/uploads/2021/11/5_Beltran.pdf Cohen, G., Afshar, S., Tapson, J., & Van Schaik, A. (2017). “EMNIST: Extending MNIST to Handwritten Letters” . In 2017 international joint conference on neural networks (IJCNN) (pp. 2921-2926). IEEE. https://doi.org/10.1109/IJCNN.2017.7966217 Diesendruck, L., Marini, L., Kooper, R., Kejriwal, M., & McHenry, K. (2012). “A Framework to Access Handwritten Information within Large Digitized Paper Pollections” . In 2012 IEEE 8th International Conference on E-Science (pp. 1-10). IEEE. https://doi.org/10.1109/eScience.2012.6404434 Johnson, S., & Acemoglu, D. (2023). “Power and progress: Our thousand-year struggle over technology and prosperity.” Hachette UK. Kim, G. et al. (2022). “OCR-Free Document Understanding Transformer” . In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13688. Springer, Cham. https://arxiv.org/abs/2111.15664 Marciano, R., Lee, M., Underwood, W., Laib, S., Diker, Z. & Singh, A. (2018), “Digital Curation of a World War II Japanese-American Incarceration Camp Collection: Implications for Sociotechnical Archival Systems”, DigitalHeritage2018, San Francisco, Oct. 27, 2018 (part of the Digital Solutions for Heritage Archives & Collections session). See: https://ai-collaboratory.net/wp-content/uploads/2020/04/DigitalHERITAGE_2018_paper_220.pdf Marciano, R., Underwood, W., Hanaee, M., Mullane, C., Singh, A., Tethong, Z. (2019), "Automating the Detection of Personally Identifiable Information (PII) in Japanese-American WWII Incarceration Camp Records", 2018 IEEE International Conference on Big Data, Jan. 24, 2019, pp.2725-2732. See: https://ai-collaboratory.net/wp-content/uploads/2020/03/2.Marciano.pdf Saha, P. K., Borgefors, G., & di Baja, G. S. (2016). “A Survey on Skeletonization Algorithms and their Applications” . Pattern recognition letters , 76 , 3-12. https://doi.org/10.1016/j.patrec.2015.04.006 Underwood, B., Marciano, R., Laib, S., Apgar C., Beteta, L., Falak, W., Gilman, M., Hardcastle, R., Holden, K., Huang, Y., Baasch, D., Ballard, B., Glaser, T., Gray, A., Plummer, L., Diker, Z., Jha, M., Singh, and A., Walanj, N. (2017), “Computational Curation of a Digitized Series of WW11 Japanese-American Internment,” IEEE Big Data 2017’ 2 nd Computational Archival Science (CAS) Workshop, Boston, MA, Dec. 13, 2017. See: https://ai-collaboratory.net/wp-content/uploads/2020/04/Underwood.pdf Footnotes https://ai-collaboratory.net/projects/ct-ja_ww2_camps/ https://terp.umd.edu/truth-in-exile https://mydigitalpublication.com/publication/?m=30305&i=798391&p=12&ver=html5 https://cases.umd.edu/github/cases-umd/Japanese-American-WWII-Incarceration/blob/main/index.ipynb https://ischool.umd.edu/news/aic-umd-student-project-featured-on-japanese-television/ https://medium.com/@ero22/data-desert-islands-and-digital-dark-ages-richard-marciano-on-records-and-data-management-13acac0219a7 https://ai-collaboratory.net/2021/12/14/dec-9-2021-digital-curation-showcase/ https://www.youtube.com/watch?v=XWrG_-O8SY4 https://www.archives.gov/research/census/1950/questions-asked https://www.archives.gov/research/census/1950/ed-maps The CASES website is available at http://cases.umd.edu https://ai-collaboratory.net/2024/08/17/saa-2024-the-impact-of-ai-on-the-future-of-archival-work/ TensorFlow Tutorial on using imbalanced training data: https://www.tensorflow.org/tutorials/structured_data/imbalanced_data https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-from-scratch-for-mnist-handwritten-digit-classification/ Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5105914","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":366805025,"identity":"0333af0d-b17f-443b-b555-48d3814adfaa","order_by":0,"name":"Greg Jansen","email":"","orcid":"","institution":"University of Maryland, College Park","correspondingAuthor":false,"prefix":"","firstName":"Greg","middleName":"","lastName":"Jansen","suffix":""},{"id":366805026,"identity":"d4acbf4c-2984-478f-b6f3-8a240350cd4f","order_by":1,"name":"Richard Marciano","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAtUlEQVRIiWNgGAWjYLCCBCDmB2IJBgZmErRINpCkBQQMDhCrRbf97MMPD2ruyBnfyE68wVBhndhASIvZmXRjiYRjz4zNbuRutmA4k06ElgNpbAwJbIcTt93I3SbB2HaYCC3nnwG1/DucuHkGSMs/YrTcANqSCDR8gwRISwNRWp4xSyT2HTaWOPN2s0XCsXRjIhyWxvjxx7fDcvztuRtvfKixliWoBRUkkKZ8FIyCUTAKRgEuAACiWEF3IoQYJwAAAABJRU5ErkJggg==","orcid":"","institution":"University of Maryland, College Park","correspondingAuthor":true,"prefix":"","firstName":"Richard","middleName":"","lastName":"Marciano","suffix":""}],"badges":[],"createdAt":"2024-09-18 00:27:08","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5105914/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5105914/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":67320220,"identity":"4aa51f79-eb7b-4c71-9a60-fecc5fff746b","added_by":"auto","created_at":"2024-10-23 15:30:15","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":3971418,"visible":true,"origin":"","legend":"\u003cp\u003eMap of the 13 Japanese American families from the southwest corner of Sacramento in the 1940 Census\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/a1c6166bb950135ee5c96b02.png"},{"id":67319025,"identity":"5087a22a-75ba-452f-a65f-d46d7f6261ce","added_by":"auto","created_at":"2024-10-23 15:22:14","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":1164715,"visible":true,"origin":"","legend":"\u003cp\u003eSacramento population schedule No. 70-55 page 32 extract showing 2 households\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/9d67bdeb8ea92fb7ba98454c.jpeg"},{"id":67320223,"identity":"5884c5c9-a6bc-4d93-a748-e6b31cd96400","added_by":"auto","created_at":"2024-10-23 15:30:15","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":1352360,"visible":true,"origin":"","legend":"\u003cp\u003eMap showing ED 55, highlighted in yellow\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/c39c27d296c1c65d35735d85.jpeg"},{"id":67319033,"identity":"a8882113-a6c2-40b9-9dde-3ea9106323a5","added_by":"auto","created_at":"2024-10-23 15:22:14","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":243502,"visible":true,"origin":"","legend":"\u003cp\u003ePage contour shown in green.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/1c41eece8f33bbddd0871619.png"},{"id":67319023,"identity":"8de259d0-1521-4202-9000-82d413403adf","added_by":"auto","created_at":"2024-10-23 15:22:14","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":221539,"visible":true,"origin":"","legend":"\u003cp\u003eInverted image, such that shapes and lines are “ones”.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/312e5ac6368f48bedde532de.png"},{"id":67320892,"identity":"fe04b450-19dc-4123-8876-a69a4cf17dd7","added_by":"auto","created_at":"2024-10-23 15:38:14","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":165756,"visible":true,"origin":"","legend":"\u003cp\u003eSkeletonized image\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/608f8d522b106903778cd859.png"},{"id":67320894,"identity":"1454bb74-9532-48e5-8354-7c3212d335b8","added_by":"auto","created_at":"2024-10-23 15:38:14","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":403207,"visible":true,"origin":"","legend":"\u003cp\u003eHLT detected lines with near horizontals in green, solution in red.\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/7e25da9059dd43a13989e590.png"},{"id":67319038,"identity":"489277d3-2b06-430f-8e07-bc73e9005b8b","added_by":"auto","created_at":"2024-10-23 15:22:15","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":967939,"visible":true,"origin":"","legend":"\u003cp\u003eIn which the notes field defies our template fitting effort.\u003c/p\u003e","description":"","filename":"floatimage8.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/73c9cf5405245c6a001745a2.png"},{"id":67320899,"identity":"ad1b1972-a80e-4a56-b145-f32eb3fd088c","added_by":"auto","created_at":"2024-10-23 15:38:15","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":293432,"visible":true,"origin":"","legend":"\u003cp\u003eMorphological transform with selected shape in red.\u003c/p\u003e","description":"","filename":"floatimage9.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/3028d0d2ce233519973d9863.png"},{"id":67322175,"identity":"5813cb24-70ce-4685-8072-ab2d3fbed5ff","added_by":"auto","created_at":"2024-10-23 15:54:15","extension":"png","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":376225,"visible":true,"origin":"","legend":"\u003cp\u003eMorphological transform with selected shape in red.\u003c/p\u003e","description":"","filename":"floatimage10.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/629bfcc7d46dc573b8d1b98a.png"},{"id":67321434,"identity":"c11af5e6-e06b-4223-88ff-7a2c085379e2","added_by":"auto","created_at":"2024-10-23 15:46:15","extension":"png","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":98676,"visible":true,"origin":"","legend":"\u003cp\u003eA sample of EMNIST training data with class labels.\u003c/p\u003e","description":"","filename":"floatimage11.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/2bf8b27f07f14a966c7e7501.png"},{"id":67320213,"identity":"58ce1f4e-8e53-49cc-bbc5-fed36c3ac131","added_by":"auto","created_at":"2024-10-23 15:30:14","extension":"png","order_by":12,"title":"Figure 12","display":"","copyAsset":false,"role":"figure","size":41570,"visible":true,"origin":"","legend":"\u003cp\u003eA handwritten ‘W’ character with a form line below it.\u003c/p\u003e","description":"","filename":"floatimage12.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/5b6f7bfef4173aba4effccde.png"},{"id":67320893,"identity":"d192eacc-095a-4025-820f-a4b69d5ceff0","added_by":"auto","created_at":"2024-10-23 15:38:14","extension":"png","order_by":13,"title":"Figure 13","display":"","copyAsset":false,"role":"figure","size":112623,"visible":true,"origin":"","legend":"\u003cp\u003eEMNIST characters modified to include noisy lines\u003c/p\u003e","description":"","filename":"floatimage13.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/621dcd50f7bde65fe2807d34.png"},{"id":67320219,"identity":"4128eb06-9ee1-416c-bb4f-2ab7151195ba","added_by":"auto","created_at":"2024-10-23 15:30:15","extension":"png","order_by":14,"title":"Figure 14","display":"","copyAsset":false,"role":"figure","size":30490,"visible":true,"origin":"","legend":"\u003cp\u003eCode defining layers in the first neural network model\u003c/p\u003e","description":"","filename":"14.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/ccb1487e81177ce1a4b945e7.png"},{"id":67320216,"identity":"f3bc1ffb-7104-49e5-9679-3a855532c85a","added_by":"auto","created_at":"2024-10-23 15:30:14","extension":"png","order_by":15,"title":"Figure 15","display":"","copyAsset":false,"role":"figure","size":17904,"visible":true,"origin":"","legend":"\u003cp\u003eCompilation of model with loss function and optimizer.\u003c/p\u003e","description":"","filename":"15.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/d66de8726a040780d88c6e51.png"},{"id":67319028,"identity":"2f96462a-a1ec-42c9-8c41-6e2ce322add2","added_by":"auto","created_at":"2024-10-23 15:22:14","extension":"png","order_by":16,"title":"Figure 16","display":"","copyAsset":false,"role":"figure","size":45240,"visible":true,"origin":"","legend":"\u003cp\u003eSetting up a CNN model\u003c/p\u003e","description":"","filename":"16.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/415b63e6bc11ac23207017db.png"},{"id":67319036,"identity":"2869be0c-051f-4f44-b960-6829c9e39b09","added_by":"auto","created_at":"2024-10-23 15:22:15","extension":"png","order_by":17,"title":"Figure 17","display":"","copyAsset":false,"role":"figure","size":953388,"visible":true,"origin":"","legend":"\u003cp\u003eReview screen shows unrecognized images for four census pages.\u003c/p\u003e","description":"","filename":"17.png","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/6a93ba94e6492b45617b06ec.png"},{"id":67322230,"identity":"d3987d8f-49f7-447e-8ecf-bed47618c979","added_by":"auto","created_at":"2024-10-23 15:54:29","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":11259236,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5105914/v1/80e9fdf8-edfd-45bb-990b-3675df1409b3.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Developing Computer Vision and Machine Learning Strategies to Unlock Government-created Records","fulltext":[{"header":"1. Background: Japanese American WWII Incarceration","content":"\u003ch2\u003e1.1 Japanese American Incarceration\u003c/h2\u003e\n\u003cp\u003eIn 1942 a network of 10 incarceration camps was created from California to Arkansas. Over 120,000 civilians of Japanese ancestry, two-thirds of whom were U.S. citizens, were deported into incarceration camps between 1942 and 1946. Major federal records associated with the War Relocation Authority (WRA), the agency established to handle the forced relocation and detention of Japanese Americans during World War II, include: (1) \u003cstrong\u003eThe \u0026ldquo;Japanese American Internee Data File, 1942 \u0026ndash; 1946\u0026rdquo;\u003c/strong\u003e, with camp intake records of evacuated Japanese Americans, also known as WRA Form 26. (2) \u003cstrong\u003eThe \u0026ldquo;Final Accountability Rosters of Evacuees at Relocation Centers, 1944-1946\u003c/strong\u003e, also known as FAR, with camp outtake records of evacuees at the time of their final release or transfer. and (3) \u003cstrong\u003eThe National Archives \u0026ldquo;Internal Security Case Reports, 1941 - 1947\u0026rdquo;\u003c/strong\u003e, a very significant WRA records series, comprising narrative reports prepared by camp investigators, police officers, and directors of internal security, relating cases of alleged \u0026ldquo;disorderly conduct, rioting, seditious behavior,\u0026rdquo; etc. at each of the 10 camps, with detailed information on the names and addresses in the camps of the persons involved, the time and place where the alleged incident occurred, an account of what happened, and a statement of the action taken by the investigating officer. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eOver the last decade, since 2015, the authors of this paper have explored computational treatments of these types of records. Our initiative is called \u003cem\u003e\u0026ldquo;Computational Thinking to Unlock the Japanese American WWII Camp Experience.\u0026rdquo;\u003ca href=\"#_ftn1\" name=\"_ftnref1\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e\u003c/em\u003e Examples of this work include: (1) \u003cstrong\u003eTerp Magazine Winter 2023\u003c/strong\u003e: \u003cem\u003e\u003cu\u003eTruth in Exile\u003c/u\u003e\u003c/em\u003e,\u003ca href=\"#_ftn2\" name=\"_ftnref2\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e2\u003c/sup\u003e (2) \u003cstrong\u003eSAA\u0026rsquo;s Archival Outlook July/August 2023\u003c/strong\u003e: \u003cem\u003e\u003cu\u003eGiving Data Back through Computational Scholarship\u003c/u\u003e\u003c/em\u003e,\u003ca href=\"#_ftn3\" name=\"_ftnref3\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e3\u003c/sup\u003e with a Jupyter Notebook,\u003ca href=\"#_ftn4\" name=\"_ftnref4\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e4\u003c/sup\u003e (3) \u003cstrong\u003eNHK World Japan television\u003c/strong\u003e \u003cstrong\u003edocumentary 2022\u003c/strong\u003e: \u003cem\u003e\u003cu\u003eDear Mr. Collins \u0026ndash; 80 Years Since the Japanese American Internment\u003c/u\u003e\u003c/em\u003e,\u003ca href=\"#_ftn5\" name=\"_ftnref5\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e5\u003c/sup\u003e and (4) \u003cstrong\u003eMellon Foundation Sawyer Seminar blog\u003c/strong\u003e \u003cstrong\u003e2019\u003c/strong\u003e: \u003cem\u003e\u003cu\u003eData, Desert Islands, and Digital Dark Ages\u003c/u\u003e\u003c/em\u003e.\u003ca href=\"#_ftn6\" name=\"_ftnref6\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e6\u003c/sup\u003e This work also includes a number of publications (Beltran et al. 2021), (Marciano et al. 2018), (Marciano et al. 2019), (Underwood et al. 2017).\u003c/p\u003e\n\u003ch2\u003e1.2 Focus on Sacramento\u003c/h2\u003e\n\u003cp\u003eIn Dec. 2021, the authors of this paper conducted a University of Maryland iSchool undergraduate seminar.\u003ca href=\"#_ftn7\" name=\"_ftnref7\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e7\u003c/sup\u003e One of the final student projects was called: \u003cem\u003e\u003cu\u003eA Tale of the Dislocation of a Neighborhood: from Sacramento to the Camps\u003c/u\u003e.\u0026nbsp;\u003c/em\u003eThis was first project where neighborhood Census data from 1940 was combined with WWII Incarceration data to identify 13 Japanese American families and\u0026nbsp;reveal the stories behind their deportations.\u003ca href=\"#_ftn8\" name=\"_ftnref8\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e8\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eFigure 1 locates the 13 families that lived in the southwest corner of Sacramento in 1940 (per the 1940 Census). The map highlights a triangular 1950 Census enumeration district in orange, ED 70-150, and suggests a spatial methodology for using both Census surveys, to snapshot community members before and after Incarceration.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis paper features a case study based in Sacramento that generalizes the Dec. 2021 study to all neighborhoods, focusing on creating a so-far unseen window into the fate of the Japanese American community. Sacramento once housed the fourth largest Japantown community on the West Coast and saw its community forced out twice in a decade: in 1942 during WWII Japanese American Incarceration (the largest single forced relocation in US history), and in 1954 during urban renewal where Japantown residential and business districts were leveled. Our project relies on computer vision and machine learning to help recover and memorialize the history of displaced residents.\u003c/p\u003e"},{"header":"2. Census and AI","content":"\u003ch2\u003e2.1 On the 1950 Census\u003c/h2\u003e\n\u003cp\u003eOur approach to the extraction of information from the 1950s US Census focused on the handwritten population schedules, which recorded the details of individual households and people. (with up to 38 data items collected for all persons including name, address, age, race, gender, marital status, place of birth, and employment).\u003cu\u003e\u0026nbsp;\u003c/u\u003e\u003ca href=\"#_ftn1\" name=\"_ftnref1\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e9\u003c/sup\u003e Since our research is entirely concerned with Japanese American households in Sacramento, we decided to create an augmented document review workflow in order to help us target the effort of manual transcription of all Japanese American households. Instead of trying to extract all of the recorded data, we isolate the Race column and, for now, only attempt to detect the presence or absence of the \u0026lsquo;W\u0026rsquo; character (which stands for White).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFigure 2 shows part of page 32 of a population schedule form for the 1950 Census enumeration district (ED 70-55) of the City of Sacramento, CA. ED 70-55 has a total of 39 pages. The first 5 lines show the handwritten information collected for 2 households: the first household is Japanese (Jap) and spans lines 1 through 3, and the second household is White (W) and spans lines 4 through 5 \u0026ndash; as shown by the highlighted red boxes in the Race column. The first household is located at \u003cstrong\u003e519 Q St.\u003c/strong\u003e (as shown in columns 2 and 3) and has 3 family members: (1) Nobujiro SAKO, head of household, age 76, born in Japan, (2) his wife, Shisa, age 57, born in Japan, and (3) his daughter, Nobuko, age 30, born in California, and never married. The second household is located at \u003cstrong\u003e1617 5\u003csup\u003eth\u003c/sup\u003e St.\u003c/strong\u003e and has 2 people: (1) Harry COLLIAS, head of household, age 27, born in California, and never married, and (2) Carl SETTLE, lodger, age 28, divorced, born in Missouri.\u003c/p\u003e\n\u003cp\u003eCensus population schedules are organized around enumeration districts (EDs). \u003cem\u003e\u0026ldquo;An enumeration district,\u003csup\u003e10\u003c/sup\u003e\u003c/em\u003e\u003ca href=\"#_ftn2\" name=\"_ftnref2\" title=\"\"\u003e\u003c/a\u003e\u003cem\u003e\u0026nbsp;as used by the Bureau of the Census, was an area that could be covered by a single enumerator (census taker) in one census period. Enumeration districts varied in size from several city blocks in densely populated urban areas to an entire county in sparsely populated rural areas\u0026rdquo;.\u003c/em\u003e Figure 3 shows part of the 1950 Sacramento ED map with its 193 areas. As part of our project, we georeferenced the 1950 ED Map for Sacramento, and vectorized each of the 193 boundaries, thus creating a GIS shapefile layer with 193 polygon features. In Figure 3, we highlight in yellow, the particular ED illustrated in Figure 2, ED 55 \u003cem\u003e(technically ED 70-55 as ED numbers had two parts, where the 70 prefix designated the county and the 55 suffix was for the specific area within the county).\u003c/em\u003e\u0026nbsp; \u0026nbsp;ED 55 comprised 5 city blocks (14, 15, 16, 22, and 23). The first household shown in the first 3 lines of the ED 55 page 32 population schedule of Figure 2, is located on Q St. at the bottom, and the second household (lines 4 through 5) is within ED 55 on 5\u003csup\u003eth\u003c/sup\u003e St in the middle.\u003c/p\u003e\n\u003ch2\u003e2.2 On the Emerging Use of AI\u003c/h2\u003e\n\u003cp\u003eOne of the earlier uses of computer vision and AI to automatically extract handwritten data from the population schedules was with the 1940 Census data. This work, sponsored by the National Archives and Records Administration (NARA) and carried out by researchers at the NCSA Supercomputing Center (Diesendruck et al. 2012) used a computer vision technique called Word Spotting to index names from nearly 4 million high-resolution scanned ED forms. Traditionally, Census data has been extracted through large-scale crowdsourcing initiatives such as those carried out by FamilySearch.org and Ancestry.com. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIt is only recently with the 1950 Census released in 2022, that the use of AI has emerged to speed up the extraction process. NARA has used an AI Optical Character Recognition (OCR) text extraction tool to extract the names from the digital images of the census schedules. This is not a trivial process due to the challenges of deciphering handwritten text (illegible handwriting, poor image quality, variations in census taker handwriting styles, etc.). Ancestry.com, in an effort to deliver a searchable index of the 1950 U.S. Census to customers faster than before, announced in 2022 that it would be using \u0026ldquo;proprietary AI handwriting recognition technology\u0026rdquo;. The strategy seems to be a combination of automated extraction followed by a traditional crowdsourcing approach to mitigate AI errors.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe innovative approach we explore in this paper focuses on the more targeted goal of identifying \u0026lsquo;W\u0026rsquo; entries in the Race column, which has led to the emergence of a two-part strategy: (1) the first phase involves the use of computer vision techniques to segment the form, using clues in the form layout to identify all the boxes within the race column (this is an algorithmic approach that was implemented with the OpenCV software package in Python code), (2) the second employs deep learning, using a set of neural networks to analyze images of individual race column boxes, a fairly classic handwritten character recognition (HCR) application of machine learning. \u0026nbsp;For the machine learning program, we used the TensorFlow framework via the Keras (version 3) Python library. The rest of the paper describes our particular data extraction workflow.\u003c/p\u003e"},{"header":"3. Data Extraction Workflow","content":"\u003cp\u003eIn this section, we outline the two-step process of extracting demographic information from the census population schedules. All of the code that implements these steps is explained within Jupyter code notebooks that are publicly available for download and reuse from the CASES (Computational Archival Science Education System) website.\u003ca href=\"#_ftn1\" name=\"_ftnref1\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e11\u003c/sup\u003e There are several code notebooks that walk the reader through the entire process step-by-step, including the download of census data and handwritten character training data. In addition, a recent August 2024 panel at the annual meeting of the Society of American Archivists (SAA) on \u0026ldquo;The Impact of AI on the Future of Archival Work,\u0026rdquo; featured a recording of this data extraction workflow by Greg Jansen.\u003ca href=\"#_ftn2\" name=\"_ftnref2\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e12\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eThe process starts with the high-resolution JPEG images of census population schedules, as photographed lying in the center of a dark mat. These are full-color image files, but the photographs that were digitized appear to be black and white. The census forms themselves are ragged on some edges, printed very faintly in places, and often photographed on top of a small sheaf of other pages. Whether a result of the scanning, photography, or the original printing, there is a faint horizontal banding visible in each image that is a consistent source of noise. Often faint lines are showing through from printing on the back of the page. These forms and the way the information was recorded were designed for human transcription into tabulating systems. Therefore, they often include handwritten notes that cover entire lines and run across many columns. So even if we can do a perfect job of detecting the demographic box and the code within it, we cannot account for these line notes and numerous other exceptional conditions. Our process reduces the need for human review of Census pages, but it cannot replace it yet due to these inherent challenges that result from the forms and recording techniques.\u003c/p\u003e\n\u003cp\u003eThe first phase of image segmentation takes in each page and, after a number of steps, returns a list of coordinates that represent the pixel boundaries of the demographic cells within that page. Using these coordinates, we extract the cell images from the page and perform a few extra computer vision steps that will clean them up for HCR. Following clean-up, the cell images are fed into two different machine learning models that are trained to look at an image and predict the likelihood that the image represents a \u0026lsquo;W\u0026rsquo; character. Each prediction is a measure of the confidence the model has that the image can be classified as a \u0026lsquo;W\u0026rsquo; character. Based on the confidence produced by two different models, we implement an efficient workflow for human review of any demographic cells that cannot be confidently classified. The workflow allows a person to review at a glance the uncertain demographic cells from up to six census pages at the same time. They may then check a box by any page that warrants later transcription, due to a demographic cell of interest. The review workflow omits any demographic cells that are confidently identified as a \u0026lsquo;W\u0026rsquo; character.\u003c/p\u003e\n\u003cp\u003eNow that we have a sense of the overall process, let us explore each step in turn and dive into how these techniques are applied to a tricky feature extraction problem.\u003c/p\u003e\n\u003ch2\u003e3.1 Image Segmentation\u003c/h2\u003e\n\u003cp\u003eThe image segmentation process was the most painstaking in the whole project. Each census form is a little different and the layouts vary slightly, as we will explore later. Computer vision is a practice that benefits from deep familiarity with diverse techniques and when to apply them to a given problem. We consulted several experts in order to formulate our approach to the census forms and even so, the project stalled at several points when we had to find a better method. Note that the description below will omit many intermediate steps too numerous to include in this overview. For the curious, we recommend consulting the Jupyter notebooks and the complete code.\u003c/p\u003e\n\u003ch3\u003e3.1.1 Resize and Remove Background Mat\u003c/h3\u003e\n\u003cp\u003eThe first order of business is to remove the border of the image completely, including the darker mat below the page, as well as the edge of the paper page itself or any of the edges of pages below the current page. The black area surrounding each page isn\u0026apos;t adding any information that helps us process page contents and will in fact distract algorithms that we want to have focus on the black ink areas. This step relied on a CV technique of shape detection, specifically a morphological transform, to find the largest contiguous white shape in the image, i.e. the page itself. We then create a mask or stencil which includes all of the areas outside of the white page shape, this is a map of which pixels are beyond the page. In the last step, we overwrite all of the pixels included in the mask with the color white.\u003c/p\u003e\n\u003ch3\u003e3.1.2 Threshold to Black and White Pixels\u003c/h3\u003e\n\u003cp\u003eIn this step, we want to reduce the pixel information in our image from 256 levels of gray-scale information to a simple 2-level image of black and white pixels. In order to do this, we have to decide on a threshold, above which a pixel will be made white and below which a pixel will be made black. We used a CV algorithm, called Otsu\u0026rsquo;s threshold, to determine a useful threshold for each image individually. We then converted the image to black and white. Finally, we inverted each image, so that you get a page with white ink and a black background. This is done because CV algorithms are generally designed to work on shapes or ink drawn in ones (white) with zero values (black) used to represent empty space.\u003c/p\u003e\n\u003ch3\u003e3.1.3 Thin or \u0026ldquo;Skeletonize\u0026rdquo; the Lines\u003c/h3\u003e\n\u003cp\u003eThis next step is one of several in which we try to align the lines printed on the page with the rows and columns of the pixels in our image. This means bringing the form into alignment with the X and Y axis of the image. This will help immensely with future steps.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSkeletonizing is a highly descriptive technical term for an image processing operation that takes a binary image like ours and removes ink from the narrower sides of shapes, thinning them until each shape is only one pixel wide. (Saha et al. 2016). It kind of looks like the bones within the shapes. If you did skeletonize the shape of a person, it would look kind of like bones except that the skull would be reduced to a vertical line, not a solid.\u003c/p\u003e\n\u003cp\u003eSingle-pixel width shapes are helpful when we want to distinguish handwritten curvy scribbles from the long straight lines that make up the paper census form. It is another way of removing extraneous information (thickness) from our image in order to perform the next step, which will be detecting long lines and then rotating our image so that those lines are mostly perfectly horizontal or vertical. The OpenCV code to skeletonize a binary (black and white) image is a single line, but we\u0026rsquo;ve also specified the use of Zhuang Suen\u0026rsquo;s algorithm to do so, following earlier census work at NCSA that provided the framework for much of the CV work. (Diesendruck et al. 2012)\u003c/p\u003e\n\u003ch3\u003e3.1.4 Detect lines\u003c/h3\u003e\n\u003cp\u003eLong, straight lines within the page image with the Hough Lines Transform (HLT) algorithm, which in essence tallies all the lines to which each white pixel might contribute. This technique yields a list of lines that are described as the combination of an offset and an angle. In this case we are only concerned with horizontal lines, so we tell the HLT algorithm to constrain itself to a range of angles within ten degrees or so off the horizontal axis. This saves us some extraneous computation.\u003c/p\u003e\n\u003ch3\u003e3.1.5 Rotate the Image to Horizontal\u003c/h3\u003e\n\u003cp\u003eGiven all of the long, near-horizontal lines printed in the image, we need to find the proper angle to align them with our image axes, as well as we can. What we can do first is throw away any lines that are outliers, meaning their angles are very different from most of the other lines. In the image, these show up as diagonal. We group our lines by their approximate angle and then discard all but the most numerous group. Taking this representative group of close-to-horizontal angles, we get the average angle. The last step is to then rotate the whole image by this angle, bringing everything into alignment. Now the white pixels for a vertical line in the image will mostly fall into a single pixel column. The same will be true for horizontal lines and pixel rows. Often the angle of rotation is vanishingly small, but across the whole image, this can cause a line to stray into a neighboring pixel row or column.\u003c/p\u003e\n\u003ch3\u003e3.1.6 Fit a Uniform Template to Each Normalized Image (slide and/or magnify)\u003c/h3\u003e\n\u003cp\u003eSo, we now have an image where the printed form lines, thinned to one pixel wide, align with our image axes. The handwriting on the page is also thinned to a single pixel wide, but it will not align with the image axes generally, as the writing is scattered into many rows and columns. This means that when we add up all of the \u0026ldquo;ink\u0026rdquo; with a row, we can see a distinct spike whenever we get to a horizontal rule in the form. The same spikes are visible at each vertical line if you add up the values in the image columns. So, there is a pretty clear signal here for where our lines fall in terms of rows and columns.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTo restate our goal here, we are looking for the \u0026ldquo;Race\u0026rdquo; column in the form, which will be marked by two vertical lines. Each form row, representing a census person, is also marked off by form lines. How do we figure out which spikes represent the lines that we need for image segmentation? The answer is that we need a map of the form lines we are expecting to find. Then we will need to fit our map onto the observed territory in each individual image. In this case, our map is more like a template of expected form line offsets. We can consider each axis in turn, but we will take the vertical lines as an example. In general, the forms share the same template, except that the forms may shift slightly under the camera. Also, the camera distance or magnification can be slightly different. So, if we know where we find the vertical lines in one image, we can expect to find them again in every image after some shifting (translation) and magnification of their expected positions or offsets. Since we have twenty-four vertical form lines, we can use every line at once to confirm evidence of the correct magnification and translation. The brute force process is to simply run through a reasonable range of translations and magnifications, until we find one that puts the spikes, I.E. the most ink, at the expected locations of lines in our template. This is accomplished in two nested loops that compare the ink found at template offsets for each value of magnitude and translation. We use the same process to apply a template of horizontal form lines, but that includes a new twist to be described next.\u003c/p\u003e\n\u003ch3\u003e3.1.7 The Hunt for the Horizontal Line Template\u003c/h3\u003e\n\u003cp\u003eAt some point, deep in the thorny thicket of computer vision analysis, we found ourselves well and truly lost, with no fitting map of horizontal lines. Our horizontal template was seen to fit well on many of the pages, but on some others, it was fitted quite poorly, obviously out of place and shoddy. It was after some consternation that it finally dawned on us that the printing on some pages was not like the others. A notes field that is printed at the bottom of some forms was inexplicably printed at the top of others.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAs if to deny us any hope of orderly processing, this devilish Notes field was sometimes large and sometimes small. How then to fit a template to these variably horizontal lines? Do we need to try three or four different templates before finding the one that matches notes field placement and size? We seized upon some hope when we realized that we might try aligning a smaller template consisting of just those horizontal lines that we care about, I.E. the lines that are consistent and can guide us to the correct rows in the population schedule. Using a template that started with the lines in the column headers and ended with the twenty-five lines used to record individuals, we were able to generally get better results. The brute force process of magnification and translation had to explore a much larger range of translation but would usually find a good fit. However, when the fit was wrong, it was generally wrong by one or two printed rows. Since the table rows are equally spaced, this makes some sense, as many different offsets produce significant overlap between template rows and the image.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe still needed a better way to home in on the correct fit to get consistent results and reduce the range of solutions being considered. While studiously ignoring the taunts of the ungovernable notes field, we spied a clue that just might work. If you squint at a page image, one line stands out from the rest. Approximately in the middle of the page, there is a heavier line that starts at the top of the table headers and continues on, down to the end of the individual record rows. Sometimes printed faintly or not at all in the lower part of the page, we see that this heavy line is generally printed well at the top of the table, and there, at the top of the heavy line, we have a landmark that we may use to as a starting place for aligning the template.\u003c/p\u003e\n\u003cp\u003eThe CV technique we can use to find the heavy line shape is another morphological transform, this time with some parameters that will restrict the shapes identified to be vertically long and wider than most lines. This restriction is imposed through the use of what is called a kernel, a pattern of pixels that must fit within all the identified shapes in one or more positions. We used a kernel that was fifty pixels high and just two pixels wide. Prior to running the morphological transform, we also included a step called \u0026ldquo;dilation\u0026rdquo; to specifically repair any small breaks in the heavy vertical line due to misprints, ensuring that we will find it more consistently. Of all the shapes that the morphological transform detects, we choose the one that is vertically the longest. In almost all cases this combination of steps will yield a shape whose upper limit can be used to initially align our small horizontal lines template, before proceeding to the magnification and translation fitting steps. In those cases where the heavy line is not found, we simply fall back to our older, less consistent approach, which is often adequate.\u003c/p\u003e\n\u003ch3\u003e3.1.8 Slice Regions of Interest Using the Adjusted Template\u003c/h3\u003e\n\u003cp\u003eNow that the template of form line offsets has been fitted to the territory, the row and column spikes that represent lines in each image, we can use that adjusted template to pick out the particular areas that are of interest to us. In our case, we want to extract images of every cell in the race column. In the template, the Race column is between the 12\u003csup\u003eth\u003c/sup\u003e and 13\u003csup\u003eth\u003c/sup\u003e vertical lines. Within the smaller horizontal template, the individual record rows begin at the 3\u003csup\u003erd\u003c/sup\u003e line and continue through the 28\u003csup\u003eth\u003c/sup\u003e with thirty roughly equal spaces between each row. Since most census recorders seemed to write demographic code on the line, rather than above it, we extended each cell down the page slightly, in order to capture the entire handwritten code.\u003c/p\u003e\n\u003ch2\u003e3.2 Handwritten Character Recognition (HCR)\u003c/h2\u003e\n\u003cp\u003eThe HCR process operates on individual demographic cell images. It attempts to predict, through two different kinds of machine learning models, whether or not a particular cell contains the \u0026lsquo;W\u0026rsquo; character. The models are trained on a large collection of handwritten characters, designated as EMNIST, that were originally collected by the US National Institute of Standards and Technology (NIST) (Cohen et al. 2017). Some subsets of the original NIST data were actually based on handwritten characters extracted from census records.\u003c/p\u003e\n\u003cp\u003eEach training image is 28 by 28 pixels with some gray levels. The NIST recommends that models based on EMNIST work best when the target characters, which must have the same image dimensions, have been centered by their bounding box, instead of their center of mass. If you look at the images above you can see that they all have margins of about the same size all around them. Unfortunately, we were unable to center our demographic cell images by the bounding box and were forced to use the center of mass technique, due to the interference of the horizontal lines on which the characters were written. Below you can observe a typical census demographic cell that was extracted by the previously described CV segmentation technique.\u003c/p\u003e\n\u003cp\u003eIn Figure 12, you may see the dashed-looking form line that remains below the \u0026lsquo;W\u0026rsquo; character. The white box represents a 28 by 28-pixel area that has been drawn at the center of mass of this image, or as close to that as the margins allow. You can see that a bounding box around just the \u0026lsquo;W\u0026rsquo; would yield a different result, especially where you can see the \u0026lsquo;W\u0026rsquo; touching the right side of the white box. This bounding box discrepancy was not yet resolved in our project and may have impacted the results. We are working on some additional CV techniques designed to remove \u0026nbsp;the form lines more completely. However, the issue of the form line was addressed in training, as explained below.\u003c/p\u003e\n\u003ch3\u003e3.2.1 Adding \u0026ldquo;census form noise\u0026rdquo; to EMNIST Training Images\u003c/h3\u003e\n\u003cp\u003eSince we could not find a CV method to remove the form lines from our target images, we decided to do the next best thing and train our machine learning models on images that included similar noisy lines. For that, we would have to modify the EMNIST images before training the models. The TensorFlow framework has some affordances that support this kind of pre-training step on your training data, and we would end up using several. We generated randomized noisy lines and added these to the lower rows of the training images.\u003c/p\u003e\n\u003cp\u003eOther pre-training operations include the translation of pixel values into normalized floating-point numbers (fractions between 0 and 1) and the re-labeling of character classes to a binary, either 1 to indicate a W or 0 indicating not W. Using these two possibilities together as one logical output, we ask the model to produce a single floating-point number between zero and one that indicates the confidence that the character in the image is a \u0026lsquo;W\u0026rsquo;. Finally, another concern about the training dataset is that the frequency of the \u0026lsquo;W\u0026rsquo; class is about one in twenty-six, as the dataset is balanced with equal representation of all letters. This means that the model will learn that \u0026lsquo;W\u0026rsquo;s are somewhat rare, about as frequent as any other letter. However, in the census data the \u0026lsquo;W\u0026rsquo; is by far the most frequent character that we see. We followed an approach in training of setting a calculated initial bias\u003ca href=\"#_ftn3\" name=\"_ftnref3\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e13\u003c/sup\u003e to address this imbalance between the training data and the census data.\u003c/p\u003e\n\u003ch3\u003e3.2.2 Feed-forward Neural Network Setup and Training\u003c/h3\u003e\n\u003cp\u003eWe define the most straightforward neural network (NN) model first and customize it for our input parameters and the output we want. This model is defined as a sequence that takes in a 28 by 28 tensor array, representing image pixels. The input is first flattened into a 784-unit long tensor array of one dimension (28 times 28). Then each of these \u003cem\u003einput features\u003c/em\u003e is connected to each of the units in the subsequent neural network layer. The dense NN layer stores weights associated with each of these connections and those weights are used to determine the value each unit will pass to the next layer in the model. These connection weights are what we adjust during training. An activation formula (ReLU) is supplied to this layer. The default activation formula is \u0026quot;linear\u0026quot;, which means that unit output pretty directly reflects the input and connection weights. Our activation formula of ReLU means that unit output never exceeds a certain maximum value and that any unit output below a certain threshold is attenuated or lessened. This privileges the output of units that are strongly weighted and also receiving input.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEach unit in the first dense layer is subsequently connected to the single unit in the last layer. The activation there is \u003cem\u003esigmoid\u003c/em\u003e, not ReLU. Sigmoid is an activation function that converts the output value to a number between zero and one, with one representing total model confidence that the image is a \u0026lsquo;W\u0026rsquo; character.\u003c/p\u003e\n\u003cp\u003eAfter we define the neural network layers and activations in the first step, we compile that model, adding an optimizer, a loss function, and a custom function for gathering metrics during training. The loss function compares each prediction to the known class in the training data. The model will be trained over many passes in order to minimize this loss measure. Our loss function is binary, which means that the model is judged on how well it predicts a single value based on the input it receives.\u003c/p\u003e\n\u003cp\u003eAfter all of this setting up, the fitting or training of the model to the training data is shockingly straightforward. In a single line of code, we can ask for the model to be fit in 15 \u003cem\u003eepochs\u003c/em\u003e, which means that the model will pass through the training data fifteen times, adjusting connection weights and improving its accuracy with each pass. We trained the model on a laptop with an Intel Core i7-12800H processor, without connecting TensorFlow to a separate GPU. Even so, the training on all 88,800 images was completed in under five minutes.\u003c/p\u003e\n\u003ch3\u003e3.2.3 Convolutional Neural Network Setup and Training\u003c/h3\u003e\n\u003cp\u003eA convolutional neural network (CNN) is one that includes so-called convolved layers that are not densely connected. Instead, connections are made between groups of units, called kernels, that take input from overlapping visual regions in an image. This pattern of bundled neuron connections that divide the visual field is inspired by the organization of the animal visual cortex. While a deeper exploration of CNN organization is beyond the scope of this article, one may find many references on the subject. The model configuration in our TensorFlow code was based on a tutorial we found that was specific to HCR the task.\u003ca href=\"#_ftn4\" name=\"_ftnref4\" title=\"\"\u003e\u003c/a\u003e\u003csup\u003e14\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eIn the setup code above, you can observe the creation of the two additional \u0026ldquo;convolved\u0026rdquo; layers along with the specification of their kernel dimensions in pixels. Unlike the first deep learning model, the pixel inputs are not flattened at the beginning into a single dimension but kept in two dimensions such that they can be divided into the overlapping visual regions used by the CNN kernels. When all neurons are connected between two layers, the two-dimensional arrangement of the input image is irrelevant.\u003c/p\u003e"},{"header":"4. Results","content":"\u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Model predictions of a \u0026lsquo;W\u0026rsquo; in census records\u003c/h2\u003e \u003cp\u003eNext, we ran both of our deep learning models on example pages from the 1950s US census and reviewed our results. Census recorders tend to develop a shorthand script for recording codes, as they work quickly to record their data. We saw that \u0026lsquo;W\u0026rsquo; characters with cleanly separated line strokes were identified readily enough by both ML models. However, any more casual scribbles would throw off our carefully trained models. In our testing of several hundred census pages, we determined that more than two-thirds of \u0026lsquo;W\u0026rsquo; codes could be reliably automatically recognized, eliminating the need for human review of those records. This includes a necessary margin of caution on our part since one could simply accept \u0026lsquo;W\u0026rsquo;s that were predicted with much lower model confidence levels.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Applying HCR to transcription workflow\u003c/h2\u003e \u003cp\u003eIn order to apply this technology to filtering census records, we built a rudimentary user interface for page review into another Jupyter notebook. The interface presents six pages of filtered demographic cells for review on each screen. Any recognized \u0026lsquo;W\u0026rsquo; codes are omitted, so the user only has to review a smaller subset of the demographic codes. If any page has codes that are interesting for further study or transcription, the user may check a box next to that page and \u0026ldquo;save the page for later\u0026rdquo;. A log file gathers these saved pages, which they can return to later. In Fig.\u0026nbsp;\u003cspan refid=\"Fig12\" class=\"InternalRef\"\u003e17\u003c/span\u003e you can see a screenshot of the interface with four pages of demographic images shown. For each page around twenty \u0026lsquo;W\u0026rsquo; characters have been filtered out, as noted in the page labels.\u003c/p\u003e \u003cp\u003eIn our interface tests, we found that a deliberate user could review one screen of six census pages in around 11 seconds. That means that one can review each census page for demographic codes in less than two seconds a significant time saving over manual review. One could plausibly expect to complete a review of the 1950s Sacramento metro area, a population of approximately 216,000 people, in four hours with this workflow, not including some necessary breaks to refresh your eyes.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Next Steps","content":"\u003cp\u003eThere are a number of follow-up questions that remain after the research we describe. For one thing, it is without a doubt possible to better tune the images that we send to the predictive models. This includes ongoing work on CV approaches to eliminating the form lines that remain, which seems feasible, given that we know their approximate location and their shape. Some kind of morphological transform can be developed that will dissolve these lines prior to the recognition task. Elimination of the form lines would also make it possible to apply a \u0026ldquo;bounding box\u0026rdquo; approach to centering the characters in the 28 by 28 input images with a margin, as recommended in the EMNIST data set notes, bringing our target images into better alignment with the training data.\u003c/p\u003e \u003cp\u003eBeyond improving the image preparation steps, there are also newer deep learning models that have been developed, especially transformer models, which may provide a more state-of-the-art approach to the recognition task. In particular, we think that the OCR-free document understanding transformer, known as Donut, offers much to be explored. (Kim et al. \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eHaving outlined these opportunities, the first step is to clean up and publish the code notebooks that were used to perform this work so that others can fully explore our work. The code notebooks document in a narrative and specific way the iterative process by which the segmentation, recognition, and user interface tasks were developed, along with the exploration of the difficult puzzles we encountered along the way.\u003c/p\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eIt is rewarding, but intrinsically challenging, to attempt to apply recent AI technologies to archival materials that were printed or handwritten before the digital era, especially given that the information was originally designed for human transcription and tabulation. The field of machine learning is moving forward by leaps and bounds, yet adoption within archives is constrained by many factors, ranging from ethical concerns to the availability of expertise and resources. We chose in this study not to use general-purpose cloud offerings but rather to evaluate and adapt more traditional computer vision techniques to our specific challenges with historical materials. In many ways to deliver a project like this one feels like dropping a little paper boat into a rushing torrent. The variety of new machine learning models and their many applications are frankly astounding. We are very grateful to be able to explore just a few of these opportunities with regard to the 1950 US Census.\u003c/p\u003e \u003cp\u003eUnfortunately, we cannot recommend that a technologist in the archives pursue such projects casually. The preparation of materials was a painstaking process that unfolded over the course of several weeks. It required consultation with outside computer scientists with whom we had unique access. In the meantime, our Japanese American household study proceeded, using more time-consuming manual review and transcription, led by Dr. Marciano and two 2023 LEADING fellows. This covered both the 1940 and 1950 US Census records for Sacramento and stretched over several weeks, in which we also crowdsourced the creation of two Japanese American household datasets. This allowed us to assess the time that would have been saved had the automated review workflow been used, thus, demonstrating its labor-saving potential.\u003c/p\u003e \u003cp\u003eDue to the time investment required, these endeavors require a strong commitment to the outcomes and a realistic cost-benefit analysis. In our case, generalizing these techniques to the quite large dataset of every city in the United States in the 1950s, would provide a substantial benefit to related research. Another substantial benefit, we think, is being able to share the code, notebooks, and narrative that explain the development process from beginning to end. We hope that the project can help to further propel the community of practice for AI and archives, giving practitioners more guidance and contacts to consult when evaluating the application of machine learning to archival records. In the long run, we must remember that new automation, when applied with due care and consideration, opens up new opportunities for our own labor (Johnson \u0026amp; Acemoglu \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). We must plan to approach new roles alongside our community of practice that engages innovative forms of computation to enhance the public good.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAll authors wrote and reviewed the manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBeltran, L., Ping O\u0026rsquo;Brien, E., Jansen, G., Marciano, R. (2021)\u003cstrong\u003e \u0026ldquo;A Framework for Unlocking and Linking WWII Japanese American Incarceration Biographical Data\u0026rdquo;\u003c/strong\u003e, 2021 IEEE International Conference on Big Data, Dec. 15, 2021, Orlando, FL. See: https://ai-collaboratory.net/wp-content/uploads/2021/11/5_Beltran.pdf\u003c/li\u003e\n\u003cli\u003eCohen, G., Afshar, S., Tapson, J., \u0026amp; Van Schaik, A. (2017). \u003cstrong\u003e\u0026ldquo;EMNIST: Extending MNIST to Handwritten Letters\u0026rdquo;\u003c/strong\u003e. In \u003cem\u003e2017 international joint conference on neural networks (IJCNN)\u003c/em\u003e (pp. 2921-2926). IEEE. https://doi.org/10.1109/IJCNN.2017.7966217 \u003c/li\u003e\n\u003cli\u003eDiesendruck, L., Marini, L., Kooper, R., Kejriwal, M., \u0026amp; McHenry, K. (2012). \u003cstrong\u003e\u0026ldquo;A Framework to Access Handwritten Information within Large Digitized Paper Pollections\u0026rdquo;\u003c/strong\u003e. In \u003cem\u003e2012 IEEE 8th International Conference on E-Science\u003c/em\u003e (pp. 1-10). IEEE. https://doi.org/10.1109/eScience.2012.6404434 \u003c/li\u003e\n\u003cli\u003eJohnson, S., \u0026amp; Acemoglu, D. (2023). \u003cstrong\u003e\u0026ldquo;Power and progress: Our thousand-year struggle over technology and prosperity.\u0026rdquo;\u003c/strong\u003e Hachette UK.\u003c/li\u003e\n\u003cli\u003eKim, G. \u003cem\u003eet al.\u003c/em\u003e (2022). \u003cstrong\u003e\u0026ldquo;OCR-Free Document Understanding Transformer\u0026rdquo;\u003c/strong\u003e. In: Avidan, S., Brostow, G., Ciss\u0026eacute;, M., Farinella, G.M., Hassner, T. (eds) Computer Vision \u0026ndash; ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13688. Springer, Cham. https://arxiv.org/abs/2111.15664 \u003c/li\u003e\n\u003cli\u003eMarciano, R., Lee, M., Underwood, W., Laib, S., Diker, Z. \u0026amp; Singh, A. (2018),\u003cstrong\u003e \u0026ldquo;Digital Curation of a World War II Japanese-American Incarceration Camp Collection: Implications for Sociotechnical Archival Systems\u0026rdquo;,\u003c/strong\u003e DigitalHeritage2018, San Francisco, Oct. 27, 2018 (part of the Digital Solutions for Heritage Archives \u0026amp; Collections session). See: https://ai-collaboratory.net/wp-content/uploads/2020/04/DigitalHERITAGE_2018_paper_220.pdf \u003c/li\u003e\n\u003cli\u003eMarciano, R., Underwood, W., Hanaee, M., Mullane, C., Singh, A., Tethong, Z. (2019), \u003cstrong\u003e\u0026quot;Automating the Detection of Personally Identifiable Information (PII) in Japanese-American WWII Incarceration Camp Records\u0026quot;, \u003c/strong\u003e2018 IEEE International Conference on Big Data, Jan. 24, 2019, pp.2725-2732. See: https://ai-collaboratory.net/wp-content/uploads/2020/03/2.Marciano.pdf \u003c/li\u003e\n\u003cli\u003eSaha, P. K., Borgefors, G., \u0026amp; di Baja, G. S. (2016). \u003cstrong\u003e\u0026ldquo;A Survey on Skeletonization Algorithms and their Applications\u0026rdquo;\u003c/strong\u003e. \u003cem\u003ePattern recognition letters\u003c/em\u003e, \u003cem\u003e76\u003c/em\u003e, 3-12. https://doi.org/10.1016/j.patrec.2015.04.006 \u003c/li\u003e\n\u003cli\u003eUnderwood, B., Marciano, R., Laib, S., Apgar C., Beteta, L., Falak, W., Gilman, M., Hardcastle, R., Holden, K., Huang, Y., Baasch, D., Ballard, B., Glaser, T., Gray, A., Plummer, L., Diker, Z., Jha, M., Singh, and A., Walanj, N. (2017), \u003cstrong\u003e\u0026ldquo;Computational Curation of a Digitized Series of WW11 Japanese-American Internment,\u0026rdquo;\u003c/strong\u003e IEEE Big Data 2017\u0026rsquo; 2\u003csup\u003end\u003c/sup\u003e Computational Archival Science (CAS) Workshop, Boston, MA, Dec. 13, 2017. See: https://ai-collaboratory.net/wp-content/uploads/2020/04/Underwood.pdf\u003c/li\u003e\n\u003c/ol\u003e\n"},{"header":"Footnotes","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ai-collaboratory.net/projects/ct-ja_ww2_camps/\u003c/span\u003e\u003cspan address=\"https://ai-collaboratory.net/projects/ct-ja_ww2_camps/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://terp.umd.edu/truth-in-exile\u003c/span\u003e\u003cspan address=\"https://terp.umd.edu/truth-in-exile\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://mydigitalpublication.com/publication/?m=30305\u0026amp;i=798391\u0026amp;p=12\u0026amp;ver=html5\u003c/span\u003e\u003cspan address=\"https://mydigitalpublication.com/publication/?m=30305\u0026amp;i=798391\u0026amp;p=12\u0026amp;ver=html5\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://cases.umd.edu/github/cases-umd/Japanese-American-WWII-Incarceration/blob/main/index.ipynb\u003c/span\u003e\u003cspan address=\"https://cases.umd.edu/github/cases-umd/Japanese-American-WWII-Incarceration/blob/main/index.ipynb\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ischool.umd.edu/news/aic-umd-student-project-featured-on-japanese-television/\u003c/span\u003e\u003cspan address=\"https://ischool.umd.edu/news/aic-umd-student-project-featured-on-japanese-television/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://medium.com/@ero22/data-desert-islands-and-digital-dark-ages-richard-marciano-on-records-and-data-management-13acac0219a7\u003c/span\u003e\u003cspan address=\"https://medium.com/@ero22/data-desert-islands-and-digital-dark-ages-richard-marciano-on-records-and-data-management-13acac0219a7\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ai-collaboratory.net/2021/12/14/dec-9-2021-digital-curation-showcase/\u003c/span\u003e\u003cspan address=\"https://ai-collaboratory.net/2021/12/14/dec-9-2021-digital-curation-showcase/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.youtube.com/watch?v=XWrG_-O8SY4\u003c/span\u003e\u003cspan address=\"https://www.youtube.com/watch?v=XWrG_-O8SY4\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.archives.gov/research/census/1950/questions-asked\u003c/span\u003e\u003cspan address=\"https://www.archives.gov/research/census/1950/questions-asked\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.archives.gov/research/census/1950/ed-maps\u003c/span\u003e\u003cspan address=\"https://www.archives.gov/research/census/1950/ed-maps\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e The CASES website is available at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://cases.umd.edu\u003c/span\u003e\u003cspan address=\"http://cases.umd.edu\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ai-collaboratory.net/2024/08/17/saa-2024-the-impact-of-ai-on-the-future-of-archival-work/\u003c/span\u003e\u003cspan address=\"https://ai-collaboratory.net/2024/08/17/saa-2024-the-impact-of-ai-on-the-future-of-archival-work/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e TensorFlow Tutorial on using imbalanced training data: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.tensorflow.org/tutorials/structured_data/imbalanced_data\u003c/span\u003e\u003cspan address=\"https://www.tensorflow.org/tutorials/structured_data/imbalanced_data\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-from-scratch-for-mnist-handwritten-digit-classification/\u003c/span\u003e\u003cspan address=\"https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-from-scratch-for-mnist-handwritten-digit-classification/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Computer Vision, Machine Learning, Artificial Intelligence, 1950 U.S. Census records, Sacramento, WWII Japanese American incarceration","lastPublishedDoi":"10.21203/rs.3.rs-5105914/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5105914/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis work explores the development of AI and ML computer vision techniques to unlock digitized handwritten US Census records from the 1950s, which includes over 6.5 million images and was only recently made available to the public on April 1, 2022, following a 72-year access restriction period. The 1950 Census offers a unique window \u003cem\u003e\"into one of the most transformative periods in modern American history, revealing a country of roughly 151 million people who had just recently emerged from the hardships and uncertainties of World War II and the Great Depression.\"\u003c/em\u003e (\u003ca href=\"http://census.gov/\"\u003ecensus.gov\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003eThis computer vision and machine learning work is part of a larger case study based in Sacramento, California focusing on creating a so-far unseen window into the fate of the Japanese American community. Sacramento once housed the fourth largest Japantown community on the West Coast and saw its community forced out twice in a decade: in 1942 during WWII Japanese American Incarceration (the largest single forced relocation in US history), and in 1954 during urban renewal where Japantown residential and business districts were leveled. Our project uses AI-based computational treatments to help recover and memorialize the history of the erased Sacramento Japantown. Moreover, we contrast these findings with the processing of the 1940 Census (released in 2012), thus producing a novel \"before-and-after\" representation.\u003c/p\u003e\n\u003cp\u003eWe demonstrate a workflow for extracting demographic information using image segmentation, computer vision techniques, and deep learning for handwritten character recognition. These techniques are generalizable to other cities, states, and communities, and demonstrate AI-assisted strategies to unlock vital demographic information. The approach highlights the potential benefits of computational techniques on social justice issues.\u003c/p\u003e\n\u003cp\u003eThe workflow represents an AI-assisted filtering process for Census records, with a user interface for computationally driven page review. The goal is to automate the culling of pages to select a smaller subset of pages that can then be further targeted or crowdsourced.\u003c/p\u003e","manuscriptTitle":"Developing Computer Vision and Machine Learning Strategies to Unlock Government-created Records","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-23 15:22:09","doi":"10.21203/rs.3.rs-5105914/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"fc0235be-dfbd-4fa7-88d2-5b4f3ffe01c4","owner":[],"postedDate":"October 23rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-05T10:23:17+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-23 15:22:09","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5105914","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5105914","identity":"rs-5105914","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}