Information Extraction System for Cargo Invoices

Information Extraction System for Cargo Invoices | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Information Extraction System for Cargo Invoices QiuXing Michelle Tan, Qi Cao, Chee Kiat Seow, Peter Chunyu Yau This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2413475/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Rapid growth in the digitization of documents, such as cargo invoices or receipts, has alleviated the demand for methods to process information accurately and efficiently. However, it has become impractical for humans to extract the data due to the size of the corpus. Furthermore, extracting the information manually can be labor-intensive and time-consuming as employees need to go through multiple digitized invoices and extract the critical information. The digital documents, such as digital cargo invoices contain various components such as tables, key-value pairs and figures. Existing optical character recognition (OCR) method are able to recognize texts. But it is challenging for them to extract the key-value pairs in unformatted digital cargo invoices. Hence, creating a smart information extraction system that can extract key-value pairs would be beneficial, as it would help organizations achieve workflow efficiency, resource utilization and eliminate costly errors. In this paper, a pipeline of the information extraction system is proposed with deep learning approaches for classifying key-value pairs first, followed by linking the key-value pairs. Two merging rules and two pairing rules are developed in the proposed pipeline. Various experiments have been conducted to illustrate the performance of the system. Information Extraction Key-Value Pairs Optical Character Recognition Natural Language Processing Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction On a daily basis, many organizations, such as logistics companies, deal with many paper documents such as receipts or cargo invoices [1]. Currently, the majority of these documents must be processed manually, which is time-consuming and expensive [2]. One of the most difficult aspects of invoice processing for logistics organizations is the time-consuming, intensive in-house procedure, requiring numerous manual works when extracting and keying data into various internal software systems. In addition, data captured from logistics invoices poses a particular difficulty because they are received in multiple formats. This is a significant issue, due to the intricacy of the invoices. Manually inspecting and correcting scanning errors greatly lengthens the correction and processing time [3]. Digital invoices contain various components such as tables, key-value pairs and figures. A key-value pair is made up of two connected data elements: a key which is a constant defining the data set; and a value which is a variable that is part of the set. A fully formed key-value pair will be like Gross Weight: 123 kg, where “Gross Weight” is the key and “123 kg” is the value. A corporation can benefit from digitising and extracting key information on a number of levels. Business owners can better track their processes, provide better customer services, increase employee productivity, and cut costs. Optical character recognition (OCR) also known as text recognition, is the process of extracting text information from scanned documents and images. Current OCR systems, such as Tesseract [4] and EasyOCR can recognize raw text in unformatted digital documents or images. But they are not capable of extracting information like key-value pairs from unformatted data. Key-value pairs are the most significant components in digital cargo invoices. Key-value pairs make raw texts more understandable. The existing solutions, such as Amazon Web Services (AWS) Textract, provide such a service [5]. However, it is a paid service in which many of the small and medium enterprises (SMEs) are reluctant to pay extra costs for going digital since the SMEs usually have a tight budget. Furthermore, the off-the-shelf solutions may not fit the specific domains, as most of the solutions lack of flexibility to be customized or tailored according to different requirements from different customers. Moreover, the AWS Textract is a service hosting on the cloud platform. Users need to upload their digital invoices and receipts to the cloud platform, in order to get them processed. It raises possible privacy concerns since some companies might treat the invoices and data as private and business sensitive. Hence, automation of extracting key-value pairs within unformatted digital invoices is proposed as it can significantly reduce manpower and costs while simultaneously ensuring the reliability of the data retrieved. In this paper, an information extraction system with the deep learning approaches that is capable of extracting key-value pairs will be presented to improve the overall performance. The proposed information extraction system would be beneficial as it would help companies achieve workflow efficiency, resource utilization and eliminate costly errors. The remaining parts of this paper are organized next. Section II presents the prior works in the literature. Section III explains the pipeline of the proposed information extraction system. Section IV analyses and discusses experiment results. Lastly, Section V concludes this paper. 2. Related Work There are three different techniques of key-value pair extraction: regular expression, natural language processing (NLP) and layout detection. This section studies the prior arts related to OCR key-value pairs extraction approaches. OCR is an important portion as it will affect the results for the key-value pairs. As mistaken words might be wrongly classified when performing key-value pair extraction. Hence, OCR plays a crucial role for key-value pair extraction. OCR is used to extract text information from scanned documents and images. Vedant Kumar et al . [6] use OCR on bill receipts images taken from a mobile phone to extract out the text information from the receipts. Before extracting the information, they have done some image pre-processing such as binarization, removing of shadows and etc. After that the processed image is inputted into the Tesseract OCR engine to extract the text. OCR has been used for scanned documents to extract the text information in [7]. Similarly, they have pre-processed the scanned invoices by sharpening the images, threshing and binarization. 2.1 Key-Value Pair Extraction using Regular Expression A key-value searching system is developed by Kaló and Sipos [8] that uses the open-source Tesseract OCR engine and post-processing techniques with regular expressions. Regular expressions are patterns used to match character combinations in strings. To find key-value pairs with regular expression, we can find by text with colon. If the regular expression manages to find the word, this means that it is a key. This method can be used to create a low-cost office automation system for invoice processing. Regular expression is very effective at identifying patterns and replacing strings. It increases efficiency and versatility by allowing constant patterns and variable values to be searched in the text. Before developing the key-value searching system with regular expression, it is necessary to learn what patterns are there in the dataset, collate all the different types of patterns and put them into a pattern dictionary. The regular extraction pattern dictionary can be expanded over time to learn a wider variety of patterns, with more datasets being included and more patterns being added. New patterns can be added based on the errors. If these errors occur during a later process, they would be stored and corrected. Using regular expressions to extract key-value pairs is efficient in finding specific patterns or text. However, when a new text or pattern is introduced, that would be an issue as the system does not understand and cannot extract them. Additionally, some keys and values may have a variety of different forms of patterns. Thus, there is a need to manually look through the dataset and update the patterns to extract the key-value pairs correctly and effectively. Furthermore, with many patterns declared, it degrades the readability and performance of the codes. Hence, with these problems being identified, extracting key-value pairs using regular expression is not efficient. Though, regular expression may be suitable for post-processing in assistant of validation of value formats. 2.2 Key-Value Pair Extraction using NLP Bidirectional Encoder Representations from Transformers (BERT) [9] utilizes Transformer, an attention mechanism that discovers contextual relationships between words (or subwords) in a text. Transformer has two independent working parts: an encoder that reads text inputs, and a decoder that generates a task prediction. BERT has two training tasks: Masked Language Model (MLM) and Next Sentence Prediction. Robustly Optimized BERT Pretraining Approach (Roberta) [10] is an optimized approach for pretraining natural language processing. Roberta has almost similar architecture to BERT. Some changes are removing the Next Sentence Prediction and training with bigger batch sizes and longer sequences. The BERT is extended to a new model, StructBERT, by incorporating language structures into pre-training [11]. StructBERT expands BERT by leveraging structural information, such as word-level and sentence-level ordering, in addition to the present masking method. Two new structural objectives are added to model pre-training, focusing on inner-sentence and inter-sentence structures. In this way, the language characteristics of pre-training phase are explicitly represented. Furthermore, StructBERT uses structural pre-training to embed dependency between words and sentences in the contextualized representation, allowing the model to be more generalizable and adaptable. This allows StructBERT to represent language structures explicitly by requiring it to reconstruct the correct order of words and sentences in order to make accurate predictions. StructBERT is compared with a few methods such as BERT, Roberta, etc. Through the experimentation results, it shows that the StructBERT model can obtain better results in a variety of downstream tasks with the improvement of the F1 scores to 93%. StructBERT framework is better than the regular expression approach, as it is trained on the textual data. It understands the key-value pairs and reconstructs the correct order of words, in order to make accurate predictions. However, the NLP approach only takes in the text from the document and does not incorporate the position of the text. In the tasks of extracting key value pairs, the position of the text is useful. However, StructBERT is still useful to understand the key-value pair structure and can be used together with layout awareness algorithms to improve the accuracy. 2.3 Key-Value Pair Extraction using Layout Detection LayoutLM [12] is the few of the models successfully reported that does labeling using positional information, text-based information, and image information. As an upgrade from LayoutLM, the LayoutLMv2 is reported in [13], which uses model architectures and pre-training tasks to pre-train text, layout, and image in a multi-modal framework. Unlike other Visually-rich Document Understanding (VrDU) approaches which aims to analyze scanned documents [13], LayoutLMv2 not only considers the text and layout information, but also integrates the new text-image alignment and text image matching tasks, which help to learn cross-modality interaction. In addition, LayoutLMv2 incorporates a spatially-aware self-attention mechanism into the Transformer design, allowing the model to completely comprehend the relative positioning relationships between different text blocks. Experiments are conducted to compare the efficiency of different types of models [13] with multiple datasets, including open sourced FUNSD dataset [14] which consists of scanned forms that differ significantly in terms of their structure and presentation. The results show that LayoutLMv2 outperforms strong baselines and achieves better performance on a variety of downstream visually rich document comprehension tasks. Another approach, LAMBERT, is reported in [15] that tackles the challenges of comprehending documents where non-trivial layout affects the local semantics. It is a Layout-Aware Language Model, which combines NLP methods with layout understanding mechanisms. The LAMBERT uses the layout information of the document and train it with the pretrained Roberta. Hence, users do not need to input any images, as the model uses the bounding box coordinates instead to understand the layout of the document. A pre-training approach, StructuralLM, utilizes cells and document layouts from scanned documents [16]. This approach is influenced by LayoutLM, but with some differences. Firstly, rather than modeling word-level 2D-position embeddings, the StructuralLM uses cell-level 2D-position embeddings to represent the layout information of cells. The StructuralLM also introduces a cell position classification which attempts to predict a cell location based on the location of cells and their semantic relationships. Lastly, the StructuralLM removes the image embeddings from LayoutLM that are only needed in the downstream tasks while keeping the 1D-position embeddings to describe the positional link between tokens from the same cell. The LayoutLMv2 framework with layout detection approach is better than models with the regular expression and NLP approaches for extracting key-value pairs. LayoutLMv2 integrates NLP processing techniques in the framework and text-image alignment where it is able to comprehend the relative positioning relationships between different text blocks. This is efficient for extracting key-value pairs as most of them are either next to each other or positioned on top of the other. However, while experimenting with LayoutLMv2 on our dataset in our research, as our dataset has several different layouts, it has been observed that the LayoutLMv2 has some limitations. While LayoutLMv2 trains well on the layout of the invoices and when a different layout of the invoice is input as test data, it will not do as well. It may not be able to accurately detect the key-value pairs. However, this can be solved by training more datasets from different varieties. 3. Proposed Methodology In this section, the overall design of the proposed information extraction pipeline for cargo invoices is described. 3.1 System Arcitecture Pipeline The flow chart of the proposed information extraction system for cargo invoices is shown in Fig. 1. The proposed pipeline of information extraction system consists of two portions: the key-value pair classification and linking the key-value pairs. The key-value pair classification portion will be to explore models to classify the text from OCR to “Question”, “Answer” and “Others”. For the second portion of linking the key-value pairs, it takes the output from the key-value pair classification to link the key-value pairs with the use of layout spatial awareness like the bounding box position. To begin the pipeline of the proposed system, data collection is performed by taking images of cargo invoices from warehouses. After the data collection, there would be a quality check to ensure that the images are not blurry, as blurry images would hinder the performance. After that, the cargo invoices will be annotated into FUNSD format [14] using an open source tool, Banksy [17]. This open source tool is an image annotation tool where the Named Entity Recognition (NER), Named Entity Linking (NEL), and a box region on the image will be outputted. The NER will be the labels such as “Question”, “Answer” and “Others”. The NEL is the tasks to link the “Question” text to the “Answer” text. Lastly, the box region is the bounding box, where the bounding box of texts is drawn. The next step is to perform data cleansing on the data. Data cleaning is a necessary process because it improves our data quality and increases overall productivity. After cleaning, the data is formatted into a model-ready form for training and split the dataset into the train set (80%), and the test set (20%). Subsequently, different algorithms for the key-value pair classification and detection are explored. The text classification algorithms are only trained on the text and labels. Whereas for linking of key-value pairs algorithms, it uses layout awareness and text embeddings to train and get the keys and values. After comparing the results of multiple key-value pair classification algorithms such as LayoutLMv2, BERT, etc., the best model will be chosen for key-value pair classification. For the algorithms to link the key-value pairs, two methods explored are the regular expression and linking key-value pairs by the bounding box location. After comparing the results of these two algorithms, the best model will be used and output the final key-value pair. The output will be evaluated with the ground truth. If the performance of the output from the proposed information extraction pipeline is low, it will go through another iteration to improve the accuracy by experimenting with different model hyperparameters. This will be an iterative process until it achieves good accuracy. Finally, if the performance is satisfied, it will go through inferencing using the optimal model chosen by the proposed pipeline, and merge the output of word-level key and value to the entire key and value pairs. 3.2 Selection of Algorithms To explore key-value pair classification, two types of different techniques will be used to compare: NLP methods, and NLP methods combined with layout spatial awareness. NLP methods are considered as they can understand context-sensitive human language. By using NLP, it can effectively extract data or information from text-based documents [16]. For the NLP methods, both BERT model and Universal Language Model (ULMFit) model are explored. The BERT model analyzes a word's left and right sides to infer its context. Additionally, The BERT model uses MLM, which covers or masks a word in a sentence. MLM enables or enforces bidirectional learning from a text by requiring BERT to predict the word on either side of the covered word [18]. With this in mind, applying it to the cargo invoice dataset, the model could understand the key-value pairs. For example, the term "Product" could be next to the word "Number". The ULMFit model is trained using a general-domain corpus to capture overall language properties at several layers and then learns task-specific features [19]. The model uses a 3-layer ASGD Weight-Dropped Long Short-Term Memory Networks (LSTM) architecture. There are three steps to the training: Pre-training a general language model on a text, fine-tuning the language model on a target task, and fine-tuning the classifier on the target task are the first two steps. The other technique for key-value pair classification is NLP with layout spatial awareness. The first algorithm is YOLOv4 combined with BERT. This algorithm is able to understand the position of the text as well as using BERT to understand the text embeddings. The second algorithm is LayoutLMv2; LayoutLMv2 not only considers the text and layout information, but also integrates the new text-image alignment and text image matching tasks, which help to learn cross-modality interaction [13]. For the two algorithms for the linking of key-value pairs, i.e., the regular expression and the pairing by the nearest bounding box, the regular expression is efficient in finding specific patterns or text which is common in cargo invoices, for example, "Product No.: 123". The patterns commonly seen for keys and values are with colons. For the regular expression algorithm, an analysis will be done to see the common key-value pairs and their patterns to extract the key and values pairing. For the pairing by the nearest bounding box algorithm, it will pair keys and values by finding the nearby bounding box. The pairing rules are to merge the text only if they have the exact vertical coordinates (i.e., y coordinates), overlapped bounding box and are close. 3.3 Datasets The datasets used for experiments and performance comparisons are the FUNSD dataset [14], and Cargo Invoices dataset. The Cargo Invoices dataset is constructed in this research which are all the cargo invoices images from the warehouses. TABLE I describes the number of training and testing data in each dataset. Table I SAMPLE SIZES OF UTILIZED TWO DATASETS Dataset Used Dataset Training Testing Total Data FUNSD [14] 149 50 199 Cargo Invoices Dataset 484 121 605 4. Experiment Results And Analysis 4.1 Evaluation Metrics The evaluation metrics used in the ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction [20] are employed to evaluate the experiment performances in this paper: the Precision, Recall and F1 Score. The extracted text will be compared to the ground truth for each test image. If the submitted content and category of the extracted text matches the ground truth, it is marked as correct; otherwise, it is labelled as inaccurate. Furthermore, the algorithms will also be further evaluated by the classification report and confusion matrix. The values of the precision, recall, F1-score, and support are the four most critical headers for classification results to pay attention in classification reports. The value of precision refers to the ability of a classifier to avoid labelling a negative instance as positive [21]. The recall is the ability of a classifier to find all positive instances. The F1 score is a weighted harmonic mean of precision and recall. The support is the number of actual class occurrences in the specified dataset. The precision value can be calculated in Eq. ( 1 ). Precision is useful when False Positives (FP) pose a greater threat than False Negatives (FN). In our research, having lots of wrong results would affect the key-value pair linking and the final key-value pair results. $$Precision= \frac{TP}{TP+FP}$$ 1 where TP denotes True Positive. The recall value is computed in Eq. ( 2 ). When FN is more important than FP, it is a valuable metric. $$Recall= \frac{TP}{TP+FN}$$ 2 The F1 score ranges between 0 and 1, that can be calculated in Eq. ( 3 ). If the F1 score scores closer to 1 it means that the model performs better. $$F1 Score= 2 \times \frac{Precision \times Recall}{Precision+Recall}$$ 3 4.2 Label Classification Comparisons Prior to classification, the dataset is first split into 80% training and 20% testing. The four classification models experimented with are BERT, ULMFit, YOLOv4 combined with BERT and LayoutLMv2, each of which will classify the text in to “Question”, “Answer” and “Others”. In the experiments in this sub-section, the dataset used in the constructed Cargo Invoice dataset in this research. 1) Results of the ULMFit Model With the Cargo Invoice dataset, the ULMFit model is utilized for key-value pairs classification. The results of precision, recall, F1 score and support for the ULMFit model are shown in TABLE II. The ULMFit model scores 93% accuracy. Overall, the performance of this model is good. However, the model does not perform well for the "Other" class, compared to the other two classes. It might because there are fewer "Other" labels as compared to those of "Question" and "Answer". Table II ULMFit Results with Cargo Invoice Dataset Class Precision Recall F1 Score Support Question 99% 98% 98% 882 Answer 88% 96% 92% 882 Other 93% 80% 86% 568 Accuracy 93% 2332 2) Results of the BERT Model The experiment results of the BERT classification model with the Cargo Invoice dataset are shown in TALBE III. The BERT model achieves 88% accuracy. But the “Answer” label does not do well for the values of precision, recall and F1 score. This model can perform better on the “Other” label as the score for precision, recall, and F1 score is higher than the others. Table III BERT Model Classification Results Class Precision Recall F1 Score Support Question 85% 91% 88% 2128 Answer 83% 76% 79% 1387 Other 96% 95% 95% 1493 Accuracy 88% 5008 3) Results of YOLOv4 Combined with BERT Model The classification results of the YOLOv4 combined with BERT model is shown in TABLE IV. It shows that the overall performance is worse than other experimented classification models. For “Other” label, it performs the worst as compared to other two labels. It might be due to the imbalanced dataset whereby “Other” has the least amount of data comparatively. The results in this experiment show that the combination of YOLOv4 and BERT algorithms does not enhance the performance, as the BERT model itself manages to achieve 88% for the cargo invoices as seen in TABLE III. Table IV Results of YOLOv4 Combined with BERT Model Precision Recall F1 Score Support Answer 52% 42% 46% 882 Other 46% 39% 40% 568 Question 49% 40% 44% 882 Accuracy 40.27% 2332 4) Results of the LayoutLMv2 Model For the LayoutLMv2 model, the labeling methods are slightly different from other three models. The labels of the LayoutLMv2 model follow the BIOES tagging, where B means beginning; I mean in the middle; E means the ending; O means others; and S means a single word representing a full sequence [22]. For example, “Product No.”, will be split into “Product” which will be B-QUESTION and “No.” is “E-QUESTION”. The experiment results of the LayoutLMv2 classification model are shown in TABLE V. It is observed that the LayoutLMv2 model achieves an accuracy of 96%, which is the highest out of other three classification models. All the labels achieve 90% and above for the precision, recall and F1 score. It shows that the LayoutLMv2 model can predict well and distinguish each labels properly. Even though there is some unbalanced data, the model is still able to predict well for all the labels. The analysis on the confusion matrix will be further conducted whether this model is the best. Table V LayoutLMv2 Model Classification Results Precision Recall F1 Score Support B-ANSWER 93% 94% 94% 331 B-QUESTION 98% 98% 98% 507 E-ANSWER 90% 96% 93% 331 E-QUESTION 98% 98% 98% 507 I-ANSWER 96% 98% 97% 915 I-QUESTION 96% 97% 97% 104 O 97% 93% 94% 1387 S-ANSWER 95% 97% 96% 502 S-QUESTION 96% 98% 97% 375 Accuracy 96% 4959 5) Experiment Results with the FUNSD Dataset Besides the classification experiments on the constructed Cargo Invoice dataset, the experiments are also performed on the FUNSD dataset to compare the performances of these four classification models. The overall performance with the FUNSD dataset is shown in TABLE VI. Table VI Classification Results with the FUNSD dataset Model Precision Recall F1 Score ULMFit 77% 76% 76% BERT 55% 67% 60% YOLO combined with BERT 49% 40% 44% LayoutLMV2 80% 85% 83% It has shown that LayoutLMv2 model performes the best out of other three models. This has been demonstrated that integrating the spatial-aware self-attention Mechanism into the Transformer architecture in the LayoutLMv2 model has fully allowed the model to understand the relative positional relationship among different text blocks. On the other hand, YOLOv4 combined with the BERT model does not perform well. Even though it is trained with the text along with the bounding box of the texts, it is not able to find the relative positional relationship accurately compared to the LayoutLMv2 model. Thus, the LayoutLMv2 model will be the model chosen by the proposed pipeline for the key-value pair classification tasks. 4.3 Linking of Key-value Pair Comparisons The two algorithms for the linking of key-value pairs to be evaluated are: the regular expression algorithm, and finding the pairs by the nearest bounding box algorithm. 1. Regular Expression Algorithm Before beginning regular expression, some analysis is done on the dataset to understand the keys and values patterns, as well how they are paired to enhance the process of extracting key-value pairs. There are various formats of the keys, such as the short forms. Hence we need to create a list of the different forms of the keys. Some examples of the types of formats can be seen in TABLE VII. Table VII Some examples of different formats of keys Word Different Formats of Keys Gross Weight G/W G/Weight Gross wt Gross wght Gross wght(kg) Net Weight Net WT Net wght Net weight(kg) Net wt(kg) Net wght(kg) N/W Dimension Dim Dims Dim(mm) DIM (CM) Dimensions(cm) After finding out the different formats of the keys from the Cargo Invoice dataset, a word cloud is created to better understand the keys in the dataset, as shown in Fig. 2. Through the wordcloud, words such as "gross weight", "net weight", "PO", "country of origin", and "dimension" stand out the most. Hence, from the common keys, the next step is understanding their values pattern. Some examples of the patterns of values are shown in TABLE VII. Table VIII Pattern of Values Key Values Patterns Gross Weight G/W 2134 23Kg/g 88,500 KG For gross weight the pattern is always an integer/float followed by the metrics (KG/lb/g). Sometimes G/W would be infront of the integer/float. Net Weight 270 lb 88,500 KG 3.840 KG For Net weight the pattern is similar to Gross weight. The pattern is always an integer/float followed by the metrics (KG/lb/g). PO PO-IMA-21007 PO2000004328 For PO the pattern is PO followed by letters then integers or just integers. Dimension 228.00 X 148.00 X 165.00 CM 89.76 X 58.26 X 64.96 INCH 3815*2150*2550mm For Dimension, the pattern is integer/float X integer/float X integer/float After doing analysis on these patterns, key value pairs will be extracted. For the work flow of the extraction of key-value pairs are as follow: Find all key-value pairs. Calculate Levenshtein distance with given identifiers to see which one is the most likely identifier. Return key-value pair with the lowest normalized Levenshtein distance. 2) Pairing Via Nearest Bounding Box The input for this part is the output from the LayoutLMv2 classification model. The words are in the word level form and the labels are in BIOES tagging [22]. Hence, we will need to combine the word level text into sentence level to make sense of the full question and answer. To combine them into sentence level, the bounding box position and the labels will be critical. There are two merging rules to get the full questions and answers as currently in the word level form. These two merging rules will be described next. The flowchart of the merging rule 1 is shown in Fig. 3. First, it gets the coordinates, labels and text from the LayoutLMv2 Predicted results and images. Then it traverses through the whole results and compare whether if both labels are same. If not the same, then it is assign as None to the variables called “neighbours”. If it is the same, then it checks whether the vertical y coordinates are the same. This is to check whether these nodes are on the same line. If they are not on the same line, then it is assign as None . If they are on the same line, then it proceeds to check whether the right side of a bounding box is the same as the left side of another bounding box. If it is the same, then the bounding box ID is assigned to each of the nodes in the “Neighbours” variable. Otherwise, it will be assign None . After that, the second part of the merging rule 1 shown in the blue box in Fig. 3 checks if there are multiple bounding boxes connected to one another. If yes, then group them into one group. If a bounding box is not connected to another bounding box, then they will be put into individual groups. Lastly, the purple box in Fig. 3, the merged results are derived and updated into the dictionary into the form of: {'text': pred[link_list[0]]['text'], 'labels': pred[link_list[0]]['labels'], 'left_neighbour': None, 'right_neighbour': None, 'width': pred[link_list[0]]['width'], 'height': pred[link_list[0]]['height']}} The second merging rule is shown in Fig. 4. For the second merging rule of the full questions and answer, it uses the data derived from the mergin rule 1 to further enhance the method of getting the full questions and answers. First, the merging rule 2 gets the center of the bounding box from the mean of the x and y coordinates. Next it finds the nearby bounding boxes and checks if both labels are same. If not the same, the nodes are assigned as None for the variable “neighbours”. If both labels are the same, it then checks if the center of the bounding box A is within the range of the bounding box B in the y direction. If not, the variable “neighbours” is assigned to None . Otherwise, it checks if the center of the bounding box B is within the range of the bounding box A in the y direction. If yes, it checks if the two bounding boxes overlap in the x direction, or if the 2 bounding boxes gap in the x direction is < 10% of the width of the image. If the condition is true, the bbox_id of these 2 bounding boxes (bboxes) are assigned as neighbours to each other's nodes. If the condition is fasle, the bbox_id of these 2 bboxes are assigned as None to the variables “neighbours”. The second part of the merging rule 2 in the blue box is to check if there are multiple bounding boxes connected to one another. If true, then group these bounding boxes into one group. If a bounding box is not connected to others, then it is put into an individual group. These results will be merged and passed to the last process in the merging rule 2 shown in the purple box. Lastly, the results are merged and updated into the dictionary in the same format as that of the merging rule 1. After merging the words into the full key and values, we need to start pairing the keys and values together. There are two pairing rules for this step. The flowchart in Fig. 5 shows how the key-value pairing rule 1 works. It takes in the results from the merging of full questions and answers. It then checks if the “Question” and ”Answer” have the right neighbour, followed by checking whether the y coordinates between both “Question” and ”Answer” are identical. Lastly, the key-value pairing rule 1 checks whether the coordinates of the right side of the “Question” bounding box is identical to the left side of the “Answer” bounding box. After performing the key-value pairing rule 1, the output is splitted into successfully paired and unsuccessfully paired. The unsuccessfully paired results are inputted into the key-value pairing rule 2 to pair the questions ans answers. The key-value pairing rule 2 is presented in Fig. 6. Firstly, it checks whether both centers of the bounding boxes are within the range of the other bounding box in the y axis. Next it checks whether they overlap in the x axis or the gap in the x axis is less than 50% of the width of the bounding box with the smaller width value. The final output consists of the label, bounding boxes, text, left_neightbour, right_neighbour, the width, the height, the variable of pair_with , and lastly the center attribute. 3) Results Comparison After performing the key-value pairing, the results are evaluated by comparing the derived final key-value pairs with the ground-truth. The experiment result comparisons are shown in TABLE IX. The metrics used to do the comparison is the precision, recall and F1 score. Table IX Results Comparison for Key Value Pairing Algorithm Precision Recall F1 Score Regular Expression 63% 60% 66% Pairing by Finding Nearby Bounding Box 73% 72% 70% It is observed from the results that the second algorithm, i.e., the pairing by finding the nearby bounding box, has done a better job with a precision of 73%, recall of 72% and F1 score of 70%. In comparison, the first algorithm, i.e., the regular expression, achieves a precision of 63%, recall of 60% and F1 score of 66%. The reason for the regular expression not perform well might because there are limited patterns introduced into the system. Even though the algorithm of the pairing by finding a nearby bounding box performs better, it can still be further improved, as on some occasions the questions and answers are very far apart in a horizontal direction. Furthermore, there are a few mistakes that are made by the OCR results which have caused the LayoutLMv2 to misclassify some labels and hence messes up the key-value pairs. 5. Conclusion In this paper, an end-to-end pipeline of information extraction system for extracting key-value pairs is presented. The proposed system uses deep learning approaches for classifying key-value pairs, and linking the key-value pairs. Its performance and effectiveness are evaluated. First, a few deep learning approaches are employed to explore key-value label classification and linking key-value pairs. After selection, experiments are conducted to evaluate and compare the performances of each model for key-value pairs. It is observed that the results from the LayoutLMv2 model performs the best. It shows that the LayoutLMv2 architecture of layout spatial awareness and words embedding improve the results of standard text classification. Afterwards, experiments for linking key-value pairs are conducted. There are two methods for linking the key-value pairs: the regular expression and a unique method by linking the key-value pairs by finding the nearby bounding boxes. After evaluating the experiment performance, the algorithm of linking by finding the nearby bounding boxes performs better with a precision of 73%, recall of 72% and F1-score of 70%. Several recommendations can be made for future improvements to the overall pipeline. For the key-value label classification, the LayoutLMv2 [13] model has recently introduced a new version known as LayoutLMv3 [23]. The new model may be implemented to investigate its efficacy in enhancing the key-value label classification results compared to the current model. Moreover, other key-value label classification approaches may be explored with sufficient time to improve the model performance further. For linking of key-value pairs, most key-value pairs are typically located beside each other on the same line. However, there are some cases where the values are below the keys. Currently, this proposed information extraction system has not explored that and can only find nearby bounding boxes horizontally. Hence, to further improve the system, it needs to be able to get key-value pairs vertically nearby. Declarations Acknowledgment The first author would like to thank her intern supervisor Mr. Eric Tan of Infocomm Media Development Authority (IMDA), Singapore for his guidance and dedicated supports in the project. References R. Turner, “The myth of the paperless office,” New Library World, vol. 104, no. 3, pp. 120–121, 2003. doi: 10.1108/03074800310467043 . B. Klein, S. Agne, and A. Dengel, “Results of a study on invoice-reading systems in Germany,” International Workshop on Document Analysis Systems , pp. 451–462, 2004. doi: 10.1007/978-3-540-28640-0_43 . R. Hatfield, “How to Capture Freight Logistics Data,” 2021. Web link: . A. Kay, “Tesseract: an open-source optical character recognition engine,” Linux Journal, 2007. Amazon Web Services, “Form Data (Key-Value Pairs),” Web link: https://docs.aws.amazon.com/textract/latest/dg/how-it-works-kvp.html . V. Kumar, P. Kaware, P. Singh, R. Sonkusare, and S. Kumar, "Extraction of information from bill receipts using optical character recognition," International Conference on Smart Electronics and Communication , 2020, pp. 72–77. doi: 10.1109/ICOSEC49089.2020.9215246 . V. N. Sai Rakesh Kamisetty, B. Sohan Chidvilas, S. Revathy, et al ., "Digitization of Data from Invoice using OCR," 6th International Conference on Computing Methodologies and Communicatio n, 2022. doi: 10.1109/ICCMC53470.2022.9754117 . Á. Z. Kaló and M. L. Sipos, "Key-Value Pair Searhing System via Tesseract OCR and Post Processing," IEEE 19th World Symposium on Applied Machine Intelligence and Informatics , 2021. doi: 10.1109/SAMI50585.2021.9378680 . J. Devlin, M. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” North American Chapter of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423 . Y. Liu, M. Ott, N. Goyal, et al ., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv:1907.11692, 2019. W. Wang, B. Bi, M. Yan, et al ., “StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding,” International Conference on Learning Representations , 2020. Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, “LayoutLM: Pre-training of Text and Layout for Document Image Understanding,” 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020. doi: 10.1145/3394486.3403172 . Y. Xu, Y. Xu, T. Lv, et al ., “LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding,” 59th Annu.. Meeting of Assoc. for Computatuional Linguistics and 11th Intl. Joint Conf. on Natural Language Processing , 2021, doi: 10.18653/v1/2021.acl-long.201 . G. Jaume, H. Kemal Ekenel and J. Thiran, "FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents," International Conference on Document Analysis and Recognition Workshops , 2019, pp. 1–6, doi: 10.1109/ICDARW.2019.10029 L, Garncarek, R. Powalski, T. Stanisławek, et al ., “LAMBERT: Layout-Aware language Modeling for information extraction,” International Conference on Document Analysis and Recognition , 2021. doi: 10.1007/978-3-030-86549-8_34 . C, Li, B. Bi, M. Yan, et al ., “StructuralLM: Structural Pre-training for Form Understanding,” 59th Annual Meeting of the Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing , 2021. doi: 10.18653/v1/2021.acl-long.493 . Banksy Annotation Tool. 2020. Weblink: https://github.com/AboutGoods/Banksy-annotation-tool B. Muller, “BERT 101 State of The Art NLP Model Explained,” 2022. Web link: https://huggingface.co/blog/bert-101 . J. Howard, and S. Ruder, “Universal Language Model Fine-tuning for Text Classification,” 56th Annual Meeting of the Association for Computational Linguistics , 2018. doi: 10.18653/v1/P18-1031 . Robust Reading Competition, “ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction,” Web link: https://rrc.cvc.uab.es/?ch=13&com=tasks . S. Agrawal, “Metrics to Evaluate your Classification Model to take the right decisions,” 2021. Web link: https://www.analyticsvidhya.com/blog/2021/07/metrics-to-evaluate-your-classification-model-to-take-the-right-decisions/ B. Johansen, “Named-Entity Recognition for Norwegian,” 22nd Nordic Conference on Computational Linguistics , 2019. Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, “LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking,” arXiv:2204.08387, 2022. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2413475","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":162886180,"identity":"35b12fb5-8327-493b-a6bb-83877384321d","order_by":0,"name":"QiuXing Michelle Tan","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"QiuXing","middleName":"Michelle","lastName":"Tan","suffix":""},{"id":162886181,"identity":"a602e059-b700-4f56-923b-a3a44bf51b48","order_by":1,"name":"Qi Cao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYDACZjBpw8CHIUZASxoDG/FaIOAwCVr4jjMwfvhRcd6ujf344xcf2xjk+Rt4jA3waZE8zMAs2XPmdnIbT46Z5cw2BsMZB3iME/BpMQA5ibftdjIbQw6bMe82BsYNDDzGBwhpYfzbdi6Zjf/5M5AWe6K0MPO2HbBjk0gwfgzUkgjSgtdhkocZm6VlziQnsEm8MWOc+U8iecZhtmK83uc7f/jgxzcVdvb8/OmPP3w4Y2Pb3968WQKfFoYDjA0gKhFIsgFVShARkQcglD0QM38gpHgUjIJRMApGJgAAbOhBpf/RI2kAAAAASUVORK5CYII=","orcid":"","institution":"University of Glasgow","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Qi","middleName":"","lastName":"Cao","suffix":""},{"id":162886182,"identity":"9e125f40-f040-4406-9a4c-9534ff8430e5","order_by":2,"name":"Chee Kiat Seow","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Chee","middleName":"Kiat","lastName":"Seow","suffix":""},{"id":162886183,"identity":"e42fcb6b-3019-453d-892b-bd0c0b4e927b","order_by":3,"name":"Peter Chunyu Yau","email":"","orcid":"","institution":"University of Glasgow","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Peter","middleName":"Chunyu","lastName":"Yau","suffix":""}],"badges":[],"createdAt":"2022-12-25 12:44:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-2413475/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-2413475/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":31045411,"identity":"1e0d7d97-925f-49d7-a8e9-48993678273b","added_by":"auto","created_at":"2023-01-03 17:55:01","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":62351,"visible":true,"origin":"","legend":"\u003cp\u003eFlow chart of the proposed information extraction system\u003c/p\u003e","description":"","filename":"Fig1.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/98c94be13624718c9a7ab0da.png"},{"id":31043718,"identity":"4b68988a-f016-469d-889e-a099d9ade746","added_by":"auto","created_at":"2023-01-03 17:47:01","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":189705,"visible":true,"origin":"","legend":"\u003cp\u003eWord Cloud Of Keys in the Cargo Invoice Dataset\u003c/p\u003e","description":"","filename":"Fig2.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/0116221ed20bfc28f09313f1.png"},{"id":31043719,"identity":"43417476-0401-40e0-9824-83cc44e76491","added_by":"auto","created_at":"2023-01-03 17:47:01","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":76826,"visible":true,"origin":"","legend":"\u003cp\u003eMerging Rule 1 to get Full Questions and Answers Flowchart\u003c/p\u003e","description":"","filename":"Fig3.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/ef6d85eea5f136afc6756b16.png"},{"id":31047315,"identity":"e312eca4-1b80-4077-8db8-79c2718df089","added_by":"auto","created_at":"2023-01-03 18:03:01","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":80746,"visible":true,"origin":"","legend":"\u003cp\u003eMerging Rule 2 to get Full Questions and Answers Flowchart\u003c/p\u003e","description":"","filename":"Fig4.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/13a0a5fe967944db31768599.png"},{"id":31043714,"identity":"7cdf1990-1ae1-4076-9231-8a156f33c55f","added_by":"auto","created_at":"2023-01-03 17:47:01","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":37857,"visible":true,"origin":"","legend":"\u003cp\u003eFlowchart of the Key-Value Pairing Rule 1\u003c/p\u003e","description":"","filename":"Fig5.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/ba6d143a377eae504ceec075.png"},{"id":31043717,"identity":"bf574b09-4c3c-4739-9752-a4bf6a3e1173","added_by":"auto","created_at":"2023-01-03 17:47:01","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":59828,"visible":true,"origin":"","legend":"\u003cp\u003eFlowchart of the Key-Value Pairing Rule 2\u003c/p\u003e","description":"","filename":"Fig6.png","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/2994c767ccd98296e24e6978.png"},{"id":31047316,"identity":"ff2001fb-7003-44c3-8812-23697947ad74","added_by":"auto","created_at":"2023-01-03 18:03:08","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":837689,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2413475/v1/d1047309-fff3-4526-ba2d-676f0ed95b35.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Information Extraction System for Cargo Invoices","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eOn a daily basis, many organizations, such as logistics companies, deal with many paper documents such as receipts or cargo invoices [1]. Currently, the majority of these documents must be processed manually, which is time-consuming and expensive [2].\u003c/p\u003e \u003cp\u003eOne of the most difficult aspects of invoice processing for logistics organizations is the time-consuming, intensive in-house procedure, requiring numerous manual works when extracting and keying data into various internal software systems. In addition, data captured from logistics invoices poses a particular difficulty because they are received in multiple formats. This is a significant issue, due to the intricacy of the invoices. Manually inspecting and correcting scanning errors greatly lengthens the correction and processing time [3].\u003c/p\u003e \u003cp\u003eDigital invoices contain various components such as tables, key-value pairs and figures. A key-value pair is made up of two connected data elements: a key which is a constant defining the data set; and a value which is a variable that is part of the set. A fully formed key-value pair will be like Gross Weight: 123 kg, where \u0026ldquo;Gross Weight\u0026rdquo; is the key and \u0026ldquo;123 kg\u0026rdquo; is the value.\u003c/p\u003e \u003cp\u003eA corporation can benefit from digitising and extracting key information on a number of levels. Business owners can better track their processes, provide better customer services, increase employee productivity, and cut costs. Optical character recognition (OCR) also known as text recognition, is the process of extracting text information from scanned documents and images. Current OCR systems, such as Tesseract [4] and EasyOCR can recognize raw text in unformatted digital documents or images. But they are not capable of extracting information like key-value pairs from unformatted data. Key-value pairs are the most significant components in digital cargo invoices. Key-value pairs make raw texts more understandable.\u003c/p\u003e \u003cp\u003eThe existing solutions, such as Amazon Web Services (AWS) Textract, provide such a service [5]. However, it is a paid service in which many of the small and medium enterprises (SMEs) are reluctant to pay extra costs for going digital since the SMEs usually have a tight budget. Furthermore, the off-the-shelf solutions may not fit the specific domains, as most of the solutions lack of flexibility to be customized or tailored according to different requirements from different customers. Moreover, the AWS Textract is a service hosting on the cloud platform. Users need to upload their digital invoices and receipts to the cloud platform, in order to get them processed. It raises possible privacy concerns since some companies might treat the invoices and data as private and business sensitive.\u003c/p\u003e \u003cp\u003eHence, automation of extracting key-value pairs within unformatted digital invoices is proposed as it can significantly reduce manpower and costs while simultaneously ensuring the reliability of the data retrieved. In this paper, an information extraction system with the deep learning approaches that is capable of extracting key-value pairs will be presented to improve the overall performance. The proposed information extraction system would be beneficial as it would help companies achieve workflow efficiency, resource utilization and eliminate costly errors.\u003c/p\u003e \u003cp\u003eThe remaining parts of this paper are organized next. Section II presents the prior works in the literature. Section III explains the pipeline of the proposed information extraction system. Section IV analyses and discusses experiment results. Lastly, Section V concludes this paper.\u003c/p\u003e"},{"header":"2. Related Work","content":"\u003cp\u003eThere are three different techniques of key-value pair extraction: regular expression, natural language processing (NLP) and layout detection. This section studies the prior arts related to OCR key-value pairs extraction approaches.\u003c/p\u003e \u003cp\u003eOCR is an important portion as it will affect the results for the key-value pairs. As mistaken words might be wrongly classified when performing key-value pair extraction. Hence, OCR plays a crucial role for key-value pair extraction. OCR is used to extract text information from scanned documents and images. Vedant Kumar \u003cem\u003eet al\u003c/em\u003e. [6] use OCR on bill receipts images taken from a mobile phone to extract out the text information from the receipts. Before extracting the information, they have done some image pre-processing such as binarization, removing of shadows and etc. After that the processed image is inputted into the Tesseract OCR engine to extract the text. OCR has been used for scanned documents to extract the text information in [7]. Similarly, they have pre-processed the scanned invoices by sharpening the images, threshing and binarization.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Key-Value Pair Extraction using Regular Expression\u003c/h2\u003e \u003cp\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eA key-value searching system is developed by Kal\u0026oacute; and Sipos [8] that uses the open-source Tesseract OCR engine and post-processing techniques with regular expressions. Regular expressions are patterns used to match character combinations in strings. To find key-value pairs with regular expression, we can find by text with colon. If the regular expression manages to find the word, this means that it is a key.\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e \u003cp\u003eThis method can be used to create a low-cost office automation system for invoice processing. Regular expression is very effective at identifying patterns and replacing strings. It increases efficiency and versatility by allowing constant patterns and variable values to be searched in the text. Before developing the key-value searching system with regular expression, it is necessary to learn what patterns are there in the dataset, collate all the different types of patterns and put them into a pattern dictionary. The regular extraction pattern dictionary can be expanded over time to learn a wider variety of patterns, with more datasets being included and more patterns being added. New patterns can be added based on the errors. If these errors occur during a later process, they would be stored and corrected.\u003c/p\u003e \u003cp\u003eUsing regular expressions to extract key-value pairs is efficient in finding specific patterns or text. However, when a new text or pattern is introduced, that would be an issue as the system does not understand and cannot extract them. Additionally, some keys and values may have a variety of different forms of patterns. Thus, there is a need to manually look through the dataset and update the patterns to extract the key-value pairs correctly and effectively. Furthermore, with many patterns declared, it degrades the readability and performance of the codes. Hence, with these problems being identified, extracting key-value pairs using regular expression is not efficient. Though, regular expression may be suitable for post-processing in assistant of validation of value formats.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Key-Value Pair Extraction using NLP\u003c/h2\u003e \u003cp\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eBidirectional Encoder Representations from Transformers (BERT) [9] utilizes Transformer, an attention mechanism that discovers contextual relationships between words (or subwords) in a text. Transformer has two independent working parts: an encoder that reads text inputs, and a decoder that generates a task prediction. BERT has two training tasks: Masked Language Model (MLM) and Next Sentence Prediction.\u003c/p\u003e \u003cp\u003eRobustly Optimized BERT Pretraining Approach (Roberta) [10] is an optimized approach for pretraining natural language processing. Roberta has almost similar architecture to BERT. Some changes are removing the Next Sentence Prediction and training with bigger batch sizes and longer sequences.\u003c/p\u003e \u003cp\u003eThe BERT is extended to a new model, StructBERT, by incorporating language structures into pre-training [11]. StructBERT expands BERT by leveraging structural information, such as word-level and sentence-level ordering, in addition to the present masking method. Two new structural objectives are added to model pre-training, focusing on inner-sentence and inter-sentence structures. In this way, the language characteristics of pre-training phase are explicitly represented.\u003c/p\u003e \u003cp\u003eFurthermore, StructBERT uses structural pre-training to embed dependency between words and sentences in the contextualized representation, allowing the model to be more generalizable and adaptable. This allows StructBERT to represent language structures explicitly by requiring it to reconstruct the correct order of words and sentences in order to make accurate predictions. StructBERT is compared with a few methods such as BERT, Roberta, etc. Through the experimentation results, it shows that the StructBERT model can obtain better results in a variety of downstream tasks with the improvement of the F1 scores to 93%.\u003c/p\u003e \u003cp\u003eStructBERT framework is better than the regular expression approach, as it is trained on the textual data. It understands the key-value pairs and reconstructs the correct order of words, in order to make accurate predictions. However, the NLP approach only takes in the text from the document and does not incorporate the position of the text. In the tasks of extracting key value pairs, the position of the text is useful. However, StructBERT is still useful to understand the key-value pair structure and can be used together with layout awareness algorithms to improve the accuracy.\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Key-Value Pair Extraction using Layout Detection\u003c/h2\u003e \u003cp\u003eLayoutLM [12] is the few of the models successfully reported that does labeling using positional information, text-based information, and image information. As an upgrade from LayoutLM, the LayoutLMv2 is reported in [13], which uses model architectures and pre-training tasks to pre-train text, layout, and image in a multi-modal framework. Unlike other Visually-rich Document Understanding (VrDU) approaches which aims to analyze scanned documents [13], LayoutLMv2 not only considers the text and layout information, but also integrates the new text-image alignment and text image matching tasks, which help to learn cross-modality interaction. In addition, LayoutLMv2 incorporates a spatially-aware self-attention mechanism into the Transformer design, allowing the model to completely comprehend the relative positioning relationships between different text blocks.\u003c/p\u003e \u003cp\u003eExperiments are conducted to compare the efficiency of different types of models [13] with multiple datasets, including open sourced FUNSD dataset [14] which consists of scanned forms that differ significantly in terms of their structure and presentation. The results show that LayoutLMv2 outperforms strong baselines and achieves better performance on a variety of downstream visually rich document comprehension tasks.\u003c/p\u003e \u003cp\u003eAnother approach, LAMBERT, is reported in [15] that tackles the challenges of comprehending documents where non-trivial layout affects the local semantics. It is a Layout-Aware Language Model, which combines NLP methods with layout understanding mechanisms. The LAMBERT uses the layout information of the document and train it with the pretrained Roberta. Hence, users do not need to input any images, as the model uses the bounding box coordinates instead to understand the layout of the document.\u003c/p\u003e \u003cp\u003eA pre-training approach, StructuralLM, utilizes cells and document layouts from scanned documents [16]. This approach is influenced by LayoutLM, but with some differences. Firstly, rather than modeling word-level 2D-position embeddings, the StructuralLM uses cell-level 2D-position embeddings to represent the layout information of cells. The StructuralLM also introduces a cell position classification which attempts to predict a cell location based on the location of cells and their semantic relationships. Lastly, the StructuralLM removes the image embeddings from LayoutLM that are only needed in the downstream tasks while keeping the 1D-position embeddings to describe the positional link between tokens from the same cell.\u003c/p\u003e \u003cp\u003eThe LayoutLMv2 framework with layout detection approach is better than models with the regular expression and NLP approaches for extracting key-value pairs. LayoutLMv2 integrates NLP processing techniques in the framework and text-image alignment where it is able to comprehend the relative positioning relationships between different text blocks. This is efficient for extracting key-value pairs as most of them are either next to each other or positioned on top of the other. However, while experimenting with LayoutLMv2 on our dataset in our research, as our dataset has several different layouts, it has been observed that the LayoutLMv2 has some limitations. While LayoutLMv2 trains well on the layout of the invoices and when a different layout of the invoice is input as test data, it will not do as well. It may not be able to accurately detect the key-value pairs. However, this can be solved by training more datasets from different varieties.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Proposed Methodology","content":"\u003cp\u003eIn this section, the overall design of the proposed information extraction pipeline for cargo invoices is described.\u003c/p\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n\u003ch2\u003e3.1 System Arcitecture Pipeline\u003c/h2\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n\u003cp\u003eThe flow chart of the proposed information extraction system for cargo invoices is shown in Fig.\u0026nbsp;1. The proposed pipeline of information extraction system consists of two portions: the key-value pair classification and linking the key-value pairs. The key-value pair classification portion will be to explore models to classify the text from OCR to \u0026ldquo;Question\u0026rdquo;, \u0026ldquo;Answer\u0026rdquo; and \u0026ldquo;Others\u0026rdquo;. For the second portion of linking the key-value pairs, it takes the output from the key-value pair classification to link the key-value pairs with the use of layout spatial awareness like the bounding box position.\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eTo begin the pipeline of the proposed system, data collection is performed by taking images of cargo invoices from warehouses. After the data collection, there would be a quality check to ensure that the images are not blurry, as blurry images would hinder the performance. After that, the cargo invoices will be annotated into FUNSD format [14] using an open source tool, Banksy [17]. This open source tool is an image annotation tool where the Named Entity Recognition (NER), Named Entity Linking (NEL), and a box region on the image will be outputted. The NER will be the labels such as \u0026ldquo;Question\u0026rdquo;, \u0026ldquo;Answer\u0026rdquo; and \u0026ldquo;Others\u0026rdquo;. The NEL is the tasks to link the \u0026ldquo;Question\u0026rdquo; text to the \u0026ldquo;Answer\u0026rdquo; text. Lastly, the box region is the bounding box, where the bounding box of texts is drawn.\u003c/p\u003e\n\u003cp\u003eThe next step is to perform data cleansing on the data. Data cleaning is a necessary process because it improves our data quality and increases overall productivity. After cleaning, the data is formatted into a model-ready form for training and split the dataset into the train set (80%), and the test set (20%).\u003c/p\u003e\n\u003cp\u003eSubsequently, different algorithms for the key-value pair classification and detection are explored. The text classification algorithms are only trained on the text and labels. Whereas for linking of key-value pairs algorithms, it uses layout awareness and text embeddings to train and get the keys and values.\u003c/p\u003e\n\u003cp\u003eAfter comparing the results of multiple key-value pair classification algorithms such as LayoutLMv2, BERT, etc., the best model will be chosen for key-value pair classification. For the algorithms to link the key-value pairs, two methods explored are the regular expression and linking key-value pairs by the bounding box location. After comparing the results of these two algorithms, the best model will be used and output the final key-value pair. The output will be evaluated with the ground truth. If the performance of the output from the proposed information extraction pipeline is low, it will go through another iteration to improve the accuracy by experimenting with different model hyperparameters. This will be an iterative process until it achieves good accuracy. Finally, if the performance is satisfied, it will go through inferencing using the optimal model chosen by the proposed pipeline, and merge the output of word-level key and value to the entire key and value pairs.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e\n\u003ch2\u003e3.2 Selection of Algorithms\u003c/h2\u003e\n\u003cp\u003eTo explore key-value pair classification, two types of different techniques will be used to compare: NLP methods, and NLP methods combined with layout spatial awareness.\u003c/p\u003e\n\u003cp\u003eNLP methods are considered as they can understand context-sensitive human language. By using NLP, it can effectively extract data or information from text-based documents [16]. For the NLP methods, both BERT model and Universal Language Model (ULMFit) model are explored. The BERT model analyzes a word's left and right sides to infer its context. Additionally, The BERT model uses MLM, which covers or masks a word in a sentence. MLM enables or enforces bidirectional learning from a text by requiring BERT to predict the word on either side of the covered word [18]. With this in mind, applying it to the cargo invoice dataset, the model could understand the key-value pairs. For example, the term \"Product\" could be next to the word \"Number\".\u003c/p\u003e\n\u003cp\u003eThe ULMFit model is trained using a general-domain corpus to capture overall language properties at several layers and then learns task-specific features [19]. The model uses a 3-layer ASGD Weight-Dropped Long Short-Term Memory Networks (LSTM) architecture. There are three steps to the training: Pre-training a general language model on a text, fine-tuning the language model on a target task, and fine-tuning the classifier on the target task are the first two steps.\u003c/p\u003e\n\u003cp\u003eThe other technique for key-value pair classification is NLP with layout spatial awareness. The first algorithm is YOLOv4 combined with BERT. This algorithm is able to understand the position of the text as well as using BERT to understand the text embeddings. The second algorithm is LayoutLMv2; LayoutLMv2 not only considers the text and layout information, but also integrates the new text-image alignment and text image matching tasks, which help to learn cross-modality interaction [13].\u003c/p\u003e\n\u003cp\u003eFor the two algorithms for the linking of key-value pairs, i.e., the regular expression and the pairing by the nearest bounding box, the regular expression is efficient in finding specific patterns or text which is common in cargo invoices, for example, \"Product No.: 123\". The patterns commonly seen for keys and values are with colons. For the regular expression algorithm, an analysis will be done to see the common key-value pairs and their patterns to extract the key and values pairing. For the pairing by the nearest bounding box algorithm, it will pair keys and values by finding the nearby bounding box. The pairing rules are to merge the text only if they have the exact vertical coordinates (i.e., y coordinates), overlapped bounding box and are close.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e\n\u003ch2\u003e3.3 Datasets\u003c/h2\u003e\n\u003cp\u003eThe datasets used for experiments and performance comparisons are the FUNSD dataset [14], and Cargo Invoices dataset. The Cargo Invoices dataset is constructed in this research which are all the cargo invoices images from the warehouses. TABLE I describes the number of training and testing data in each dataset.\u003c/p\u003e\n\u003cp\u003eTable I SAMPLE SIZES OF UTILIZED TWO DATASETS\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Taba\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth colspan=\"4\" align=\"left\"\u003e\n\u003cp\u003eDataset Used\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eDataset\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eTraining\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eTesting\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eTotal Data\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eFUNSD [14]\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e149\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e50\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e199\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eCargo Invoices Dataset\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e484\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e121\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"char\" char=\".\"\u003e\n\u003cp\u003e605\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003c/div\u003e"},{"header":"4. Experiment Results And Analysis","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e\n\u003ch2\u003e4.1 Evaluation Metrics\u003c/h2\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n\u003cp\u003eThe evaluation metrics used in the ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction [20] are employed to evaluate the experiment performances in this paper: the Precision, Recall and F1 Score. The extracted text will be compared to the ground truth for each test image. If the submitted content and category of the extracted text matches the ground truth, it is marked as correct; otherwise, it is labelled as inaccurate. Furthermore, the algorithms will also be further evaluated by the classification report and confusion matrix.\u003c/p\u003e\n\u003cp\u003eThe values of the precision, recall, F1-score, and support are the four most critical headers for classification results to pay attention in classification reports. The value of precision refers to the ability of a classifier to avoid labelling a negative instance as positive [21]. The recall is the ability of a classifier to find all positive instances. The F1 score is a weighted harmonic mean of precision and recall. The support is the number of actual class occurrences in the specified dataset.\u003c/p\u003e\n\u003cp\u003eThe precision value can be calculated in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). Precision is useful when False Positives (FP) pose a greater threat than False Negatives (FN). In our research, having lots of wrong results would affect the key-value pair linking and the final key-value pair results.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ1\" class=\"mathdisplay\"\u003e$$Precision= \\frac{TP}{TP+FP}$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere TP denotes True Positive.\u003c/p\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n\u003cp\u003eThe recall value is computed in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e). When FN is more important than FP, it is a valuable metric.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ2\" class=\"mathdisplay\"\u003e$$Recall= \\frac{TP}{TP+FN}$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n\u003cp\u003eThe F1 score ranges between 0 and 1, that can be calculated in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e). If the F1 score scores closer to 1 it means that the model performs better.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ3\" class=\"mathdisplay\"\u003e$$F1 Score= 2 \\times \\frac{Precision \\times Recall}{Precision+Recall}$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec12\" class=\"Section2\"\u003e\n\u003ch2\u003e4.2 Label Classification Comparisons\u003c/h2\u003e\n\u003cp\u003ePrior to classification, the dataset is first split into 80% training and 20% testing. The four classification models experimented with are BERT, ULMFit, YOLOv4 combined with BERT and LayoutLMv2, each of which will classify the text in to \u0026ldquo;Question\u0026rdquo;, \u0026ldquo;Answer\u0026rdquo; and \u0026ldquo;Others\u0026rdquo;. In the experiments in this sub-section, the dataset used in the constructed Cargo Invoice dataset in this research.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e1) Results of the ULMFit Model\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWith the Cargo Invoice dataset, the ULMFit model is utilized for key-value pairs classification. The results of precision, recall, F1 score and support for the ULMFit model are shown in TABLE II. The ULMFit model scores 93% accuracy. Overall, the performance of this model is good. However, the model does not perform well for the \"Other\" class, compared to the other two classes. It might because there are fewer \"Other\" labels as compared to those of \"Question\" and \"Answer\".\u003c/p\u003e\n\u003cp\u003eTable II ULMFit Results with Cargo Invoice Dataset\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabb\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eClass\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003ePrecision\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRecall\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eF1 Score\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eSupport\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eQuestion\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e99%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e882\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eAnswer\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e88%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e92%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e882\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOther\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e93%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e80%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e86%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e568\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e93%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e2332\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e2) Results of the BERT Model\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe experiment results of the BERT classification model with the Cargo Invoice dataset are shown in TALBE III. The BERT model achieves 88% accuracy. But the \u0026ldquo;Answer\u0026rdquo; label does not do well for the values of precision, recall and F1 score. This model can perform better on the \u0026ldquo;Other\u0026rdquo; label as the score for precision, recall, and F1 score is higher than the others.\u003c/p\u003e\n\u003cp\u003eTable III BERT Model Classification Results\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabc\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eClass\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003ePrecision\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRecall\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eF1 Score\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eSupport\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eQuestion\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e85%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e91%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e88%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e2128\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eAnswer\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e83%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e76%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e79%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1387\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOther\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e95%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e95%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1493\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e88%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e5008\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e3) Results of YOLOv4 Combined with BERT Model\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe classification results of the YOLOv4 combined with BERT model is shown in TABLE IV. It shows that the overall performance is worse than other experimented classification models. For \u0026ldquo;Other\u0026rdquo; label, it performs the worst as compared to other two labels. It might be due to the imbalanced dataset whereby \u0026ldquo;Other\u0026rdquo; has the least amount of data comparatively. The results in this experiment show that the combination of YOLOv4 and BERT algorithms does not enhance the performance, as the BERT model itself manages to achieve 88% for the cargo invoices as seen in TABLE III.\u003c/p\u003e\n\u003cp\u003eTable IV Results of YOLOv4 Combined with BERT Model\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabd\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003ePrecision\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eRecall\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eF1 Score\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eSupport\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eAnswer\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e52%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e42%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e46%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e882\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eOther\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e46%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e39%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e40%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e568\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eQuestion\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e49%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e40%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e44%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e882\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eAccuracy\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e40.27%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e2332\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e4) Results of the LayoutLMv2 Model\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFor the LayoutLMv2 model, the labeling methods are slightly different from other three models. The labels of the LayoutLMv2 model follow the BIOES tagging, where \u003cem\u003eB\u003c/em\u003e means beginning; \u003cem\u003eI\u003c/em\u003e mean in the middle; \u003cem\u003eE\u003c/em\u003e means the ending; \u003cem\u003eO\u003c/em\u003e means others; and \u003cem\u003eS\u003c/em\u003e means a single word representing a full sequence [22]. For example, \u0026ldquo;Product No.\u0026rdquo;, will be split into \u0026ldquo;Product\u0026rdquo; which will be B-QUESTION and \u0026ldquo;No.\u0026rdquo; is \u0026ldquo;E-QUESTION\u0026rdquo;.\u003c/p\u003e\n\u003cp\u003eThe experiment results of the LayoutLMv2 classification model are shown in TABLE V. It is observed that the LayoutLMv2 model achieves an accuracy of 96%, which is the highest out of other three classification models. All the labels achieve 90% and above for the precision, recall and F1 score. It shows that the LayoutLMv2 model can predict well and distinguish each labels properly. Even though there is some unbalanced data, the model is still able to predict well for all the labels. The analysis on the confusion matrix will be further conducted whether this model is the best.\u003c/p\u003e\n\u003cp\u003eTable V LayoutLMv2 Model Classification Results\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabe\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003ePrecision\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRecall\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eF1 Score\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eSupport\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eB-ANSWER\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e93%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e331\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eB-QUESTION\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e507\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eE-ANSWER\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e90%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e93%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e331\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eE-QUESTION\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e507\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eI-ANSWER\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e915\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eI-QUESTION\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e104\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eO\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e93%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e94%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e1387\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eS-ANSWER\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e95%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e502\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eS-QUESTION\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e98%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e97%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e375\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003e96%\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e4959\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e5) Experiment Results with the FUNSD Dataset\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eBesides the classification experiments on the constructed Cargo Invoice dataset, the experiments are also performed on the FUNSD dataset to compare the performances of these four classification models. The overall performance with the FUNSD dataset is shown in TABLE VI.\u003c/p\u003e\n\u003cp\u003eTable VI Classification Results with the FUNSD dataset\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabf\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eModel\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003ePrecision\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eRecall\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eF1 Score\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eULMFit\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e77%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e76%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e76%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eBERT\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e55%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e67%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e60%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eYOLO combined with BERT\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e49%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e40%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e44%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eLayoutLMV2\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e80%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e85%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e83%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eIt has shown that LayoutLMv2 model performes the best out of other three models. This has been demonstrated that integrating the spatial-aware self-attention Mechanism into the Transformer architecture in the LayoutLMv2 model has fully allowed the model to understand the relative positional relationship among different text blocks. On the other hand, YOLOv4 combined with the BERT model does not perform well. Even though it is trained with the text along with the bounding box of the texts, it is not able to find the relative positional relationship accurately compared to the LayoutLMv2 model. Thus, the LayoutLMv2 model will be the model chosen by the proposed pipeline for the key-value pair classification tasks.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec13\" class=\"Section2\"\u003e\n\u003ch2\u003e4.3 Linking of Key-value Pair Comparisons\u003c/h2\u003e\n\u003cp\u003eThe two algorithms for the linking of key-value pairs to be evaluated are: the regular expression algorithm, and finding the pairs by the nearest bounding box algorithm.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e1. Regular Expression Algorithm\u003c/em\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eBefore beginning regular expression, some analysis is done on the dataset to understand the keys and values patterns, as well how they are paired to enhance the process of extracting key-value pairs. There are various formats of the keys, such as the short forms. Hence we need to create a list of the different forms of the keys. Some examples of the types of formats can be seen in TABLE VII.\u003c/p\u003e\n\u003cp\u003eTable VII Some examples of different formats of keys\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabg\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eWord\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eDifferent Formats of Keys\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eGross Weight\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eG/W\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eG/Weight\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eGross wt\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eGross wght\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eGross wght(kg)\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eNet Weight\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNet WT\u003c/p\u003e\n\u003cp\u003eNet wght\u003c/p\u003e\n\u003cp\u003eNet weight(kg)\u003c/p\u003e\n\u003cp\u003eNet wt(kg)\u003c/p\u003e\n\u003cp\u003eNet wght(kg)\u003c/p\u003e\n\u003cp\u003eN/W\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eDimension\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eDim\u003c/p\u003e\n\u003cp\u003eDims\u003c/p\u003e\n\u003cp\u003eDim(mm)\u003c/p\u003e\n\u003cp\u003eDIM (CM)\u003c/p\u003e\n\u003cp\u003eDimensions(cm)\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eAfter finding out the different formats of the keys from the Cargo Invoice dataset, a word cloud is created to better understand the keys in the dataset, as shown in Fig.\u0026nbsp;2. Through the wordcloud, words such as \"gross weight\", \"net weight\", \"PO\", \"country of origin\", and \"dimension\" stand out the most. Hence, from the common keys, the next step is understanding their values pattern. Some examples of the patterns of values are shown in TABLE VII.\u003c/p\u003e\n\u003cp\u003eTable VIII Pattern of Values\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabh\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eKey\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eValues\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003ePatterns\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGross Weight\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eG/W 2134\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e23Kg/g\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e88,500 KG\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFor gross weight the pattern is always an integer/float followed by the metrics (KG/lb/g). Sometimes G/W would be infront of the integer/float.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNet Weight\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e270 lb\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e88,500 KG\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e3.840 KG\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFor Net weight the pattern is similar to Gross weight. The pattern is always an integer/float followed by the metrics (KG/lb/g).\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003ePO\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003ePO-IMA-21007\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003ePO2000004328\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFor PO the pattern is PO followed by letters then integers or just integers.\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eDimension\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e228.00 X 148.00 X 165.00 CM\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e89.76 X 58.26 X 64.96 INCH\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003e3815*2150*2550mm\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFor Dimension, the pattern is integer/float X integer/float X integer/float\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter doing analysis on these patterns, key value pairs will be extracted. For the work flow of the extraction of key-value pairs are as follow:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eFind all key-value pairs.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eCalculate Levenshtein distance with given identifiers to see which one is the most likely identifier.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eReturn key-value pair with the lowest normalized Levenshtein distance.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cem\u003e2) Pairing Via Nearest Bounding Box\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe input for this part is the output from the LayoutLMv2 classification model. The words are in the word level form and the labels are in BIOES tagging [22].\u003c/p\u003e\n\u003cp\u003eHence, we will need to combine the word level text into sentence level to make sense of the full question and answer. To combine them into sentence level, the bounding box position and the labels will be critical.\u003c/p\u003e\n\u003cp\u003eThere are two merging rules to get the full questions and answers as currently in the word level form. These two merging rules will be described next.\u003c/p\u003e\n\u003cp\u003eThe flowchart of the merging rule 1 is shown in Fig.\u0026nbsp;3. First, it gets the coordinates, labels and text from the LayoutLMv2 Predicted results and images. Then it traverses through the whole results and compare whether if both labels are same. If not the same, then it is assign as \u003cem\u003eNone\u003c/em\u003e to the variables called \u0026ldquo;neighbours\u0026rdquo;. If it is the same, then it checks whether the vertical \u003cem\u003ey\u003c/em\u003e coordinates are the same. This is to check whether these nodes are on the same line. If they are not on the same line, then it is assign as \u003cem\u003eNone\u003c/em\u003e. If they are on the same line, then it proceeds to check whether the right side of a bounding box is the same as the left side of another bounding box. If it is the same, then the bounding box ID is assigned to each of the nodes in the \u0026ldquo;Neighbours\u0026rdquo; variable. Otherwise, it will be assign \u003cem\u003eNone\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eAfter that, the second part of the merging rule 1 shown in the blue box in Fig.\u0026nbsp;3 checks if there are multiple bounding boxes connected to one another. If yes, then group them into one group. If a bounding box is not connected to another bounding box, then they will be put into individual groups.\u003c/p\u003e\n\u003cp\u003eLastly, the purple box in Fig.\u0026nbsp;3, the merged results are derived and updated into the dictionary into the form of:\u003c/p\u003e\n\u003cp\u003e{'text': pred[link_list[0]]['text'],\u003c/p\u003e\n\u003cp\u003e'labels': pred[link_list[0]]['labels'],\u003c/p\u003e\n\u003cp\u003e'left_neighbour': None,\u003c/p\u003e\n\u003cp\u003e'right_neighbour': None,\u003c/p\u003e\n\u003cp\u003e'width': pred[link_list[0]]['width'],\u003c/p\u003e\n\u003cp\u003e'height': pred[link_list[0]]['height']}}\u003c/p\u003e\n\u003cp\u003eThe second merging rule is shown in Fig.\u0026nbsp;4. For the second merging rule of the full questions and answer, it uses the data derived from the mergin rule 1 to further enhance the method of getting the full questions and answers.\u003c/p\u003e\n\u003cp\u003eFirst, the merging rule 2 gets the center of the bounding box from the mean of the \u003cem\u003ex\u003c/em\u003e and \u003cem\u003ey\u003c/em\u003e coordinates. Next it finds the nearby bounding boxes and checks if both labels are same. If not the same, the nodes are assigned as \u003cem\u003eNone\u003c/em\u003e for the variable \u0026ldquo;neighbours\u0026rdquo;. If both labels are the same, it then checks if the center of the bounding box A is within the range of the bounding box B in the \u003cem\u003ey\u003c/em\u003e direction. If not, the variable \u0026ldquo;neighbours\u0026rdquo; is assigned to \u003cem\u003eNone\u003c/em\u003e. Otherwise, it checks if the center of the bounding box B is within the range of the bounding box A in the \u003cem\u003ey\u003c/em\u003e direction. If yes, it checks if the two bounding boxes overlap in the \u003cem\u003ex\u003c/em\u003e direction, or if the 2 bounding boxes gap in the \u003cem\u003ex\u003c/em\u003e direction is \u0026lt;\u0026thinsp;10% of the width of the image. If the condition is true, the \u003cem\u003ebbox_id\u003c/em\u003e of these 2 bounding boxes (bboxes) are assigned as \u003cem\u003eneighbours\u003c/em\u003e to each other's nodes. If the condition is fasle, the \u003cem\u003ebbox_id\u003c/em\u003e of these 2 bboxes are assigned as \u003cem\u003eNone\u003c/em\u003e to the variables \u0026ldquo;neighbours\u0026rdquo;.\u003c/p\u003e\n\u003cp\u003eThe second part of the merging rule 2 in the blue box is to check if there are multiple bounding boxes connected to one another. If true, then group these bounding boxes into one group. If a bounding box is not connected to others, then it is put into an individual group. These results will be merged and passed to the last process in the merging rule 2 shown in the purple box. Lastly, the results are merged and updated into the dictionary in the same format as that of the merging rule 1.\u003c/p\u003e\n\u003cp\u003eAfter merging the words into the full key and values, we need to start pairing the keys and values together. There are two pairing rules for this step.\u003c/p\u003e\n\u003cp\u003eThe flowchart in Fig.\u0026nbsp;5 shows how the key-value pairing rule 1 works. It takes in the results from the merging of full questions and answers. It then checks if the \u0026ldquo;Question\u0026rdquo; and \u0026rdquo;Answer\u0026rdquo; have the right neighbour, followed by checking whether the \u003cem\u003ey\u003c/em\u003e coordinates between both \u0026ldquo;Question\u0026rdquo; and \u0026rdquo;Answer\u0026rdquo; are identical. Lastly, the key-value pairing rule 1 checks whether the coordinates of the right side of the \u0026ldquo;Question\u0026rdquo; bounding box is identical to the left side of the \u0026ldquo;Answer\u0026rdquo; bounding box.\u003c/p\u003e\n\u003cp\u003eAfter performing the key-value pairing rule 1, the output is splitted into successfully paired and unsuccessfully paired. The unsuccessfully paired results are inputted into the key-value pairing rule 2 to pair the questions ans answers. The key-value pairing rule 2 is presented in Fig.\u0026nbsp;6. Firstly, it checks whether both centers of the bounding boxes are within the range of the other bounding box in the \u003cem\u003ey\u003c/em\u003e axis. Next it checks whether they overlap in the \u003cem\u003ex\u003c/em\u003e axis or the gap in the \u003cem\u003ex\u003c/em\u003e axis is less than 50% of the width of the bounding box with the smaller width value.\u003c/p\u003e\n\u003cp\u003eThe final output consists of the label, bounding boxes, text, left_neightbour, right_neighbour, the width, the height, the variable of \u003cem\u003epair_with\u003c/em\u003e, and lastly the center attribute.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e3) Results Comparison\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eAfter performing the key-value pairing, the results are evaluated by comparing the derived final key-value pairs with the ground-truth. The experiment result comparisons are shown in TABLE IX. The metrics used to do the comparison is the precision, recall and F1 score.\u003c/p\u003e\n\u003cp\u003eTable IX Results Comparison for Key Value Pairing\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tabi\" border=\"1\"\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eAlgorithm\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003ePrecision\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eRecall\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003eF1 Score\u003c/span\u003e\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003eRegular Expression\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e63%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e60%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e66%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"BoldUnderline\"\u003ePairing by Finding Nearby Bounding Box\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e73%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e72%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cspan class=\"Underline\"\u003e70%\u003c/span\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eIt is observed from the results that the second algorithm, i.e., the pairing by finding the nearby bounding box, has done a better job with a precision of 73%, recall of 72% and F1 score of 70%. In comparison, the first algorithm, i.e., the regular expression, achieves a precision of 63%, recall of 60% and F1 score of 66%. The reason for the regular expression not perform well might because there are limited patterns introduced into the system. Even though the algorithm of the pairing by finding a nearby bounding box performs better, it can still be further improved, as on some occasions the questions and answers are very far apart in a horizontal direction. Furthermore, there are a few mistakes that are made by the OCR results which have caused the LayoutLMv2 to misclassify some labels and hence messes up the key-value pairs.\u003c/p\u003e"},{"header":"5. Conclusion","content":"\u003cp\u003eIn this paper, an end-to-end pipeline of information extraction system for extracting key-value pairs is presented. The proposed system uses deep learning approaches for classifying key-value pairs, and linking the key-value pairs. Its performance and effectiveness are evaluated.\u003c/p\u003e \u003cp\u003eFirst, a few deep learning approaches are employed to explore key-value label classification and linking key-value pairs. After selection, experiments are conducted to evaluate and compare the performances of each model for key-value pairs. It is observed that the results from the LayoutLMv2 model performs the best. It shows that the LayoutLMv2 architecture of layout spatial awareness and words embedding improve the results of standard text classification.\u003c/p\u003e \u003cp\u003eAfterwards, experiments for linking key-value pairs are conducted. There are two methods for linking the key-value pairs: the regular expression and a unique method by linking the key-value pairs by finding the nearby bounding boxes. After evaluating the experiment performance, the algorithm of linking by finding the nearby bounding boxes performs better with a precision of 73%, recall of 72% and F1-score of 70%.\u003c/p\u003e \u003cp\u003eSeveral recommendations can be made for future improvements to the overall pipeline. For the key-value label classification, the LayoutLMv2 [13] model has recently introduced a new version known as LayoutLMv3 [23]. The new model may be implemented to investigate its efficacy in enhancing the key-value label classification results compared to the current model. Moreover, other key-value label classification approaches may be explored with sufficient time to improve the model performance further.\u003c/p\u003e \u003cp\u003eFor linking of key-value pairs, most key-value pairs are typically located beside each other on the same line. However, there are some cases where the values are below the keys. Currently, this proposed information extraction system has not explored that and can only find nearby bounding boxes horizontally. Hence, to further improve the system, it needs to be able to get key-value pairs vertically nearby.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgment\u003c/h2\u003e \u003cp\u003eThe first author would like to thank her intern supervisor Mr. Eric Tan of Infocomm Media Development Authority (IMDA), Singapore for his guidance and dedicated supports in the project.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eR. Turner, \u0026ldquo;The myth of the paperless office,\u0026rdquo; New Library World, vol.\u0026nbsp;104, no. 3, pp.\u0026nbsp;120\u0026ndash;121, 2003. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1108/03074800310467043\u003c/span\u003e\u003cspan address=\"10.1108/03074800310467043\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eB. Klein, S. Agne, and A. Dengel, \u0026ldquo;Results of a study on invoice-reading systems in Germany,\u0026rdquo; \u003cem\u003eInternational Workshop on Document Analysis Systems\u003c/em\u003e, pp.\u0026nbsp;451\u0026ndash;462, 2004. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-540-28640-0_43\u003c/span\u003e\u003cspan address=\"10.1007/978-3-540-28640-0_43\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eR. Hatfield, \u0026ldquo;How to Capture Freight Logistics Data,\u0026rdquo; 2021. Web link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003c/span\u003e\u003cspan address=\"http://www.cloud-trade.com/blogs/how-to-capture-freight-logistics-data\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA. Kay, \u0026ldquo;Tesseract: an open-source optical character recognition engine,\u0026rdquo; Linux Journal, 2007.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmazon Web Services, \u0026ldquo;Form Data (Key-Value Pairs),\u0026rdquo; Web link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://docs.aws.amazon.com/textract/latest/dg/how-it-works-kvp.html\u003c/span\u003e\u003cspan address=\"https://docs.aws.amazon.com/textract/latest/dg/how-it-works-kvp.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eV. Kumar, P. Kaware, P. Singh, R. Sonkusare, and S. Kumar, \"Extraction of information from bill receipts using optical character recognition,\" \u003cem\u003eInternational Conference on Smart Electronics and Communication\u003c/em\u003e, 2020, pp.\u0026nbsp;72\u0026ndash;77. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/ICOSEC49089.2020.9215246\u003c/span\u003e\u003cspan address=\"10.1109/ICOSEC49089.2020.9215246\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eV. N. Sai Rakesh Kamisetty, B. Sohan Chidvilas, S. Revathy, \u003cem\u003eet al\u003c/em\u003e., \"Digitization of Data from Invoice using OCR,\" \u003cem\u003e6th International Conference on Computing Methodologies and Communicatio\u003c/em\u003en, 2022. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/ICCMC53470.2022.9754117\u003c/span\u003e\u003cspan address=\"10.1109/ICCMC53470.2022.9754117\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026Aacute;. Z. Kal\u0026oacute; and M. L. Sipos, \"Key-Value Pair Searhing System via Tesseract OCR and Post Processing,\" \u003cem\u003eIEEE 19th World Symposium on Applied Machine Intelligence and Informatics\u003c/em\u003e, 2021. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/SAMI50585.2021.9378680\u003c/span\u003e\u003cspan address=\"10.1109/SAMI50585.2021.9378680\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Devlin, M. Chang, K. Lee, K. Toutanova, \u0026ldquo;BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,\u0026rdquo; North American Chapter of the Association for Computational Linguistics, 2019. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/n19-1423\u003c/span\u003e\u003cspan address=\"10.18653/v1/n19-1423\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eY. Liu, M. Ott, N. Goyal, \u003cem\u003eet al\u003c/em\u003e., \u0026ldquo;RoBERTa: A Robustly Optimized BERT Pretraining Approach,\u0026rdquo; arXiv:1907.11692, 2019.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eW. Wang, B. Bi, M. Yan, \u003cem\u003eet al\u003c/em\u003e., \u0026ldquo;StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding,\u0026rdquo; \u003cem\u003eInternational Conference on Learning Representations\u003c/em\u003e, 2020.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eY. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, \u0026ldquo;LayoutLM: Pre-training of Text and Layout for Document Image Understanding,\u0026rdquo; \u003cem\u003e26th ACM SIGKDD International Conference on Knowledge Discovery \u0026amp; Data Mining\u003c/em\u003e, 2020. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1145/3394486.3403172\u003c/span\u003e\u003cspan address=\"10.1145/3394486.3403172\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eY. Xu, Y. Xu, T. Lv, \u003cem\u003eet al\u003c/em\u003e., \u0026ldquo;LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding,\u0026rdquo; \u003cem\u003e59th Annu.. Meeting of Assoc. for Computatuional Linguistics and 11th Intl. Joint Conf. on Natural Language Processing\u003c/em\u003e, 2021, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/2021.acl-long.201\u003c/span\u003e\u003cspan address=\"10.18653/v1/2021.acl-long.201\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eG. Jaume, H. Kemal Ekenel and J. Thiran, \"FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents,\" \u003cem\u003eInternational Conference on Document Analysis and Recognition Workshops\u003c/em\u003e, 2019, pp.\u0026nbsp;1\u0026ndash;6, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/ICDARW.2019.10029\u003c/span\u003e\u003cspan address=\"10.1109/ICDARW.2019.10029\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eL, Garncarek, R. Powalski, T. Stanisławek, \u003cem\u003eet al\u003c/em\u003e., \u0026ldquo;LAMBERT: Layout-Aware language Modeling for information extraction,\u0026rdquo; \u003cem\u003eInternational Conference on Document Analysis and Recognition\u003c/em\u003e, 2021. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/978-3-030-86549-8_34\u003c/span\u003e\u003cspan address=\"10.1007/978-3-030-86549-8_34\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eC, Li, B. Bi, M. Yan, \u003cem\u003eet al\u003c/em\u003e., \u0026ldquo;StructuralLM: Structural Pre-training for Form Understanding,\u0026rdquo; \u003cem\u003e59th Annual Meeting of the Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing\u003c/em\u003e, 2021. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/2021.acl-long.493\u003c/span\u003e\u003cspan address=\"10.18653/v1/2021.acl-long.493\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBanksy Annotation Tool. 2020. Weblink: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/AboutGoods/Banksy-annotation-tool\u003c/span\u003e\u003cspan address=\"https://github.com/AboutGoods/Banksy-annotation-tool\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eB. Muller, \u0026ldquo;BERT 101 State of The Art NLP Model Explained,\u0026rdquo; 2022. Web link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://huggingface.co/blog/bert-101\u003c/span\u003e\u003cspan address=\"https://huggingface.co/blog/bert-101\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Howard, and S. Ruder, \u0026ldquo;Universal Language Model Fine-tuning for Text Classification,\u0026rdquo; \u003cem\u003e56th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e, 2018. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/P18-1031\u003c/span\u003e\u003cspan address=\"10.18653/v1/P18-1031\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRobust Reading Competition, \u0026ldquo;ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction,\u0026rdquo; Web link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://rrc.cvc.uab.es/?ch=13\u0026amp;com=tasks\u003c/span\u003e\u003cspan address=\"https://rrc.cvc.uab.es/?ch=13\u0026amp;com=tasks\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eS. Agrawal, \u0026ldquo;Metrics to Evaluate your Classification Model to take the right decisions,\u0026rdquo; 2021. Web link: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.analyticsvidhya.com/blog/2021/07/metrics-to-evaluate-your-classification-model-to-take-the-right-decisions/\u003c/span\u003e\u003cspan address=\"https://www.analyticsvidhya.com/blog/2021/07/metrics-to-evaluate-your-classification-model-to-take-the-right-decisions/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eB. Johansen, \u0026ldquo;Named-Entity Recognition for Norwegian,\u0026rdquo; \u003cem\u003e22nd Nordic Conference on Computational Linguistics\u003c/em\u003e, 2019.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eY. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, \u0026ldquo;LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking,\u0026rdquo; arXiv:2204.08387, 2022.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Information Extraction, Key-Value Pairs, Optical Character Recognition, Natural Language Processing","lastPublishedDoi":"10.21203/rs.3.rs-2413475/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2413475/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRapid growth in the digitization of documents, such as cargo invoices or receipts, has alleviated the demand for methods to process information accurately and efficiently. However, it has become impractical for humans to extract the data due to the size of the corpus. Furthermore, extracting the information manually can be labor-intensive and time-consuming as employees need to go through multiple digitized invoices and extract the critical information. The digital documents, such as digital cargo invoices contain various components such as tables, key-value pairs and figures. Existing optical character recognition (OCR) method are able to recognize texts. But it is challenging for them to extract the key-value pairs in unformatted digital cargo invoices. Hence, creating a smart information extraction system that can extract key-value pairs would be beneficial, as it would help organizations achieve workflow efficiency, resource utilization and eliminate costly errors. In this paper, a pipeline of the information extraction system is proposed with deep learning approaches for classifying key-value pairs first, followed by linking the key-value pairs. Two merging rules and two pairing rules are developed in the proposed pipeline. Various experiments have been conducted to illustrate the performance of the system.\u003c/p\u003e","manuscriptTitle":"Information Extraction System for Cargo Invoices","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-01-03 17:46:56","doi":"10.21203/rs.3.rs-2413475/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4dcfff76-4a36-496f-9c22-a1a5ff176bbf","owner":[],"postedDate":"January 3rd, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-01-03T17:54:58+00:00","versionOfRecord":[],"versionCreatedAt":"2023-01-03 17:46:56","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-2413475","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2413475","identity":"rs-2413475","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}