9  PDF Load (PyMuPDF)

PyMuPDFLoader: https://python.langchain.com/docs/integrations/document_loaders/pymupdf/

from langchain_community.document_loaders import PyMuPDFLoader

loader = PyMuPDFLoader("pdf/layout-parser-paper.pdf")

docs = loader.load()
docs[0]

Document(metadata={'producer': 'pdfTeX-1.40.21', 'creator': 'LaTeX with hyperref', 'creationdate': '2021-06-22T01:27:10+00:00', 'source': 'pdf/layout-parser-paper.pdf', 'file_path': 'pdf/layout-parser-paper.pdf', 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'moddate': '2021-06-22T01:27:10+00:00', 'trapped': '', 'modDate': 'D:20210622012710Z', 'creationDate': 'D:20210622012710Z', 'page': 0}, page_content='LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 (\x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n5 University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model configurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\nefforts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser, an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io.\nKeywords: Document Image Analysis · Deep Learning · Layout Analysis\n· Character Recognition · Open Source library · Toolkit.\n1\nIntroduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocument image analysis (DIA) tasks including document image classification [11,\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021')

docs[0].metadata

{'producer': 'pdfTeX-1.40.21',
 'creator': 'LaTeX with hyperref',
 'creationdate': '2021-06-22T01:27:10+00:00',
 'source': 'pdf/layout-parser-paper.pdf',
 'file_path': 'pdf/layout-parser-paper.pdf',
 'total_pages': 16,
 'format': 'PDF 1.5',
 'title': '',
 'author': '',
 'subject': '',
 'keywords': '',
 'moddate': '2021-06-22T01:27:10+00:00',
 'trapped': '',
 'modDate': 'D:20210622012710Z',
 'creationDate': 'D:20210622012710Z',
 'page': 0}

9.1 Extract Table

9.1.1 Doc 1: layout-parser-paper.pdf

loader = PyMuPDFLoader(
    "pdf/layout-parser-paper.pdf",
    mode="page",
    extract_tables="markdown",
)
docs = loader.load()
print(docs[4].page_content)

LayoutParser: A Unified Toolkit for DL-Based DIA
5
Table 1: Current layout detection models in the LayoutParser model zoo
Dataset
Base Model1 Large Model
Notes
PubLayNet [38]
F / M
M
Layouts of modern scientific documents
PRImA [3]
M
-
Layouts of scanned modern magazines and scientific reports
Newspaper [17]
F
-
Layouts of scanned US newspapers from the 20th century
TableBank [18]
F
F
Table region on modern scientific and business document
HJDataset [31]
F / M
-
Layouts of history Japanese documents
1 For each dataset, we train several models of different sizes for different needs (the trade-offbetween accuracy
vs. computational cost). For “base model” and “large model”, we refer to using the ResNet 50 or ResNet 101
backbones [13], respectively. One can train models of different architectures, like Faster R-CNN [28] (F) and Mask
R-CNN [12] (M). For example, an F in the Large Model column indicates it has a Faster R-CNN model trained
using the ResNet 101 backbone. The platform is maintained and a number of additions will be made to the model
zoo in coming months.
layout data structures, which are optimized for efficiency and versatility. 3) When
necessary, users can employ existing or customized OCR models via the unified
API provided in the OCR module. 4) LayoutParser comes with a set of utility
functions for the visualization and storage of the layout data. 5) LayoutParser
is also highly customizable, via its integration with functions for layout data
annotation and model training. We now provide detailed descriptions for each
component.
3.1
Layout Detection Models
In LayoutParser, a layout model takes a document image as an input and
generates a list of rectangular boxes for the target content regions. Different
from traditional methods, it relies on deep convolutional neural networks rather
than manually curated rules to identify content regions. It is formulated as an
object detection problem and state-of-the-art models like Faster R-CNN [28] and
Mask R-CNN [12] are used. This yields prediction results of high accuracy and
makes it possible to build a concise, generalized interface for layout detection.
LayoutParser, built upon Detectron2 [35], provides a minimal API that can
perform layout detection with only four lines of code in Python:
1 import
layoutparser as lp
2 image = cv2.imread("image_file") # load
images
3 model = lp. Detectron2LayoutModel (
4
"lp:// PubLayNet/ faster_rcnn_R_50_FPN_3x /config")
5 layout = model.detect(image)
LayoutParser provides a wealth of pre-trained model weights using various
datasets covering different languages, time periods, and document types. Due to
domain shift [7], the prediction performance can notably drop when models are ap-
plied to target samples that are significantly different from the training dataset. As
document structures and layouts vary greatly in different domains, it is important
to select models trained on a dataset similar to the test samples. A semantic syntax
is used for initializing the model weights in LayoutParser, using both the dataset
name and model name lp://<dataset-name>/<model-architecture-name>.


|Dataset|Base Model1|Large Model|Notes|
|---|---|---|---|
|PubLayNet [38] PRImA [3] Newspaper [17] TableBank [18] HJDataset [31]|F / M M F F F / M|M &amp;#45; &amp;#45; F &amp;#45;|Layouts of modern scientific documents Layouts of scanned modern magazines and scientific reports Layouts of scanned US newspapers from the 20th century Table region on modern scientific and business document Layouts of history Japanese documents|

9.1.2 Doc 2: Abnormal Liver Function Tests.pdf

loader2 = PyMuPDFLoader(
    "pdf/Abnormal Liver Function Tests.pdf",
    mode="page",
    extract_tables="markdown",
)
docs2 = loader2.load()
print(docs2[0].page_content)

New 2023 
ACR Appropriateness Criteria® 
1 
Abnormal Liver Function Tests 
American College of Radiology 
ACR Appropriateness Criteria®
Abnormal Liver Function Tests 
Variant 1: 
Abnormal liver function tests. Hepatocellular predominance with mild aminotransferase 
increase. Initial imaging. 
Procedure 
Appropriateness Category 
Relative Radiation Level 
US abdomen 
Usually Appropriate 
O
US duplex Doppler abdomen 
Usually Appropriate 
O
US shear wave elastography abdomen 
May Be Appropriate 
O 
MR elastography abdomen 
May Be Appropriate 
O
MRI abdomen without and with IV contrast 
with MRCP 
May Be Appropriate 
O
MRI abdomen without IV contrast with 
MRCP 
May Be Appropriate 
O
CT abdomen and pelvis without IV contrast 
May Be Appropriate 
☢☢☢
US abdomen with IV contrast 
Usually Not Appropriate 
O
CT abdomen and pelvis with IV contrast 
Usually Not Appropriate 
☢☢☢
CT abdomen and pelvis without and with IV 
contrast 
Usually Not Appropriate 
☢☢☢☢
Variant 2: 
Abnormal liver function tests. Hepatocellular predominance with moderate or severe 
aminotransferase increase. Initial imaging. 
Procedure 
Appropriateness Category 
Relative Radiation Level 
US abdomen 
Usually Appropriate 
O
US duplex Doppler abdomen 
Usually Appropriate 
O
CT abdomen and pelvis with IV contrast 
Usually Appropriate 
☢☢☢
MRI abdomen without and with IV contrast 
with MRCP 
May Be Appropriate 
O
MRI abdomen without IV contrast with 
MRCP 
May Be Appropriate 
O
CT abdomen and pelvis without IV contrast 
May Be Appropriate 
☢☢☢
US abdomen with IV contrast 
Usually Not Appropriate 
O
US shear wave elastography abdomen 
Usually Not Appropriate 
O 
MR elastography abdomen 
Usually Not Appropriate 
O
CT abdomen and pelvis without and with IV 
contrast 
Usually Not Appropriate 
☢☢☢☢


|Procedure|Appropriateness Category|Relative Radiation Level|
|---|---|---|
|US abdomen|Usually Appropriate|O|
|US duplex Doppler abdomen|Usually Appropriate|O|
|US shear wave elastography abdomen|May Be Appropriate|O|
|MR elastography abdomen|May Be Appropriate|O|
|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|
|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|
|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|
|US abdomen with IV contrast|Usually Not Appropriate|O|
|CT abdomen and pelvis with IV contrast|Usually Not Appropriate|☢☢☢|
|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|


|Procedure|Appropriateness Category|Relative Radiation Level|
|---|---|---|
|US abdomen|Usually Appropriate|O|
|US duplex Doppler abdomen|Usually Appropriate|O|
|CT abdomen and pelvis with IV contrast|Usually Appropriate|☢☢☢|
|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|
|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|
|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|
|US abdomen with IV contrast|Usually Not Appropriate|O|
|US shear wave elastography abdomen|Usually Not Appropriate|O|
|MR elastography abdomen|Usually Not Appropriate|O|
|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|

docs2[0].model_dump()

{'id': None,
 'metadata': {'producer': 'Adobe PDF Library 17.11.238; modified using iTextSharp™ 5.5.13.3 ©2000-2022 iText Group NV (AGPL-version)',
  'creator': 'Acrobat PDFMaker 17 for Word',
  'creationdate': '2022-12-06T08:43:10-05:00',
  'source': 'pdf/Abnormal Liver Function Tests.pdf',
  'file_path': 'pdf/Abnormal Liver Function Tests.pdf',
  'total_pages': 16,
  'format': 'PDF 1.4',
  'title': 'Abnormal Liver Function Tests',
  'author': 'dobrien',
  'subject': '',
  'keywords': '',
  'moddate': '2024-12-09T11:13:44-05:00',
  'trapped': '',
  'modDate': "D:20241209111344-05'00'",
  'creationDate': "D:20221206084310-05'00'",
  'page': 0},
 'page_content': 'New 2023 \nACR Appropriateness Criteria® \n1 \nAbnormal Liver Function Tests \nAmerican College of Radiology \nACR Appropriateness Criteria®\nAbnormal Liver Function Tests \nVariant 1: \nAbnormal liver function tests. Hepatocellular predominance with mild aminotransferase \nincrease. Initial imaging. \nProcedure \nAppropriateness Category \nRelative Radiation Level \nUS abdomen \nUsually Appropriate \nO\nUS duplex Doppler abdomen \nUsually Appropriate \nO\nUS shear wave elastography abdomen \nMay Be Appropriate \nO \nMR elastography abdomen \nMay Be Appropriate \nO\nMRI abdomen without and with IV contrast \nwith MRCP \nMay Be Appropriate \nO\nMRI abdomen without IV contrast with \nMRCP \nMay Be Appropriate \nO\nCT abdomen and pelvis without IV contrast \nMay Be Appropriate \n☢☢☢\nUS abdomen with IV contrast \nUsually Not Appropriate \nO\nCT abdomen and pelvis with IV contrast \nUsually Not Appropriate \n☢☢☢\nCT abdomen and pelvis without and with IV \ncontrast \nUsually Not Appropriate \n☢☢☢☢\nVariant 2: \nAbnormal liver function tests. Hepatocellular predominance with moderate or severe \naminotransferase increase. Initial imaging. \nProcedure \nAppropriateness Category \nRelative Radiation Level \nUS abdomen \nUsually Appropriate \nO\nUS duplex Doppler abdomen \nUsually Appropriate \nO\nCT abdomen and pelvis with IV contrast \nUsually Appropriate \n☢☢☢\nMRI abdomen without and with IV contrast \nwith MRCP \nMay Be Appropriate \nO\nMRI abdomen without IV contrast with \nMRCP \nMay Be Appropriate \nO\nCT abdomen and pelvis without IV contrast \nMay Be Appropriate \n☢☢☢\nUS abdomen with IV contrast \nUsually Not Appropriate \nO\nUS shear wave elastography abdomen \nUsually Not Appropriate \nO \nMR elastography abdomen \nUsually Not Appropriate \nO\nCT abdomen and pelvis without and with IV \ncontrast \nUsually Not Appropriate \n☢☢☢☢\n\n\n|Procedure|Appropriateness Category|Relative Radiation Level|\n|---|---|---|\n|US abdomen|Usually Appropriate|O|\n|US duplex Doppler abdomen|Usually Appropriate|O|\n|US shear wave elastography abdomen|May Be Appropriate|O|\n|MR elastography abdomen|May Be Appropriate|O|\n|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|\n|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|\n|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|\n|US abdomen with IV contrast|Usually Not Appropriate|O|\n|CT abdomen and pelvis with IV contrast|Usually Not Appropriate|☢☢☢|\n|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|\n\n\n|Procedure|Appropriateness Category|Relative Radiation Level|\n|---|---|---|\n|US abdomen|Usually Appropriate|O|\n|US duplex Doppler abdomen|Usually Appropriate|O|\n|CT abdomen and pelvis with IV contrast|Usually Appropriate|☢☢☢|\n|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|\n|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|\n|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|\n|US abdomen with IV contrast|Usually Not Appropriate|O|\n|US shear wave elastography abdomen|Usually Not Appropriate|O|\n|MR elastography abdomen|Usually Not Appropriate|O|\n|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|',
 'type': 'Document'}

docs2[0].page_content

'New 2023 \nACR Appropriateness Criteria® \n1 \nAbnormal Liver Function Tests \nAmerican College of Radiology \nACR Appropriateness Criteria®\nAbnormal Liver Function Tests \nVariant 1: \nAbnormal liver function tests. Hepatocellular predominance with mild aminotransferase \nincrease. Initial imaging. \nProcedure \nAppropriateness Category \nRelative Radiation Level \nUS abdomen \nUsually Appropriate \nO\nUS duplex Doppler abdomen \nUsually Appropriate \nO\nUS shear wave elastography abdomen \nMay Be Appropriate \nO \nMR elastography abdomen \nMay Be Appropriate \nO\nMRI abdomen without and with IV contrast \nwith MRCP \nMay Be Appropriate \nO\nMRI abdomen without IV contrast with \nMRCP \nMay Be Appropriate \nO\nCT abdomen and pelvis without IV contrast \nMay Be Appropriate \n☢☢☢\nUS abdomen with IV contrast \nUsually Not Appropriate \nO\nCT abdomen and pelvis with IV contrast \nUsually Not Appropriate \n☢☢☢\nCT abdomen and pelvis without and with IV \ncontrast \nUsually Not Appropriate \n☢☢☢☢\nVariant 2: \nAbnormal liver function tests. Hepatocellular predominance with moderate or severe \naminotransferase increase. Initial imaging. \nProcedure \nAppropriateness Category \nRelative Radiation Level \nUS abdomen \nUsually Appropriate \nO\nUS duplex Doppler abdomen \nUsually Appropriate \nO\nCT abdomen and pelvis with IV contrast \nUsually Appropriate \n☢☢☢\nMRI abdomen without and with IV contrast \nwith MRCP \nMay Be Appropriate \nO\nMRI abdomen without IV contrast with \nMRCP \nMay Be Appropriate \nO\nCT abdomen and pelvis without IV contrast \nMay Be Appropriate \n☢☢☢\nUS abdomen with IV contrast \nUsually Not Appropriate \nO\nUS shear wave elastography abdomen \nUsually Not Appropriate \nO \nMR elastography abdomen \nUsually Not Appropriate \nO\nCT abdomen and pelvis without and with IV \ncontrast \nUsually Not Appropriate \n☢☢☢☢\n\n\n|Procedure|Appropriateness Category|Relative Radiation Level|\n|---|---|---|\n|US abdomen|Usually Appropriate|O|\n|US duplex Doppler abdomen|Usually Appropriate|O|\n|US shear wave elastography abdomen|May Be Appropriate|O|\n|MR elastography abdomen|May Be Appropriate|O|\n|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|\n|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|\n|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|\n|US abdomen with IV contrast|Usually Not Appropriate|O|\n|CT abdomen and pelvis with IV contrast|Usually Not Appropriate|☢☢☢|\n|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|\n\n\n|Procedure|Appropriateness Category|Relative Radiation Level|\n|---|---|---|\n|US abdomen|Usually Appropriate|O|\n|US duplex Doppler abdomen|Usually Appropriate|O|\n|CT abdomen and pelvis with IV contrast|Usually Appropriate|☢☢☢|\n|MRI abdomen without and with IV contrast with MRCP|May Be Appropriate|O|\n|MRI abdomen without IV contrast with MRCP|May Be Appropriate|O|\n|CT abdomen and pelvis without IV contrast|May Be Appropriate|☢☢☢|\n|US abdomen with IV contrast|Usually Not Appropriate|O|\n|US shear wave elastography abdomen|Usually Not Appropriate|O|\n|MR elastography abdomen|Usually Not Appropriate|O|\n|CT abdomen and pelvis without and with IV contrast|Usually Not Appropriate|☢☢☢☢|'

9.2 Extract images

9.2.1 Tesseract

from langchain_community.document_loaders.parsers import TesseractBlobParser

loader = PyMuPDFLoader(
    "pdf/layout-parser-paper.pdf",
    mode="page",
    images_inner_format="html-img",
    images_parser=TesseractBlobParser(),
)

docs = loader.load()
print(docs[5].page_content)

6
Z. Shen et al.
Fig. 2: The relationship between the three types of layout data structures.
Coordinate supports three kinds of variation; TextBlock consists of the co-
ordinate information and extra features like block text, types, and reading orders;
a Layout object is a list of all possible layout elements, including other Layout
objects. They all support the same set of transformation and operation APIs for
maximum flexibility.
Shown in Table 1, LayoutParser currently hosts 9 pre-trained models trained
on 5 different datasets. Description of the training dataset is provided alongside
with the trained models such that users can quickly identify the most suitable
models for their tasks. Additionally, when such a model is not readily available,
LayoutParser also supports training customized layout models and community
sharing of the models (detailed in Section 3.5).
3.2
Layout Data Structures
A critical feature of LayoutParser is the implementation of a series of data
structures and operations that can be used to efficiently process and manipulate
the layout elements. In document image analysis pipelines, various post-processing
on the layout analysis model outputs is usually required to obtain the final
outputs. Traditionally, this requires exporting DL model outputs and then loading
the results into other pipelines. All model outputs from LayoutParser will be
stored in carefully engineered data types optimized for further processing, which
makes it possible to build an end-to-end document digitization pipeline within
LayoutParser. There are three key components in the data structure, namely
the Coordinate system, the TextBlock, and the Layout. They provide different
levels of abstraction for the layout data, and a set of APIs are supported for
transformations or operations on these classes.




<img alt="Coordinate

textblock

x-interval

[eAsaqui-A

Coordinate
+

Extra features

Rectangle

Quadrilateral

Block
Text

Block
Type

Reading
Order

layout

[ coordinatel1 textblockl |
&#x27;

“y textblock2 , layout1 ]

A list of the layout elements

The same transformation and operation APIs src="#" />

9.2.2 LLM Extract image

from langchain_community.document_loaders.parsers import LLMImageBlobParser
from langchain_openai import ChatOpenAI

loader = PyMuPDFLoader(
    "pdf/layout-parser-paper.pdf",
    mode="page",
    images_inner_format="markdown-img",
    images_parser=LLMImageBlobParser(model=ChatOpenAI(model="gpt-4o", max_tokens=1024)),
)
docs = loader.load()
print(docs[5].page_content)

6
Z. Shen et al.
Fig. 2: The relationship between the three types of layout data structures.
Coordinate supports three kinds of variation; TextBlock consists of the co-
ordinate information and extra features like block text, types, and reading orders;
a Layout object is a list of all possible layout elements, including other Layout
objects. They all support the same set of transformation and operation APIs for
maximum flexibility.
Shown in Table 1, LayoutParser currently hosts 9 pre-trained models trained
on 5 different datasets. Description of the training dataset is provided alongside
with the trained models such that users can quickly identify the most suitable
models for their tasks. Additionally, when such a model is not readily available,
LayoutParser also supports training customized layout models and community
sharing of the models (detailed in Section 3.5).
3.2
Layout Data Structures
A critical feature of LayoutParser is the implementation of a series of data
structures and operations that can be used to efficiently process and manipulate
the layout elements. In document image analysis pipelines, various post-processing
on the layout analysis model outputs is usually required to obtain the final
outputs. Traditionally, this requires exporting DL model outputs and then loading
the results into other pipelines. All model outputs from LayoutParser will be
stored in carefully engineered data types optimized for further processing, which
makes it possible to build an end-to-end document digitization pipeline within
LayoutParser. There are three key components in the data structure, namely
the Coordinate system, the TextBlock, and the Layout. They provide different
levels of abstraction for the layout data, and a set of APIs are supported for
transformations or operations on these classes.




![**Image Summary:**

Diagram illustrating layout transformation using coordinates, text blocks, and layout elements with extra features. Includes coordinate types (x-interval, y-interval, rectangle, quadrilateral) and text block features (block text, block type, reading order). Shows integration into a layout list format with transformation APIs. 

**Extracted Text:**

Coordinate

x-interval
y-interval

Rectangle

Quadrilateral

textblock
Coordinate + Extra features
Block Text
Block Type
Reading Order
...

layout
[ coordinate1, textblock1, ..., textblock2, layout1 \\]
A list of the layout elements

The same transformation and operation APIs](#)