PDF documents represent the bulk of newer content stored in ECM systems and cloud storage worldwide. PDF is replacing TIFF as the output format of choice for imaging systems, and theres simply no substitute for PDF when it comes to creating a shareable record of static electronic content.
Documents are often used in lawsuits and similar expensive activities. In these cases, the ability to reliably search for keywords and phrases, personally identifiable information (PII), trade-secrets or any sort of sensitive material is fundamental to understanding your risks.
Consider the following examples:
Discovery: During the document discovery process that precedes litigation, text-extraction and search processes are used to locate responsive documents, i.e., those that must be turned over to opposing counsel. Failure to turn over a responsive document can create numerous expensive problems if discovered later in the proceedings. Attorneys like to know what they've got.
FOIA: Freedom of Information Act procedures includes searches of extracted text to speed the identification and removal of sensitive information ranging from PII to classified data. Even so, FOIA requests are a typical channel for data breaches, as this otherwise unremarkable example makes clear.
Most of todays text-extraction tools have serious limitations when it comes to supporting reliable searching of PDF files because they dont actually account for the document they simply provide the ability to search the file. Theres a huge difference, as Ill explain.
At its heart, the PDF file-format is all about painting things onto pages, also known as "rendering". Theres far more to PDF, of course, but thats the core of it. As a result, PDF documents dont always work precisely as one might expect when they are used for purposes other than rendering... such as searching.
Lets look at one common problem. It requires a brief dip into the guts of PDF, but not too far, and its illustrative.
PDF is extremely flexible. Its commonplace (and sometimes essential) for software to create PDF files in which the ordering of text in the file (the "content order") differs from the texts appearance on the page (the "logical reading order").
In Example 1, the content order is portrayed to the left. The orange arrow shows the order in which the text occurs in the file itself, which obviously differs from the way a human reads the page.
When reading the page visually, the logical reading order is what matters. When extracting text or searching, however, if the software uses the content order (how the PDF file is made) instead of the logical reading order (what's displayed on the page), your results may be affected.
Using most currently-available tools, including Adobe Acrobat, other PDF viewers and the server-side text extraction and search features in ECM systems, searching this PDF for final check will not find that phrase even though it's plainly present on the page.
The reason is straight-forward: the text extraction and/or search software is using the file's content order, not the logical reading order.
Example 2 shows a document that's visually identical to the previous example. In this case the content order matches the human readers expectations. Accordingly, a content-order search for final check or "final check list" will work.
Both PDF documents look identical on-screen and in-print but the guts of the files are quite different.
Few organizations can control the technical details of the documents they receive or create. PDF documents can and will include variations in the way text is encoded into the file, and sometimes these variations affect searchability.
If reviewers and other downstream procedures rely on text-extraction or search that uses only the content order, some hits will be missed.
To address this problem ECM customers and others who need to trust their text extraction and search systems can take the following steps:
Text extraction and search software developers have many choices about how they index PDF files. It's up to ECM and other customers to demand software that does the right thing.... or risk the reality that some searches will fail to find each instance.
Although often defeating simpler text-ordering problems, reducing PDF documents to images in order to OCR them for will introduce new errors and cause more text to be missed in many relatively common circumstances:
Comparing text-extraction from PDF vs. OCR is easy enough to check for yourself. Take an arbitrary collection of PDF documents, extract all the text and count the words that result. Next, convert the PDFs to images and run OCR. The OCR process will (unless all pages in the example were from simple business documents) find significantly fewer words.
Today, both desktop and server-side ECM text extraction and search technology typically rests on assumptions rooted in the era of TIFF images and plain-text.
To manage exposure to the release of PII, company secrets, national security data and many other types of sensitive information, ECM customers who rely on PDF for electronic records should strongly consider reviewing the ability of their ECM systems to capture and search text from PDF documents.
Founder of Document Solutions, Inc. in 1996, Duff Johnson is a 23 year veteran of the electronic document space and a recognized leader in the electronic document technology industry. Now an independent consultant, Duff serves the PDF industry as ISO Project co-Leader (and US TAG chair) for ISO 32000 and ISO 14289. Previously Vice Chairman of the Board, Duff Johnson …
Founder of Document Solutions, Inc. in 1996, Duff Johnson is a 23 year veteran of the electronic document space and …