How we test PDF-to-Word conversion

Purpose and claim boundary

The public PDFCore 8.0 sample answers a narrow question: does the production converter create valid, editable Word packages containing the structures its interface promises on a controlled engineering corpus? The current answer is yes for the published fixtures. It does not answer the broader question of whether PDFCore is one of the five best converters on the market.

A competitive conclusion requires enough varied real documents to expose different failure modes, the same input and settings discipline for every product, reviewers who do not know which converter produced each file, and a scoring rule written before results are seen. Until that gate is completed, the Quality Lab reports engineering readiness and known limitations only.

Corpus design

The current downloadable corpus is synthetic and rights-cleared. It deliberately combines portrait and landscape sections, multi-column text, aligned rows, graphics, headings and repeated page-edge content. Synthetic fixtures are useful because expected text and structure are known exactly, regressions are repeatable, and the files can be redistributed publicly.

The next competitive corpus must contain at least 30 rights-cleared PDFs and must not be selected only because PDFCore handles them well. It should include:

• Born-digital reports with common and uncommon embedded fonts
• Clean scans and degraded scans at several resolutions
• Mixed native-text and image-only pages
• Tables with borders, without borders and with merged cells
• Forms, invoices, academic papers, manuals and multi-column newsletters
• Portrait and landscape sections, rotated text and rotated pages
• Footnotes, endnotes, headers, footers and page numbers
• Photographs, charts, vector diagrams and layered artwork
• CJK, Arabic/RTL and mixed-script content
• Password-protected inputs used with explicit authorization
• Large, malformed and partially recoverable PDFs

Each document receives an immutable ID, source hash, page count, language label, content category, known accessibility or structure information, and a record of the rights that permit benchmarking and publication.

Controlled conversion environment

Every converter should run on the same Windows version and hardware class, with the same installed fonts and Microsoft Word build used for output rendering. Network access, OCR language packs, hardware acceleration and background processes must be recorded. A warm-cache run and a cold-start run should not be silently mixed.

Settings must map to declared goals rather than to whichever option happens to win a document. We use three goals:

1. Editable: retain pages and graphics while producing useful Word-native paragraphs, tables and repeated regions.
2. Precise: keep visual objects near their original coordinates while retaining selectable text when possible.
3. Flowing: prioritize reading order, paragraphs, headings and rewriting over fixed page placement.

If a competing product exposes a materially different model, the mapping and reason are documented before the run. OCR language is set from the corpus label. Page ranges, image retention and table detection are kept consistent wherever equivalent controls exist.

Package validity and safety checks

First, the output must exist, finish within the timeout and open as an Office Open XML document. Automated validation checks the package relationships and required parts. The document is also opened in desktop Word because a package can satisfy a schema while still trigger a repair warning or display incorrectly.

For each output we record:

• Conversion success, failure, timeout or crash
• Open XML validation errors and Word repair prompts
• Unexpected overwrite or partial-output behavior
• Output page count and section orientation
• Whether requested page ranges were honored
• Conversion time, peak memory and output size

A visually impressive file that crashes on one document in ten must not outrank a stable converter without that operational penalty appearing in the score.

Visual similarity

All DOCX files are rendered to PDF with the same Word build, then rasterized at a fixed resolution. Page dimensions are normalized only when the converter clearly preserved the intended paper size. We compare source and output renders using image similarity measurements plus human review.

Automatic image distance is not sufficient by itself. It can penalize a correctly reconstructed font substitution or reward a full-page screenshot that looks perfect but contains no editable text. Reviewers therefore score text placement, line wrapping, column geometry, table boundaries, graphics, headers/footers and pagination separately. Visual scores never replace the editability score.

Text accuracy and reading order

Born-digital PDFs are evaluated against extracted reference text; scans use manually verified transcripts for selected regions. Character error rate and word error rate are useful for OCR, while ordered-block accuracy captures a different problem: whether the converter mixed columns, inserted a sidebar mid-sentence or placed a footer in the body.

Critical fields receive additional checks. Names, dates, monetary totals, account codes, negative values and footnote markers can matter more than ordinary prose. The benchmark reports both overall recognition and errors in these high-consequence fields.

Semantic Word structure

We inspect what Word objects were actually created, not what the page merely resembles. Checks include:

• Paragraphs versus hundreds of isolated text boxes
• Heading styles visible in Word’s Navigation pane
• Real tables, rows, cells and merged-cell relationships
• Word header and footer parts rather than repeated body text
• Images and drawing objects with sensible anchors
• Page and section breaks, margins and orientation
• Absence of invisible duplicate text behind a page image

No output model is always best. Positioned text is a legitimate precise-layout technique; it simply receives a lower free-rewrite score than coherent paragraphs. A full-page image with hidden OCR can be a useful searchable archive, but it is not counted as an editable Word reconstruction.

Standard edit script

A reviewer performs the same bounded edits in every file: replace a sentence near the top, add a paragraph in the middle, change a heading, edit one table value, insert a table row, replace an image caption and update a header or footer. We record completion time, visible damage and manual repairs.

This measures the cost a user actually feels. A file may look excellent before editing but collapse when one sentence wraps. Another may initially differ from the PDF yet remain stable and easy to revise. Both outcomes should be visible rather than compressed into one opaque score.

Blind review and scoring

Candidate letters are randomized independently for every document. Before reviewers receive a DOCX, the framework removes core, application and custom producer metadata, records separate hashes for the untouched raw output and sanitized review copy, and scans package entry names and readable package parts for product/vendor markers. Any remaining marker stops the run. Reviewers receive only source pages, anonymized outputs and a task sheet; the secret mapping remains locked until their ratings are final.

These 25/20/20/20/15 weights are frozen before a competitive run. Successful outputs require recorded edit-script seconds, manual repair count and any Word repair prompt. Results include per-category scores, failures and edit-cost evidence—not only an overall number.

Public PDFCore 8.0.0 sample evidence

The sample checkpoint's fixed release suite passed 42 of 42 tests, and five consecutive runs passed 210 of 210. All three published sample DOCX files produced zero Open XML validation errors. The current 8.0.1 release suite passes 47 of 47 tests, including batch and single-file OCR-language selection, zero-language safeguards and accessibility-state coverage, but the downloadable DOCX samples remain explicitly tied to 8.0.0.

These are repeatable production-service tests, but the corpus breadth remains the important limitation. The public files allow anyone to inspect what is proven today while the larger rights-cleared benchmark is built.