Skip to content

fix(pdf): preserve PDF metadata dates and UTC offsets - #2531

Open
wToTw123 wants to merge 1 commit into
unclecode:developfrom
wToTw123:codex/1110-pdf-metadata-dates
Open

wToTw123 wants to merge 1 commit into
unclecode:developfrom
wToTw123:codex/1110-pdf-metadata-dates

Conversation

@wToTw123

Copy link
Copy Markdown

Summary

The handwritten PDF date parser drops the UTC offset and returns None for partial dates such as D:202404. Read the two public pypdf date properties and recover independently from an invalid date field. Dates with an explicit offset now carry tzinfo. Callers comparing these dates with naive datetimes may need to normalize them.

List of files changed and why

  • crawl4ai/processors/pdf/processor.py contains the behavior fix.
  • tests/unit/test_pdf_metadata_dates.py exercises the public entry point with offline regression inputs.

How Has This Been Tested?

The tests generate real PDFs and exercise PDFContentScrapingStrategy.scrap(). They cover UTC, positive and negative offsets, partial dates, missing dates and malformed metadata.

On Windows with Python 3.11 and pypdf 7.0.0 where applicable, the targeted tests report 3 failed, 2 passed on develop and 5 passed with the patch. The related run reports 30 passed. git diff --check, Python compilation and Black for the changed test file pass. The scoped source-file Black check reports pre-existing whitespace-only formatting. The complete source file has baseline Black failures. Validation covers the named local suites.

Checklist:

  • My code follows the style guidelines of this project. The targeted Black results are listed above.
  • I have performed a self-review of my own code. Contributor approval is tracked with the review materials.
  • I have commented my code, particularly in hard-to-understand areas. The patch uses the existing extraction flow and focused tests.
  • I have made corresponding changes to the documentation. The patch applies the existing extraction or citation contract.
  • I have added/updated unit tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes. See the targeted results above for the exact coverage.

Prepared with AI assistance. The problem was discovered and reproduced from develop source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant