See repo: PDF to Markdown Wrapper (pdftomd.sh) is a RAG workflow-friendly enhancement of Marker that converts a PDF into a single markdown file. It handles GPU and PyTorch configuration, document splitting and chunking, image BASE64 embedding, LLM post-processing and cleanup, and consolidation of output

https://github.com/ngpepin/pdftomd-RAG

• Splits large PDFs into chunks (100 pages by default, 10 pages when -l/--llm is enabled) and runs Marker once on the chunk folder (avoids repeated model loads).
• Consolidates all chunk markdown into a single .md file.
• Optionally embeds images as Base64 (no external asset folders needed).
• Optional text-only output that strips image links from the final markdown.
• Optional OCR pass via bundled ocr-pdf/ocr-pdf.sh before conversion.
• Optional LLM helper via a built-in Marker --use_llm.
• Automatically uses GPU when available and installs CUDA-enabled torch when needed.
• Cleans up intermediate files and attempts to stop spawned processes on exit.
• Optional supplemental LLM post-processing step with --clean.
• The overall result can be a much cleaner more streamlined end product more suited to RAG pipeline ingestion.

posted 8 months ago

CC BY-SA 4.0

ngpepin‭

-7 reputation 0 1 -4 0