PDF-Summary: Pulling Readable Summaries Out of Scientific Papers
If you’ve ever had a folder of papers you meant to read, you know the problem this came out of. I wanted something that would take a PDF and hand me twenty sentences worth reading, so I could decide whether the paper deserved an hour.
PDF-Summary is that tool. It’s a small Python package — two pieces, one that gets text out of a PDF and one that summarizes it. No model training, no GPU, and this predates everyone reaching for an LLM the moment they see a document.
Getting the text out¶
PDFs are a terrible source of text. Line breaks land mid-sentence, headers and
footers show up in the middle of paragraphs, and ligatures come through as
whatever the font felt like. The ReadPDF class runs the file through
textract and then does the cleanup nobody warns you about:
from pdfsum.readpdf.readpdf import ReadPDF
pdf = ReadPDF("paper.pdf")
text = pdf.get_text()
sentences = pdf.clean_sentences(text, return_sent_list=True)
Two bits of cleanup earn their keep. The first collapses runs of full stops —
tables of contents and reference lists leave long strings of dots that
otherwise read as hundreds of empty sentences. The second is clean_sentences,
which splits with spaCy’s sentencizer and then throws away anything shorter
than 40 characters or longer than 400. Short fragments are almost always page
numbers, headings or figure labels that happen to end in a period, and very
long ones are usually a table that got flattened into a line. Dropping both
ends of that distribution does more for summary quality than anything clever I
tried.
Summarizing it¶
The summarizer is TextRank,
by way of pytextrank as a spaCy pipeline component. It’s PageRank on a graph
of phrases: build a graph where nodes are key phrases and edges are
co-occurrence, run the ranking, then pick the sentences carrying the
highest-ranked phrases. The summary is extractive — every sentence is one the
author actually wrote, which is exactly what I wanted. Nothing gets invented.
The other reason to use it here is the vocabulary. It loads
SciSpacy’s en_core_sci_lg, a spaCy
model trained on biomedical text, so it tokenizes and segments scientific
writing without falling over on chemical names and citation clutter:
from pdfsum.sumpdf.textrank import TextSum
ts = TextSum()
ts.load_dictionary("en_core_sci_lg")
ts.add_pipe("textrank")
doc = ts.get_text(text) # strips references, drops [12]-style citations
summary = ts.get_text_rank_summary(doc, limit_sentences=20)
get_text cuts everything after the last “references” heading before
summarizing. Reference lists are dense with exactly the terms TextRank thinks
are important, and leaving them in gives you a summary made of bibliography.
There are a few pipes beyond TextRank, from SciSpacy: an abbreviation detector that ties “CNN” back to where the paper defined it, a UMLS entity linker for mapping mentions to medical concepts, and rule-based tokenizer and sentence segmenter tuned for scientific text. There’s also phrase and token matching, so you can pull out every sentence mentioning a term you care about — I used that to search a stack of papers for specific bacteria and food terms without reading them.
Installing it¶
The pinned versions tell you when this was written — spaCy 2.2, SciSpacy 0.2.4:
python -m pip install -r requirements.txt
That also pulls en_core_sci_lg straight from AI2’s S3 bucket, which is a
large download. textract is the fussiest dependency; on some systems it wants
system packages before it will install.
Looking back¶
I’d build parts of this differently now. The spaCy 2.x pipeline API it uses was
reworked in 3.x, so nlp.add_pipe calls need updating, and I’d probably reach
for a transformer summarizer for the abstractive version.
But the extractive approach has held up better than I expected. When you’re triaging papers, sentences the author actually wrote beat a fluent paraphrase that might be subtly wrong — and it runs on a laptop, offline, in seconds.