When building a B2B document extractor, choosing between a rule-based system and a large language model (LLM) can be challenging. This article presents five key questions and detailed answers based on a practical experiment where both approaches were implemented for extracting order details from PDFs. The rule-based system used pytesseract for OCR and regex patterns, while the LLM leveraged Ollama with LLaMA 3. We explore performance, flexibility, and real-world trade-offs.

1. What were the main steps in building the rule-based document extractor?

The rule-based extractor relied on pytesseract to perform optical character recognition (OCR) on scanned PDFs. Once text was extracted, a set of hard-coded regular expressions and pattern-matching rules were applied to identify key fields like vendor name, order number, line items, and totals. The approach assumed a consistent document layout—for example, expecting the invoice number to appear after a specific label such as "Invoice #:". Each field was extracted sequentially, with fallback logic for missing data. While this method was fast and deterministic, it required significant manual effort to design rules, and any layout variation (e.g., different invoice templates) would break the extraction.

2. How did the LLM-based extractor using Ollama and LLaMA 3 differ?

Instead of fixed rules, the LLM extractor used Ollama to run LLaMA 3 locally. The entire PDF was first converted to raw text via OCR (same pytesseract pipeline) but then fed as context to the LLM with a prompt like: "Extract the vendor, order number, and line items from this invoice text." The model returned structured JSON. This approach required no hand-crafted rules because the LLM understood semantics and handled varied layouts inherently. However, it was slower per document (typically 5–10 seconds versus <1 second for rules) and sometimes hallucinated fields when the text was ambiguous. The LLM also required a GPU for reasonable performance.

3. Which approach performed better on accuracy for a standard B2B order form?

On a controlled test set of 50 standard invoices from one vendor, the rule-based system achieved 98% field-level accuracy—it confidently extracted exactly what the patterns captured. The LLM achieved only 92% on the same set, occasionally misinterpreting dates or merging line items. However, when tested on a mixed set of 20 invoices from different vendors, the rule-based accuracy dropped to 75% (owing to layout changes), while the LLM maintained 84% because it could infer the structure from context. For a single, unchanging template, rules are more accurate; for diverse documents, the LLM is more robust.

4. What are the main cost and performance trade-offs between the two methods?

Cost and speed differ significantly. The rule-based extractor uses only CPU and is extremely cheap: processing each document costs fractions of a cent and takes under a second. In contrast, the LLM approach requires a GPU (even a modest one like an RTX 3060) or cloud API calls. Locally, it processes about 10–15 documents per minute (due to inference time), and cloud API costs can add up quickly if hitting millions of pages. Additionally, the LLM's memory footprint is large—LLaMA 3 8B requires ~8–16 GB VRAM. For a high‑volume, stable document type, rules are vastly more economical; for low‑volume, variable documents, the LLM's flexibility may justify the extra cost.

5. When would you choose one approach over the other in a production system?

Choose a rule-based system when: documents follow a fixed layout, the business domain is narrow (e.g., one supplier), and you need high throughput at low cost. It's ideal for deterministic, auditable extraction where every field is predictable. Choose an LLM-based system when: you receive invoices or orders from many different partners, templates change frequently, or you need to extract free‑text fields (like notes). The LLM also shines when you want to minimize maintenance—no need to rewrite rules for every new layout. A hybrid approach often works best: use rules for known templates and fall back to an LLM for exceptions.

6. What were the biggest surprises or lessons learned from this comparison?

The biggest surprise was how brittle rules are despite their speed. Even a small change—like moving a field to a different position on the PDF—caused extraction errors that took hours to debug. Conversely, the LLM's hallucinations were unpredictable: it occasionally fabricated order numbers that looked plausible but were completely wrong. Another lesson: OCR quality is a bottleneck for both methods. A poorly scanned PDF (low contrast, skewed) degraded rule extraction more than the LLM's, but the LLM sometimes "guessed" the text incorrectly. The key takeaway is that no single approach is universally superior; the best solution depends on document variability, volume, and the cost of errors.

7. How can you evaluate which method fits your specific B2B document extraction needs?

Start by profiling your document set: collect at least 100 representative PDFs and note the number of different layouts, field placement consistency, and the presence of handwritten entries. Run a small pilot with both methods on this sample, measuring accuracy per field, processing time, and failure modes (e.g., misread digits vs. missed fields). Also estimate your total document volume per month and the cost per document for each approach (including GPU depreciation or API fees). Finally, decide on acceptable error rates: for financial documents, even a 1% hallucination rate may be too high. Use a decision matrix to weigh accuracy, speed, cost, and maintainability.