Tagging Documents with a Tiered Taxonomy
An exploration of 23 low-cost methods of applying a taxonomy to EU laws and humanitarian reports
TL;DR
I tested 23 ways of tagging documents with labels from a tiered taxonomy, from prompting an LLM through to small models trained on labeled examples, on EU laws and humanitarian excerpts that people had already tagged. A combined method, which weighs the suggestions of several small trained models, beat the published benchmarks on both datasets and every low-cost LLM method I tried. LLMs did earn their place on rare labels, and when shown labeled examples, and more powerful and costly LLMs would certainly perform better. Everything is in an interactive report and a GitHub repo (opens in a new tab) you can run yourself.

I love a good taxonomy! 🤓
In an earlier post I looked at building a taxonomy from documents automatically, using TaxoAdapt. That’s half of the job. The other half is applying one: going through a pile of documents and giving each the right labels, at the right level of detail.
My hope was that in 2026 is that an LLM can just, well, magically do this. 🤞
Our test data
EU laws. MultiEURLEX has 65,000 EU laws in 23 languages (Chalkidis et al., EMNLP, 2021 (opens in a new tab); Dataset (opens in a new tab)). The EU Publications Office tagged each one with labels from EuroVoc (opens in a new tab), the thesaurus it runs for the EU’s own work and the vocabulary EU documents are indexed with. EuroVoc is a big one: 7,372 labels in eight tiers. The test laws are newer than the training laws, as they would be in real life.
Humanitarian excerpts. HumSet has excerpts from crisis reports in English, French and Spanish (Fekih et al., Findings of EMNLP, 2022 (opens in a new tab); Dataset (opens in a new tab), Dataset 2.0 (opens in a new tab)). Humanitarian analysts in 46 response projects picked and tagged them on DEEP, a shared analysis platform. Those projects used 11 different analysis frameworks between them, which the HumSet authors merged into one: sectors, pillars and subpillars, so two tiers. I used version 2.0.0 of the dataset, which includes 12,896 test set excerpts.
For each method I scored the exact label a person chose, and also each tier for that label, because a label can land in the right branch with the wrong tier. If a person tagged a law labelling and a method chose packaging, both are tier 4 labels under the same tier 3 label, preparation for market, so it’s right at the top three tiers and wrong on the exact label. This is common when dealing with multi-tiered taxonomies.
The are many ways to tag a document
The methods I tried fell into a few families:
- Prompting an LLM, from simple to clever: write keywords and match them to labels, choose from the full list, search the taxonomy first and then choose, or walk down the tree one level at a time.
- LLM with examples: KI-FSPrompt shows the LLM similar documents that people have already tagged.
- Trained models: Omikuji, a similar-documents lookup and an embedding classifier, which run on a laptop, plus XR-Transformer.
- Fine-tuned models: Legal-BERT, HumBERT, a reranker, GLiNER2.5 and a small LLM, each fine-tuned on a rented GPU for a few dollars.
- A combined method: Orchestration where several of the trained models each suggest labels, and a scorer trained on documents they never saw decides which to keep. Which models it uses was chosen for each dataset.
The LLM methods used open models, gpt-oss-120b and gpt-oss-20b to align with research papers and keep costs low, through Hugging Face Inference Providers. Most methods are simplified from their papers, especially where some details of method were hard to figure out from the published information.
Do we have a winner?
Sigh. I wasn’t able to find any magic AI solution yet, at least not at scale and low-cost. Careful orchestration won the day in this test, a combined method with multiple models.

On the EU laws it scored 0.566 by exact labels, against 0.497 for the best single model, a fine-tuned Legal-BERT. Against the published benchmarks, on the full test sets …
- EU laws: 0.610 mean R-Precision trained on the training laws only, as BERT was (0.640 trained on newer laws as well), against 0.484 for BERT in the MultiEURLEX paper (Chalkidis et al., EMNLP, 2021 (opens in a new tab)).
- HumSet 2.0.0: 0.663 mean macro F1 across the five label groups, against 0.648 for HumBERT, the mean of the five scores its paper prints (Tamagnone et al., IJCAI, 2023 (opens in a new tab)).

Where LLMs did help
On the EU laws, every fine-tuned model beat every LLM method. The best LLM method, KI-FSPrompt, scored 0.340 by exact labels, against 0.566 for the combined method. On HumSet it was closer: KI-FSPrompt beat three of the fine-tuned models, though it still trailed every trained model without an LLM inside it.
But LLMs were good at something the trained models weren’t. A plain LLM keyword pipeline found 49% of the correct labels that never appeared in training; the combined method found 1%. That matters for a taxonomy that keeps growing. The catch is that the pipeline suggested about 15 labels per law when a law has about 6.5, so most of its suggestions were wrong.
A few other things stood out:
- Showing the LLM labeled examples helps a lot. KI-FSPrompt beat the plain keyword pipeline by 0.13 on the laws, at about half the cost.
- Search-then-choose needs a trained model in the shortlist. Searching the taxonomy alone put only 41% of the correct labels in the top 100 for the LLM to choose from. Adding a trained model’s suggestions raised that to 75%.
- More LLMs isn’t better. A five-LLM ensemble did worse than the plain pipeline and cost about eight times as much.
All said, this was with low-cost LLMs. In a brief pilot on 50 of the laws, swapping Anthropic’s Opus 5.5 into KI-FSPrompt raised its exact-label F1 from 0.342 to 0.463, at about $68 per 1,000 laws against $0.40 with the open models. A real gain, but still below fine-tuned Legal-BERT on the same laws, 0.530.
Things that didn’t work
Walking down the tree. Asking an LLM to pick a tier-1 label, then a child of it, and so on, scored 0.02 on the laws. By design, one wrong turn near the top rules out everything below it. Keeping three branches open helped, but not much.
Asking the LLM to check its own work. A second call to check each label lowered F1 by 0.02 on both datasets and doubled the number of calls but didn’t significantly improve performance in this study.
A little on the research
This lines up nicely with a recent benchmark on subject indexing at the German National Library. Kähler and colleagues found that supervised extreme multi-label methods using transformer-based features gave the best overall results, while LLM-based methods did better on graded relevance and in the long tail of rarely used subjects (Kähler et al., arXiv preprint, 2026 (opens in a new tab)). Different documents and a different taxonomy, but a similar picture: trained models for the bulk of the work, LLMs for the long tail.
Yes, there are caveats
- Results vary between runs. One of the trained models, Omikuji, can’t be seeded, and it affects which models the combined method picks. In two full reruns of mine, the headline figures moved by up to about 0.01: 0.630 and 0.632 on the EU laws, 0.670 and 0.672 on HumSet. Every finding held in both.
- The EU laws comparison uses a sample of 600 test laws, so small differences between neighboring methods may not hold on other documents.
- Stronger LLMs would absolutely change the order of the LLM methods and narrow the gap, but at higher cost.
This is just a quick review of techniques, no doubt with more feature engineering and tuning, results will change.
Try it yourself
The interactive report has every result, a card for each method with the papers behind it, and tables you can sort.
The repo (opens in a new tab) runs the whole thing with one command: it downloads the data, runs the notebooks, launches the GPU fine-tuning on Hugging Face Jobs and builds the report. At November 2026 prices a full run costs about $100 in LLM calls and $37 of GPU time, or about $13 if you stop after the LLM experiments.
Wrapping up
So, on two very different taxonomies, a combination of small trained models beat the published benchmarks and every LLM method I tried. LLMs earned their place on rare and unseen labels, and when shown labeled examples, but not as the main tagger, at least not with the models I used.
I’d love to hear if you’ve had a different experience applying a taxonomy, especially with stronger models.
Please follow me on Medium (opens in a new tab) or LinkedIn (opens in a new tab) for more noodling about with AI.
Unless otherwise noted, all images are by the author.
References
- Chalkidis, Fergadiotis and Androutsopoulos (2021). MultiEURLEX — A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer, Proceedings of EMNLP 2021, pages 6974–6996: aclanthology.org/2021.emnlp-main.559 (opens in a new tab)
- Fekih, Tamagnone, Minixhofer, Shrestha, Contla, Oglethorpe and Rekabsaz (2022). HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crises Response, Findings of EMNLP 2022, pages 4379–4389: aclanthology.org/2022.findings-emnlp.321 (opens in a new tab)
- Tamagnone, Fekih, Contla, Orozco and Rekabsaz (2023). Leveraging Domain Knowledge for Inclusive and Bias-aware Humanitarian Response Entry Classification, Proceedings of IJCAI 2023, pages 6219–6227: ijcai.org/proceedings/2023/690 (opens in a new tab)
- Kähler, Konermann, Kluge and Schumacher (2026). Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature, arXiv preprint 2607.14882: arxiv.org/abs/2607.14882 (opens in a new tab)