Which taxonomy taggers work best
We tested 23 ways of tagging documents with labels from a tiered taxonomy, on EU laws and on humanitarian excerpts. The most accurate was our combined method: several trained models each suggest labels, and a scorer trained on documents they never saw decides which to keep. Which models it draws on is chosen for each dataset. It led on the EU laws, and on HumSet 2.0.0 by the HumBERT paper's measure, though XR-Transformer edges it by exact labels. Methods that prompt an LLM came well behind, though on the EU laws they found more of the rare labels.
Mean R-Precision on the full test set, trained on the training laws only, as BERT was; trained on newer laws as well, it scores 0.640. BERT's score is from the MultiEURLEX paper (Chalkidis et al., 2021 (opens in a new tab)).
Mean macro F1 over five label groups on the full test set; HumBERT's is the mean of the five in its paper (Tamagnone et al., 2023 (opens in a new tab)). On that paper's own average it is 0.739 against 0.73.
KI-FSPrompt, which prompts an LLM, by exact-label F1 on the EU laws.
Every score runs from 0 to 1, and higher is better.
The results
Every method was run on the same 600 EU laws and on all 12,896 test excerpts of HumSet 2.0.0, the HumBERT paper's test set, and scored the same way. None of them saw these documents during training or tuning. Open a method's card, by its name, to read what it does and the papers behind it. Sort the table by any column heading.
What the numbers mean
- F1 score
- Balances finding the correct labels against adding wrong ones. 1.00 is perfect.
- Exact labels
- Credit only for the very label a person assigned. The main measure, and the hardest. On HumSet, a pillar given together with one of its own subpillars is dropped first, since the subpillar already names it; a pillar given alone stays.
- Tier 1, 2, 3
- Credit for landing in the right branch at that level of the taxonomy, even with the wrong label inside it. Tier 1 is the broadest, so the easiest.
- Macro F1
- HumSet only. The measure the HumBERT paper (Tamagnone et al., 2023 (opens in a new tab)) reports: F1 worked out for each label, so rare labels count as much as common ones, averaged in each of five groups of labels and then over the groups. Trained models get a threshold per label, chosen on development excerpts; LLM methods, which give labels without scores, count every label they return.
- Rare labels
- Recall, from 0 to 1: the share each method found of the correct labels in the least-used quarter of labels in training, and of those never given in training.
- Cost
- LLM methods: what the calls cost per 1,000 documents. Trained methods: one-off GPU time, or free on a laptop.
Kinds of method
- LLM
- Prompts a general-purpose LLM. Needs no labelled documents.
- LLM + examples
- Prompts an LLM and shows it labelled training documents.
- LLM + trained model
- An LLM chooses from a shortlist that a trained model helped make.
- No training
- Embedding search only. Needs no labelled documents and no LLM.
- Trained
- Learns from the dataset's labelled documents. Trains on a laptop, except XR-Transformer.
- Fine-tuned
- A pretrained neural model further trained on the dataset, on a rented GPU.
- Combined
- Our method. Several trained models each suggest labels, and a scorer trained on documents they never saw decides which to keep. Which models it draws on is chosen for each dataset.
Why the exact score is lower than the tier scores
Every box below is a label in our copy of EuroVoc, which arranges its labels in 8 tiers. It is not quite a tree: 244 labels have more than one parent. Tiers 1 and 2 only group the labels below them. Laws are tagged from tier 3 down (Chalkidis et al., 2021 (opens in a new tab)), and none of these test laws has a tier 1 or 2 label.
Say a person gave a law the tier 4 label labelling
, and a method chose
packaging
, another tier 4 label. Both sit under the same tier 3 label,
preparation for market
.
- Tier 1tradeboth match
- Tier 2marketingboth match
- Tier 3preparation for marketboth match
- Tier 4 labelling person packaging method no match
A tier score replaces each label with the label or labels above it at that tier, through every parent, then compares those. At tiers 1, 2 and 3 both labels become the same label, so this near miss counts as correct. The exact score compares the labels themselves, at whatever tier they sit, here tier 4, so it scores nothing.
- EU laws: our copy of EuroVoc has 7,372 labels, only 693 of them in tiers 1 to 3; the MultiEURLEX paper counts 7,390 in its version. Of the labels people assigned to these test laws, 94% sit deeper than tier 3, so every method's exact score is below all three of its tier scores.
- Humanitarian excerpts: the taxonomy has only two tiers, and 44% of the labels people assigned are tier 1 sectors and pillars with no subpillar. The exact score mixes the two tiers, so for 20 of the 22 methods it falls between the tier 1 and tier 2 scores.
EU laws (MultiEURLEX): 600 test laws, EuroVoc's top three tiers
Where the labels come from. The laws and their labels are MultiEURLEX (Chalkidis et al., 2021 (opens in a new tab); Dataset (opens in a new tab)). The EU Publications Office tagged each law with labels from EuroVoc (opens in a new tab), the thesaurus it manages for the EU's work, in the 24 official EU languages and three more. Because the EU publishes and maintains it, and its own institutions use it, EuroVoc is the vocabulary EU documents are indexed with. Its users also include national and regional parliaments in Europe, national governments and private users.
The laws are split by date, so the test laws are newer than the training laws. Rows marked "with newer laws" were also trained on newer laws from the development split, which brings them closer in time to the test laws. The published BERT benchmark (Chalkidis et al., 2021 (opens in a new tab)) used training laws only. A rare label here was given to at most 3 of the training laws.
Tiers 1 to 3 here come from moving each label up our copy of EuroVoc, which is newer than the dataset (notebook 01). MultiEURLEX publishes its own labels for its levels 1 to 3 (Chalkidis et al., 2021 (opens in a new tab)), and ours match them for 99.4% of test laws at tier 1, 84.8% at tier 2 and 72.4% at tier 3. So these tier scores cannot be set beside the paper's results by level.
| Method | Kind | F1 score, higher is better | Recall, higher is better | Labels per law | Cost | ||||
|---|---|---|---|---|---|---|---|---|---|
| Exact labels | Tier 1 | Tier 2 | Tier 3 | Rarest quarter | Never seen | ||||
| Adds GLiNER2.5 and each label's recent history | Combined | 0.57 | 0.75 | 0.71 | 0.71 | 0.17 | 0.01 | 3.9 | $10.92 GPU, once |
| Adds each label's recent history | Combined | 0.55 | 0.77 | 0.72 | 0.72 | 0.09 | 0.00 | 5.1 | $3.85 GPU, once |
| Weighs fine-tuned Legal-BERT, similar documents, label search, fine-tuned small LLM and fine-tuned reranker, chosen on development laws | Combined | 0.55 | 0.73 | 0.70 | 0.70 | 0.12 | 0.02 | 3.8 | $8.46 GPU, once |
| Weighs fine-tuned Legal-BERT, embedding classifier, Omikuji and XR-Transformer, chosen on development laws | Combined | 0.53 | 0.73 | 0.69 | 0.68 | 0.05 | 0.00 | 3.9 | $3.85 GPU, once |
| A BERT-style model with one output per label | Fine-tuned | 0.50 | 0.75 | 0.70 | 0.69 | 0.13 | 0.01 | 5.2 | $1.09 GPU, once |
| A BERT-style model with one output per label | Fine-tuned | 0.47 | 0.73 | 0.68 | 0.67 | 0.06 | 0.00 | 4.6 | $1.03 GPU, once |
| Reads text and label names together | Fine-tuned | 0.47 | 0.74 | 0.68 | 0.67 | 0.19 | 0.06 | 4.6 | $2.46 GPU, once |
| Word counts, label trees | Trained | 0.46 | 0.69 | 0.65 | 0.63 | 0.01 | 0.00 | 4.8 | laptop |
| Qwen2.5-7B, used as a classifier | Fine-tuned | 0.46 | 0.70 | 0.66 | 0.66 | 0.02 | 0.00 | 4.1 | $3.37 GPU, once |
| BGE-M3 embeddings, one output per label | Trained | 0.44 | 0.71 | 0.65 | 0.64 | 0.01 | 0.00 | 5.5 | laptop |
| The leading trained model in a 2026 benchmark (Kähler et al., 2026 (opens in a new tab)) | Trained | 0.43 | 0.70 | 0.65 | 0.64 | 0.00 | 0.00 | 4.0 | $2.82 GPU, once |
| Word counts, label trees | Trained | 0.43 | 0.67 | 0.62 | 0.61 | 0.00 | 0.00 | 4.2 | laptop |
| Labels of the closest training documents | Trained | 0.43 | 0.71 | 0.64 | 0.60 | 0.11 | 0.00 | 5.0 | laptop |
| BGE-M3 embeddings, one output per label | Trained | 0.42 | 0.71 | 0.64 | 0.62 | 0.00 | 0.00 | 5.5 | laptop |
| Reads text and label together | Fine-tuned | 0.40 | 0.68 | 0.62 | 0.61 | 0.12 | 0.07 | 4.2 | $4.00 GPU, once |
| Labels of the closest training documents | Trained | 0.39 | 0.63 | 0.57 | 0.53 | 0.02 | 0.00 | 3.4 | laptop |
| LLM shown similar labelled documents | LLM + examples | 0.34 | 0.68 | 0.59 | 0.54 | 0.25 | 0.25 | 9.8 | $0.40 per 1,000 |
| Shortlist from both | LLM + trained model | 0.24 | 0.54 | 0.42 | 0.36 | 0.15 | 0.20 | 4.4 | $0.61 per 1,000 |
| LLMs write keywords; keywords are matched to labels | LLM | 0.21 | 0.61 | 0.46 | 0.38 | 0.35 | 0.49 | 15.3 | $0.87 per 1,000 |
| An LLM picks from the 100 closest labels | LLM | 0.19 | 0.43 | 0.32 | 0.27 | 0.15 | 0.21 | 3.9 | $0.46 per 1,000 |
| Five LLMs write keywords, one rates them | LLM | 0.17 | 0.50 | 0.37 | 0.30 | 0.18 | 0.26 | 10.0 | $6.69 per 1,000 |
| As above, matching written definitions | LLM | 0.17 | 0.41 | 0.31 | 0.26 | 0.18 | 0.23 | 3.8 | $0.59 per 1,000 |
| No extra call | LLM | 0.15 | 0.37 | 0.27 | 0.23 | 0.14 | 0.18 | 2.7 | $0.59 per 1,000 |
| A second call checks each label | LLM | 0.15 | 0.36 | 0.26 | 0.22 | 0.15 | 0.20 | 3.0 | $0.80 per 1,000 |
| Starts lower, looks ahead | LLM | 0.14 | 0.39 | 0.25 | 0.17 | 0.12 | 0.17 | 2.9 | $0.88 per 1,000 |
| Closest label names, no training | No training | 0.12 | 0.50 | 0.33 | 0.25 | 0.10 | 0.19 | 9.8 | laptop |
| Top-down, several paths | LLM | 0.05 | 0.32 | 0.14 | 0.08 | 0.08 | 0.10 | 4.2 | $1.35 per 1,000 |
| Top-down, one path | LLM | 0.02 | 0.18 | 0.06 | 0.03 | 0.01 | 0.03 | 1.4 | $0.59 per 1,000 |
Humanitarian excerpts (HumSet 2.0.0): all 12,896 test excerpts, the HumBERT paper's test set
Where the labels come from. The excerpts and their labels are HumSet (Fekih et al., 2022 (opens in a new tab); Dataset (opens in a new tab); Project page (opens in a new tab)). Humanitarian analysts in 46 response projects picked them from crisis documents on DEEP, an analysis platform run by Data Friendly Space, a US-based international NGO, whose board includes OCHA, UNHCR, UNICEF and the IFRC. Each project tagged with its own analysis framework, 11 between them, and the HumSet authors merged these into one common framework of sectors, pillars and subpillars. Unlike EuroVoc, it is not a standard published on its own, but a merger of the frameworks these projects worked with.
Excerpts in English, French and Spanish. Tier 1 holds the sectors and pillars, and tier 2 the subpillars. A rare label here was given to at most 596 of the training excerpts. Every label in these test excerpts was given in training, so the never-seen column reads none. Every method, LLM methods included, was trained, tuned and run on this version's own splits (notebooks 15 and 17), so the macro F1 columns score the paper's own test set by its measure (Tamagnone et al., 2023 (opens in a new tab)) and can be set beside its printed figures; the note on the recipe's row says what the paper reported and how our cut-offs differ from its trainer's.
How the labels differ from HumSet 1.0.0
Version 2.0.0 uses 77 labels. Only it has
Capacities & Response
, Capacities & Response->People Reached/Response Gaps
, Capacities & Response->Red Cross/Red Crescent
, Casualties
, Context
, Covid-19
, Covid-19->Prevention Campaign
, Covid-19->Research And Outlook
, Displacement
, Humanitarian Access
, Humanitarian Access->People Facing Humanitarian Access Constraints/Humanitarian Access Gaps
, Humanitarian Conditions
, Impact
, Information And Communication
, Priority Needs
, Shock/Event
and Shock/Event->Mitigating Factors
; only 1.0.0 has Capacities & Response->Number Of People Reached/Response Gaps
and Humanitarian Access->Number Of People Facing Humanitarian Access Constraints/Humanitarian Access Gaps
. Version 2.0.0
also tags some excerpts with a pillar but none of its subpillars; that pillar is then a label of its own.
The label browser and examples are from 1.0.0.
| Method | Kind | F1 score, higher is better | Recall, higher is better | Labels per excerpt | Cost | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Macro F1, sectors and pillars (paper's Avg) | Exact labels | Tier 1 | Tier 2 | Macro F1 | Rarest quarter | Never seen | ||||
| The leading trained model in a 2026 benchmark (Kähler et al., 2026 (opens in a new tab)) | Trained | 0.49 | 0.72 | 0.76 | 0.69 | 0.51 | 0.44 | none | 2.1 | $1.38 GPU, once |
| Weighs fine-tuned HumBERT and similar documents, chosen on development excerpts | Combined | 0.74 | 0.72 | 0.78 | 0.71 | 0.66 | 0.58 | none | 2.4 | $0.67 GPU, once |
| The paper's own trainer and settings, rerun (Tamagnone et al., 2023 (opens in a new tab)) | Fine-tuned | 0.72paper 0.73 | 0.69 | 0.77 | 0.68 | 0.63paper 0.65 | 0.33 | none | 2.3 | $1.26 GPU, once |
| Labels of the closest training documents | Trained | 0.65 | 0.69 | 0.73 | 0.65 | 0.59 | 0.41 | none | 2.2 | laptop |
| A BERT-style model with one output per label | Fine-tuned | 0.72 | 0.68 | 0.76 | 0.68 | 0.64 | 0.45 | none | 2.2 | $0.67 GPU, once |
| Word counts, label trees | Trained | 0.63 | 0.67 | 0.71 | 0.64 | 0.57 | 0.44 | none | 2.3 | laptop |
| BGE-M3 embeddings, one output per label | Trained | 0.49 | 0.63 | 0.68 | 0.59 | 0.47 | 0.47 | none | 2.2 | laptop |
| LLM shown similar labelled documents | LLM + examples | 0.58 | 0.58 | 0.65 | 0.57 | 0.53 | 0.59 | none | 2.8 | $0.15 per 1,000 |
| Reads text and label together | Fine-tuned | 0.61 | 0.54 | 0.63 | 0.55 | 0.51 | 0.24 | none | 3.0 | $3.00 GPU, once |
| Reads text and label names together | Fine-tuned | 0.58 | 0.50 | 0.62 | 0.45 | 0.46 | 0.11 | none | 2.6 | $2.80 GPU, once |
| Qwen2.5-7B, used as a classifier | Fine-tuned | 0.57 | 0.49 | 0.62 | 0.44 | 0.45 | 0.12 | none | 2.4 | $1.67 GPU, once |
| Top-down, several paths | LLM | 0.40 | 0.41 | 0.47 | 0.26 | 0.30 | 0.42 | none | 1.8 | $0.16 per 1,000 |
| The LLM sees every label at once | LLM | 0.43 | 0.39 | 0.48 | 0.32 | 0.35 | 0.44 | none | 1.9 | $0.15 per 1,000 |
| As above, matching written definitions | LLM | 0.42 | 0.37 | 0.45 | 0.33 | 0.35 | 0.46 | none | 1.7 | $0.29 per 1,000 |
| An LLM picks from the 100 closest labels | LLM | 0.43 | 0.36 | 0.46 | 0.30 | 0.35 | 0.54 | none | 2.4 | $0.18 per 1,000 |
| Top-down, one path | LLM | 0.36 | 0.36 | 0.42 | 0.22 | 0.26 | 0.24 | none | 1.0 | $0.13 per 1,000 |
| LLMs write keywords; keywords are matched to labels | LLM | 0.38 | 0.34 | 0.44 | 0.23 | 0.29 | 0.28 | none | 1.4 | $0.33 per 1,000 |
| A second call checks each label | LLM | 0.40 | 0.34 | 0.42 | 0.30 | 0.33 | 0.39 | none | 1.5 | $0.39 per 1,000 |
| No extra call | LLM | 0.37 | 0.32 | 0.40 | 0.28 | 0.31 | 0.37 | none | 1.4 | $0.29 per 1,000 |
| Five LLMs write keywords, one rates them | LLM | 0.37 | 0.32 | 0.43 | 0.27 | 0.29 | 0.27 | none | 2.7 | $2.84 per 1,000 |
| Starts lower, looks ahead | LLM | 0.20 | 0.22 | 0.33 | 0.30 | 0.20 | 0.40 | none | 1.5 | $0.22 per 1,000 |
| Closest label names, no training | No training | 0.33 | 0.11 | 0.25 | 0.07 | 0.25 | 0.52 | none | 7.5 | laptop |
We first ran every method on 1,500 sampled test excerpts of HumSet 1.0.0, an earlier version with different splits. There the combined method had the top score by exact labels, 0.662 against XR-Transformer's 0.642, and on all 15,147 of that version's test excerpts it beat HumBERT's recipe, rerun by us, by macro F1, 0.574 to 0.560.
LLM methods used openai/gpt-oss-120b:novita and
openai/gpt-oss-20b:deepinfra through Hugging Face Inference Providers, except
LLM-Ensemble, which
used the five open models listed in configs/experiment.yaml. GPU training ran on Hugging Face
Jobs at
November 2026 prices.
Findings
- The combined method is the most accurate. On the laws it scores 0.566 by exact labels, against 0.497 for the best single model, the fine-tuned Legal-BERT. On HumSet 2.0.0 it leads by the HumBERT paper's macro F1, 0.663 against 0.506 for XR-Transformer, the best single model, though XR-Transformer edges it by exact labels, 0.722 to 0.716. A version of the combined method also has the top score at every tier.
- Trained models beat LLM methods, with exceptions on HumSet. On the laws every fine-tuned model scores above every LLM method; the best LLM method, KI-FSPrompt, scores 0.340. On HumSet 2.0.0 KI-FSPrompt scores 0.585: below every trained model that has no LLM inside it, but above the fine-tuned reranker (0.542), GLiNER2.5 (0.502) and fine-tuned small LLM (0.493). KI-FSPrompt also needs labelled documents, because it shows them to the LLM as examples.
- LLM methods fall further behind on finer labels. On the laws, KI-FSPrompt trails the best single model by 0.07 at tier 1 (0.68 against 0.75), and by 0.15 at tier 3 (0.54 against 0.69).
- LLM methods find labels the training data lacks. The LLM keyword pipeline finds 49% of the correct labels never seen in training; the combined method finds 1%. But the pipeline suggests 15 labels per law, where a law has about 6.5, so most of its suggestions are wrong.
- Showing the LLM labelled examples is the best LLM recipe. KI-FSPrompt beats the plain LLM keyword pipeline by 0.13 (95% interval 0.12 to 0.15) on the laws and by 0.17 (95% interval 0.16 to 0.19) on the HumSet 1.0.0 sample, at about half the cost. LLM-Ensemble, with five LLMs, did worse than the plain pipeline and cost about eight times as much.
- Search-then-choose needs a trained model in the shortlist. On the laws, search alone put only 41% of the correct labels in its top 100, so the LLM could not choose the rest. Adding the trained model's suggestions raised that to 75%, and F1 by 0.08 (95% interval 0.06 to 0.09).
- Walking down the taxonomy one branch at a time fails. It scores 0.02 on the laws. Keeping three branches open helps on both datasets. Starting from search results instead of from the top helps on the laws but hurts on the excerpts.
- Asking the LLM to double-check its labels made things worse. F1 fell by 0.02 (95% interval 0.01 to 0.02) on the laws and by 0.02 (95% interval 0.02 to 0.03) on the HumSet 1.0.0 sample, and the number of calls doubled. Keeping only labels with a supporting quote also lowered F1.
Each difference with an interval was tested on the same documents: the 600 laws, or the sample of HumSet 1.0.0 where every method was first run. Every one of those intervals excludes zero, so none of these differences is likely to be chance.
Against the published benchmarks
The published benchmarks use the full test sets: 5,000 EU laws, and 12,896 humanitarian excerpts in HumSet 2.0.0. Every method was run on all of HumSet 2.0.0's, as its table above shows; on the laws only methods without an LLM call per law were run on all of them.
EU laws: mean R-Precision
For each law, take as many of the top-ranked labels as the law really has, and count the share that are correct. 1.00 is perfect. The MultiEURLEX paper prints it as a percentage: 48.4 there is 0.484 here. That figure is from its Table 4: BERT on English, all of the original labels, trained on the training laws only, so our rows marked "training laws only" are the like-for-like comparison. The paper scores against all 7,390 of its labels; we leave out the 72 of the 31,178 labels given to these laws that are missing from our copy of EuroVoc. Counting them lowers our scores by less than 0.001, in a check made outside the notebooks.
- Combined method, second attemptWith newer laws0.640
- Combined method, first attemptWith newer laws0.624
- Combined method, second attemptTraining laws only0.610
- Combined method, first attemptTraining laws only0.600
- Fine-tuned Legal-BERTWith newer laws0.529
- Fine-tuned Legal-BERTTraining laws only0.500
- BERT, publishedThe benchmark to beat (Chalkidis et al., 2021 (opens in a new tab))0.484
- XR-TransformerTraining laws only0.474
- OmikujiTraining laws only0.471
- Embedding classifierTraining laws only0.459
- Similar documentsTraining laws only0.424
- Label searchNo training0.139
The second attempt added GLiNER2.5 and each label's recent history. It raised the score by 0.015 (95% interval 0.013 to 0.018) with newer laws, and by 0.010 (0.007 to 0.013) with training laws only. It was made after seeing the first attempt's test scores, so we report it as a second attempt.
Humanitarian excerpts: mean macro F1
F1 is worked out for each label and then averaged, so rare labels count as much as common ones. This is the measure the HumBERT paper (Tamagnone et al., 2023 (opens in a new tab)) reports, over five groups of labels, on all 12,896 test excerpts of HumSet 2.0.0.
- Combined methodWeighs fine-tuned HumBERT and similar documents, chosen on development excerpts0.663
- HumBERT, publishedThe benchmark to beat (Tamagnone et al., 2023 (opens in a new tab))0.648
- HumBERT recipe, rerunThe authors' code and settings0.643
- Fine-tuned HumBERT, oursThe best single source on development excerpts0.639
| Label group | HumBERT, published | HumBERT recipe, rerun | Combined method | 95% interval |
|---|---|---|---|---|
| Sectors | 0.78 | 0.778 | 0.790 | 0.780–0.799 |
| Pillars 1D | 0.70 | 0.685 | 0.705 | 0.696–0.718 |
| Subpillars 1D | 0.54 | 0.548 | 0.574 | 0.562–0.588 |
| Pillars 2D | 0.71 | 0.705 | 0.721 | 0.707–0.735 |
| Subpillars 2D | 0.51 | 0.501 | 0.527 | 0.512–0.541 |
| Five-group mean | 0.648 | 0.643 | 0.663 | 0.657–0.670 |
| Tier-1 average | 0.73 | 0.722 | 0.739 | – |
On HumSet 2.0.0, the version HumBERT's published figures come from, the combined method's 95% interval for the five-group mean lies wholly above the published figure. Its interval lies wholly above the published figure in Subpillars 1D and Subpillars 2D. It beats HumBERT's recipe, rerun with the authors' own code and settings, in every group.
Whether that rerun reproduces the published figures depends on how subpillar scores are averaged, which the paper (Tamagnone et al., 2023 (opens in a new tab)) does not say. Averaging every label once, as every score here does, it comes within 0.015 of every published group. Averaging subpillars within each pillar first, as the authors' code does, it falls short in Subpillars 1D (by 0.041), so notebook 13 records it as not reproducing the paper. The published figures have two decimals; their mean is worked out from them.
The HumBERT paper (Tamagnone et al., 2023 (opens in a new tab)) prints no mean of the five groups. Its own "Avg" column is the tier-1 average in the table's last row: the mean of Sectors, Pillars 1D and Pillars 2D, leaving out both subpillar groups, which score lowest.
The recipe rerun is one run with one seed, so it has no interval.
Labelling long documents in pieces
Long documents were cut into 400-word pieces, an LLM method labelled each piece, and four rules combined the pieces' labels into labels for the whole document. Weighting each piece by how sure the model was gave the same result as keeping any label found in any piece. Which rule worked best depended on the dataset.
| Documents | First piece only | Label in a share of pieces | Label in any piece | Weighted by certainty | LLM cost |
|---|---|---|---|---|---|
| 120 long EU laws | 0.164 labels | 0.223 labels | 0.1155 labels | 0.1155 labels | $3.12 |
| 200 humanitarian reports | 0.162 labels | 0.273 labels | 0.5412 labels | 0.5412 labels | $1.31 |
Exact-label F1, with the average number of labels each rule gave a document. The certainty-weighted rule follows (Wu et al., 2026b (opens in a new tab)), listed below.
Limits
- Most methods are simplified from their papers, so these results say little about the originals. Each method's card says how ours differs.
- The EU laws table uses 600 laws, a sample of the test set; HumSet 2.0.0's table uses its whole test set. Small differences between neighbouring rows may not hold on other documents.
- Some findings, marked with the HumSet 1.0.0 sample, were measured only on that earlier version, where every method was first run.
- Results depend on the LLMs named above. Other LLMs could change the order of the LLM methods.
- The second attempt was made after seeing the first attempt's test scores. The pass marks it was judged against were not changed.
Papers and software
- MultiEURLEX – A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer (opens in a new tab)Ilias Chalkidis, Manos Fergadiotis and Ion Androutsopoulos (2021). arXiv preprint.
- LEGAL-BERT: The Muppets straight out of Law School (opens in a new tab)Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis and others (2020). arXiv preprint.
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (opens in a new tab)Jianlv Chen, Shitao Xiao, Peitian Zhang and others (2024). arXiv preprint.
- CTI2TTP: Evidence-Grounded TTP Extraction from Cyber Threat Intelligence via Diverge-then-Converge Verification (opens in a new tab)Yutong Cheng, Changze Li, Raihan Sultan Pasha Basuki and others (2026). arXiv preprint.
- Introducing GLiNER2.5: Efficient Span-Free Information Extraction with Schema-Driven Interface (opens in a new tab)Fastino (2026). Fastino blog.
- HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crisis Response (opens in a new tab)Selim Fekih, Nicolò Tamagnone, Benjamin Minixhofer and others (2022). arXiv preprint.
- LoRA: Low-Rank Adaptation of Large Language Models (opens in a new tab)Edward J. Hu, Yelong Shen, Phillip Wallis and others (2021). arXiv preprint.
- DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing (opens in a new tab)Lisa Kluge and Maximilian Kähler (2025). SemEval-2025.
- Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature (opens in a new tab)Maximilian Kähler, Katja Konermann, Lisa Kluge and Markus Schumacher (2026). arXiv preprint.
- DNB-AI-Project at the GermEval-2025 LLMs4Subjects Task: KIFSPrompt – Knowledge-Injected Few-Shot Prompting (opens in a new tab)Maximilian Kähler, Lisa Kluge and Katja Konermann (2025). KONVENS 2025 Workshops.
- MATCH: Multi-stage Article Tagging to Concept Hierarchies (opens in a new tab)Daniele Raimondi, Feichi Lu, Oliver Grun, Mariia Eremina and Andrea Perlato (2026). arXiv preprint.
- Graph-Constrained Policy Learning for Extreme Clinical Code Prediction (opens in a new tab)Amritpal Singh, Sebastian Torres, Khawar Shakeel and Syed Ahmad Chan Bukhari (2026). arXiv preprint.
- classifier-trainer, the trainer behind HumBERT's classifier (opens in a new tab)Data Friendly Space, the team behind DEEP (2023). GitHub.
- Leveraging Domain Knowledge for Inclusive and Bias-aware Humanitarian Response Entry Classification (opens in a new tab)Nicolò Tamagnone, Selim Fekih, Ximena Contla, Nayid Orozco and Navid Rekabsaz (2023). IJCAI 2023.
- Constraint-Aware Hierarchical Search for Regulation-Driven Fine-Grained Classification (opens in a new tab)Siyu Wang, Wei Tan and Lulu Chen (2026). arXiv preprint.
- Label-Centric Knowledge Retrieval for Zero-Shot Text Classification (opens in a new tab)Daoping Wu, Dong Wang, Siyu Liu and others (2026a). arXiv preprint.
- Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement (opens in a new tab)Yuhe Wu, Rui Qian, Guangyu Wang and others (2026b). arXiv preprint.
- Travel on the ICD Tree: Benchmarking Agentic Reasoning for ICD Coding from Chinese Electronic Medical Records (opens in a new tab)Xinjie Xu, Yongqi Fan, Shuang-shuang Chen and others (2026). Findings of ACL 2026.
- LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification (opens in a new tab)Hui Ye, Jing Zhang, Xiulong Yang and Rajshekhar Sunderraman (2026). arXiv preprint.
- Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification (opens in a new tab)Jiong Zhang, Wei-cheng Chang, Hsiang-fu Yu and Inderjit S. Dhillon (2021). arXiv preprint.
- Qwen2.5 Technical Report (opens in a new tab)Qwen team (2024). arXiv preprint.
- PECOS: Prediction for Enormous and Correlated Output Spaces (opens in a new tab)Amazon. Software, GitHub.
- bge-reranker-v2-m3 (opens in a new tab)BAAI. Model card, Hugging Face.
- Omikuji: an efficient implementation of Partitioned Label Trees (opens in a new tab)Tom Dong. Software, GitHub.
Paper details checked against arXiv and the ACL Anthology on 10 November 2026. Built by
scripts/make_report.py from outputs/results/*.json.