· Data Science Collective

Automatically Identifying Taxonomies from Documents: Using the TaxoAdapt Algorithm with Humanitarian Evaluation Reports

In this article, we explore using a new algorithm called TaxoAdapt for automatically creating new taxonomies from documents. TaxoAdapt is interesting in that it provides a way to guide the generated categories and taxonomy structure, while maintaining discovery of new categories as supported by the corpus. To make it a little easier to work with, the paper GitHub repository was forked to add some new features, and TaxoAdapt was run on AI-generated summaries of humanitarian reports. Though processing time could be improved, the algorithm is able to generate new categories, assigned to each text in the example corpus, and could be a useful tool when designing taxonomies as well as monitoring how they change over time.

Document Tagging and Taxonomies

Tagging documents by topic is, of course, very common. It allows us to find content on a particular subject as well as the frequency subjects occur across a group of documents. We live in an age where advanced tools like semantic search can find what we need quickly, but it’s often the case that filtering by topic can improve this process further, especially when using search results with AI.

Deciding on a system of tags/categories, referred to as a taxonomy, isn’t always as easy as it sounds. Humans can come up with a taxonomy — a sometimes fraught exercise resulting in stand-up arguments! — but what if the content is changing all the time and new topics keep popping up? What if we don’t actually know the topics in the first place? What if we have a taxonomy, but maybe we’re missing something?

It soon becomes a very time-consuming process to try and go through thousands of documents by hand.

Topic Modeling

There are a lot of techniques for automatically identifying topics in text, and choosing one depends on requirements. Performance, topic structure, and level of automation all play a role in this decision, and as always, no one size fits all.

In this article, we will focus on an algorithm that allows the user to guide topics, but as we shall see, it may not be suitable for every situation.

Landscape of topic modeling. Generated by providing a bullet list to Google’s Nano Banana, expect some fuzzy bits.

Landscape of topic modeling. Generated by providing a bullet list to Google’s Nano Banana, expect some fuzzy bits.

TaxoAdapt

In the preprint TaxoAdapt: Aligning LLM-Based Multidimensional Taxonomy Construction to Evolving Research Corpora the authors presented a new way to use Large Language Models (LLMs) as part of a semi-guided framework that dynamically adapts an LLM-generated taxonomy to a given corpus. It isn’t alone in being an LLM-powered taxonomy discovery algorithm, but the authors maintain that TaxoAdapt is generalizable and provides state-of-the-art performance with “26.51% more granularity-preserving and 50.41% more coherent than the most competitive baselines judged by LLMs”.

Those are some VERY specific percentages. 🙂

TAXOADAPT, a framework that dynamically constructs a LLM-enhanced, corpus-specific taxonomy using classification-based expansion signals. The diagram demonstrates a width expansion example, but the same logic is applied to depth expansion (Source).

TAXOADAPT, a framework that dynamically constructs a LLM-enhanced, corpus-specific taxonomy using classification-based expansion signals. The diagram demonstrates a width expansion example, but the same logic is applied to depth expansion (Source).

TaxoAdapt starts from an initial set of provided dimensions, i.e. a set of top-level categories. Using an LLM, it then aligns that taxonomy to the corpus, performing iterative hierarchical classification and clustering, where it tests if categories are supported by the content, adding new sub-categories or siblings as required.

Put another way, TaxoAdapt allows you to guide the taxonomy and generate multilayer tags for your documents, where the tags are shown to apply to multiple documents.

The Code

The research paper comes with a really nice GitHub repo. As with such repos it’s designed to support the paper, so I forked it and made the following minor modifications:

  1. Support for LLM Providers: Huggingface, Azure OpenAI, and Anthropic. This extends the existing support for OpenAI and local LLMs.
  2. Ability to read input datasets from Excel. Initially, datasets were in fixed format for the paper, but Excel can be handy. I also added a simple sampling script to generate a smaller category-balanced sample sheet if needed.
  3. Ability to specify taxonomy dimensions (the starting taxonomy types) in a configuration file. These were originally hard-coded for a corpus related to scientific research papers.
  4. Additional support for LLM output JSON parsing. With the new LLMs, some extra handling of JSON was required
  5. Output Results to Excel. Handy for this article!

You can find the forked repo here, and some more information on the above enhancements here.

Our Corpus: AI-Generated Summaries of United Nations Humanitarian Evaluation Reports

To test the algorithm, we need text, basically titles and document summaries. In a previous blog post, I generated summaries on 2,743 humanitarian evaluation reports, creating a nice little dataset for this article. The sharp-eyed may have noticed that in that previous work, the summarization LLM prompt included instructions for capturing topics and themes.

It’s almost as if I knew I was going to write this article or something. 🙂

The AI-generated summaries can be found here, see the very last column ‘Abstractive Summary (map reduced)’. Here is an example for a report from the United Nations Development Coordination Office (DCO) called ‘Evaluation of the Government of Seychelles & United Nations Strategic Partnership Framework (SPF) 2019–2023’ …

**Summary:**

This report evaluates the Government of Seychelles and United Nations Strategic Partnership Framework (SPF) 2019-2023. The evaluation aimed to assess the SPF's contributions to Seychelles' development priorities, identify gaps and lessons, and provide recommendations for the next SPF cycle. The report highlights the SPF's alignment with national development priorities and SDGs, as well as its effectiveness in implementing programming principles and adapting to the COVID-19 pandemic. However, the report also notes challenges, including inadequate data and high government official turnover, which hindered the SPF's implementation.

**Main Topics:**

1. **Evaluation of the Seychelles SPF 2019-2023**: The report evaluates the SPF's relevance, effectiveness, and adaptability in the context of Seychelles' development priorities and SDGs.
2. **Methodology**: The evaluation was conducted in line with the UN Evaluation Group (UNEG) Norms and Standards for Evaluation (2021) guidelines.
3. **Seychelles' Development Priorities**: The report highlights Seychelles' national development priorities, including economic transformation, environmental sustainability, human capital development, and science, technology, and innovation development.
4. **SDGs and International Obligations**: The report emphasizes the importance of aligning the SPF with SDGs and international obligations, such as the African Union Agenda 2063.

**Key Terms & Concepts:**

1. **Strategic Partnership Framework (SPF)**: A framework for cooperation between the Government of Seychelles and the United Nations.
2. **SDGs (Sustainable Development Goals)**: A set of 17 goals adopted by the United Nations to achieve sustainable development.
3. **UN Common Country Analysis (CCA)**: A report that provides an analysis of a country's development priorities and challenges.
4. **LNOB (Leave No One Behind)**: A principle that emphasizes the importance of inclusivity and leaving no one behind in development efforts.
5. **GEWE (Gender Equality, Women's Empowerment)**: A principle that emphasizes the importance of promoting gender equality and women's empowerment in development efforts.

**Methods / Approach:**

1. **Evaluation Framework**: The evaluation was conducted in line with the UN Evaluation Group (UNEG) Norms and Standards for Evaluation (2021) guidelines.
2. **Data Analysis**: The report uses data analysis to assess the SPF's effectiveness and relevance.
3. **Stakeholder Engagement**: The report highlights the importance of stakeholder engagement in the evaluation process.

**Findings / Conclusions:**

1. **Relevance**: The Seychelles SPF was strongly aligned to national development priorities and SDGs.
2. **Effectiveness**: The UN in Seychelles successfully implemented the SPF, adapting to the COVID-19 pandemic.
3. **Challenges**: The report notes challenges, including inadequate data and high government official turnover, which hindered the SPF's implementation.
4. **Recommendations**: The report provides recommendations for the next SPF cycle, including the need for more effective data management and stakeholder engagement.

**Related Concepts / Themes:**

1. **Development Cooperation**: The report highlights the importance of development cooperation between governments and international organizations.
2. **Institutional Capacity Building**: The report emphasizes the need for institutional capacity building to support effective development cooperation.
3. **Data-Driven Decision Making**: The report highlights the importance of data-driven decision making in development efforts.
4. **Stakeholder Engagement**: The report emphasizes the importance of stakeholder engagement in development efforts.

As you can see, there is a lot of nice information in the summary that might be useful for a taxonomy.

There is an argument to say this may actually not be the best approach, as we are influencing the taxonomy already, but for the purposes of this article, we’ll use what we have.

Running TaxoAdapt

Here are some instructions …

  1. First, check out the TaxoAdapt fork GitHub repo.

2. Then configure your environment …

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Copy ‘env.example’ to .env

4. In ‘.env’ set LLM_PROVIDER to one of ‘azure’, ‘openai’, ‘huggingface’ as appropriate. Then set associated API variables as instructed by comments in .env.

For this article, we used

LLM_PROVIDER=huggingface
HUGGINGFACE_MODEL=Qwen/Qwen2.5–72B-Instruct

5. Save this data sheet into the ‘./datasets’ folder in the checked out repo, as file ‘pdf_metadata_results_2023_2025.xlsx’

6. For faster execution, we will sub-sample the spreadsheet data with …

python sample_humanitarian_data.py \
  --dataset_sheet "./datasets/pdf_metadata_results_2023_2025.xlsx" \
  --dataset_sheet_tabname "PDF Metadata" \
  --dataset_title_fieldname "Title" \
  --dataset_abstract_fieldname "Abstractive Summary (map reduced)" \
  --agency_fieldname "Agency" \
  --sample_size 100 \
  --min_reports 10 \
  --random_seed 42

This generates a file ‘pdf_metadata_results_2023_2025_sampled.xlsx’ in ‘./datasets’, which has been balanced to have roughly the same number of records for each United Nations agency, in order to improve taxonomy coverage.

7. Next, edit ‘dimension_definitions.txt’ to define our starting taxonomy dimensions. This is where you can guide the taxonomy discovery.

For this article, you can use the default included in the forked repo, which looks like this (using the paper’s original format) …

# ===== HUMANITARIAN EVALUATION DIMENSIONS (EXAMPLES) =====
Sector|||Sector: the specific humanitarian sector or domain that the research addresses, such as health, nutrition, shelter, education, protection, water and sanitation, or food security. Papers are classified based on the primary sector they contribute to or evaluate.
Modality|||Modality: the type or form of humanitarian intervention being evaluated, such as cash transfers, in-kind assistance, capacity building, service delivery, advocacy, or protection services. This dimension captures how aid is delivered and what mechanisms are used.
OutcomeDomain|||OutcomeDomain: the specific area of impact or change being measured in humanitarian evaluation, such as livelihoods, resilience, well-being, program effectiveness, behavioral change, or system strengthening. This focuses on what outcomes are being assessed.
EvaluationType|||EvaluationType: the methodological approach used for evaluation, such as impact evaluation, process evaluation, formative evaluation, summative evaluation, real-time evaluation, cost-effectiveness analysis, or mixed-methods evaluation. This dimension categorizes papers by their evaluation methodology.
CrossCuttingThemes|||CrossCuttingThemes: overarching themes that cut across multiple sectors, such as gender equality, disability inclusion, climate adaptation, conflict sensitivity, localization, accountability to affected populations, or environmental sustainability. These are horizontal concerns that apply across different interventions.
GeographicContext|||GeographicContext: the geographic setting or context of the evaluation, such as urban vs. rural, fragile states, protracted crises, natural disasters, conflict zones, or specific regional contexts (e.g., Sub-Saharan Africa, Middle East, Asia-Pacific). This captures where evaluations take place.
ActorType|||ActorType: the type of organization or actor conducting or being evaluated, such as UN agencies, international NGOs, local NGOs, government agencies, private sector actors, community-based organizations, or consortia. This identifies who is involved in the humanitarian response.
ResourceInputs|||ResourceInputs: the types of resources being provided or evaluated, such as financial resources, human resources, material goods, technical assistance, information systems, or infrastructure. This dimension focuses on what inputs are being delivered.
EvaluationCriteria|||EvaluationCriteria: the evaluation criteria being applied, based on frameworks like OECD-DAC criteria (relevance, effectiveness, efficiency, impact, sustainability, coherence) or humanitarian-specific criteria (coverage, appropriateness, protection, participation). This captures the lens through which evaluations are conducted.
CoordinationArchitecture|||CoordinationArchitecture: the coordination mechanisms and structures in place, such as cluster system, inter-agency coordination, cash working groups, local coordination forums, or government-led coordination. This examines how different actors work together.
EvidenceType|||EvidenceType: the type of evidence being generated or used in the evaluation, such as quantitative data, qualitative insights, participatory data, administrative data, satellite imagery, social media data, or indigenous knowledge. This categorizes papers by their evidence base.
RecommendationType|||RecommendationType: the nature of recommendations or findings produced, such as policy recommendations, programmatic adjustments, methodological innovations, scaling strategies, or advocacy priorities. This captures the actionable outputs of evaluations.

These categories and descriptions are a quick example, when configuring this file for your own taxonomy, you might want to elaborate further.

8. Run TaxoAdapt!

Now you have set up your LLM, sampled some input text data, and configured starting taxonomy dimensions, you can run TaxoAdapt with this command …

python main.py \
  --topic "Humanitarian evaluation" \
  --llm api \
  --dataset_sheet "./datasets/pdf_metadata_results_2023_2025_sampled.xlsx" \
  --dataset_sheet_tabname "PDF Metadata" \
  --dataset_title_fieldname "Title" \
  --dataset_abstract_fieldname "Abstractive Summary (map reduced)" \
  --dimensions_file "dimensions_definitions.txt"

There are other parameters you can experiment with …

  --max_depth MAX_DEPTH
                        Maximum depth of each taxonomy
  --init_levels INIT_LEVELS
                        Number of initial levels in the taxonomy
  --max_density MAX_DENSITY
                        Maximum density of papers per node

Refer to the paper for more information on these, but basically they control how deep and dense you want your taxonomy to be. For this article we took the defaults of …

max_depth=2
init_levels=1
max_density=40

The values you choose will depend on your use case, but also bear in mind that there are performance and cost implications, as adding more layers quickly explodes the number of times the LLM must be called.

Test Results

For our test, the script took about 2 hours to generate a taxonomy, pretty slow, more on performance below.

This generates a set of taxonomy files …

Output of TaxoAdapt

Output of TaxoAdapt

You’ll notice they correspond with the starting dimensions we specified.

Within these files, you can find a breakdown of each taxonomy sub-category and associated texts. For example, for dimension ‘Recommendation Type’ we have …

{
    "label": "policy_recommendations",
    "description": "Recommendations aimed at shaping or revising policies to improve the effectiveness and efficiency of humanitarian responses.",
    "level": 1,
    "source": "initial",
    "example_papers": [
        [
            2,
            "Strategic evaluation on WFP's support to refugees, internally displaced persons, and migrants - https://www.wfp.org/publications/strategic-evaluation-wfps-support-refugees-internally-displaced-persons-and-migrants (from special page)"
        ],
        [
            5,
            "Evaluation of Philippines WFP Country Strategic Plan 2018-2023 - https://www.wfp.org/publications/evaluation-philippines-wfp-country-strategic-plan-2018-2023 (from special page)"
        ],
        [
            6,
            "UNICEF Turkmenistan Country Programme 2021-2025 Evaluation"
        ],
        [
            8,
            "EVALUATION OF IOM'S MIGRATION DATA STRATEGY - Report 4"
        ],
        [
            12,
            "Evaluation of Philippines WFP Country Strategic Plan 2018-2023 - https://www.wfp.org/publications/evaluation-philippines-wfp-country-strategic-plan-2018-2023 (from special page)"
        ],
        [
            17,
            "Evaluation of Resident Coordinator systems in complex settings"
        ],
        [
            19,
            "UN Women-WHO Global Joint Programme on Violence against Women Data: End Term Evaluation"
        ],
        [
            21,
            "Independent evaluation of the relevance and effectiveness of the GCF's investments in the African States - 230406-afr-final-report-3rd-ed-top-web-isbn_0"
        ],
        [
            34,
            "Evaluation of WFP's emergency response to the prolonged crisis in the Sahel and other countries of Central Africa (2018-2023) - https://www.wfp.org/publications/evaluation-wfps-emergency-response-prolonged-crisis-sahel-and-other-countries-central (from special page)"
        ],
        [
            35,
            "Evaluation of UNICEF work in Early Childhood Development (2018-2023) - Serbia Country Report"
        ]
    ],
    "paper_ids": [
        2,
        5,
        6,
        8,
        12,
        17,
        19,
        21,
        34,
        35,
        38,
        41,
        44,
        45,
        46,
        50,
        53,
        54,
        55,
        64,
        66,
        67,
        68,
        73,
        74,
        85,
        87,
        88,
        89,
        92,
        94
    ]
},
... etc

The forked repo also produces an output Excel file with the taxonomy and example, a copy of which can be found here.

Since we wanted a quick pass, the generated taxonomies in our test probably aren’t that earth-shattering to anybody working in the humanitarian community. Also, they aren’t terribly deep, the main reason being TaxoAdapt is quite slow because it calls LLMs in serial (the original code comments suggest parallelizing this, but this wasn’t explored further).

That said, it is interesting to see that we were able to guide a taxonomy generation where categories were supported by the documents we provided.

Future Work

For this article, we did a very quick pass with some minor adjustments to the paper GitHub repository, which I hope will make it easier for others to explore further. That said, there are some areas that weren’t tackled just yet.

As mentioned above, in its current form, the code executes quite slowly, especially using an open model provider inference API. Due to serial calling of the LLM at multiple stages in the process, this limits the number of documents and taxonomy depth quite a bit. As the authors note in the code comments, this should be relatively easy to fix through techniques like multithreading.

Another factor to consider is that because TaxoAdapt uses LLMs, the taxonomy generated may be slightly different each time. Not a problem if doing a one-time analysis, but if you wanted to use TaxoAdapt to identify taxonomy drift as content changes, to include new categories, some work would be needed to align with the existing taxonomy. Not insurmountable, but not something currently supported in the current code base.

Finally, the provided code is understandably geared towards the research paper. TaxoAdapt seems like a nice candidate for a new Python package if one of you wants to have a go.

As always, I hope you found this interesting, and I’d love to hear from you!

Please follow me on Medium or LinkedIn for more noodling about with AI. 🙂

References

TaxoAdapt: Aligning LLM-Based Multidimensional Taxonomy Construction to Evolving Research Corpora, Kargupta et al 2025.

The code for this article was forked from the paper repo. The fork contains some minor enhancements and can be found here.


Automatically Identifying Taxonomies from Documents: Using the TaxoAdapt Algorithm with… was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.