Benchmarking methods for measuring biosynthetic gene cluster similarity and determination of gene cluster families.

{"sci_version": "1.0", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:41269115", "kind": "science-record", "revision": 2, "content_hash": "sha256:16eac42931601b0dc3237778b967ebed1068c1cb9a2f1c25c83cd34e9acc85ae", "created_at": "2026-11-01T20:31:18Z", "updated_at": "2026-11-01T21:36:04Z", "generator": {"name": "corpus-assay-engine", "run_id": "reading-sci-2026-11-01"}, "paper": {"state": "complete", "title": "Benchmarking methods for measuring biosynthetic gene cluster similarity and determination of gene cluster families.", "doi": "10.1093/bioinformatics/btaf636", "pmid": "41269115", "authors": ["Abiodun S Oyedele", "Allison S Walker"], "journal": "Bioinformatics (Oxford, England)", "year": 2025, "abstract": "MOTIVATION: Natural products are often produced by a set of biosynthetic enzymes that are encoded by genes clustered together in the producer's genome, referred to as a biosynthetic gene cluster (BGC). The ability to compare and cluster BGCs is essential for several applications, including predicting which bacteria will make a known product and assessing the potential diversity of natural products produced by a set of bacteria. There are multiple methods for comparing and clustering BGCs based on their similarity, but there has been a lack of investigation into how strongly BGC similarity relates to product structural similarity and how these methods perform relative to each other.\nRESULTS: Using publicly available databases, we developed a benchmark dataset to assess how well different BGC similarity metrics correlate with the structural similarity of their products and how well these methods cluster BGCs. We found that all methods showed moderate correlation between BGC and structural similarity, with correlations improving for more similar BGCs and varying significantly by BGC biosynthetic class. Analysis of outliers revealed some outliers were due to mistakes or omissions in public datasets, while others represented deviation between BGC similarity and product structural similarity. All methods generally performed better on clustering metrics, with BiG-SCAPE performing the best after errors in the public datasets had been corrected.\nAVAILABILITY AND IMPLEMENTATION: Scripts and data required to reproduce the results are available at https://github.com/aswalker-lab/BGC-clustering-benchmark and processed similarity, clusters, and scaffolds are also available at https://huggingface.co/datasets/allie-walker/BGC-clustering-benchmark. Code is also available at Zenodo: 10.5281/zenodo.17373546.", "claims": [{"id": "claim-method", "statement": "Correlation between BGCs and Tanimoto similarity using Spearman’s rho calculated with a Python script using scipy. Clustering metrics were calculated using scikit-learn’s metrics.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:41269115", "locator": "sections[9] \"2.7 Calculation of benchmark metrics\" paragraph 2", "quote": "Correlation between BGCs and Tanimoto similarity using Spearman’s rho calculated with a Python script using scipy. Clustering metrics were calculated using scikit-learn’s metrics."}], "state": "complete"}, {"id": "claim-data-access", "statement": "The data required to reproduce the results are available from Github at https://github.com/aswalker-lab/BGC-clustering-benchmark and processed similarity, clusters, and scaffolds are also available from Hugging Face at https://huggingface.co/datasets/allie-walker/BGC-clustering-benchmark", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:41269115", "locator": "sections[22] \"Data availability\" paragraph 2", "quote": "The data required to reproduce the results are available from Github at https://github.com/aswalker-lab/BGC-clustering-benchmark and processed similarity, clusters, and scaffolds are also available from Hugging Face at https://huggingface.co/datasets/allie-walker/BGC-clustering-benchmark"}], "state": "complete"}], "gaps": []}, "experiment": {"state": "partial", "objective": "", "steps": [{"id": "read_published_estimate", "order": 1, "name": "Read the published estimate from the source", "kind": "other", "state": "complete", "blocked_on": null, "description": "The claim, method and evidence span published with the source article.", "duration_hours": null, "parameters": {}, "inputs": ["source:pmid:41269115"], "outputs": ["the published estimate and its evidence"], "instruments": [], "reagents": []}, {"id": "obtain_source_data", "order": 2, "name": "Obtain the source data from the named repository", "kind": "data-preparation", "state": "blocked", "blocked_on": "data:accession_absent", "description": "The GEO series accession resolves and its sample accessions are listed. This lane did not download the dataset or read its licence.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the source data with its repository provenance"], "instruments": [], "reagents": []}, {"id": "run_analysis", "order": 3, "name": "Run the published analysis", "kind": "analysis", "state": "complete", "blocked_on": null, "description": "Apply the paper's statistical method to the resolved series and compare the result with the published target.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the replicated result and its comparison"], "instruments": [], "reagents": []}], "materials": [], "instruments": [], "reagents": [], "conditions": {}, "gaps": ["step:obtain_source_data blocked on data:accession_absent"]}, "data": {"state": "partial", "assets": [{"id": "source:pmid:41269115", "role": "manuscript", "format": "xml", "media_type": "application/xml", "uri": "https://doi.org/10.1093/bioinformatics/btaf636", "sha256": "sha256:ae8061e4c59b71cfe562d45dd3d7480956d50bfe2d985655cdca43a770551d8c", "bytes": 41891, "access": "open", "state": "resolved", "reader": null, "reader_status": "reader_required", "identifier": "10.1093/bioinformatics/btaf636", "version": null, "note": "Full article fetched from PMC; the checksum covers the rendered section body, not the original XML bytes.", "structure": {"pmcid": "PMC12701797", "source": "europe_pmc_fulltext", "coverage": "whole_body_without_references", "text_hash_scope": "rendered_section_body"}}], "gaps": []}, "analysis": {"state": "partial", "methods": [{"id": "method-1", "family": null, "specific": null, "label": "Spearman’s rho", "dsl_operations": [], "missing_operations": [], "expressible": null, "parameters": {}, "data_shapes": [], "evidence": {"score": 1.0, "band": "verbatim", "quote": "Correlation between BGCs and Tanimoto similarity using Spearman’s rho calculated with a Python script using scipy. Clustering metrics were calculated using scikit-learn’s metrics."}}], "targets": [], "gaps": ["method.reader_named_unmapped"]}, "capability": {"state": "complete", "requirements": [{"capability_id": "capability:computational_analysis", "label": "Computational analysis platform", "assay_families": ["meta_analysis", "machine_learning", "survival_analysis", "statistical_regression", "pca", "molecular_docking", "network_analysis", "differential_expression", "gwas"], "availability_state": "owned", "tier_unlocked": "R1", "candidate": "Automated Assays Laravel application and simulated adapters"}], "gaps": []}, "readiness": {"state": "blocked", "tier": "R4", "blocking_axis": "data", "blocked_on": "data:accession_absent", "axes": {"method": {"tier": "R4", "expressible": false, "required_operations": [], "missing_operations": [], "basis": "The reader quoted the method the paper names (Spearman’s rho), but the corpus's method extractor could not map it onto a DSL operation, so the method is published but not expressible."}, "capability": {"tier": "R1", "capability_id": "capability:computational_analysis", "availability_state": "owned", "basis": "The computational analysis is owned. The physical bench the paper used is not, so a physical validation cannot be run here."}, "data": {"tier": "R4", "reachable": false, "accessions": [], "resolved_accessions": [], "statements": ["The data required to reproduce the results are available from Github at https://github.com/aswalker-lab/BGC-clustering-benchmark and processed similarity, clusters, and scaffolds are also available from Hugging Face at https://huggingface.co/datasets/allie-walker/BGC-clustering-benchmark"], "basis": "The paper names no accession to its source data, so the data has to come from the authors."}}, "can_run": {"dsl_operations": [], "assets": ["source:pmid:41269115"], "steps": ["run_analysis"]}, "must_build": {"dsl_operations": [], "converters": [], "readers": [], "capabilities": [], "data_gaps": ["data:geo:None"]}, "gaps": ["data:accession_absent"]}, "review": {"state": "partial", "machine_critique": {"kind": "extraction_review", "reader": {"model": "MiniMax-M3.1-Flash", "provider": "minimax", "provider_family": "minimax", "read_at": "2026-11-01T20:06:13Z", "independence": "single_reader"}, "evidence_check": {"method": "verbatim", "availability": "verbatim"}, "calls": 5, "usage": {"prompt_tokens": 11498, "completion_tokens": 1333, "cached_tokens": 264}}, "human_critique": null, "stages": [{"id": "extraction", "state": "complete", "reader": "MiniMax-M3.1-Flash", "provider": "minimax"}], "gaps": ["review.independent_reader", "review.machine_critique", "review.human_critique"]}, "execution": {"state": "absent", "status": "not_run", "mode": null, "runs": [], "gaps": ["execution.not_run"]}, "provenance": {"entities": [{"id": "pmid:41269115", "type": "paper", "label": "pmid:41269115"}, {"id": "source:pmid:41269115", "type": "manuscript", "label": "PMID 41269115 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "activities": [{"id": "activity-reading-sci-2026-11-01", "type": "classification", "used": [{"id": "pmid:41269115", "type": "paper", "label": "pmid:41269115"}, {"id": "source:pmid:41269115", "type": "manuscript", "label": "PMID 41269115 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "agent": "agent:corpus-assay-engine"}, {"id": "activity-read-41269115", "type": "extraction", "used": ["source:pmid:41269115"], "agent": "agent:minimax-reader"}, {"id": "activity-resolve-None", "type": "accession_resolution", "used": ["data:geo:None"], "agent": "agent:corpus-assay-engine"}], "agents": [{"id": "agent:corpus-assay-engine", "type": "software", "label": "Corpus assay engine"}, {"id": "agent:minimax-reader", "type": "software", "label": "MiniMax literature reader"}], "source_run_id": "reading-sci-2026-11-01", "dsl_catalogue_fingerprint": "343fb30756ba6ec3", "capability_table_fingerprint": "2cb3e42f047d2ecf", "snapshot_id": "ba34b49490d9bfaf"}, "lineage": {"parent_id": "pmid:41269115", "parent_hash": "sha256:ef1ede4cb84ec2363c93b634f5a4e732b77ddf8e65b419da1f07a08f026cf6ac", "fork_of": null, "generation": 1}, "gaps": ["execution.not_run", "method.reader_named_unmapped", "review.human_critique", "review.independent_reader", "review.machine_critique", "step:obtain_source_data blocked on data:accession_absent"]}