Evaluating the feasibility of automating dataset retrieval for biodiversity monitoring.

{"sci_version": "1.2", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:39897501", "kind": "science-record", "revision": 4, "content_hash": "sha256:668b2765665f52c7e81bb8d2a1221f0c5377b30edef96df5a1fe49767e63a064", "created_at": "2026-11-01T20:30:10Z", "updated_at": "2026-11-01T22:56:04Z", "generator": {"name": "corpus-assay-engine", "run_id": "reading-sci-2026-11-01"}, "paper": {"state": "complete", "title": "Evaluating the feasibility of automating dataset retrieval for biodiversity monitoring.", "doi": "10.7717/peerj.18853", "pmid": "39897501", "authors": ["Alexandre Fuster-Calvo", "Sarah Valentin", "William C Tamayo", "Dominique Gravel"], "journal": "PeerJ", "year": 2025, "abstract": "Aim: Effective management strategies for conserving biodiversity and mitigating the impacts of global change rely on access to comprehensive and up-to-date biodiversity data. However, manual search, retrieval, evaluation, and integration of this information into databases present a significant challenge to keeping pace with the rapid influx of large amounts of data, hindering its utility in contemporary decision-making processes. Automating these tasks through advanced algorithms holds immense potential to revolutionize biodiversity monitoring.\nInnovation: In this study, we investigate the potential for automating the retrieval and evaluation of biodiversity data from Dryad and Zenodo repositories. We have designed an evaluation system based on various criteria, including the type of data provided and its spatio-temporal range, and applied it to manually assess the relevance for biodiversity monitoring of datasets retrieved through an application programming interface (API). We evaluated a supervised classification to identify potentially relevant datasets and investigate the feasibility of automatically ranking the relevance. Additionally, we applied the same appraoch on a scientific literature source, using data from Semantic Scholar for reference. Our evaluation centers on the database utilized by a national biodiversity monitoring system in Quebec, Canada.\nMain conclusions: We retrieved 89 (55%) relevant datasets for our database, showing the value of automated dataset search in repositories. Additionally, we find that scientific publication sources offer broader temporal coverage and can serve as conduits guiding researchers toward other valuable data sources. Our automated classification system showed moderate performance in detecting relevant datasets (with an F-score up to 0.68) and signs of overfitting, emphasizing the need for further refinement. A key challenge identified in our manual evaluation is the scarcity and uneven distribution of metadata in the texts, especially pertaining to spatial and temporal extents. Our evaluative framework, based on predefined criteria, can be adopted by automated algorithms for streamlined prioritization, and we make our manually evaluated data publicly available, serving as a benchmark for improving classification techniques.", "claims": [{"id": "claim-method", "statement": "We used five-folds cross-validation and calculated the average precision, recall and F-score as previously defined.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:39897501", "locator": "sections[14] \"Automatic relevance classification\" paragraph 2", "quote": "We used five-folds cross-validation and calculated the average precision, recall and F-score as previously defined."}], "state": "complete"}, {"id": "claim-data-access", "statement": "The following information was supplied regarding data availability:\nThe data and script are available at GitHub and Zenodo:\n- https://github.com/Alex-Fuster/automated_datset_retrieval.git.\n- Fuster-Calvo, A., Valentin, S., C Tamayo, W., & Gravel, D. (2024). Evaluating the feasibility of automating dataset retrieval for biodiversity monitoring. https://doi.org/10.5281/zenodo.13881796.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:39897501", "locator": "sections[28] \"Data Availability\" paragraphs 1-5", "quote": "The following information was supplied regarding data availability:\nThe data and script are available at GitHub and Zenodo:\n- https://github.com/Alex-Fuster/automated_datset_retrieval.git.\n- Fuster-Calvo, A., Valentin, S., C Tamayo, W., & Gravel, D. (2024). Evaluating the feasibility of automating dataset retrieval for biodiversity monitoring. https://doi.org/10.5281/zenodo.13881796."}], "state": "complete"}, {"id": "claim-target-result", "statement": "The automated classification system showed moderate performance in detecting relevant datasets, with an F-score up to 0.68, and showed signs of overfitting.", "kind": "qualitative", "target": null, "evidence": [{"asset": "source:pmid:39897501", "locator": "sections[0] \"Abstract\" paragraph 3", "quote": "Our automated classification system showed moderate performance in detecting relevant datasets (with an F-score up to 0.68) and signs of overfitting, emphasizing the need for further refinement."}], "state": "complete"}], "gaps": []}, "experiment": {"state": "partial", "objective": "The automated classification system showed moderate performance in detecting relevant datasets, with an F-score up to 0.68, and showed signs of overfitting.", "steps": [{"id": "read_published_estimate", "order": 1, "name": "Read the published estimate from the source", "kind": "other", "state": "complete", "blocked_on": null, "description": "The claim, method and evidence span published with the source article.", "duration_hours": null, "parameters": {}, "inputs": ["source:pmid:39897501"], "outputs": ["the published estimate and its evidence"], "instruments": [], "reagents": []}, {"id": "obtain_source_data", "order": 2, "name": "Obtain the source data from the named repository", "kind": "data-preparation", "state": "blocked", "blocked_on": "data:accession_absent", "description": "The GEO series accession resolves and its sample accessions are listed. This lane did not download the dataset or read its licence.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the source data with its repository provenance"], "instruments": [], "reagents": []}, {"id": "run_analysis", "order": 3, "name": "Run the published analysis", "kind": "analysis", "state": "blocked", "blocked_on": "method:machine_learning_random_forest", "description": "Apply the paper's statistical method to the resolved series and compare the result with the published target.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the replicated result and its comparison"], "instruments": [], "reagents": []}], "materials": [], "instruments": [], "reagents": [], "conditions": {}, "gaps": ["step:run_analysis blocked on method:machine_learning_random_forest", "step:obtain_source_data blocked on data:accession_absent"]}, "data": {"state": "partial", "assets": [{"id": "source:pmid:39897501", "role": "manuscript", "format": "xml", "media_type": "application/xml", "uri": "https://doi.org/10.7717/peerj.18853", "sha256": "sha256:7b4f22490371f4fc6eb4f049aafe6b4f3220e070dca4bd04c129597b9ef41bb7", "bytes": 49076, "access": "open", "state": "resolved", "reader": null, "reader_status": "reader_required", "identifier": "10.7717/peerj.18853", "version": null, "note": "Full article fetched from PMC; the checksum covers the rendered section body, not the original XML bytes.", "structure": {"pmcid": "PMC11786708", "source": "europe_pmc_fulltext", "coverage": "whole_body_without_references", "text_hash_scope": "rendered_section_body"}}], "gaps": []}, "analysis": {"state": "partial", "methods": [{"id": "method-1", "family": "machine_learning", "specific": "random_forest", "label": "Machine learning", "dsl_operations": [], "missing_operations": ["machine_learning_random_forest"], "expressible": false, "parameters": {}, "data_shapes": ["labelled_feature_matrix"], "evidence": {"score": 1.0, "band": "verbatim", "quote": "We used five-folds cross-validation and calculated the average precision, recall and F-score as previously defined."}}], "targets": [], "gaps": ["method.machine_learning_random_forest"]}, "capability": {"state": "complete", "requirements": [{"capability_id": "capability:computational_analysis", "label": "Computational analysis platform", "assay_families": ["meta_analysis", "machine_learning", "survival_analysis", "statistical_regression", "pca", "molecular_docking", "network_analysis", "differential_expression", "gwas"], "availability_state": "owned", "tier_unlocked": "R1", "candidate": "Automated Assays Laravel application and simulated adapters"}], "gaps": []}, "readiness": {"state": "blocked", "tier": "R4", "blocking_axis": "data", "blocked_on": "data:accession_absent", "axes": {"method": {"tier": "R1", "expressible": false, "required_operations": [], "missing_operations": ["machine_learning_random_forest"], "basis": "The paper names the method it used and the corpus's method extractor maps it onto the DSL catalogue: random_forest."}, "capability": {"tier": "R1", "capability_id": "capability:computational_analysis", "availability_state": "owned", "basis": "The computational analysis is owned. The physical bench the paper used is not, so a physical validation cannot be run here."}, "data": {"tier": "R4", "reachable": false, "accessions": [], "resolved_accessions": [], "statements": ["The following information was supplied regarding data availability:\nThe data and script are available at GitHub and Zenodo:\n- https://github.com/Alex-Fuster/automated_datset_retrieval.git.\n- Fuster-Calvo, A., Valentin, S., C Tamayo, W., & Gravel, D. (2024). Evaluating the feasibility of automating dataset retrieval for biodiversity monitoring. https://doi.org/10.5281/zenodo.13881796."], "basis": "The paper names no accession to its source data, so the data has to come from the authors."}}, "can_run": {"dsl_operations": [], "assets": ["source:pmid:39897501"], "steps": []}, "must_build": {"dsl_operations": [{"operation": "machine_learning_random_forest", "unlocks_papers": null, "priority": null}], "converters": [], "readers": [], "capabilities": [], "data_gaps": ["data:geo:None"]}, "gaps": ["data:accession_absent"]}, "review": {"state": "partial", "machine_critique": {"kind": "extraction_review", "reader": {"model": "MiniMax-M3.1-Flash", "provider": "minimax", "provider_family": "minimax", "read_at": "2026-11-01T19:45:31Z", "independence": "single_reader"}, "evidence_check": {"method": "verbatim", "availability": "verbatim"}, "calls": 5, "usage": {"prompt_tokens": 11497, "completion_tokens": 1262, "cached_tokens": 264}}, "human_critique": null, "stages": [{"id": "extraction", "state": "complete", "reader": "MiniMax-M3.1-Flash", "provider": "minimax"}], "gaps": ["review.independent_reader", "review.machine_critique", "review.human_critique"]}, "execution": {"state": "absent", "status": "not_run", "mode": null, "runs": [], "gaps": ["execution.not_run"]}, "provenance": {"entities": [{"id": "pmid:39897501", "type": "paper", "label": "pmid:39897501"}, {"id": "source:pmid:39897501", "type": "manuscript", "label": "PMID 39897501 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "activities": [{"id": "activity-reading-sci-2026-11-01", "type": "classification", "used": [{"id": "pmid:39897501", "type": "paper", "label": "pmid:39897501"}, {"id": "source:pmid:39897501", "type": "manuscript", "label": "PMID 39897501 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "agent": "agent:corpus-assay-engine"}, {"id": "activity-read-39897501", "type": "extraction", "used": ["source:pmid:39897501"], "agent": "agent:minimax-reader"}, {"id": "activity-resolve-None", "type": "accession_resolution", "used": ["data:geo:None"], "agent": "agent:corpus-assay-engine"}], "agents": [{"id": "agent:corpus-assay-engine", "type": "software", "label": "Corpus assay engine"}, {"id": "agent:minimax-reader", "type": "software", "label": "MiniMax literature reader"}], "source_run_id": "reading-sci-2026-11-01", "dsl_catalogue_fingerprint": "343fb30756ba6ec3", "capability_table_fingerprint": "2cb3e42f047d2ecf", "snapshot_id": "ba34b49490d9bfaf"}, "lineage": {"parent_id": "pmid:39897501", "parent_hash": "sha256:ea13556e0bd9ff43fa5fc205ed16fb998365b40efaf02ea3dd334a3d20fa39f3", "fork_of": null, "generation": 3}, "gaps": ["execution.not_run", "method.machine_learning_random_forest", "review.human_critique", "review.independent_reader", "review.machine_critique", "step:obtain_source_data blocked on data:accession_absent", "step:run_analysis blocked on method:machine_learning_random_forest"], "assays": [{"role": "analysis", "definition_id": "machine_learning", "definition_version": "f2132e96ae6c7a4e", "definition_digest": null, "required_operations": [], "support_state": "partial", "source_hashes": {"dsl_catalogue": "f2132e96ae6c7a4e"}, "confidence": 1.0, "evidence": ["We used five-folds cross-validation and calculated the average precision, recall and F-score as previously defined."], "order_basis": "analysis_only", "id": "assay:1"}]}