A benchmark database of ten years of prospective next-day earthquake forecasts in California from the Collaboratory for the Study of Earthquake Predictability.

{"sci_version": "1.0", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:40866376", "kind": "science-record", "revision": 2, "content_hash": "sha256:bb4b1f4343a7667d821ab9ee97059b31f7c76216cb2ecef3f623333e0fc2b4ac", "created_at": "2026-11-01T20:30:59Z", "updated_at": "2026-11-01T21:35:42Z", "generator": {"name": "corpus-assay-engine", "run_id": "reading-sci-2026-11-01"}, "paper": {"state": "complete", "title": "A benchmark database of ten years of prospective next-day earthquake forecasts in California from the Collaboratory for the Study of Earthquake Predictability.", "doi": "10.1038/s41597-025-05766-3", "pmid": "40866376", "authors": ["Francesco Serafini", "José A Bayona", "Fabio Silva", "William Savran", "Samuel Stockman", "Philip J Maechling", "Maximilian J Werner"], "journal": "Scientific data", "year": 2025, "abstract": "Short-term seismicity forecasting models are increasingly developed and deployed for Operational Earthquake Forecasting (OEF) by government agencies and research institutions worldwide. To ensure their reliability, these forecasts must be rigorously tested against future observations in a fully prospective manner, allowing researchers to quantify model performance and build confidence in their predictive capabilities. The Collaboratory for the Study of Earthquake Predictability (CSEP) operated twenty-five fully automated M ≥ 3.95 seismicity models developed by nine research groups from Italy, California, New Zealand, the United Kingdom, and Japan. Between August 2007 and August 2018, these models produced over 50,000 daily forecasts for California, each specifying expected earthquake rates on a predefined space-magnitude grid over 24-hour periods. In this article, we describe the forecast database, summarize the underlying models, and demonstrate how to access and evaluate the forecasts using the open-source pyCSEP Python toolkit. The forecast data are publicly available through Zenodo, and the pyCSEP software is openly available on GitHub. This unprecedented dataset of fully prospective earthquake forecasts provides a critical benchmark for developing and testing next-generation OEF models, fostering advancements in earthquake predictability research.", "claims": [{"id": "claim-method", "statement": "Here, we provide functions to calculate the Kagan information score27 for each day, and to visualise the temporal evolution of the absolute and cumulative differences (Fig. 6).", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:40866376", "locator": "sections[14] \"Comparison with other models\" paragraph 2", "quote": "Here, we provide functions to calculate the Kagan information score27 for each day, and to visualise the temporal evolution of the absolute and cumulative differences (Fig. 6)."}], "state": "complete"}, {"id": "claim-data-access", "statement": "All forecast HDF5 files along with the code and tutorial are openly accessible under the open access Creative Commons license. They can be downloaded from Zenodo53 (https://zenodo.org/records/15076187) where we provide a single zip folder with the forecasts, the code, and the tutorial. Alternatively, the files can be downloaded separately from the CSEP website (https://cseptesting.org/grid-based-forecasts/). The h5py47 Python package version 3.12.1 was used to create the HDF5 files from the original forecasts files stored in the testing center. The code and tutorial rely heavily on the pyCSEP4–6 toolkit to handle earthquakes forecasts and catalogues which is openly accessible on GitHub (https://github.com/SCECcode/pycsep). Additional pyCSEP tutorials can be found on the pyCSEP website (https://docs.cseptesting.org/tutorials/), while material from previous pyCSEP workshops is available on GitHub (https://github.com/cseptesting). The CSEP community have also developed a library to create reproducibility packages50 called floatCSEP5,51 also available on GitHub (https://github.com/cseptesting/floatcsep). The package can be used to create docker containers storing the code and data needed to reproduce the results reported in a publication. In this way, the interested reader only needs to execute the container to exactly reproduce all the figures in a manuscript. Examples are the reproducibility packages provided by Savran et al.4 and Graham et al.5 (respectively available at https://zenodo.org/records/6626265 and https://github.com/KennyGraham1/pyCSEP_followup_paper). We encourage the reader to use the database and code we provide in combination with floatCSEP to increase the reproducibility of the results.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:40866376", "locator": "sections[17] \"Code availability\" paragraph 2", "quote": "All forecast HDF5 files along with the code and tutorial are openly accessible under the open access Creative Commons license. They can be downloaded from Zenodo53 (https://zenodo.org/records/15076187) where we provide a single zip folder with the forecasts, the code, and the tutorial. Alternatively, the files can be downloaded separately from the CSEP website (https://cseptesting.org/grid-based-forecasts/). The h5py47 Python package version 3.12.1 was used to create the HDF5 files from the original forecasts files stored in the testing center. The code and tutorial rely heavily on the pyCSEP4–6 toolkit to handle earthquakes forecasts and catalogues which is openly accessible on GitHub (https://github.com/SCECcode/pycsep). Additional pyCSEP tutorials can be found on the pyCSEP website (https://docs.cseptesting.org/tutorials/), while material from previous pyCSEP workshops is available on GitHub (https://github.com/cseptesting). The CSEP community have also developed a library to create reproducibility packages50 called floatCSEP5,51 also available on GitHub (https://github.com/cseptesting/floatcsep). The package can be used to create docker containers storing the code and data needed to reproduce the results reported in a publication. In this way, the interested reader only needs to execute the container to exactly reproduce all the figures in a manuscript. Examples are the reproducibility packages provided by Savran et al.4 and Graham et al.5 (respectively available at https://zenodo.org/records/6626265 and https://github.com/KennyGraham1/pyCSEP_followup_paper). We encourage the reader to use the database and code we provide in combination with floatCSEP to increase the reproducibility of the results."}], "state": "complete"}], "gaps": []}, "experiment": {"state": "partial", "objective": "", "steps": [{"id": "read_published_estimate", "order": 1, "name": "Read the published estimate from the source", "kind": "other", "state": "complete", "blocked_on": null, "description": "The claim, method and evidence span published with the source article.", "duration_hours": null, "parameters": {}, "inputs": ["source:pmid:40866376"], "outputs": ["the published estimate and its evidence"], "instruments": [], "reagents": []}, {"id": "obtain_source_data", "order": 2, "name": "Obtain the source data from the named repository", "kind": "data-preparation", "state": "blocked", "blocked_on": "data:accession_absent", "description": "The GEO series accession resolves and its sample accessions are listed. This lane did not download the dataset or read its licence.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the source data with its repository provenance"], "instruments": [], "reagents": []}, {"id": "run_analysis", "order": 3, "name": "Run the published analysis", "kind": "analysis", "state": "complete", "blocked_on": null, "description": "Apply the paper's statistical method to the resolved series and compare the result with the published target.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the replicated result and its comparison"], "instruments": [], "reagents": []}], "materials": [], "instruments": [], "reagents": [], "conditions": {}, "gaps": ["step:obtain_source_data blocked on data:accession_absent"]}, "data": {"state": "partial", "assets": [{"id": "source:pmid:40866376", "role": "manuscript", "format": "xml", "media_type": "application/xml", "uri": "https://doi.org/10.1038/s41597-025-05766-3", "sha256": "sha256:2154ef8d823d693ad24d87e50f313f45597f06c75be81f136c468e777186b7e4", "bytes": 52137, "access": "open", "state": "resolved", "reader": null, "reader_status": "reader_required", "identifier": "10.1038/s41597-025-05766-3", "version": null, "note": "Full article fetched from PMC; the checksum covers the rendered section body, not the original XML bytes.", "structure": {"pmcid": "PMC12391313", "source": "europe_pmc_fulltext", "coverage": "whole_body", "text_hash_scope": "rendered_section_body"}}], "gaps": []}, "analysis": {"state": "partial", "methods": [{"id": "method-1", "family": null, "specific": null, "label": "Kagan information score", "dsl_operations": [], "missing_operations": [], "expressible": null, "parameters": {}, "data_shapes": [], "evidence": {"score": 1.0, "band": "verbatim", "quote": "Here, we provide functions to calculate the Kagan information score27 for each day, and to visualise the temporal evolution of the absolute and cumulative differences (Fig. 6)."}}], "targets": [], "gaps": ["method.reader_named_unmapped"]}, "capability": {"state": "complete", "requirements": [{"capability_id": "capability:computational_analysis", "label": "Computational analysis platform", "assay_families": ["meta_analysis", "machine_learning", "survival_analysis", "statistical_regression", "pca", "molecular_docking", "network_analysis", "differential_expression", "gwas"], "availability_state": "owned", "tier_unlocked": "R1", "candidate": "Automated Assays Laravel application and simulated adapters"}], "gaps": []}, "readiness": {"state": "blocked", "tier": "R4", "blocking_axis": "data", "blocked_on": "data:accession_absent", "axes": {"method": {"tier": "R4", "expressible": false, "required_operations": [], "missing_operations": [], "basis": "The reader quoted the method the paper names (Kagan information score), but the corpus's method extractor could not map it onto a DSL operation, so the method is published but not expressible."}, "capability": {"tier": "R1", "capability_id": "capability:computational_analysis", "availability_state": "owned", "basis": "The computational analysis is owned. The physical bench the paper used is not, so a physical validation cannot be run here."}, "data": {"tier": "R4", "reachable": false, "accessions": [], "resolved_accessions": [], "statements": ["All forecast HDF5 files along with the code and tutorial are openly accessible under the open access Creative Commons license. They can be downloaded from Zenodo53 (https://zenodo.org/records/15076187) where we provide a single zip folder with the forecasts, the code, and the tutorial. Alternatively, the files can be downloaded separately from the CSEP website (https://cseptesting.org/grid-based-forecasts/). The h5py47 Python package version 3.12.1 was used to create the HDF5 files from the original forecasts files stored in the testing center. The code and tutorial rely heavily on the pyCSEP4–6 toolkit to handle earthquakes forecasts and catalogues which is openly accessible on GitHub (https://github.com/SCECcode/pycsep). Additional pyCSEP tutorials can be found on the pyCSEP website (https://docs.cseptesting.org/tutorials/), while material from previous pyCSEP workshops is available on GitHub (https://github.com/cseptesting). The CSEP community have also developed a library to create reproducibility packages50 called floatCSEP5,51 also available on GitHub (https://github.com/cseptesting/floatcsep). The package can be used to create docker containers storing the code and data needed to reproduce the results reported in a publication. In this way, the interested reader only needs to execute the container to exactly reproduce all the figures in a manuscript. Examples are the reproducibility packages provided by Savran et al.4 and Graham et al.5 (respectively available at https://zenodo.org/records/6626265 and https://github.com/KennyGraham1/pyCSEP_followup_paper). We encourage the reader to use the database and code we provide in combination with floatCSEP to increase the reproducibility of the results."], "basis": "The paper names no accession to its source data, so the data has to come from the authors."}}, "can_run": {"dsl_operations": [], "assets": ["source:pmid:40866376"], "steps": ["run_analysis"]}, "must_build": {"dsl_operations": [], "converters": [], "readers": [], "capabilities": [], "data_gaps": ["data:geo:None"]}, "gaps": ["data:accession_absent"]}, "review": {"state": "partial", "machine_critique": {"kind": "extraction_review", "reader": {"model": "MiniMax-M3.1-Flash", "provider": "minimax", "provider_family": "minimax", "read_at": "2026-11-01T20:00:38Z", "independence": "single_reader"}, "evidence_check": {"method": "verbatim", "availability": "verbatim"}, "calls": 5, "usage": {"prompt_tokens": 13482, "completion_tokens": 2005, "cached_tokens": 256}}, "human_critique": null, "stages": [{"id": "extraction", "state": "complete", "reader": "MiniMax-M3.1-Flash", "provider": "minimax"}], "gaps": ["review.independent_reader", "review.machine_critique", "review.human_critique"]}, "execution": {"state": "absent", "status": "not_run", "mode": null, "runs": [], "gaps": ["execution.not_run"]}, "provenance": {"entities": [{"id": "pmid:40866376", "type": "paper", "label": "pmid:40866376"}, {"id": "source:pmid:40866376", "type": "manuscript", "label": "PMID 40866376 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "activities": [{"id": "activity-reading-sci-2026-11-01", "type": "classification", "used": [{"id": "pmid:40866376", "type": "paper", "label": "pmid:40866376"}, {"id": "source:pmid:40866376", "type": "manuscript", "label": "PMID 40866376 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "agent": "agent:corpus-assay-engine"}, {"id": "activity-read-40866376", "type": "extraction", "used": ["source:pmid:40866376"], "agent": "agent:minimax-reader"}, {"id": "activity-resolve-None", "type": "accession_resolution", "used": ["data:geo:None"], "agent": "agent:corpus-assay-engine"}], "agents": [{"id": "agent:corpus-assay-engine", "type": "software", "label": "Corpus assay engine"}, {"id": "agent:minimax-reader", "type": "software", "label": "MiniMax literature reader"}], "source_run_id": "reading-sci-2026-11-01", "dsl_catalogue_fingerprint": "343fb30756ba6ec3", "capability_table_fingerprint": "2cb3e42f047d2ecf", "snapshot_id": "ba34b49490d9bfaf"}, "lineage": {"parent_id": "pmid:40866376", "parent_hash": "sha256:fadc5ca55a6f38e4280a46ec96973f0fd7a3d42a3d475db8554df02cb6d98fc8", "fork_of": null, "generation": 1}, "gaps": ["execution.not_run", "method.reader_named_unmapped", "review.human_critique", "review.independent_reader", "review.machine_critique", "step:obtain_source_data blocked on data:accession_absent"]}