Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation.

{"sci_version": "1.2", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:38235453", "kind": "science-record", "revision": 3, "content_hash": "sha256:2659f5f034aa8446144977dc6ee3e0967b13dee60d10bb4290f9408e0c5036d8", "created_at": "2026-11-01T19:20:08Z", "updated_at": "2026-11-01T22:11:03Z", "generator": {"name": "corpus-assay-engine", "run_id": "reading-sci-2026-11-01"}, "paper": {"state": "complete", "title": "Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation.", "doi": "10.3389/fnrgo.2022.838342", "pmid": "38235453", "authors": ["Raphaëlle N Roy", "Marcel F Hinss", "Ludovic Darmet", "Simon Ladouce", "Emilie S Jahanpour", "Bertille Somon", "Xiaoqi Xu", "Nicolas Drougard", "Frédéric Dehais", "Fabien Lotte"], "journal": "Frontiers in neuroergonomics", "year": 2022, "abstract": "As is the case in several research domains, data sharing is still scarce in the field of Brain-Computer Interfaces (BCI), and particularly in that of passive BCIs-i.e., systems that enable implicit interaction or task adaptation based on a user's mental state(s) estimated from brain measures. Moreover, research in this field is currently hindered by a major challenge, which is tackling brain signal variability such as cross-session variability. Hence, with a view to develop good research practices in this field and to enable the whole community to join forces in working on cross-session estimation, we created the first passive brain-computer interface competition on cross-session workload estimation. This competition was part of the 3rd International Neuroergonomics conference. The data were electroencephalographic recordings acquired from 15 volunteers (6 females; average 25 y.o.) who performed 3 sessions-separated by 7 days-of the Multi-Attribute Task Battery-II (MATB-II) with 3 levels of difficulty per session (pseudo-randomized order). The data -training and testing sets-were made publicly available on Zenodo along with Matlab and Python toy code (https://doi.org/10.5281/zenodo.5055046). To this day, the database was downloaded more than 900 times (unique downloads of all version on the 10th of December 2021: 911). Eleven teams from 3 continents (31 participants) submitted their work. The best achieving processing pipelines included a Riemannian geometry-based method. Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy. These results clearly underline the real challenge that is cross-session estimation. Moreover, they confirmed once more the robustness and effectiveness of Riemannian methods for BCI. On the contrary, chance level results were obtained by one third of the methods-4 teams- based on Deep Learning. These methods have not demonstrated superior results in this contest compared to traditional methods, which may be due to severe overfitting. Yet this competition is the first step toward a joint effort to tackle BCI variability and to promote good research practices including reproducibility.", "claims": [{"id": "claim-method", "statement": "In terms of classification accuracy, the obtained performances ranged from 31.32 to 54.26%.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:38235453", "locator": "sections[11] \"3.2. Performances Obtained and Methods Used\" paragraph 4", "quote": "In terms of classification accuracy, the obtained performances ranged from 31.32 to 54.26%."}], "state": "complete"}, {"id": "claim-data-access", "statement": "The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://doi.org/10.5281/zenodo.5055046.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:38235453", "locator": "sections[13] \"Data Availability Statement\" paragraph 2", "quote": "The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://doi.org/10.5281/zenodo.5055046."}], "state": "complete"}, {"id": "claim-target-result", "statement": "Although better than the adjusted chance level, the best cross-session workload classification accuracy remained under 60%.", "kind": "qualitative", "target": null, "evidence": [{"asset": "source:pmid:38235453", "locator": "sections[0] \"Abstract\" paragraph 1", "quote": "Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy."}], "state": "complete"}], "gaps": []}, "experiment": {"state": "partial", "objective": "Although better than the adjusted chance level, the best cross-session workload classification accuracy remained under 60%.", "steps": [{"id": "read_published_estimate", "order": 1, "name": "Read the published estimate from the source", "kind": "other", "state": "complete", "blocked_on": null, "description": "The claim, method and evidence span published with the source article.", "duration_hours": null, "parameters": {}, "inputs": ["source:pmid:38235453"], "outputs": ["the published estimate and its evidence"], "instruments": [], "reagents": []}, {"id": "obtain_source_data", "order": 2, "name": "Obtain the source data from the named repository", "kind": "data-preparation", "state": "blocked", "blocked_on": "data:accession_absent", "description": "The GEO series accession resolves and its sample accessions are listed. This lane did not download the dataset or read its licence.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the source data with its repository provenance"], "instruments": [], "reagents": []}, {"id": "run_analysis", "order": 3, "name": "Run the published analysis", "kind": "analysis", "state": "blocked", "blocked_on": "method:machine_learning_random_forest", "description": "Apply the paper's statistical method to the resolved series and compare the result with the published target.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the replicated result and its comparison"], "instruments": [], "reagents": []}], "materials": [], "instruments": [], "reagents": [], "conditions": {}, "gaps": ["step:run_analysis blocked on method:machine_learning_random_forest", "step:obtain_source_data blocked on data:accession_absent"]}, "data": {"state": "partial", "assets": [{"id": "source:pmid:38235453", "role": "manuscript", "format": "xml", "media_type": "application/xml", "uri": "https://doi.org/10.3389/fnrgo.2022.838342", "sha256": "sha256:9dceb011e8eb205592becf5cdcd2e1048239c131b3729c3790b096d035cad187", "bytes": 27170, "access": "open", "state": "resolved", "reader": null, "reader_status": "reader_required", "identifier": "10.3389/fnrgo.2022.838342", "version": null, "note": "Full article fetched from PMC; the checksum covers the rendered section body, not the original XML bytes.", "structure": {"pmcid": "PMC10790860", "source": "europe_pmc_fulltext", "coverage": "whole_body_without_references", "text_hash_scope": "rendered_section_body"}}], "gaps": []}, "analysis": {"state": "partial", "methods": [{"id": "method-1", "family": "machine_learning", "specific": "random_forest", "label": "Machine learning", "dsl_operations": [], "missing_operations": ["machine_learning_random_forest"], "expressible": false, "parameters": {}, "data_shapes": ["labelled_feature_matrix"], "evidence": {"score": 0.8, "band": "high", "quote": "Random Forest"}}, {"id": "method-2", "family": "machine_learning", "specific": "support_vector_machine", "label": "Machine learning", "dsl_operations": ["train_classifier", "cross_validate", "roc_auc"], "missing_operations": [], "expressible": true, "parameters": {}, "data_shapes": ["labelled_feature_matrix"], "evidence": {"score": 0.8, "band": "high", "quote": "Support Vector Machine"}}, {"id": "method-3", "family": null, "specific": null, "label": "classification accuracy", "dsl_operations": [], "missing_operations": [], "expressible": null, "parameters": {}, "data_shapes": [], "evidence": {"score": 1.0, "band": "verbatim", "quote": "In terms of classification accuracy, the obtained performances ranged from 31.32 to 54.26%."}}], "targets": [], "gaps": ["method.reader_named_unmapped", "method.machine_learning_random_forest"]}, "capability": {"state": "complete", "requirements": [{"capability_id": "capability:computational_analysis", "label": "Computational analysis platform", "assay_families": ["meta_analysis", "machine_learning", "survival_analysis", "statistical_regression", "pca", "molecular_docking", "network_analysis", "differential_expression", "gwas"], "availability_state": "owned", "tier_unlocked": "R1", "candidate": "Automated Assays Laravel application and simulated adapters"}], "gaps": []}, "readiness": {"state": "blocked", "tier": "R4", "blocking_axis": "data", "blocked_on": "data:accession_absent", "axes": {"method": {"tier": "R1", "expressible": false, "required_operations": [], "missing_operations": ["machine_learning_random_forest"], "basis": "The paper names the method it used and the corpus's method extractor maps it onto the DSL catalogue: random_forest."}, "capability": {"tier": "R1", "capability_id": "capability:computational_analysis", "availability_state": "owned", "basis": "The computational analysis is owned. The physical bench the paper used is not, so a physical validation cannot be run here."}, "data": {"tier": "R4", "reachable": false, "accessions": [], "resolved_accessions": [], "statements": ["The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://doi.org/10.5281/zenodo.5055046."], "basis": "The paper names no accession to its source data, so the data has to come from the authors."}}, "can_run": {"dsl_operations": [], "assets": ["source:pmid:38235453"], "steps": []}, "must_build": {"dsl_operations": [{"operation": "machine_learning_random_forest", "unlocks_papers": null, "priority": null}], "converters": [], "readers": [], "capabilities": [], "data_gaps": ["data:geo:None"]}, "gaps": ["data:accession_absent"]}, "review": {"state": "partial", "machine_critique": {"kind": "extraction_review", "reader": {"model": "MiniMax-M3.1-Flash", "provider": "minimax", "provider_family": "minimax", "read_at": "2026-11-01T19:05:21Z", "independence": "single_reader"}, "evidence_check": {"method": "verbatim", "availability": "verbatim"}, "calls": 5, "usage": {"prompt_tokens": 7827, "completion_tokens": 1046, "cached_tokens": 188}}, "human_critique": null, "stages": [{"id": "extraction", "state": "complete", "reader": "MiniMax-M3.1-Flash", "provider": "minimax"}], "gaps": ["review.independent_reader", "review.machine_critique", "review.human_critique"]}, "execution": {"state": "absent", "status": "not_run", "mode": null, "runs": [], "gaps": ["execution.not_run"]}, "provenance": {"entities": [{"id": "pmid:38235453", "type": "paper", "label": "pmid:38235453"}, {"id": "source:pmid:38235453", "type": "manuscript", "label": "PMID 38235453 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "activities": [{"id": "activity-reading-sci-2026-11-01", "type": "classification", "used": [{"id": "pmid:38235453", "type": "paper", "label": "pmid:38235453"}, {"id": "source:pmid:38235453", "type": "manuscript", "label": "PMID 38235453 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "agent": "agent:corpus-assay-engine"}, {"id": "activity-read-38235453", "type": "extraction", "used": ["source:pmid:38235453"], "agent": "agent:minimax-reader"}, {"id": "activity-resolve-None", "type": "accession_resolution", "used": ["data:geo:None"], "agent": "agent:corpus-assay-engine"}], "agents": [{"id": "agent:corpus-assay-engine", "type": "software", "label": "Corpus assay engine"}, {"id": "agent:minimax-reader", "type": "software", "label": "MiniMax literature reader"}], "source_run_id": "reading-sci-2026-11-01", "dsl_catalogue_fingerprint": "343fb30756ba6ec3", "capability_table_fingerprint": "2cb3e42f047d2ecf", "snapshot_id": "ba34b49490d9bfaf"}, "lineage": {"parent_id": "pmid:38235453", "parent_hash": "sha256:a6b2504c652e41acf969f640fb1abce57e557ca799adef22500a0ca79afffa03", "fork_of": null, "generation": 2}, "gaps": ["execution.not_run", "method.machine_learning_random_forest", "method.reader_named_unmapped", "review.human_critique", "review.independent_reader", "review.machine_critique", "step:obtain_source_data blocked on data:accession_absent", "step:run_analysis blocked on method:machine_learning_random_forest"], "assays": [{"role": "analysis", "definition_id": "machine_learning", "definition_version": "f2132e96ae6c7a4e", "definition_digest": null, "required_operations": [], "support_state": "partial", "source_hashes": {"dsl_catalogue": "f2132e96ae6c7a4e"}, "confidence": 0.8, "evidence": ["Random Forest"], "order_basis": "analysis_only", "id": "assay:1"}]}