High dimensional biological data retrieval optimization with NoSQL technology.

{"sci_version": "1.2", "schema": "https://corpus.bntrg.co.uk/sci/1.0/sci.schema.json", "id": "pmid:25435347", "kind": "science-record", "revision": 3, "content_hash": "sha256:07492fd34b6a6637f2940cdd78035f747a25119fa1f4e21d8390061b4817aa4b", "created_at": "2026-11-01T15:40:07Z", "updated_at": "2026-11-01T18:47:15Z", "generator": {"name": "corpus-assay-engine", "run_id": "reading-sci-2026-11-01"}, "paper": {"state": "complete", "title": "High dimensional biological data retrieval optimization with NoSQL technology.", "doi": "10.1186/1471-2164-15-S8-S3", "pmid": "25435347", "authors": ["Shicai Wang", "Ioannis Pandis", "Chao Wu", "Sijin He", "David Johnson", "Ibrahim Emam", "Florian Guitton", "Yike Guo"], "journal": "BMC genomics", "year": 2014, "abstract": "BACKGROUND: High-throughput transcriptomic data generated by microarray experiments is the most abundant and frequently stored kind of data currently used in translational medicine studies. Although microarray data is supported in data warehouses such as tranSMART, when querying relational databases for hundreds of different patient gene expression records queries are slow due to poor performance. Non-relational data models, such as the key-value model implemented in NoSQL databases, hold promise to be more performant solutions. Our motivation is to improve the performance of the tranSMART data warehouse with a view to supporting Next Generation Sequencing data.\nRESULTS: In this paper we introduce a new data model better suited for high-dimensional data storage and querying, optimized for database scalability and performance. We have designed a key-value pair data model to support faster queries over large-scale microarray data and implemented the model using HBase, an implementation of Google's BigTable storage system. An experimental performance comparison was carried out against the traditional relational data model implemented in both MySQL Cluster and MongoDB, using a large publicly available transcriptomic data set taken from NCBI GEO concerning Multiple Myeloma. Our new key-value data model implemented on HBase exhibits an average 5.24-fold increase in high-dimensional biological data query performance compared to the relational model implemented on MySQL Cluster, and an average 6.47-fold increase on query performance on MongoDB.\nCONCLUSIONS: The performance evaluation found that the new key-value data model, in particular its implementation in HBase, outperforms the relational model currently implemented in tranSMART. We propose that NoSQL technology holds great promise for large-scale data management, in particular for high-dimensional biological data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new data model as a basis for migrating tranSMART's implementation to a more scalable solution for Big Data.", "claims": [{"id": "claim-data-access", "statement": "In our experiment we took a database dump of the DEAPP schema that tranSMART uses to store patient microarray data and extracted a public Multiple Myeloma (MULTMYEL) [20,21] dataset [GEO:GSE24080] [22]. The reason we chose MULTMYEL as our test dataset was that it is one of the largest datasets loaded into our tranSMART database instance, consisting of 559 samples and 54,675 probesets for each sample, totalling approximately 30.5 million records.", "kind": "methodological", "target": null, "evidence": [{"asset": "source:pmid:25435347", "locator": "sections[9] \"Querying transcriptomic data in tranSMART\" paragraph 2", "quote": "In our experiment we took a database dump of the DEAPP schema that tranSMART uses to store patient microarray data and extracted a public Multiple Myeloma (MULTMYEL) [20,21] dataset [GEO:GSE24080] [22]. The reason we chose MULTMYEL as our test dataset was that it is one of the largest datasets loaded into our tranSMART database instance, consisting of 559 samples and 54,675 probesets for each sample, totalling approximately 30.5 million records."}], "state": "complete"}, {"id": "claim-target-result", "statement": "The HBase key-value model outperformed the relational models, with average query-performance increases of 5.24-fold over MySQL Cluster and 6.47-fold over MongoDB.", "kind": "qualitative", "target": null, "evidence": [{"asset": "source:pmid:25435347", "locator": "sections[0] \"Abstract\" paragraph 2", "quote": "An experimental performance comparison was carried out against the traditional relational data model implemented in both MySQL Cluster and MongoDB, using a large publicly available transcriptomic data set taken from NCBI GEO concerning Multiple Myeloma."}, {"asset": "source:pmid:25435347", "locator": "sections[0] \"Abstract\" paragraph 2", "quote": "Our new key-value data model implemented on HBase exhibits an average 5.24-fold increase in high-dimensional biological data query performance compared to the relational model implemented on MySQL Cluster, and an average 6.47-fold increase on query performance on MongoDB."}], "state": "complete"}], "gaps": []}, "experiment": {"state": "partial", "objective": "The HBase key-value model outperformed the relational models, with average query-performance increases of 5.24-fold over MySQL Cluster and 6.47-fold over MongoDB.", "steps": [{"id": "read_published_estimate", "order": 1, "name": "Read the published estimate from the source", "kind": "other", "state": "complete", "blocked_on": null, "description": "The claim, method and evidence span published with the source article.", "duration_hours": null, "parameters": {}, "inputs": ["source:pmid:25435347"], "outputs": ["the published estimate and its evidence"], "instruments": [], "reagents": []}, {"id": "obtain_source_data", "order": 2, "name": "Obtain the source data from the named repository", "kind": "data-preparation", "state": "blocked", "blocked_on": "data:accession_absent", "description": "The GEO series accession resolves and its sample accessions are listed. This lane did not download the dataset or read its licence.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the source data with its repository provenance"], "instruments": [], "reagents": []}, {"id": "run_analysis", "order": 3, "name": "Run the published analysis", "kind": "analysis", "state": "complete", "blocked_on": null, "description": "Apply the paper's statistical method to the resolved series and compare the result with the published target.", "duration_hours": null, "parameters": {}, "inputs": [], "outputs": ["the replicated result and its comparison"], "instruments": [], "reagents": []}], "materials": [], "instruments": [], "reagents": [], "conditions": {}, "gaps": ["step:obtain_source_data blocked on data:accession_absent"]}, "data": {"state": "partial", "assets": [{"id": "source:pmid:25435347", "role": "manuscript", "format": "xml", "media_type": "application/xml", "uri": "https://doi.org/10.1186/1471-2164-15-S8-S3", "sha256": "sha256:08fd168e89747cf96ff0934cccc9eef1b5db8c89254a722e30875fd826311d38", "bytes": 30390, "access": "open", "state": "resolved", "reader": null, "reader_status": "reader_required", "identifier": "10.1186/1471-2164-15-S8-S3", "version": null, "note": "Full article fetched from PMC; the checksum covers the rendered section body, not the original XML bytes.", "structure": {"pmcid": "PMC4248814", "source": "europe_pmc_fulltext", "coverage": "whole_body_without_references", "text_hash_scope": "rendered_section_body"}}, {"id": "asset:table:pmid:25435347:1", "role": "table", "format": "jats-table", "media_type": "application/vnd.jats.table+json", "uri": null, "sha256": "sha256:9db628d9be29595aac396ec77261917d7887888bc798d4f74f50aaa2b124d144", "bytes": 358, "access": "open", "state": "verified", "reader": "read_table", "reader_status": "available", "identifier": "section[1]/table-wrap[1]#T1", "version": null, "note": "Relational microarray data schema.", "structure": {"pmcid": "PMC4248814", "section_index": 1, "section_title": "Background", "row_count": 2, "column_count": 7}, "table": {"label": "Table 1", "caption": "Relational microarray data schema.", "rows": [["GENE_SYMBOL", "PROBESET_ID", "PATIENT_ID", "TRIAL_NAME", "RAW", "LOG", "ZSCORE"], ["VARCHAR2(100)", "VARCHAR2(100)", "NUMBER(18)", "VARCHAR2(50)", "NUMBER(18)", "NUMBER(18)", "NUMBER(18)"]], "row_count": 2, "column_count": 7, "numeric_cells": 7, "header_rows": 1, "locator": "section[1]/table-wrap[1]#T1"}}, {"id": "asset:table:pmid:25435347:2", "role": "table", "format": "jats-table", "media_type": "application/vnd.jats.table+json", "uri": null, "sha256": "sha256:48442b559b69dbe678afd6deca1bb2b232b0d20e5fc4e5acfc83f2d6616326a5", "bytes": 366, "access": "open", "state": "verified", "reader": "read_table", "reader_status": "available", "identifier": "section[2]/table-wrap[1]#T2", "version": null, "note": "Example of a relational model representation of a patient record.", "structure": {"pmcid": "PMC4248814", "section_index": 2, "section_title": "Related work", "row_count": 2, "column_count": 7}, "table": {"label": "Table 2", "caption": "Example of a relational model representation of a patient record.", "rows": [["GENE_SYMBOL", "PROBESET_ID", "PATIENT_ID", "TRIAL_NAME", "RAW", "LOG", "ZSCORE"], ["LDOC1", "204454_at", "79622", "MULTMYEL", "71.900002", "6.16791991", "8.3069731"]], "row_count": 2, "column_count": 7, "numeric_cells": 5, "header_rows": 1, "locator": "section[2]/table-wrap[1]#T2"}}, {"id": "asset:table:pmid:25435347:3", "role": "table", "format": "jats-table", "media_type": "application/vnd.jats.table+json", "uri": null, "sha256": "sha256:1e2d04aa2605dec38e22e146e1dff60be5ebce65d57b5b475800fc99f20b3a58", "bytes": 488, "access": "open", "state": "verified", "reader": "read_table", "reader_status": "available", "identifier": "section[8]/table-wrap[1]#T3", "version": null, "note": "Example in DEAPP", "structure": {"pmcid": "PMC4248814", "section_index": 8, "section_title": "Key-value data model example", "row_count": 6, "column_count": 7}, "table": {"label": "Table 3", "caption": "Example in DEAPP", "rows": [["GENE_SYMBOL", "PROBESET_ID", "PATIENT_ID", "TRIAL_NAME", "RAW", "LOG", "ZSCORE"], ["LDOC1", "204454_at", "79622", "MULTMYEL", "71.900002", "6.16791991", "8.3069731"], [""], ["EI24", "216396_s_at", "79622", "MULTMYEL", "917.20001", "9.84109256", "9.57629541"], [""], ["GRID1", "1555267_at", "79737", "MULTMYEL", "608.29999", "9.24863917", "8.88386512"]], "row_count": 6, "column_count": 7, "numeric_cells": 16, "header_rows": 1, "locator": "section[8]/table-wrap[1]#T3"}}, {"id": "asset:table:pmid:25435347:4", "role": "table", "format": "jats-table", "media_type": "application/vnd.jats.table+json", "uri": null, "sha256": "sha256:976e6b60fccd6522f2e576b64ae02537120e7470b40d30b17f1f719e091930ae", "bytes": 1176, "access": "open", "state": "verified", "reader": "read_table", "reader_status": "available", "identifier": "section[8]/table-wrap[2]#T4", "version": null, "note": "Example data model in BigTable transformed from table 3.", "structure": {"pmcid": "PMC4248814", "section_index": 8, "section_title": "Key-value data model example", "row_count": 40, "column_count": 4}, "table": {"label": "Table 4", "caption": "Example data model in BigTable transformed from table 3.", "rows": [["Key", "Value"], ["Row Key", "Column Key", ""], ["", "", ""], ["", "Family", "Qualifier", ""], [""], ["MULTMYEL + 79622", "LOG", "EI24 + 216396_s_at", "9.84109256"], [""], ["MULTMYEL + 79622", "LOG", "LDOC1 + 204454_at", "6.16791991"], [""], ["MULTMYEL + 79737", "LOG", "GRID1 + 1555267_at", "9.24863917"], [""], ["4a. LOG Family StoreFile"], [""], ["Key", "Value"], [""], ["Row Key", "Column Key", ""], ["", "", ""], ["", "Family", "Qualifier", ""], [""], ["MULTMYEL + 79622", "RAW", "EI24 + 216396_s_at", "917.20001"], [""], ["MULTMYEL + 79622", "RAW", "LDOC1 + 204454_at", "71.900002"], [""], ["MULTMYEL + 79737", "RAW", "GRID1 + 1555267_at", "608.29999"], [""], ["4b. RAW Family StoreFile"], [""], ["Key", "Value"], [""], ["Row Key", "Column Key", ""], ["", "", ""], ["", "Family", "Qualifier", ""], [""], ["MULTMYEL + 79622", "ZSCORE", "EI24 + 216396_s_at", "9.57629541"], [""], ["MULTMYEL + 79622", "ZSCORE", "LDOC1 + 204454_at", "8.306973"], [""], ["MULTMYEL + 79737", "ZSCORE", "GRID1 + 1555267_at", "8.88386512"], [""], ["4c. ZSCORE Family StoreFile"]], "row_count": 40, "column_count": 4, "numeric_cells": 30, "header_rows": 1, "locator": "section[8]/table-wrap[2]#T4"}}], "gaps": []}, "analysis": {"state": "absent", "methods": [], "targets": [], "gaps": ["method", "operations"]}, "capability": {"state": "complete", "requirements": [{"capability_id": "capability:computational_analysis", "label": "Computational analysis platform", "assay_families": ["meta_analysis", "machine_learning", "survival_analysis", "statistical_regression", "pca", "molecular_docking", "network_analysis", "differential_expression", "gwas"], "availability_state": "owned", "tier_unlocked": "R1", "candidate": "Automated Assays Laravel application and simulated adapters"}], "gaps": []}, "readiness": {"state": "blocked", "tier": "R4", "blocking_axis": "data", "blocked_on": "data:accession_absent", "axes": {"method": {"tier": "R4", "expressible": false, "required_operations": [], "missing_operations": [], "basis": "The reader quoted the method the paper names (not named), but the corpus's method extractor could not map it onto a DSL operation, so the method is published but not expressible."}, "capability": {"tier": "R1", "capability_id": "capability:computational_analysis", "availability_state": "owned", "basis": "The computational analysis is owned. The physical bench the paper used is not, so a physical validation cannot be run here."}, "data": {"tier": "R4", "reachable": false, "accessions": ["[GEO:GSE24080]"], "resolved_accessions": [], "statements": ["In our experiment we took a database dump of the DEAPP schema that tranSMART uses to store patient microarray data and extracted a public Multiple Myeloma (MULTMYEL) [20,21] dataset [GEO:GSE24080] [22]. The reason we chose MULTMYEL as our test dataset was that it is one of the largest datasets loaded into our tranSMART database instance, consisting of 559 samples and 54,675 probesets for each sample, totalling approximately 30.5 million records."], "basis": "The paper names [GEO:GSE24080], but the accession does not resolve in the repository (no reason recorded), so the data cannot be reached from here. No dataset was downloaded and no licence was read."}}, "can_run": {"dsl_operations": [], "assets": ["source:pmid:25435347", "asset:table:pmid:25435347:1", "asset:table:pmid:25435347:2", "asset:table:pmid:25435347:3", "asset:table:pmid:25435347:4"], "steps": ["run_analysis"]}, "must_build": {"dsl_operations": [], "converters": [], "readers": [], "capabilities": [], "data_gaps": ["data:geo:None"]}, "gaps": ["data:accession_absent"]}, "review": {"state": "partial", "machine_critique": {"kind": "extraction_review", "reader": {"model": "MiniMax-M3.1-Flash", "provider": "minimax", "provider_family": "minimax", "read_at": "2026-10-31T18:23:10Z", "independence": "same_provider"}, "evidence_check": {"method": null, "availability": null}, "calls": 8, "usage": {"prompt_tokens": 56208, "completion_tokens": 2575, "cached_tokens": 14600}}, "human_critique": null, "stages": [{"id": "extraction", "state": "complete", "reader": "MiniMax-M3.1-Flash", "provider": "minimax"}], "gaps": ["review.independent_reader", "review.machine_critique", "review.human_critique"]}, "execution": {"state": "absent", "status": "not_run", "mode": null, "runs": [], "gaps": ["execution.not_run"]}, "provenance": {"entities": [{"id": "pmid:25435347", "type": "paper", "label": "pmid:25435347"}, {"id": "source:pmid:25435347", "type": "manuscript", "label": "PMID 25435347 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "activities": [{"id": "activity-reading-sci-2026-11-01", "type": "classification", "used": [{"id": "pmid:25435347", "type": "paper", "label": "pmid:25435347"}, {"id": "source:pmid:25435347", "type": "manuscript", "label": "PMID 25435347 full text"}, {"id": "data:geo:None", "type": "dataset", "label": "GEO None"}], "agent": "agent:corpus-assay-engine"}, {"id": "activity-read-25435347", "type": "extraction", "used": ["source:pmid:25435347"], "agent": "agent:minimax-reader"}, {"id": "activity-resolve-None", "type": "accession_resolution", "used": ["data:geo:None"], "agent": "agent:corpus-assay-engine"}], "agents": [{"id": "agent:corpus-assay-engine", "type": "software", "label": "Corpus assay engine"}, {"id": "agent:minimax-reader", "type": "software", "label": "MiniMax literature reader"}], "source_run_id": "reading-sci-2026-11-01", "dsl_catalogue_fingerprint": "343fb30756ba6ec3", "capability_table_fingerprint": "2cb3e42f047d2ecf", "snapshot_id": "ba34b49490d9bfaf"}, "lineage": {"parent_id": "pmid:25435347", "parent_hash": "sha256:3f6b213de671defa51ab73b28a65e5931994b4ebd4a35cd041978c629272e835", "fork_of": null, "generation": 2}, "gaps": ["execution.not_run", "method", "operations", "review.human_critique", "review.independent_reader", "review.machine_critique", "step:obtain_source_data blocked on data:accession_absent"], "assays": [{"role": "data_generation", "definition_id": "microarray", "definition_version": "99e1195c7d6f", "definition_digest": null, "required_operations": [], "support_state": "partial", "source_hashes": {"assay_lexicon": "99e1195c7d6f"}, "confidence": 0.99, "evidence": ["-throughput transcriptomic data generated by microarray experiments is the most abundant and frequen", "in translational medicine studies. Although microarray data is supported in data warehouses such as", "l to support faster queries over large-scale microarray data and implemented the model using HBase,"], "order_basis": "paper_text_order:95", "id": "assay:1"}]}