End-to-end data development for the ON-LiMiT data packages

Luke Johnston

2026-11-12

Aims

  • Problems with data in research
  • Seedcase: What do we do?
  • Seedcase and ON-LiMiT (developing the data)
  • Relevant operational and procedural aspects
  • Future plans

Problems with data in research

Have you ever…

spent way too much time getting data ready for analysis?

Have you ever…

had no idea what a particular column name meant?

Have you ever…

had questions about data that only past researchers could answer?

Problems with data in research

  • Data processing and cleaning takes far too long
  • Data is rarely (and often poorly) documented

Seedcase

https://seedcase-project.org

What do we do?

  • Use practices from software development lifecycle and data engineering
  • Develop FAIR-compliant data (Findable, Accessible, Interoperable, Reusable)
  • Reproducibly create usable data from raw data
  • Extract and display metadata on a website
  • Build formal, versioned packages of data
  • Develop software tools to simplify work in a broader sense
  • Build data packages with collaborators (ON-LiMiT and DP-Next)

Seedcase and ON-LiMiT

From raw data to analysis-ready

flowchart LR
  redcap[("REDCap")]
  myfood24[("MyFood24")]
  others[("...<br>other<br>sources")]

  package_source["Data package<br>development:<br>metadata and<br>data"]
  data_package[(Data package)]
  website[Website with metadata]

  subset1[("Data subset:<br> Research analysis 1")]
  subset2[("Data subset:<br> Research analysis 2")]

  redcap & others & myfood24 -->
    package_source -->
    data_package & website

  data_package --> subset1 & subset2

Building the critical metadata

flowchart TB
  raw[("Raw (meta)data<br>source")]
  new_version[Create<br>new version]
  data_package[(Data package)]
  website[Rebuild<br>website]

  subgraph metadata ["Steps for metadata"]
    extract[Extracting] ~~~
      clean[Cleaning]
    edit[Editing] ~~~
      check[Checking]
  end

  raw -->
    metadata -->
    new_version --> data_package & website

  new_version -->|repeat| metadata

  style metadata fill:none

Raw-to-ready data

flowchart TB
  raw[("Raw data<br>source")]
  new_version[Create<br>new version]
  data_package[(Data package)]

  subgraph data ["Steps for data"]
    split[Splitting] ~~~
      join[Joining]
    clean[Cleaning] ~~~
      pivot[Pivoting]
    check[Checking]
  end

  raw -->
    data -->
    new_version -->
    data_package

  new_version -->|repeat| data

  style data fill:none

Result: Reproducible, well-documented, and versioned data

feasibility-data_v1.0.0.zip/
├── datapackage.json
├── README.md
├── LICENSE.md
├── CHANGELOG.md
└── resources/
    ├── sociodemographics.parquet
    ├── ...
    └── blood.parquet

Relevant operational and procedural aspects

  • Need strong coordination from end-to-end to create analysis-ready data
  • Some things make sense for data collection—but not for later analysis (as we experienced in feasibility)
  • Data package website as “first stop”
    https://onlimit-study.github.io/feasibility-data/

Future plans

Data request flow

  subgraph researcher ["Researcher"]
    request_page["Data request<br>webpage"]
    request["Standardised request"]
  end
  subgraph data_manager ["Data manager"]
    create_subset["Run a single command"]
    subset_data[("Requested data<br>subset")]
  end

  request_page -->|Select columns<br>and rows| request
  request -->|Email<br>request| create_subset
  create_subset -->|Generate<br>subset| subset_data

  style researcher fill:none
  style data_manager fill:none

ON-LiMiT data on GenomeDK

Data package location

Restricted access

sdca-onlimit-study/
└── main-data/
    └── releases/
        └── main-data_v1.0.0.zip

Analysis project location

Researcher access

sdca-onlimit-analyses/
└── project1/
    └── main-data-<timestamp>.tar

Workshop on doing analysis in GenomeDK

Being developed in the DP-Next project