Overview

UniteLabs provides a layered data infrastructure so workflows can ingest raw instrument files, persist structured records, and expose both to downstream analysis — without managing storage backends manually. Three components work together: Object Storage, the Data Warehouse, and File System Connectors.

How the data layer fits together

Data flows in one direction through the stack. Instruments write result files to local directories. File System Connectors detect new files via change subscriptions and forward them upstream. Object Storage (S3-compatible MinIO) holds every raw file as an immutable archive. ETL workflows — orchestrated by Prefect — pull files from Object Storage, parse them, and write structured records into the Data Warehouse (PostgreSQL).

Instrument output
      │
      ▼
File System Connector  ──────────────────►  Object Storage (S3/MinIO)
(subscribe_changes)                               │
                                                  ▼
                                         ETL Workflow (Prefect)
                                                  │
                                                  ▼
                                       Data Warehouse (PostgreSQL)

Each layer is independently accessible — analysts can query the Warehouse directly with DBeaver, and raw files in Object Storage are always available for reprocessing.

Components

Credentials and secrets

All data layer credentials are stored in the UniteLabs secrets manager — never hardcoded. Two key secrets the data layer depends on:

• datalake-admin — MinIO access key, secret key, endpoint URL, and bucket name
• warehouse-admin — PostgreSQL connection string for the managed warehouse

See Secrets for setup steps.

A typical data pipeline

Here is what happens end-to-end when a LabChip GXII Touch runs a protein analysis:

1. The instrument writes three CSV files (PeakTable, SizeTable, WellTable) to a local directory.
2. A File System Connector monitors that directory and emits a change event for each new file.
3. A Prefect ETL flow receives the event, uploads the raw file to Object Storage under data/instruments/labchip/, and records the file path in the Warehouse.
4. Once all three files for a plate are present, the flow parses them, maps wells to Benchling sample IDs, and writes structured results to the labchip_results table.
5. Downstream workflows and BI tools query the Warehouse directly.