Skip to contents

Extract Slim Features and Blobs from IFCB Data

Source: R/ifcb_extract_features.R
ifcb_extract_features.Rd

This function computes the "slim" feature set (version 4) and blob masks from raw Imaging FlowCytobot (IFCB) data by calling the WHOI ifcb-features Python package. For each bin it writes a feature table (<bin>_features_v4.csv, 30 morphological features per region of interest) and an archive of binary blob masks (<bin>_blobs_v4.zip, one 1-bit PNG per ROI). Features and blobs are written to separate, user-specified directories.

Usage

ifcb_extract_features(
  data_folder,
  features_folder,
  blobs_folder,
  bins = NULL,
  parallel = FALSE,
  n_cores = NULL,
  overwrite = FALSE,
  feature_tag = c("features", "fea"),
  multiblob = FALSE,
  backend = NULL,
  verbose = TRUE
)

Arguments

data_folder

The path to a directory containing raw IFCB data (.roi, .adc and .hdr files). The directory is searched recursively by the raw-data reader, so nested data structures are supported.

features_folder

The path to the directory where the <bin>_features_v4.csv files will be written. Created if it does not exist.

blobs_folder

The path to the directory where the <bin>_blobs_v4.zip files will be written. Created if it does not exist.

bins

An optional character vector of bin names (e.g. "D20220522T003051_IFCB134") to restrict processing to a subset of bins. If NULL (default), all bins found in data_folder are processed.

parallel

A logical indicating whether to process bins in parallel. Default is FALSE.

n_cores

An integer specifying the number of parallel workers to use when parallel = TRUE (worker processes on Linux, threads on Windows and macOS; see Details). If NULL (default), parallel::detectCores() - 1 workers are used. Ignored when parallel = FALSE.

overwrite

A logical indicating whether to overwrite existing feature and blob files. If FALSE (default), bins whose outputs already exist are skipped.

feature_tag

A string controlling the token between the bin lid and the version in the feature file name. "features" (default) writes <bin>_features_v4.csv (the upstream ifcb-features convention); "fea" writes <bin>_fea_v4.csv, the name the IFCB Dashboard (ifcbdb / pyifcb's FeaturesDirectory) searches for. Use "fea" when the output is destined for an IFCB Dashboard instance; remember the dataset directory there must be registered with product version 4 to match the _v4 suffix. The blob archive name (<bin>_blobs_v4.zip) is unaffected.

multiblob

A logical indicating whether to additionally write multiblob/<bin>_multiblob_v4.csv files (per-blob features for regions of interest with more than one blob) inside features_folder. Bins without multi-blob ROIs get no sidecar file, as in upstream ifcb-features. Requires ifcb-features v1.2.0 or later; see Details. Default is FALSE.

backend

An optional string forcing the raw-data reader, either "ifcbkit" or "pyifcb". If NULL (default), the IRFCB_IFCB_BACKEND environment variable is used when set, otherwise the preferred available reader (ifcbkit when both are installed). See Details for the cases in which the two readers differ.

verbose

A logical indicating whether to print progress messages, including a progress bar that advances as each bin is processed. Default is TRUE.

Value

Invisibly returns a tibble with one row per bin and the columns bin, status ("processed", "skipped" or "error") and message. The function is primarily called for its side effect of writing feature and blob files to disk.

Details

This function wraps the extract_slim_features workflow from the ifcb-features Python repository, which can be found at https://github.com/WHOIGit/ifcb-features.

Python and the ifcb-features package must be installed to use this function. The required Python packages can be installed in a virtual environment using ifcb_py_install(features = TRUE), which additionally installs ifcb-features and its dependencies (a raw-data reader, phasepack, scikit-image, scikit-learn).

Supported ifcb-features versions: raw data is read through whichever reader the installed ifcb-features release provides - ifcbkit for v1.1.0 and later, pyifcb for v1.0.0 and earlier. Both are supported and may be installed side by side, with ifcbkit preferred when both are present. Use the backend argument (or the IRFCB_IFCB_BACKEND environment variable) to force a particular reader.

The feature code itself is unchanged between these releases, so the choice of reader does not affect how a region of interest is measured. The readers do not agree in every case, however: pyifcb skips a ROI whose recorded width is zero, while ifcbkit skips one whose width or height is zero, and ifcbkit additionally stitches overlapping ROI pairs in older I-style bins, which pyifcb returns separately. For the D-style bins produced by current instruments the two agree on ROI numbering and pixel data, and outputs are interchangeable; for I-style data, pin a reader with backend if you need results comparable to an earlier run.

Python version requirement: ifcb-features requires Python >= 3.10. Installing v1.0.0 or earlier additionally pulls in pyifcb, which needs a binary h5py wheel (available for Python 3.10-3.13). See https://github.com/WHOIGit/ifcb-features for current requirements, and use ifcb_py_install(features = TRUE) to install into a compatible environment.

Multiblob output: the slim feature table describes each ROI's largest blob (plus summed* columns over all blobs). With multiblob = TRUE, the per-blob features of every blob in a multi-blob ROI are additionally written to multiblob/<bin>_multiblob_v4.csv inside features_folder, one row per blob with roi_number, blob_number and 18 morphological columns - the sidecar output ifcb-features introduced in v1.2.0, which is also the minimum version required (older releases never compute per-blob rows, and the function stops with an error if one is installed; update with ifcb_py_install(features = TRUE)). As upstream, a bin in which no ROI has more than one blob gets no sidecar file at all, so the presence of a <bin>_multiblob_v4.csv means that bin genuinely contains multi-blob ROIs. The skip logic accounts for this by reading the numBlobs column of a bin's existing feature CSV to tell whether a sidecar is expected: re-running with multiblob = TRUE over a directory previously extracted without it therefore skips the bins with single-blob ROIs only and re-extracts just those that need a sidecar, without overwrite = TRUE.

Bins are processed sequentially by default. When parallel = TRUE, bins are distributed across n_cores workers, which can substantially reduce runtime for large datasets. Existing outputs are skipped unless overwrite = TRUE, so the function can be re-run to resume an interrupted extraction.

The parallel backend depends on the platform. On Linux, bins run in separate worker processes, giving true multi-core parallelism. On Windows and macOS, where the embedded Python interpreter cannot reliably spawn worker processes, a thread pool is used instead; because of Python's Global Interpreter Lock the speedup there is smaller and depends on how much of the work runs in native (numpy / scikit-image) code. A further consequence of the thread backend is that interrupting a run (ESC / Stop) does not halt a bin already being processed: it finishes and writes its outputs before the run stops.

See also

ifcb_py_install, ifcb_read_features, https://github.com/WHOIGit/ifcb-features

Examples

if (FALSE) { # \dontrun{
# Install the Python environment including ifcb-features
ifcb_py_install(features = TRUE)

# Extract features and blobs from all bins in a data folder
ifcb_extract_features(
  data_folder = "path/to/data",
  features_folder = "path/to/features",
  blobs_folder = "path/to/blobs"
)

# Process a subset of bins in parallel using 4 cores
ifcb_extract_features(
  data_folder = "path/to/data",
  features_folder = "path/to/features",
  blobs_folder = "path/to/blobs",
  bins = c("D20220522T003051_IFCB134", "D20220522T000439_IFCB134"),
  parallel = TRUE,
  n_cores = 4
)

# Write IFCB Dashboard-compatible feature names (<bin>_fea_v4.csv)
ifcb_extract_features(
  data_folder = "path/to/data",
  features_folder = "path/to/features",
  blobs_folder = "path/to/blobs",
  feature_tag = "fea"
)

# Also write per-blob features for multi-blob ROIs
# (path/to/features/multiblob/<bin>_multiblob_v4.csv;
# requires ifcb-features >= 1.2.0)
ifcb_extract_features(
  data_folder = "path/to/data",
  features_folder = "path/to/features",
  blobs_folder = "path/to/blobs",
  multiblob = TRUE
)
} # }