Institute of Fundamental Technological Research

In this study, we introduce ‘‘instance loss functions’’, a new family of loss functions designed to enhance the
training of neural networks in the instance-level segmentation and detection of objects in biomedical image
data, particularly those of varied numbers and sizes. Intended to be utilized conjointly with traditional loss
functions, these proposed functions, prioritize object instances over pixel-by-pixel comparisons. The specific
functions, the instance segmentation loss (instance), the instance center loss (center), the false instance rate
loss (false), and the instance proximity loss (proximity), serve distinct purposes. Specifically, instance improves
instance-wise segmentation quality, center enhances segmentation quality of small instances, false minimizes
the rate of false and missed detections across varied instance sizes, and proximity improves detection quality
by pulling predicted instances towards the ground truth instances. Through the task of segmenting white
matter hyperintensities (WMH) in brain MRI, we benchmarked our proposed instance loss functions, both
individually and in combination via an ensemble inference models approach, against traditional pixel-level
loss functions. Data were sourced from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and the
WMH Segmentation Challenge datasets, which exhibit significant variation in WMH instance sizes. Empirical
evaluations demonstrate that combining two instance-level loss functions through ensemble inference models
outperforms models using other loss function on both the ADNI and WMH Segmentation Challenge datasets for
the segmentation and detection of WMH instances. Further, applying these functions to the segmentation of
nuclei in histopathology images demonstrated their effectiveness and generalizability beyond WMH, improving
performance even in contexts with less severe instance imbalance.

Instance-level segmentation loss, Instance-level detection loss, White matter hyperintensities, Brain lesions