A framework including data selection, task assignment and annotation combination stages for a confidence based benchmark dataset for retinal image processing is proposed.
A novel task assignment is used to remove data and reader biases.
The annotation of readers is combined based on their accuracy and performance.
The framework is used to build a confidence based benchmark dataset for cyst segmentation.
The generated benchmark can be used to reliably evaluate cyst segmentation methods.