A framework for semantic egocentric photo streams segmentation is presented. Contextual and semantic information is extracted by employing a CNN approach. A vocabulary of concepts is defined by relying on linguistic information. Images sharing contextual and semantic attributes are grouped together. Our framework is suited for event recognition, semantic indexing and summarization.