Video supervoxel segmentation has recently been established and applied to large-scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of video content: they capture object shape and motion well. Yet, it is not known if the supervoxel segmentation retains the semantics of the underlying video content. Here, we study the preliminary human perception of video supervoxel segmentation and its utilities in guiding the design of machine vision algorithms. We have conducted a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation. Our study has human observers watching supervoxel segmentation videos and trying to discriminate both actor and action. We have gathered and analyzed a large set of 640 human perceptions over 96 videos at 3 different supervoxel scales. The ultimate findings have suggested that a significant amount of semantics have been well retained in the video supervoxel hierarchies. The results have further inspired our design of a new supervoxel feature.