Background: There is no objective way to measure the amount of manipulation and retraction of neural tissue by the surgeon. Our objective is to develop metrics quantifying dynamic retraction of cerebral tissue and the manipulation of instruments during a neurosurgical intervention. Methods: We trained a convolutional neural network to analyze microscopic footage of neurosurgical procedures and thereby generate metrics evaluating the surgeon’s dynamic retraction of brain tissue and the surgeon’s manipulation of the instruments themselves. U-Net image segmentation is used to output bounding polygons around cerebral parenchyma of interest, as well as the vascular structures and cranial nerves. Results: On the validation set, our network achieves a state of the art Intersection over Union (IoU) of 70.1% (Recall = 89%) and 74.3% (Recall = 91%) for surgical tools and biological structures respectively. Multivariate statistical analysis was used to evaluate dynamic retraction and tissue handling. Conclusions: We describe a semantic segmentation model for surgical instruments and intracranial structures to evaluate dynamic retraction of soft tissue and manipulation of instruments during a surgical procedure, while accounting for movement of the operative microscope. Using the intraoperative footage, this model can potentially provide the surgeon with objective feedback.