COCO-Bridge: Structural Detail Data Set for Bridge Inspections

The purpose of this research is to propose a means to address two issues faced by unmanned aerial vehicles (UAVs) during bridge inspection. The first issue is that UAVs have a notoriously difficult time operating near bridges. This is because of the potential for the navigation signal to be lost between the operator and the UAV. Therefore, there is a push to automate or semiautomate the UAV inspection process. One way to improve automation is by improving UAVs’ ability to contextualize their environment through object detection and object avoidance. The second issue is that, to the best of the authors’ knowledge, no method has been developed to automatically contextualize detected defects to a structural bridge detail during or after UAV flight. Significant research has been conducted on UAVs’ ability to detect defects, like cracks and corrosion. However, detecting the presence of a defect alone does not contextualize its significance or help with an inspector’s job to rate specific structural bridge details. This paper outlines a use case for a data set and model to detect critical structural bridge details, providing context and vision for enhancing the autonomous UAV bridge inspection process. Identifying these structural bridge details that require inspection may assist an UAV in path planning and object avoidance in GPS-denied environments. The detection of structural details adds an ability to contextualize defect detection and localize issues to a bridge detail. This also has implications for providing cues to inspectors, in real time, on defect-susceptible areas while UAVs are in flight. The image data set, Common Objects in Context for bridge inspection (COCO-Bridge), for UAV object detection was collected and then trained using deep learning techniques. This data set consists of 774 images and over 2,500 object instances to detect 4 structural bridge details: bearings, cover plate terminations, gusset plate connections, and out-of-plane stiffeners. These details were chosen because they either must be rated by an inspector or checked because they are prone to failure. Methods to economize the predictive capabilities of the model through image augmentation were investigated to extend the performance of the training images. It was concluded that for this domain of data, structural bridge detail images, the mean average precision, and F1 score performance were improved by mirroring the training images along their -axis. The outcome of this paper was an open-source annotated data set, which can be used in computer vision applications for visual inspection, growing the capabilities of artificial intelligence in structural engineering.