There is a problem of real-time detection difficulty in road surface damage detection. This paper proposes an improved lightweight model based on you only look once version 5 (YOLOv5). Firstly, this paper fully utilized the convolutional neural network (CNN) + ghosting bottleneck (G_bneck) architecture to reduce redundant feature maps. Afterwards, we upgraded the original upsampling algorithm to content-aware reassembly of features (CARAFE) and increased the receptive field. Finally, we replaced the spatial pyramid pooling fast (SPPF) module with the basic receptive field block (BasicRFB) pooling module and added dilated convolution. After comparative experiments, we can see that the number of parameters and model size of the improved algorithm in this paper have been reduced by nearly half compared to the YOLOv5s. The frame rate per second (FPS) has been increased by 3.25 times. The mean average precision (mAP@0.5:0.95) has increased by 8%—17% compared to other lightweight algorithms.