The lateral wave in ultrasonic TOFD (time of flight diffraction) image has a tail in transit time, which disturbs the detection and evaluation of shallow weld defect. Meanwhile, the lateral wave and back-wall echo that act as background add redundant data in digital image processing. In order to separate defect wave from lateral wave and prepare the way for following image processing, an algorithm of background removal method named as mean-subtraction is developed. Based on this, an improved method by statistic of the energy distribution in the image is proposed. The results show that by choosing proper threshold value according to the axial energy distribution of the image, the background can be removed automatically and the defect section becomes predominant. Meanwhile, diffractive wave of shallow weld defect can be separated from lateral wave effectively.