In this paper, Rauch-Tung-Striebel(RTS) optimal smoothing algorithm is applied to improving off-line satellite attitude determination accuracy. The measurements are first processed by the forward filter, and then, a separate backward smoothing pass is used for obtaining the smoothing solution. The performances between RTS and extended Kalman filter(EKF) in gyro/star-sensor integrated system are compared. RTS smoother exhibits better accuracy and robustness than EKF under the situation of lower sensors precision, different length of data and larger initial attitude error. The results verify the effectiveness and practicability of RTS smoothing algorithm in satellite attitude determination.