Abstract Thin-walled frame parts are widely used in aerospace and other fields. However, part deformation caused by cutting load and residual stress poses significant challenges to manufacturing accuracy and parts performance. To enhance the machining accuracy and part performance, this research proposes a novel clamping layout optimization method on the basis of deconvolution neural network (DCNN) and particle swarm optimization algorithm (PSO). The method integrates simulation model to simulate deformation and uses DCNN to predict the deformation field efficiently. The PSO is then applied to optimize the clamping layout while meeting physical and interference constraints. The case study shows that this method effectively reduces the deformation in the web area of thin-walled frame parts. Compared with the uniform layout, the optimization objective function value is reduced by 21.4%, and the maximal deformation and average deformation are separately decreased by 16.0% and 39.8%, which verifies the effectiveness of the presented method. This research not only addresses the critical issue of machining deformation in thin-walled parts but also demonstrates a novel combination of artificial intelligence based modeling and simulation techniques, which can be expanded to other advanced manufacturing challenges.