To solve the problem of data imbalance between source domain and target domain of pedestrian behavior in transfer learning process, a multi-source domain transfer learning pedestrian behavior recognition algorithm based on EMD (Earth Mover's Distance) measurement is proposed. Firstly, the feature information of pedestrian behavior image data is extracted under the condition of limited sample resources and mapped to a unified feature space. secondly, the distribution difference between different data domains (source domain and target domain) is quantified by EMD. Finally, transfer learning is carried out by making full use of similar source domain data, so as to effectively reduce the distribution distance between different data domains, make up for the problem of insufficient target domain data, and reduce the impact of uneven data distribution on recognition accuracy. Compared with other models, the proposed algorithm achieves accuracy of 76.7%, which improves the recognition effect of target domain and pedestrian behavior. The research results can effectively solve the problems of insufficient data and uneven data distribution in pedestrian behavior target domain.