This paper aims to study the consensus approach for the decision making problems with hesitant fuzzy linguistic decision matrices. The distance measure is proposed for the hesitant fuzzy linguistic term sets. Then based on the distance among the individual decision matrices, the group consensus level and some other consensus levels are defined for the hesitant fuzzy decision matrices, and the consensus reaching process is introduced. The model is to make the adjusted elements as little as possible and make the information provided for the experts as much as possible, and is helpful for the experts to determinate the adjusted elements. Finally, a numerical example is used to illustrate the feasibility and practicability of the proposed consensus model.