Most high-order computational fluid dynamics methods for compressible flows are based on Riemann solver for the flux evaluation and Runge-Kutta time stepping technique for temporal evolution. Different from these traditional high-order methods, the flux of the gas-kinetic scheme (GKS) is based on a time-dependent evolution solution of the kinetic equation, such as the Bhatnagar-Gross-Krook model, and it targets on the Euler and N-S solutions. High-order GKS (HGKS) has been systemically developed over the past decades. The recently proposed two-stage fourth-order HGKS is efficient, accurate and robust. In this thesis, we mainly focus on the efficient implementations of HGKS, especially on the construction of compact schemes. The first two parts of the thesis focus on the temporal and spatial discretizations of HGKS. By combining the second-order or third-order GKS flux functions with the multi-stage multi-derivative technique, a family of HGKSs can be constructed. As an extension of the previous two-stage fourth-order GKS, the fifth-order schemes with two and three stages have been developed. It provides a framework to construct HGKS with temporal accuracy towards to an arbitrary order. Then the weighted essentially non-oscillatory with adaptive order (WENO-AO) method is adopted for the initial reconstruction in HGKS. With the help of the new reconstruction method, HGKS can achieve uniform high-order for equilibrium and non-equilibrium states. Moreover, the new scheme becomes simpler in reconstruction, more flexible on numerical treatment at the Gaussian points, and more robust than the previous HGKS with standard WENO reconstruction. We present the fifth-order WENO-AO GKS for validation, while this reconstruction procedure can be easily extended to arbitrary order in space. Then, a fourth-order compact GKS is developed. It is composed of three ingredients, which are the two-stage fourth-order framework for temporal discretization, the higher-order gas evolution model for the evaluations of interface values and fluxes, and the fifth-order Hermite WENO reconstruction for the cell-averaged values updated through fluxes and their first order derivatives obtained through the differences of flow variables at the cell interfaces. These ingredients make the scheme to achieve higher-order of accuracy and better robustness with lower computational cost. The CFL number used in this compact GKS is on the order of 0.5 instead of 0.14 in the popular discontinuous Galerkin method. Furthermore, targeting on simulating more challenging flow problems, such as the aeroacoustic generation and propagation, and turbulence, up to eighth-order compact schemes have been developed. These higher-order schemes show spectral-like resolution for acoustic waves, and capture shocks crisply by the special designed WENO-Z reconstructions on a compact stencil. Lastly, the two-stage compact scheme has been successfully extended to unstructured meshes. Accurate solutions have been obtained for both inviscid and viscous flows without sensitive dependence on the quality of the mesh. The robustness of the scheme has been validated as well through many cases, including strong shocks in the hypersonic viscous flow simulations.