High harmonic generation (HHG) from solids shows great application prospects in compact short-wavelength light sources and as a tool for imaging the dynamics in crystals with subnanometer spatial and attosecond temporal resolution. However, the underlying collision dynamics behind solid HHG is still intensively debated and no direct mapping relationship between the collision dynamics with band structure has been built. Here, we show that the electron and its associated hole can be elastically scattered by neighboring atoms when their wavelength approaches the atomic size. We reveal that the elastic scattering of electron/hole from neighboring atoms can dramatically influence the electron recombination with its left-behind hole, which turns out to be the fundamental reason for the anisotropic interband HHG observed recently in bulk crystals. Our findings link the electron/hole backward scattering with Van Hove singularities and forward scattering with critical lines in the band structure and thus build a clear mapping between the band structure and the harmonic spectrum. Our work provides a unifying picture for several seemingly unrelated experimental observations and theoretical predictions, including the anisotropic harmonic emission in MgO, the atomic-like recollision mechanism of solid HHG, and the delocalization of HHG in ZnO. This strongly improved understanding will pave the way for controlling the solid-state HHG and visualizing the structure-dependent electron dynamics in solids.