Cloud computing has been envisioned as the next-generation architecture of IT enterprise. However the security problem of Cloud computing hinder itself from large-scale application. One major challenge is to verify the integrity of data at untrusted server under the condition of supporting public auditability and dynamic data operation. This paper analyzed the problems of existing schemes, introduced commutative hash into Merkle hash tree, aggregated basic block into subblock and block, constructed a kind of hierarchical Merkle hash tree by using the concept of hierarchical structure in granular computing and proposed an improved scheme to offer effective, multi-granular dynamic operation. We analyzed the efficiency and security of the protocol. The analysis result showed that the protocol is efficient and security.