Electricity is the key to the proper functioning of modern human society. Ever-increasing electricity consumption gives rise to recent regulations and significant endeavors to improve the energy efficiency in all kinds of human activity from manufacturing to commerce, from transportation to digital communication, from entertainment to laptops and portable devices. An important technology for helping reduce energy consumption is the ability to store any excess electrical energy for long periods of time and efficiently retrieve the stored energy. The design and management of electrical energy storage systems is the focus of the present paper, which starts off by reviewing and comparing various types of electrical energy storage elements in terms of various metrics of interest ranging from power and energy density to output power rating and from self-leakage rate to cost per unit of stored energy, and from life cycle of the storage element to the efficiency of the charge/discharge cycle. Next the paper reviews various energy storage systems while motivating the need for a hybrid energy storage system comprised of heterogeneous types of energy storage elements organized in a hierarchical manner so as to hide the weaknesses of each storage element while eliciting their strengths. The paper continues with a detailed explanation of key challenges that one faces when dealing with the optimal design and runtime management of a hybrid energy storage system targeting some specific application scenario; for example, grid-scale energy management, household peak power shaving, mobile platform power saving, and more. A survey of some existing solutions to these problems is also included.