High-performance wavelength-selective infrared (IR) sensors require small pixel structures, a low-thermal mass, and operation in the middle-wavelength infrared (MWIR) and long-wavelength infrared (LWIR) regions for multicolor IR imaging. All-metal-based mushroom plasmonic metamaterial absorbers (MPMAs) were investigated theoretically and were designed to enhance the performance of wavelength-selective uncooled IR sensors. All components of the MPMAs are based on thin layers of metals such as Au without oxide insulators for increased absorption. The absorption properties of the MPMAs were investigated by rigorous coupled-wave analysis. Strong wavelength-selective absorption is realized over a wide range of MWIR and LWIR wavelengths by the plasmonic resonance of the micropatch and the narrow-gap resonance, without disturbance from the intrinsic absorption of oxide insulators. The absorption wavelength is defined mainly by the micropatch size and is longer than its period. The metal post width has less impact on the absorption properties and can maintain single-mode operation. Through-holes can be formed on the plate area to reduce the thermal mass. A small pixel size with reduced thermal mass and wideband single-mode operation can be realized using all-metal-based MPMAs.