A mathematical model for heat and mass transfer in a miniature heat pipe with a grooved wick structure is developed and solved analytically to yield the maximum heat transport rate and the overall thermal resistance under steady-state conditions. The effects of the liquid–vapor interfacial shear stress, the contact angle, and the amount of initial liquid charge have been considered in the proposed model. In particular, a novel method called a modified Shah method is suggested and validated; this method is an essential feature of the proposed model and accounts for the effect of the liquid–vapor interfacial shear stress. In order to verify the model, experiments for measuring the maximum heat transport rate and the overall thermal resistance are conducted. The analytical results for the maximum heat transport rate and the total thermal resistance based on the proposed model are shown to be in close agreement with the experimental results. From the proposed model, numerical optimization is performed to enhance the thermal performance of the miniature heat pipe. It is estimated that the maximum heat transport rate of outer diameter 3 and 4 mm heat pipes can be enhanced up to 48%and 73%, respectively, when the groove wick structure is optimized from the existing configurations. Similarly, the total thermal resistance of these heat pipes can be reduced by 7%and 11%, respectively, as a result of optimization.