Background: With the increasing use of social media platforms for health communication, the quality of health educational short videos (HESVs) has become a key concern. However, no standardized framework exists to evaluate the quality of health videos on social media, highlighting the need for a comprehensive evaluation system. Objective: The aim of this study is to develop a valid and structured evaluation tool for assessing the quality of HESVs on social media. Methods: The initial evaluation indicators obtained from the literature review and brainstorming undertaken in the study group were provided to the nominal group reference Lasswell’s 5W communication model, and two rounds of nominal group technique (NGT) were carried out to screen, add, revise, and adjust indicators, and reach a consensus of evaluation system. The indicators were then ranked based on their significance, as scored by the experts using the analytic hierarchy process. The content validity was assessed by experts who rated the relevance of each indicator on a 4-point Likert scale. Results: The primary indicators include communicator, communication content, communication channel, and communication effect, along with 13 secondary indicators and 34 tertiary indicators. 11 experts were enrolled in the NGT, 45% of experts had a doctoral degree, 80% of them were ranked associate professor or professor. The average familiarity coefficient of each key indicator of the NGT was 0.85. The average values of the expert judgment coefficient and authority coefficient were 0.93 and 0.85, respectively. In Round 1 of NGT, the “Communication target” of 5 primary indicators, 7 of 20 secondary indicators, and 66 of 94 tertiary indicators did not reach a consensus, and therefore, they were not deleted and will proceed to the next round of NGT. In Round 2 NGT, 1 primary indicator, 7 secondary indicators, and 59 tertiary indicators were deleted based on the consensus criteria. After the two rounds of NGT, 4 primary indicators, 13 secondary indicators, and 34 tertiary indicators finally reached a consensus. Among primary indicators, communication content was found to be the most influential, accounting for 45.68%. Among secondary indicators, credibility, scientificity, availability, and social attention were the most influential indicators, with priorities of 56.67%, 24.26%, 74.62%, and 39.89% in their respective categories. Among tertiary indicators, ‘Become a hot search recommended by the platform’ was the most influential indicator with a weight of 0.07. The content validity of all the evaluation indicators were 0.73 – 1.0, and the scale-level content validity index (average) was 0.87, which was indicated as acceptable. Conclusions: The evaluation system for the quality of HESVs on social media (LassVQ) was developed, and its validity was acceptable. The proposed evaluation system can be used in conjunction with qualitative methods to gain a holistic perspective on the multidimensional quality of HESVs on social media.