In the era of the epidemic, people’s way of life has undergone tremendous changes compared to before. For students, online courses have become the main teaching method. Due to the barrier of the screen, it is difficult for the instructors to observe the students’ reactions and adjust the teaching agenda according to the students’ responses. Thus, it could be hard to assess the teaching quality of the online courses for a normal instructor and it requires significant time and efforts of a professional-pedagogical expert to evaluate the online course teaching quality. In order to solve this problem, a five-aspect pedagogy evaluating framework has been designed, including clarity, classroom interaction, technical management of online teaching, empathy, and time management. Based on the pedagogy protocol, a multimodal machine learning model has been proposed to automate the assessment process. This method extracts mid-level behavior features, high-level interpretable features and classifies the videos based on supervised learning.