In this paper, we propose an algorithm that distributes processor time to the tasks decoding encoded frames with a goal maximizing total QoS (quality of services) of video conference system. An encoded frame has such a characteristic that the QoS of recovered frame image also increases as the processor time given for decoding the frame gets to increase. Thus, the quality of decoded image for each frame can be represented as a QoS function of the amount of service time given to decode. In addition, every stream of video conference has close time-dependency between continuous frames belonging to the same stream. Based on the time-dependency and QoS functions, we propose an on-line frame scheduling algorithm which does not schedule all frames in the system but just a few frames while maximizing total QoS of video streams in the conference. The simulation results show that, as the system load gets to increase, the proposed algorithm compared to the existing EDF algorithm can reduce the quality of decoded frame images more smoothly and show the movements of conference attendees more naturally without short cutting.