In this paper, a new visual telephone system which has a differential task interrupt transfer feature for real time video phone service is presented. Owing to the result of Interrupt transfer of different speed according to the time critical degree of tasks, the flow of audio and video data stream can be kept as constant speed in other word that means video phone services are carried out in real time. The ITU-T H.32x visual telephone recommendations are first analyzed, and the unsatisfactory items of existing systems are second inquired the cause, such as performance, quality. And then the design concept and ideas which enable it to solve them are third devised, the next, the new architecture of visual telephone system for real time video phone source are designed, which make it possible to solve the existing problems by means of different tasks interrupt transfer method.