This paper proposes tip position control system which uses a visual marker to determine the tip position of a robot manipulator. The main idea of this paper is to introduce visual marker for the tracking control of a robot manipulator. Existing researches utilize stationary markers to get pattern information from them. Unlike existing researches, we introduce visual markers to get the coordinates of them in addition to their pattern information. Markers need not be stationary and the extracted coordinate of markers are used as a reference trajectory for the tracking control of a robot manipulator. To build the proposed control scheme, we first obtain intrinsic parameters through camera calibration and evaluate their validity. Secondly, we present a procedure to obtain the relative coordinate of a visual marker with respect to a camera. Thirdly, we derive the equation for the kinematics of the SCORBOTER 4pc manipulator which we use for control of manipulator. Also, we provide a flow diagram of entire visual marker tracking system. The feasibility of the proposed scheme will be demonstrated through real experiments.