This paper proposes an algorithm to estimate the wind velocity by tracking free flying balloons. Balloons used in this method are expendable but inexpensive, which increases the usefulness of the method. Also we can obtain accurate 3D information by using multiple cameras and estimate the wind velocity of the local field. The proposed system consists of aerodynamic modeling of the balloon, a tracking algorithm using image processing, and the velocity estimation algorithm. We performed unit tests of each algorithm for the verification. The method is validated using a system simulation and sources of error case identified.