The visibility polygon of a simple polygon P is the set of points which are visible from a visibility source in P such as a point or an edge. Since a visibility polygon is the set of points, the set operations such as intersection, union, or difference can be executed on them. The intersection (resp. union) of two visibility polygons is the set of points which are visible from both (resp. either) of the corresponding two visibility sources. The difference of two visibility polygons is the set of points which are visible from only a visibility source. Previously, the best known algorithm for the set operations of two polygons with total n vertices takes O(nlogn ＋ k) time, where k is the output size. In this paper, we present O(n) time algorithms for computing the intersection, the union, and the difference of given two visibility polygons, which are optimal.