We investigated the surface structure and wetting behavior of octaneselenolate self-assembled monolayers (SAMs) on Au(111) formed in a 50 ${mu}M$ ethanol solution according to immersion time, using scanning tunneling microscopy (STM) and an automatic contact angle (CA) goniometer. Closely-packed, well-ordered alkanethiol SAMs would form as the immersion time increased; unexpectedly, however, we observed the structural transition of octaneselenolate SAMs from a molecular row phase with a long-range order to a disordered phase with a high density of vacancy islands (VIs). Molecularly resolved STM imaging revealed that the missing-row ordered phase of the SAMs could be assigned as a $(6{ imes}{surd}3)R30^{circ}$ superlattice containing three molecules in the rectangular unit cell. In addition, CA measurements showed that the structural order and defect density of VIs are closely related to the wetting behaviors of octaneselenolate SAMs on gold. In this study, we clearly demonstrate that interactions between the headgroups and gold surfaces play an important role in determining the physical properties and surface structure of SAMs.