Objective We investigated which factors in psychological changes positively or negatively affect the quality of life to suggest desirable directions in the period of the COVID-19 pandemic. Methods Online survey was conducted with 1,011 adults more than 19 years old in Busan, South Korea. Quality of life was measured using the World Health Organization Quality of Life-BREF. Questions regarding the psychological changes were about COVID-19-related concerns, distress in complying with quarantine guidelines, and interest in seeking something to do alone. Results Quality of life was perceived to be below average (mean±SD, 2.86±0.53). The more economic worries there were, the lower the quality of life in psychological, social relationships and environmental domains. The more distress one experienced when maintaining their personal hygiene, the higher the quality of life related to their physical health. Likewise, the more difficult it was to maintain social distances, the higher the quality of life associated with psychological and social relationships. The more interested someone was in how to spend time alone, the higher the quality of life in all domains significantly. Conclusion We can minimize the negative impacts of COVID-19 by maintaining economic stability, maintaining prosocial behaviors related to personal hygiene and making good use of personal time.