Objective This study’s aim is to verify the longitudinal effect of depressive symptoms on all-cause mortality depending on the age group. Methods The Korean Longitudinal Study of Aging’s data from 2006 to 2016 were assessed using longitudinal data analysis, and 10,145 participants (age [mean±standard deviation], 61.7±11.1 years; males, 4,426 [43.6%]; females, 5,719 [56.4%]; middle-aged adults, 6,036 [59.5%]; older adults, 4,109 [40.5%]) were included at baseline. The chi-square test, log-rank test, factor analysis, and Cox proportional hazards models were used to investigate the association between depressive symptoms and mortality. To verify that which feelings and behaviors are associated with mortality, factor analysis was used to Center for Epidemiologic Studies Depression (CES-D) scale and it was divided into two factors: negative and positive affects. Results The hazard ratio (HR) of all-cause mortality for the Q1 (high) of the negative affect factor was higher than the Q3 (low) of this factor (HR, 1.489; 95% confidence interval, 1.284–1.728). Additionally, the association between negative affect and all-cause mortality was stronger in middle-aged adults than older adults. Conclusion The study provided evidence of the longitudinal effect of depressive symptoms on all-cause mortality regardless of age. However, middle-aged adults could be more sensitive to negative feelings and behaviors than older adults.