The present study aimed to compare the acceptable noise levels (ANLs) of normal-hearing listeners (N = 10) and cochlear implant (CI, N = 10) users, depending on the target-talker gender and the number of background competing talkers. The profile of hearing disability of CI users was also determined by the Korean-translated speech, spatial, and qualities of hearing scale (K-SSQ) for CI users. Results showed that the ANLs did not significantly differ between CI users and normal-hearing listeners. For both groups, the effect of target-talker gender (male vs. female) was not significant, whereas the number of background competing talkers (1 to 8 talkers) significantly influenced ANL results. The ANLs of both groups significantly increased as the number of competing talkers was reduced. The greatest distraction from noise was observed when the gender of target and background talkers was matched. This suggests that, regardless of listener groups, all the listeners are more annoyed by background noise when the background speech becomes more meaningful or when the voice of target and competing talkers gets similar. Correlational analyses revealed that the ANLs of CI users were significantly related to the K-SSQ responses on the quality of hearing domain, indicating that CI users who accepted more amount of noise actually reported more improvement on sound quality with CI use.