This paper is to understand the relationship between poverty and school education through investigating the characteristics of poor family children's lives in their community, schools, and families. For this purpose, I interviewed elementary, middle, high school students who live in permanent rental apartments which are in middle class area of a city. Their parents, homeroom teachers, school principles, and local residents are included as interviewees. The main idea from the analysis of those data is that the influences of poverty on education are multi-level. Poor children have been segregated from middle class children in their community and upper schools which poor children attend recognize them as threats against the schools' fame. Poor children with even the same physical family conditions have experienced different parents' upbringing styles which have some significance to them from the diverse families' life history. Poor children recognize that they are from poor class by school achievement scores, especially Math scores. Schools do not notice their own interests. This result requires reconsidering a thesis that poverty is reproduced through the elements of individual or family backgrounds, such as family structure, parents' upbringing style, and peer relationship. The thesis which overemphasize the influence of family background and do not consider that of community and schools can justify poverty.