- 동적 스케줄링 기반 웹 크롤러의 성능분석
- Preliminary Performance Evaluation of a Web Crawler with Dynamic Scheduling Support
- ㆍ 저자명
- Lee. Yong-Doo,Chae. Soo-Hwan
- ㆍ 간행물명
- 한국산업정보학회논문지
- ㆍ 권/호정보
- 2003년|8권 3호|pp.12-18 (7 pages)
- ㆍ 발행정보
- 한국산업정보학회
- ㆍ 파일정보
- 정기간행물| PDF텍스트
- ㆍ 주제분야
- 기타
A web crawler is used widely in a variety of Internet applications such as search engines. As the Internet continues to grow, high performance web crawlers become more essential. Crawl scheduling which manages the allocation of web pages to each process for downloading documents is one of the important issues. In this paper, we identify issues that are important and challenging in the crawl scheduling. To address the issues, we propose a dynamic owl scheduling framework and subsequently a system architecture for a web crawler subject to the framework. This paper presents the architecture of a web crawler with dynamic scheduling support. The result of our preliminary performance evaluation made to the proposed crawler architecture is also presented.