- Conference Article
10
- 10.1109/ideas.2007.4318093
CINDI Robot: an Intelligent Web Crawler Based on Multi-level Inspection
- Sep 01, 2007
- Rui Chen + 2 more +2
With the explosion of the Web, focused Web crawlers are gaining attention. Focused Web crawlers aim at finding Web pages related to the pre-defined topic. CINDI Robot is a focused Web crawler devoted to finding computer science and software engineering academic documents. We propose a multi-level inspection scheme to discover relevant Web pages. Through this multi-level inspection scheme, the text feature of the content contributes to the classification; furthermore other Web characteristics, such as URL pattern, anchor text and so on, assist the decision process. The experiment result demonstrates this multi-level inspection method outperforms other traditional methods.
Read more