Our search software consists of two parts. The first is an indexing mechanism (indexer). The purpose of the indexer is to walk through HTTP, FTP, NEWS servers or local files, recursively grabbing all the documents and storing words meta-data about those documents in an SQL database in a smart and efficient manner.
Crawler / spider / meta