This defines how often the Crawler will follow links embedded in websites.
A minimum of 0 is recommended and means that the page you enter under "Starting Point" will be added to the index, but no linked content is indexed. 2-4 is good for normal indexing. Be careful with the depth. Consider a branching factor of average 20; A prefetch-depth of 8 would index 25.600.000.000 pages, maybe this is the whole WWW.

Crawling Depth:

This is an emacs-like regular expression that must match with the URLs which are used to be crawled. Use this i.e. to crawl a single domain. If you set this filter it makes sense to increase the crawling depth.

Crawling Filter:

If you use this option, web pages that are already existent in your database are crawled and indexed again. It depends on the age of the last crawl if this is done or not: if the last crawl is older than the given date, the page is crawled again, otherwise it is treated as 'double' and not loaded or indexed again.

Use Re-Crawl Option:

Interval:

Year(s)   Month(s)   Day(s)   Hour(s)   Minute(s)

This option will automatically create a domain-filter which limits the crawl on domains the crawler will find on the given depth. You can use this option i.e. to crawl a page with bookmarks while restricting the crawl on only those domains that appear on the bookmark-page. The adequate depth for this example would be 1.
The default value 0 gives no restrictions.

Use Auto-Dom Filter:

Depth:

You can limit the maxmimum number of pages that are fetched and indexed from a single domain with this option. You can combine this limitation with the 'Auto-Dom-Filter', so that the limit is applied to all the domains within the given depth. Domains outside the given depth are then sorted-out anyway.

Limit max. pages per domain:

Page-Count:

A questionmark is usually a hint for a dynamic page. URLs pointing to dynamic content should usually not be crawled. However, there are sometimes web pages with static content that is accessed with URLs containing question marks. If you are unsure, do not check this to avoid crawl loops.

Accept URLs with '?' / dynamic URLs:

This option is used by default for proxy prefetch, but is not needed for explicit crawling. We recommend to leave this switched off unless you want to control the crawl results with the Cache Monitor.

Store to Proxy Cache:

This enables indexing of the wepages the crawler will download. This should be switched on by default, unless you want to crawl only to fill the Proxy Cache without indexing.

Index text:

Index media:

If checked, the crawler will contact other peers and use them as remote indexers for your crawl. If you need your crawling results locally, you should switch this off. Only senior and principal peers can initiate or receive remote crawls. A YaCyNews message will be created to inform all peers about a global crawl, so they can omit starting a crawl with the same start point.

Do Remote Indexing:

Intention to start this global crawl (optional):

This message will appear in the 'Other Peer Crawl Start' table of other peers.

This can be useful to circumvent that extremely common words are added to the database, i.e. "the", "he", "she", "it"... To exclude all words given in the file yacy.stopwords from indexing, check this box.

Exclude static Stop-Words:

Existing start URLs are re-crawled. Other already visited URLs are sorted out as "double". A complete re-crawl will be available soon.

From File:

From URL: