Help


Spider tab


The Spider tab enables you to determine how DeepTrawl "spiders" through the structure of a website.



Settings


Number of threads

How many pages DeepTrawl trawls at the same time. The lower this value, the slower the trawl. However, higher values take up more bandwidth.


Cache size for image download information

DeepTrawl keeps information about downloaded images cached in memory. This prevents the same image being downloaded many times, so speeding up Trawls. The initial setting of 5,000 items is sufficient for most websites, but if your website is very large, you may want to increase the cache size.


Replace least use items: On reaching capacity, delete the least used items to make more room.


Clear before re-checking: Empty the image cache when images are re-checked in the DeepTrawl Editor window. Leave this option switched on if the structure of your website is likely to change between uses of the re-check feature. Otherwise you may wish to switch it off to increase the speed of the re-check feature.


Cache size for page download information

DeepTrawl keeps information about downloaded pages and other linked files cached in memory in order to prevent the same urls being downloaded many times. This speeds up trawling but also uses up memory. The initial setting of 5,000 items is sufficient for most web sites, but if your web site is very large you may want to increase the cache size.


Replace least used items: On reaching capacity, delete the least used items to make more room.


Clear before re-checking: Empty the page cache when links are re-checked in the DeepTrawl Editor window. Leave this option switched on if the structure of your website is likely to change between uses of the re-check feature. Otherwise you may wish to switch it off to increase the speed of the re-check feature.