Help


Skip URLs tab


This tab allows you to enter URLs which DeepTrawl should never try to download. Full or part URLs should be entered one per line.


This is useful, for example, for instructing DeepTrawl to never check specific links or images, or if you are using a Form Action to login to your website and you don't want DeepTrawl to follow a logout URL.



Settings


URL List

Enter full or part urls which should never be downloaded,


Example:

http://www.MySite.com/myPage.htm

http://www.MySite.com/myImage.jpg

This stops DeepTrawl from downloading the two individual resources.


Example:

http://www.MySite.com/myPage.htm

http://www.MySite.com/myImage.jpg

/myDir/

This blocks downloading of the first two resources and also blocks anything in the myDir directory.