Help


Connection tab


Use the Connection tab to change the advanced settings relating to download errors and timeouts.



Settings


User-Agent string

Enter or select a string to identify DeepTrawl in web server logs.


Timeout

How long DeepTrawl waits before assuming an inactive connection should be treated as a timeout error.


Retry for non-contactable servers/timeouts

How many times DeepTrawl retries connections when they timeout or are non-contactable, and how long to wait between each retry.


The default of retrying 5 times with a 3 second delay ensures that DeepTrawl checks your network connection once during the retry cycle. DeepTrawl can only test the connection a maximum of once every 10 seconds. For this reason, reducing the retry count / delay is possible, but not recommended.


Cookies

Whether DeepTrawl accepts cookies, and if so, whether to use the original cookie data when re-requesting the page for operations such as opening the page in the Editor window.


Send Referer

If selected DeepTrawl will send the referer field in http request as a browser would. Many crawlers don't do this (e.g. Googlebot) so having it disabled is normal behavior.


Redirection

The maximum number of redirections DeepTrawl follows when trying to download a page. You can also send post data (from form actions) to all redirected URLs as well as to the original request URL.


Send domains as lower case

Select to transparently convert all host names to lower case when a request is made (case change won't show up in the interface).


Trust all SSL certs

When selected DeepTrawl will trust all SSL certificates (including self signed).