5 Commits

Author	SHA1	Message	Date
Guo Xiang Tan	214dac05de	Update annotations.	2018-07-16 14:19:07 +08:00
Arpit Jalan	9ca6ebe8fe	FEATURE: enforce tagging on categories	2018-04-11 07:15:24 +05:30
Neil Lalonde	2bd44bbf13	WebCrawlerRequest.clear_cache needs to clear user agent list too	2018-03-27 15:11:48 -04:00
Neil Lalonde	4d12ff2e8a	when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.	2018-03-27 13:44:14 -04:00
Neil Lalonde	ced7e9a691	FEATURE: control which web crawlers can access using a whitelist or blacklist	2018-03-22 15:41:02 -04:00