7 Commits

Author	SHA1	Message	Date
Sam Saffron	30990006a9	DEV: enable frozen string literal on all files ... This reduces chances of errors where consumers of strings mutate inputs and reduces memory usage of the app. Test suite passes now, but there may be some stuff left, so we will run a few sites on a branch prior to merging	2019-05-13 09:31:32 +08:00
Sam Saffron	3dc4ab905c	DEV: annotate models ... (also looks like renaming of bigint(8) -> bigint in annotate which seems fine)	2019-05-03 08:34:12 +10:00
Guo Xiang Tan	214dac05de	Update annotations.	2018-07-16 14:19:07 +08:00
Arpit Jalan	9ca6ebe8fe	FEATURE: enforce tagging on categories	2018-04-11 07:15:24 +05:30
Neil Lalonde	2bd44bbf13	WebCrawlerRequest.clear_cache needs to clear user agent list too	2018-03-27 15:11:48 -04:00
Neil Lalonde	4d12ff2e8a	when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.	2018-03-27 13:44:14 -04:00
Neil Lalonde	ced7e9a691	FEATURE: control which web crawlers can access using a whitelist or blacklist	2018-03-22 15:41:02 -04:00