I even possess wrote a spider and it's engaged on the entire.
I even possess do up USER_AGENT in settings.
Nevertheless after I deployed on PythonAnywhere now it provides me 403 error:
DEBUG: Crawled (403)
(referer: None) DEBUG: Rule at line 12 without any user agent to position into stamp it on.
What might perchance or no longer it’s? What is this line 12?
UPDATE: tested on Heroku and it provides me 200. So whats up with PythonAnywhere?
UPDATE2: they most attention-grabbing enable walk domains: https://www.pythonanywhere.com/boards/matter/27574/