Modify Scrapy user agent
suggest changeSometimes the default Scrapy user agent ("Scrapy/VERSION (+http://scrapy.org)"
) is blocked by the host. To change the default user agent open settings.py, uncomment and edit the following line to what ever you want.
#USER_AGENT = 'projectName (+http://www.yourdomain.com)'
For example
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
Found a mistake? Have a question or improvement idea?
Let me know.
Table Of Contents