In settings.py, set

DOWNLOAD_DELAY = 5

to tell Scrapy that it should wait at least 5 seconds before making a new request.

Reason: treat the website to scrape gently so that it won't rate limit you or even have your IP address banned.

results matching ""

    No results matching ""