Output

For testing, I usually output a small amount of scraped data to a json file. Scrapy appends to a given file instead of overwriting its contents. So if we run the command twice, we will end up with a broken json. One of the solutions is to use json lines:

scrapy crawl spiderName -o output.jl