Using Scrapy to crawl websites

A wonderful tool Scrapy came to my notice that can be conveniently used to crawl websites. It has many good features and I tested it successfully on Fedora 25. I am using it currently to do a bit of data mining and data analysis.