Scrapy

Scrapy is a web-crawling framework you can use for web scraping or extract data through APIs as well as a general-purpose web crawler. The way it works is fairly simple – you only need to give some instructions to a set of spiders (or crawlers) and put them to work. Scrapy lets you reuse your code and provides you with a web-crawling shell to test how your instructions will work on a real scenario.

Scrapy ( SKRAY-peye) is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler.