scrapy

March 16, 2024 | seedling, permanent

tags :

Python Apps #

Scrapy, a fast high-level web crawling & scraping framework for Python. github

Used it in KSAFlyer (first version) and KSAPrice

It uses Twisted #

Starting a project #

scrapy startproject tutorial

tutorial/
    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here
        __init__.py

        items.py          # project items definition file

        middlewares.py    # project middlewares file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders
            __init__.py

Links to this note

html2text
trafilatura