Python library.
selectors
Play with selectors using scrapy shell. Can use CSS or XPaths.
scrapy shell "URL"
using spider
scrapy crawl NAME
storing spider
scrapy crawl NAME -O FILENAME.csv
Can use any file type. -O overrides, -o appends. If using json, jsonl is appendable and stream-like.
resources
- Common Practices — Scrapy 2.13.0 documentation
- How to Use a Proxy with Scrapy in 2025 - ZenRows
- Render JavaScript with Scrapy: 5 Best Middlewares - ZenRows
- How To Set Scrapy Delays/Sleeps Between Requests | ScrapeOps
- Advanced Web Scraping with Scrapy: A Python Tutorial for Efficient Data Extraction | Scrape.do
- The Python Scrapy Playbook | ScrapeOps
- Optimizing Crawl Speed and Performance in Scrapy - Sling Academy
- How To Solve Scrapy 403 Unhandled or Forbidden Errors | ScrapeOps
- Automated Rotating Proxy Solution | Smart Proxy Manager | Zyte