"Scrapy到底该怎么学？"今天，我将用这篇万字长文，带你从零开始掌握Scrapy框架的核心用法，并分享我在实际项目中的实战经验！建议收藏⭐！

一、Scrapy简介：为什么选择它？

1.1 Scrapy vs Requests+BeautifulSoup

很多新手会问：“我已经会用Requests+BeautifulSoup了，为什么还要学Scrapy？”

对比项	Requests+BS4	Scrapy
性能	同步请求，速度慢	异步IO，高性能
扩展性	需要手动实现	内置中间件、管道系统
功能完整性	仅基础爬取	自带去重、队列管理、异常处理
适用场景	小规模数据采集	企业级爬虫项目

👉 结论：如果是小型项目，Requests够用；但如果是商业级爬虫，Scrapy是更好的选择！

1.2 Scrapy核心架构

（图解Scrapy架构，建议配合流程图理解）

---

二、手把手实战：开发你的第一个Scrapy爬虫

2.1 环境准备

# 推荐使用虚拟环境
python -m venv scrapy_env
source scrapy_env/bin/activate  # Linux/Mac
scrapy_env\Scripts\activate  # Windowspip install scrapy

2.2 创建项目

scrapy startproject book_crawler
cd book_crawler
scrapy genspider books books.toscrape.com

2.3 编写爬虫代码

# spiders/books.py
import scrapyclass BooksSpider(scrapy.Spider):name = "books"def start_requests(self):urls = ['http://books.toscrape.com/']for url in urls:yield scrapy.Request(url=url, callback=self.parse)def parse(self, response):# 提取书籍信息for book in response.css('article.product_pod'):yield {'title': book.css('h3 a::attr(title)').get(),'price': book.css('p.price_color::text').get(),'rating': book.css('p.star-rating::attr(class)').get().split()[-1]}# 翻页逻辑next_page = response.css('li.next a::attr(href)').get()if next_page:yield response.follow(next_page, callback=self.parse)

2.4 运行爬虫

scrapy crawl books -o books.csv

---

三、Scrapy高级技巧（企业级应用）

3.1 突破反爬：随机UserAgent+代理IP

# middlewares.py
from fake_useragent import UserAgent
import randomclass RandomUserAgentMiddleware:def process_request(self, request, spider):request.headers['User-Agent'] = UserAgent().randomclass ProxyMiddleware:PROXY_LIST = ['http://proxy1.example.com:8080','http://proxy2.example.com:8080']def process_request(self, request, spider):proxy = random.choice(self.PROXY_LIST)request.meta['proxy'] = proxy

在settings.py中启用：

DOWNLOADER_MIDDLEWARES = {'book_crawler.middlewares.RandomUserAgentMiddleware': 543,'book_crawler.middlewares.ProxyMiddleware': 544,
}

3.2 数据存储：MySQL+Pipeline

# pipelines.py
import pymysqlclass MySQLPipeline:def __init__(self):self.conn = pymysql.connect(host='localhost',user='root',password='123456',db='scrapy_data',charset='utf8mb4')self.cursor = self.conn.cursor()def process_item(self, item, spider):sql = """INSERT INTO books(title, price, rating) VALUES (%s, %s, %s)"""self.cursor.execute(sql, (item['title'],item['price'],item['rating']))self.conn.commit()return itemdef close_spider(self, spider):self.conn.close()

---

四、常见问题Q&A

Q1：如何爬取JavaScript渲染的页面？

方案一：Scrapy+Splash

# 安装：docker run -p 8050:8050 scrapinghub/splash
yield scrapy.Request(url,self.parse,meta={'splash': {'args': {'wait': 2.5}}}
)

方案二：Scrapy+Playwright（推荐）

# settings.py
DOWNLOAD_HANDLERS = {"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler","https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Q2：如何实现分布式爬虫？

使用scrapy-redis：

# settings.py
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = 'redis://:password@localhost:6379/0'

---

五、性能优化技巧

1. 并发控制：

   # settings.py
   CONCURRENT_REQUESTS = 32  # 默认16
   DOWNLOAD_DELAY = 0.25  # 防止被封
   

2. 缓存请求：

   HTTPCACHE_ENABLED = True
   HTTPCACHE_EXPIRATION_SECS = 86400  # 缓存1天
   

3. 自动限速：

   AUTOTHROTTLE_ENABLED = True
   AUTOTHROTTLE_START_DELAY = 5.0