Web Scraping using Python Scrapy_BS4 - Introduction
Posted 一蓑烟雨
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Web Scraping using Python Scrapy_BS4 - Introduction相关的知识，希望对你有一定的参考价值。
What is Web Scraping
This is also referred to as web harvesting and web data extraction.
This is the process of automatically downloading a web page‘s data and extracting information from it.
Benefits of Web Scraping
Component of applications used for web indexing. e.g. Google
Web and data mining
Online price monitoring
Online price comparison
Product review to watch the competition
Gather real estate listing
Weather data monitoring
Website change detection
Research
Basic Rules for Web Scraping
Always check a website‘s Terms and Conditions before you scape it to avoid legal issues.
Do not request data from a website too aggressively(spamming) with your program as this may overload and break the website.
Tools used for Web Scraping
- Scrapy
- Scrapy is a free open source application framework.
- It is used for crawling web sites and extracting data.
- Can be installed using pip: pip install scrapy
- Beautiful Soup
- This is a python library used to extract data from HTML and XML files.
- Can be installed using pip: pip install beautifualsoup4(bs4)
Target Website:https://bluelimelearning.github.io/my-fav-quotes/
以上是关于Web Scraping using Python Scrapy_BS4 - Introduction的主要内容，如果未能解决你的问题，请参考以下文章
阅读OReilly.Web.Scraping.with.Python.2015.6笔记---Crawl
Python的基本Web Scraping（Beautifulsoup和Requests）
阅读OReilly.Web.Scraping.with.Python.2015.6笔记---找出网页中所有的href
使用 Python 和 asyncio 进行 Web Scraping
Step by Step of "Web scraping with Python" ----Richard Lawson ---3/n