Working with Data Sources 2

Posted 2020-08-19 阿难的机器学习计划

Web Scriping:

1. We can also use requests.get to get the html file form a webpage.

2. If we would like to extract the content from the webpage, we can use BeautifulSoup Library.

　　from bs4 import BeautifulSoup

　　parser = BeautifulSoup(content, ‘html.parser‘) #initial the parser, pass the content by using BeautifulSoup

　　body = parser.body # extract the <p></p> from the parser 

　　p = body.p #Get body from <p></p>

　　head = parser.head

　　title_text = head.title.text #Get the content from <title></title>

3. We can use find_all function to find all the relevant content in the webpage. The find_all function can only being usd to bs4 elements.(tag)

　　head = parser.find_all("head") # Find all the files with tag head and save them as a list into variable head.

　　title = head[0].find_all("title")

　　title_text = title[0].text 

4. Find_all function can also find the content by its id. Find_all always return a list.

　　second_paragraph_text = parser.find_all("p", id ="second")[0].text

5. Find_all function can also find the content by class. 

　　second_inner_paragraph_text = parser.find_all("p", class_= "inner-text")[1].text # "p" indicates the tag of the class.

6. We can also use CSS selector to find the specific content. Same as find_all method. selector method also works on the sb4 format and return a list.

　　first_outer_text = parser.select(".outer-text")[0].text 

　　second_text = parser.select("#second")[0].text