Deep web data scraping with python in google colaboratory

The information that you want is being generated dynamically. So, you won’t get it with requests. I suggest you use selenium for that.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time


link = 'https://www.birdsnest.com.au/brands/boho-bird/73067-amore-wrap-dress'
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome('C:/Users/../Downloads/../chromedriver.exe', options=options)
driver.get(link)
time.sleep(3)

soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.close()
page_new = soup.find('div', class_='model-info clearfix')
results = page_new.find_all('p')
for result in results:
    print(result.text)

Output

Marnee usually wears a size 8.
                She is wearing a size 10 in this style.
              
Her height is 178 cm.

Show Marnee’s body measurements

Marnee’s body measurements are:
Bust 81 cm
Waist 64 cm
Hips 89 cm

<div class="model-info-header">
              <p>
                <strong><span class="model-info__name">Marnee</span></strong> usually wears a size <strong><span class="model-info__standard-size">8</span></strong>.
                She is wearing a size <strong><span class="model-info__wears-size">10</span></strong> in this style.
              </p>
              <p class="model-info-header__height">Her height is <strong><span class="model-info__height">178 cm</span></strong>.</p>
              <p>
                <span class="js-model-info-more model-info__link model-info-header__more">Show <span class="model-info__name">Marnee</span>’s body measurements</span>
              </p>
            </div>

With requests you will miss all the data in BOLD which is what you want.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top