WebScraping Requests Status gives me 200 without content

When no jobs are found, the website shows this message: No jobs were found that matched your search. You can use this in order to find out whether the page contains any jobs or not. Here is the full code:

import time
import requests
from bs4 import BeautifulSoup

website = 'https://www.daijob.com/en/jobs/search_result?job_search_form_hidden=1&keywords=Data+Analyst'

page = 0

while True:

        time.sleep(1)

        r = requests.get(website, params = {"page" : page+1})
        if 'No jobs were found that matched your search.' in r.text:
            break
        else:
            html = r.content
            soup = BeautifulSoup(html, "lxml")
            print('\033[1m' + 'Web 1, page {0}'.format(page+1) + '\033[0m')
            page += 1

Output:

Web 1, page 1
Web 1, page 2
Web 1, page 3
Web 1, page 4
Web 1, page 5

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top