failing to scrape dynamic webpages using selenium in python

Try below approach using python – requests simple, straightforward, reliable, fast and less code is required when it comes to requests. I have fetched the API URL from website itself after inspecting the network section of google chrome browser.

What exactly below script is doing:

1. First it will take the API URL and do GET request.

2. After getting the data script will parse the JSON data using json.loads library.

3. Finally it will iterate all over the list of companies list and print them for ex:- Rank, Company name, Social account links, CEO name etc.

   import json
   import requests
   from urllib3.exceptions import InsecureRequestWarning
   requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
   
   def scrap_inc_5000():
   
   URL = 'https://www.inc.com/rest/companyprofile/nuleaf-naturals/withlist'
   
   response = requests.get(URL,verify = False)
   result = json.loads(response.text) #Parse result using JSON loads
   extracted_data = result['fullList']['listCompanies']
   for data in extracted_data:
       print('-' * 100)
       print('Rank : ',data['rank'])
       print('Company : ',data['company'])
       print('Icon : ',data['icon'])
       print('CEO Name : ',data['ifc_ceo_name'])
       print('Facebook Address : ',data['ifc_facebook_address'])
       print('File Location : ',data['ifc_filelocation'])
       print('Linkedin Address : ',data['ifc_linkedin_address'])
       print('Twitter Handle : ',data['ifc_twitter_handle'])
       print('Secondary Link : ',data['secondary_link'])
       print('-' * 100)
   scrap_inc_5000()