can you list the best ways to make this code move faster?

Try the async library to asynchronously attempt to do the same thing. The advantage of using async python is that you do not need to wait for one http call to finish before calling the next. This is a fantastic article on how to write concurrent/parallel code in python, give it a read if the syntax here is confusing.

refactored to run in async mode:

import asyncio
import aiohttp
from bs4 import BeautifulSoup

async def find_account(id, session):
    async with session.get(f'{id}/profile') as r:
        if r.status == 200:
            response_text = await
            soup = BeautifulSoup(response_text, 'html.parser')
            stuff = soup.find_all('h2')
            special = stuff[0]
            special = list(special)
            special = special[0]
            print(f'{id-9999} out of 1000 done')
            return str(special) + '      ID: {}'.format(id)
            return None

async def crawl_url_id_range(min_id, max_id):
    tasks = []
    async with aiohttp.ClientSession() as session:
        for id in range(min_id, max_id):
            tasks.append(asyncio.ensure_future(find_account(id=id, session=session)))
        return await asyncio.gather(*tasks)

event_loop = asyncio.get_event_loop()
users = event_loop.run_until_complete(crawl_url_id_range(min_id=10000, max_id=11000))

I tested and the above code works fairly well.

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top