How to Scroll using Selenium in Python

Auto-scrolling and scraping dynamically loaded pages like Instagram, Reddit, Twitter

Photo by Tomasz Frankowski on Unsplash

TL;DR

def scrolldown(driver):
    """A method for scrolling the page."""
    # Get scroll height.
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        # Scroll down to the bottom.
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        # Wait to load the page.
        sleep(2)
        # Calculate new scroll height and compare with last scroll height.
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
            print("END OF PAGE")
        last_height = new_height

and for a short scroll, pin your y value in here:

driver.execute_script(“window.scrollTo(0, 900)”)

Websites like Instagram load content dynamically, so without an API, you’ll need to use Selenium and scroll pages to access response data.

Setup

First, you’ll want to set up your webdriver so that you can use it as a class, and you certainly want to be using chromedriver, not firefox’s webdriver.

Set up your webdriver config