Useful Selenium snippets in Python
I’m not an expert in Selenium, nor in Python, but it looks like I’m using it more often every month. Selenium is in most cases used to test web applications – it provides a framework to run the website in the real browser and to interact with it programmatically. It can be also helpful to reproduce actions taken or to provide step-by-step instructions on how to perform given action on the website.
My case
In my case, I’m mostly using Selenium with Python to retrieve the data from the websites. Sometimes it is directly related to the websites of our clients, sometimes I’m gathering data for statistical reasons, in some cases for my pet projects or Machine Learning exercises.
I’m also using Selenium to perform CSS regressions tests. As you probably noticed, changing CSS can be tricky – sometimes you change something in one place and this change affects multiple other pages. This is why I’m using CSS regression tests for – I’m comparing the “old” page layout with the current one and check if there are any unwanted changes.
Useful snippets
There are various things I use regularly and I want this article to gather the most important things. Starting from the beginning, I prefer to use Firefox as the browser from one simple reason: the web driver for Firefox doesn’t change that often as a Chrome driver. Chrome driver is strictly tied with the browser version and for me it causes the driver to change every other month. It is not the end of the world but requires additional work. For places where the Selenium-based jobs are performed automatically, it is way better to use Firefox and don’t think of drivers.
Starting the browser
The first thing I do when working with Selenium is to start the browser. It is rather stright-forward:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://your-website.com')
# perform other actions
driver.quit()
The web driver can be downloaded and located in the same folder as your script or in another place within the system path. The code above will open the browser and load the page. The last thing (once you perform any other actions required) is to close the browser. Otherwise, it will stay open but will not be useful for future scripts – there is no connection to such an open browser.
Browser options and size
You can set various options when opening the browser. The one that is used mostly by me is to open it in headless mode. It means that the browser is not visible, all operations are performed in the background. This is not convenient when you are learning and you want to see what Selenium is really doing on the site. On the other hand, it is good for automation and actions performed on the servers. You don’t really need to render the browser there – you simply need the results.
In order to set the browser options, you should change the previous code like this:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.set_window_size(1000,800)
driver.get('https://your-website.com')
# perform other actions
driver.quit()
Please note that I also added one more thing – the size of the browser window. It is useful when you want to check how the site behaves in various sizes or you want to test it in a particular size.
Retrieving the data
Once you have your website loaded, you can reach to the page elements. The simplest examples I use regularly are:
element_by_id = driver.find_element_by_id('id_of_element')
element_by_class = driver.find_element_by_class_name('name_of_class')
element_by_xpath = driver.find_element_by_xpath('//div/span')
element_by_css = driver.find_element_by_css_selector('p.content')
Please note that these functions are written in singular form (“find element”) but they are also existing in the plural (except “find by id” which is reasonable). So, the “find element” will return the first element found. The “find elements” will return iterable containing all elements found.
The example below shows how I work with “find elements” results – displaying the text of each element found:
elements = driver.find_elements_by_class_name('content')
for element in elements:
print(element.text)
Retrieving the parent element
Sometimes I need to reach the parent element of the one I just found. In such a case, I use this method:
elements = driver.find_elements_by_class_name('content')
for element in elements:
parent_element = element.find_element_by_xpath('..')
As you can see, you can use “find” functions not only on the “driver” level but also starting form any element of the page. This gives you more control over the search results.
Waiting for something to apperar
Right now, when the websites are loading the data dynamically, there is a possibility that the content will not be present on the page once it is loaded. You may want to wait until the content is visible. You can do it this way:
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.XPATH, "//span[contains(text(), 'text you are waiting for')]"))
)
element = WebDriverWait(driver, 30).until(
EC.presence_of_element_located((By.ID, "menu-root"))
)
In the above examples, first, the WebDriver is waiting for up to 15 seconds to see if there is an element with “text you are waiting for”. If such an element is present, it will be placed in the “element” variable. Otherwise, an exception is raised.
In the second example, Web Driver is waiting for up to 30 seconds to see if the element of ID “menu-root” is visible. Again – once it becomes present, it will be placed in the “element” variable. Otherwise, an exception is raised. In both cases, the “element” variable is filled as soon as the element is available. The driver is waiting for the 15 or 30 seconds only if the element is not present.
There is much more…
The above covers only the very basic operations in Selenium using Python. There is much more. You can fill forms, download files, basically – do everything the user is able to do in the browser. I will add more snippets, but right now you can also try to google for the things you need 🙂 Enjoy!