Posted by Marta on December 18, 2020 Viewed 1099 times
Step by step tutorial. Learn to create a selenium bot that will automate any data entry task you like. Learn how to log in to a site with a bot.
Hey there! This article is a step by step tutorial where you will learn to create a selenium bot that will automate any data entry task you like. Instead of opening a website and enter your login details, you can create a selenium bot to do this for you. In this instance, you will learn how to log in to a site with a bot.
I will show step by step how to get your bot working. Plus some aspects you should avoid, to prevent a website from blocking your bot.
You will create the bot using selenium. In case you are not familiar with Selenium, Selenium is a library that will help you to launch and control a web browser. Using Selenium you can automatically fill in forms and simulate mouse movements and clicks. Find out more about selenium
Le’s get started with our bot!
First step to automate data entry is installing the selenium library. You can install the library using the
pip tool, which will manage all installed libraries.
You can install selenium executing the following command from your terminal:
>> pip install selenium
To run selenium you also need a driver, which is basically a browser. You could just use your already installed browser or download a driver. You can download a driver following the link below:
Make a note where you save this file, you will need it in the following steps
By now, you should have selenium installed and the driver file. Time to start adding some code to our bot. First step is opening the login site. In this instance, I will use the aliexpress login site: https://login.aliexpress.com/.
from selenium import webdriver browser = webdriver.Chrome('./chromedriver') #Replace with driver filename browser.get("https://login.aliexpress.com/")
Please note the driver file and your script should be in the same folder, just for simplicity. The code in line 3 is creating the browser using the
Chrome class, and as a parameter you need to pass the location of your driver file. Therefore, you should replace
./chromedriver‘ with the your driver filename.
Once you get your bot to open the login site, next step is entering your details, username and password, in the webform. There are two steps to achieve this:
See these steps in action below:
user_input = browser.find_element_by_id('fm-login-id') user_input.send_keys("<replace with your email> ") password_input = browser.find_element_by_id('fm-login-password') time.sleep(1) password_input.send_keys("<replace with your password> ")
As you can see, in the code above, I used the method
find_element_by_id, and as a parameter you need to pass the html widget id. This method will return the widget from the HTML DOM document with the given passed id.
In case you don’t know the component id, you can find out using the “Inspect” function of the Chrome Browser. You can access it just by double clicking the component, and select Inspect from the context menu:
Is this enough to login? I am afraid not. The reason is that most websites want to avoid bots login in and extracting information. Therefore website owners put mechanisms in place to detect bots and block them. Usually adding captchas and other widget that require specific human intervention.
What can you do then? One option is simulating human behaviour. For example, if you run your script, you will notice that the keys were entered really fast. The website will pick up this, detect that a bot could be accessing the site, and automatically add more widgets to block it. So you need to enter the keys as a human would do, one by one and with a small milliseconds delay between each key. Check out how to do that below:
user_input = browser.find_element_by_id('fm-login-id') username = "<replace with your username>" for letter in username: user_input.send_keys(letter) wait_time = random.randint(0,1000)/1000 time.sleep(wait_time) password_input = browser.find_element_by_id('fm-login-password') time.sleep(1) password = "<replace with your password>" for letter in password: password_input.send_keys(letter) wait_time = random.randint(0,1000)/1000 time.sleep(wait_time)
Keep in mind that each website has different mechanisms, and therefore behaves differently when it comes to block a bot. This specific website will include a slider if it suspects a bot is accessing the website.
However that shouldn’t stop the bot. Using selenium you can also simulate sliding movement, which basically involves press a widget, hold, move and release. Here are the steps to slide the slide bar:
slidebar_width = browser.find_element_by_css_selector("#nc_1__scale_text .nc-lang-cnt").size['width'] time.sleep(2) slider_button = browser.find_element_by_css_selector('#nc_1_wrapper #nc_1_n1z') sliding_move = ActionChains(browser) sliding_move.click_and_hold(slider_button).move_by_offset(slidebar_width,0).release().perform() time.sleep(2)
As you can see in the code above, another selenium method available to find components is
find_element_by_css_selector. This method allows you to search component by their classes as well, which is really useful when the widget doesn’t have an id.
And lastly, to log in in this website, the bot will need to press the sign in button. First, as before, you need to find the button component in the html. Once you found it, call the
click() method and selenium will click the button.
signin_button = browser.find_element_by_css_selector(".fm-login .fm-button") signin_button.click()
To summarise, in this tutorial we have learned how to automate data entry into a log in web form using selenium. The principle to create a bot is quite simple, first find the widget on which you want to perform an action, using
find_element_by_css method, and then click and enter your text.
Keep in mind some website have mechanisms in place to block bots, so you should simulate human behaviour as closely as you can.
Hope you enjoy the article. Thanks for reading and supporting this blog! 🙂
Steady pace book with lots of worked examples. Starting with the basics, and moving to projects, data visualisation, and web applications
Unique lay-out and teaching programming style helping new concepts stick in your memory
Great guide for those who want to improve their skills when writing python code. Easy to understand. Many practical examples
Perfect Boook for anyone who has an alright knowledge of Java and wants to take it to the next level.
Excellent read for anyone who already know how to program and want to learn Best Practices
Perfect book for anyone transitioning into the mid/mid-senior developer level
Great book and probably the best way to practice for interview. Some really good information on how to perform an interview. Code Example in Java