Table of Contents
Browser Automation with Selenium: Fingerprints …
I want to use selenium/webdriver to simulate a browser and scrape some website-content with it. Even if its not the fastest method, for me it has many advantages such as executing scripts etc.
For many websites it is forbidden to access them via an automated method, for example search engines like google or bing.
For one tool i need to scrape the estimated resultstat from google for several keywords. This will look like the following: simulate the browser that visits and types in a keyword and scrapes the results, then after a little pause type in the next keyword, scrape the results and so on…
My question is: Is it possible for a website to recognize that I’m using selenium to simulate the browser instead of using the browser by hand? Especially the google case gives me some doubts. I know selenium is partly developed by google or at least by some guys working for google. So does leave selenium some fingerprints or isn’t it possible to decide if I’m using the browser by myself or simulated by selenium, even for google?
Artjom B. 59k24 gold badges115 silver badges202 bronze badges
asked Jul 15 ’13 at 11:53
No, nobody can actually see that you’re using Selenium and not hand-operating the browser yourself with WebDriver. I’m not sure about the old Selenium RC, but it should be the same way. Here’s how it works:
Selenium opens up a browser with a clean profile (or with a profile you selected)
Selenium is hooked up to the browser so it can steer it, control it. But the browser still does most of the work. Basically, Selenium replaces the user inputs to the browser, but not more.
You can easily verify this by reading the contents of the HTTP headers sent by your browser.
If you ever actually needed Selenium to be recognized by your server, you can use Browsermob-proxy and add a custom header to your requests.
All that said, there is one thing you must be aware of. While there’s no way to detect Selenium directly, there can be some indirect clues picked up by the website you’re visiting. Those usually include scanning for too many requests made in virtually no time – this might be an issue for you. Make sure your Selenium is behaving like a user.
EDIT 2016/04:
Apparanetly it is possible as states that a company can do it. My guess – and it is nothing but a guess – is that they can run some JS that Selenium installs into the browser to operate.
answered Jul 15 ’13 at 12:26
Petr JanečekPetr Janeček35. 9k10 gold badges118 silver badges142 bronze badges
4
Signs point to yes, sites are able to regonize that you are using Selenium.
Counter Example: detects and blocks my browser instance launched using Selenium while “normal” browsing done manually (not using the browser launched by the Selenium web driver) work with out issue.
See this stackoverflow question for additional details
Can a website detect when you are using selenium with chromedriver?
answered Oct 23 ’15 at 21:38
Brian CainBrian Cain8981 gold badge7 silver badges20 bronze badges
Not the answer you’re looking for? Browse other questions tagged selenium-webdriver web-scraping browser-automation or ask your own question.
Browser Fingerprinting and Concealing Selenium – stafford …
I recently discovered a website that would respond to requests normally in Chrome, but failed to display anything when Chrome was being controlled by Selenium. Upon investigation, I discovered both Browser Fingerprinting and a way to have the site display correctly when accessed via Selenium WebDriver.
What is Browser Fingerprinting?
When you browse to a website, JavaScript executing on that website can query a number of things that, when aggregated, can produce a unique identifier, or fingerprint, that represents your browser or device. By querying for data that is often persistent over time, this same fingerprint can be generated determinatively regardless of clearing cookies or accessing via private/incognito modes, and can be used to track your activity over time and across websites. Examples include querying browser version, detecting installed browser extensions and, (for older browsers) determining history by inspecting the:visited CSS selector.
Fingerprint generation can also work cross-browser by using techniques that produce the fingerprint from the underlying device. For example, Canvas Fingerprinting uses WebGL to produce fingerprint variations from GPU hardware and graphics drivers. Similarly, querying for device font-availability could also return similar results cross-browser.
What is FingerprintJS?
It’s an open-source JavaScript library that will perform the fingerprinting for you. A commercial version exists which adds further functionality including Bot Detection which amongst others attempts to identify;
Selenium combinations (FF, Chrome, IE etc)
Chrome Headless
Crawl engines/bots
Accessing a site that implements FingerprintJS Pro via Selenium displays nothing, and running the same request in Postman throws a 429 – Too many requests error. Inspecting requests when we run Chrome ourselves seems to Just Work, but if we enable Preserve Logs we can see the fingerprinting requests (though not the response content).
The first request fails with 429 as it is absent a validated cookie. This request however still includes content, in the form of script references to, and, and a pre-validated cookie. The call to (using the pre-validated cookie) includes a second cookie which is used in the XHR to fingerprint whose response includes a third, validated cookie. This validated cookie is then used in a reloaded request to the original address which now succeeds with 200.
Subsequent requests no longer throw 429 or attempt to fingerprint for as long as the validated cookie remains cached. Clearing the cookie triggers the fingerprinting process again on the next request.
Solving for automated processes
The findings above mean any automated process accessing the site must;
Be able to execute JavaScript and, in the case of Selenium;
Conceal the presence of (Selenium) WebDriver
Selenium obviously already executes JavaScript so all we need to do is conceal the presence of WebDriver. The following displays the solution for C#, the same can be seen for Python here.
var driver = new ChromeDriver();
driver. ExecuteChromeCommand(“dScriptToEvaluateOnNewDocument”,
new neric. Dictionary
{
{ “source”, @”
fineProperty(navigator, ‘webdriver’, {
get: () => undefined})”}});
Prior to the above, the fingerprinting process will call nagivator. webdriver, receive true, and fail. With the call returning undefined, the fingerprinting process succeeds and the site loads.
Happy scraping!
Spoofing Browser Fingerprint in Selenium? | BlackHatWorld
Spoofing Browser Fingerprint in Selenium? | BlackHatWorld
Hi Guest, our system has detected that an AdBlocker is installed in your browser which may block essential functions in BlackHatWorld. Please consider disabling the AdBlocker for all pages on BHW so you can use the core functionality of BHW.
You may not be aware, but any visitor supports our site by just viewing ads. All ad’s are directly related to Internet Marketing.
Home
Forums
Programming & Web Design
Scripting
General Scripting Chat
You are using an out of date browser. It may not display this or other websites should upgrade or use an alternative browser.
#1
Joined
Dec 15, 2016
Messages
134
Reaction score
31
Im currently trying to use, a firefox extension, to randomize my fingerprint. By default though a lot of its better spoofing settings like timezone spoofing and setting my user agent to mobile are disabled.
I cant find any way to edit the extension settings from selenium webdriver (python). Does anyone know where extensions save their settings? Or if not a better way of spoofing my browser fingerprint in selenium?
Last edited: Feb 16, 2017
#2
Dec 22, 2016
119
38
I am currently using MultiLoginApp, it appears to be an independent spoofing and proxy app which supports chrome, firefox and many more.
#3
Short answer: you cannot edit add-on settings through selenium webdriver.
If you want to automate browser fingerprint management, you need software like Multiloginapp. Here you can get a feeling of its API and here see some automation examples
turns out selenium can actually edit extension settings actually. Extension settings are stored in the about:config menu where firefox stores all its other settings.
in selenium you can edit about:config options including those of the extensions
#4
Oct 28, 2013
663
737
Sorry for misinforming you then. Frankly didn’t know about this possibility.
#5
No problem. The information about extension settings being stored there was rather hard to find. And your Multiloginapp suggestion is interesting I’ll have to look into that.
#6
Oct 22, 2015
188
125
Website
create a profile where you store the user-agent you want (and other stuff):
profile = refoxProfile(‘/home/user/. mozilla/firefox/’)
t_preference(“eragent. override”, “Mozilla/5. 0 (Windows NT 6. 1; WOW64; rv:45. 0) Gecko/20100101 Firefox/45. 0”)
t_preference(“places. history. enabled”, False)
t_preference(“lineApps”, True)
t_preference(“sswords”, True)
t_preference(“teSettings”, True)
t_preference(“nitizeOnShutdown”, True)
profile. update_preferences()
driver = refox(firefox_profile=profile)
#7
#8
Aug 14, 2016
60
17
It’s trustable for sure. I was skeptical at first as to how well it can generate fresh fingerprints and handle all the browser side of things for me. Before I started using them, I was wasting time researching fingerprints and ways sites detect if you are running with selenium (gecko driver / chome driver). They even got their own browser call mimic which is their version of google chrome which helped me A LOT. Feel free to hit me up if you need any help with automating with 3rd party apps.
#9
Aug 9, 2019
14
9
i wrote a universal function to your projects to change random useragent in selenium. Create file and put different useragents in the column. Forum hasn’t function to add python code, so i add this code to php quote
from random import choice
useragents = open(“”)()(“\n”)
user_agent = choice(useragents) # choice random useragent from file
def FF_config(USERAGENT): #update useragent function
fp = refoxProfile()
t_preference(“eragent. override”, USERAGENT) # choice useragent
t_preference(“erconnection. enabled”, False) # disable webrtc
t_preference(“”, 0) # disable flash
t_preference(“”, “en”)
fp. update_preferences() #save settings
return refox(firefox_profile=fp)
driver = FF_config(user_agent) # init web driver with random useragent
#10
Mar 12, 2017
197
46
Thanks for sharing, but also keep in mind that if your file with useragents contains totally random stuff like mix of windows, linux and mobile UAs, then it’s not gonna work well because each UA has bunch of other properties that you need to maintain to make it look legit.
#11
You can make separate files with different useragents from different operating systems and call the desired user-agent if necessary. It depends on what the task is? Oh, sorry for my language, i’m trying to teach English))
#12
Apr 1, 2015
261
54
Is it able to automate in multilogin?
I am struggling to hide fingerprints on browsers like chrome or firefox.
#13
Oct 15, 2018
715
420
Multiloginapp has an API that you can use for browser automation
#14
Apr 4, 2018
138
48
This is useful stuff!
Multilogin costs $450 per month for rest api and cli access
Is it possible to automate a browser with selenium, in such a way that it can’t be identified as being selenium driven?
#15
Mar 16, 2014
307
86
My developer did this, I am not able to give you specific hints but what I can say is yes, you can do that with selenium and you don’t have to pay 450$ a month for multilogin app.
And btw multilogin canvas noise from what I’ve seen, is detectable.
Remember that you will have to figure out proper combinations of user agents and all other parameters in order to make to look legit. You cannot change randomly every parameter. You can change fonts, resolution, user agents, webgl and everything else. We avoided only canvas as it was too difficult
#16
Yes.
User agent
Webrtc!
Screen size
Fonts
Addon variety
Also some websites use on screen heat mapping so if you can emulate hovering over random links etc it doesn’t hurt.
There’s a start
#17
#18
#19
Feb 4, 2016
308
43
Isnt possible to hack the api tho? We only need the url and parameters passed in the request to the webserver
#20
Oct 31, 2019
7
0
Sorry for bringing up this older topic, but its indeed very very interesting. So unf. Multiloginapp is for me no option, bc. of the high monthly price. So im currently looking for a solution how to spoof the following Prints
Audio Printing
Canvas Fingerprint
Web GL Vendor
Can this be done directly in the About config or is it possible to manipulate this in a header? I currently use python. If anybody has another option or a link to read it u would be awesome! As im pretty new to this topic im currently kind of helpless and hopeless
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
Frequently Asked Questions about selenium browser fingerprint
Is browser fingerprinting possible?
Your browser fingerprint is likely to be extremely unique. Websites can use your unique fingerprint to gather and generate an in-depth personal file of websites that you’ve visited or target you with very personalized ads. There are various methods you can employ to cover up your prints on the internet.Jul 27, 2021
How do I get around my browser fingerprint?
How to prevent browser fingerprintingDisable Flash. If you are a Chrome user, then you should not worry about flash because Chrome will stop using it by the end of 2020. … Revisit extensions and plugins. … Keep all of your software updated. … Use Incognito or private mode. … Use Tor. … Use a VPN. … Ditch the smartphone.Oct 14, 2020
Can a website tell if you’re using selenium?
The answer is YES! Websites can detect the automation using JavaScript experimental technology navigator. webdriver in the navigator interface. If the website is loaded with automation tools like Selenium, the value of navigator.