Description
For anyone that hasn't been following #2842, CF pushed an update that prevented UC Mode from easily bypassing CAPTCHA Turnstiles on Linux servers. Additionally, uc_click() was rendered ineffective for clicking Turnstile CAPTCHA checkboxes when clicking the checkbox was required. I've been working on solutions to these situations.
As I mentioned earlier in #2842 (comment), if CF detects either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, then they don't let the click through. (The JS-detection part is new.) I read online that CF employees borrowed ideas from https://github.com/kaliiiiiiiiii/brotector (a Selenium detector) in order to improve their CAPTCHA. Naturally, I was skeptical at first, but I have confirmed that the two algorithms do appear to get similar results. (Brotector was released 6 weeks ago, while the Cloudflare update happened 2 weeks ago.)
The solution to bypassing the improved CAPTCHAs requires using pyautogui to stay undetected. There was also the matter of how to make pyautogui work well on headless Linux servers. (Thanks to some ideas by @EnmeiRyuuDev in #2842 (comment), that problem was overcome by setting pyautogui._pyautogui_x11._display to Xlib.display.Display(os.environ['DISPLAY']) on Linux in order to sync up pyautogui with the X11 virtual display.)
The improved SeleniumBase UC Mode will have these new methods:
driver.uc_gui_press_key(key) # Use PyAutoGUI to press the keyboard key
driver.uc_gui_press_keys(keys) # Use PyAutoGUI to press a list of keys
driver.uc_gui_write(text) # Similar to uc_gui_press_keys(), but faster
driver.uc_gui_handle_cf(frame="iframe") # PyAutoGUI click CF TurnstileIt'll probably be easier to understand how those work via examples. Here's one for uc_gui_handle_cf based on the example in #2842 (comment):
import sys
from seleniumbase import SB
agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0.0.0"
if "linux" in sys.platform:
agent = None # Use the default UserAgent
with SB(uc=True, test=True, rtf=True, agent=agent) as sb:
url = "https://www.virtualmanager.com/en/login"
sb.uc_open_with_reconnect(url, 4)
sb.uc_gui_handle_cf() # Ready if needed!
sb.assert_element('input[name*="email"]')
sb.assert_element('input[name*="login"]')
sb.set_messenger_theme(location="bottom_center")
sb.post_message("SeleniumBase wasn't detected!")Above, I deliberately gave it an incomplete UserAgent so that CAPTCHA-clicking is required to advance. On macOS and Windows, the default UserAgent that SeleniumBase gives you is already enough to bypass the CAPTCHA screen entirely. The uc_gui_handle_cf() method is designed such that if there's no CAPTCHA that needs to be clicked on the page you're on, then nothing happens. Therefore, you can add the line whenever you think you'll encounter a CAPTCHA or not. In case there's more than one iframe on a website, you can specify the CSS Selector of the iframe as an arg when calling uc_gui_handle_cf(). There will be new examples in the SeleniumBase/examples/ folder for all the new UC Mode methods. To sum up, you may need to use the newer uc_gui_* methods in order to get past some CAPTCHAs on Linux where uc_click() worked previously.
On the topic of Brotector, (which is the open source bot-detector library that CF borrowed ideas from), there is a huge opportunity: Now that effective bot-detection software is available to the general public (all the code is open source!), anyone can now build their own CAPTCHA services (or just add CAPTCHAs to sites without the "service" part). I've already jumped on this with the Brotector CAPTCHA: https://seleniumbase.io/apps/brotector. I've also created a few test sites that utilize it:
I did make some improvements to the original Brotector algorithm in order to be suitable for CAPTCHAs: I needed a definite Allow/Block answer, rather than a number between 0 and 1 determining the likelihood of a bot, etc. I've been using these new test sites for testing the improved UC Mode.
That covers the major updates from 4.28.0 (with the exception of Brotector CAPTCHA test sites, which were already available to the public at the URLs listed above).
There will also be some other improvements:
- More
sbmethods added directly into thedriver. - An improvement to the Recorder to better handle autogenerated IDs in selectors.
- Python dependency updates.
- Some method simplifications.
- Some timing updates.
- Some updates to default settings.
Now, when using UC Mode on Linux, the default setting is NOT using headless mode. If for some reason you decide to use UC Mode and Headless Mode together, note that although Chrome will launch, you'll definitely be detected by anti-bots, and on top of that, pyautogui methods won't work. Use xvfb=True / --xvfb in order to be sure that the improved X11 virtual display on Linux activates. You'll need that for the uc_gui_* methods to work properly.
Much of that will get covered in the 3rd UC Mode video tutorial on YouTube (expected sometime in the near future).
In case anyone has forgotten, SeleniumBase is still a Test Automation Framework at heart, (which includes an extremely popular feature for stealth called "UC Mode"). UC Mode has gathered a lot of the attention, but SeleniumBase is more than just that.