【爬虫】 selenium 异步并发方案

正文

selenium 的一些替代

puppeteer
https://github.com/puppeteer/puppeteer

playwright
https://github.com/microsoft/playwright
https://github.com/microsoft/playwright-python

微软最强 Python 浏览器自动化工具开源了！可录制脚本
微软最强 Python 自动化工具开源了！不用写一行代码！
https://mp.weixin.qq.com/s/5QboULcxmcUsf-Itm12iVA

selenium异步并发方案
http://www.guimaizi.com/archives/550

# coding: utf-8
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.binary_location = r'%s'%"/Applications/Chromium.app/Contents/MacOS/Chromium"
driver = webdriver.Chrome(executable_path="/Users/guimaizi/hack-tool/chromedriver",options=chrome_options)
list_url=['https://www.w3school.com.cn/js/js_loop_for.asp','http://www.voidcn.com/article/p-stvclize-bte.html',
          'https://xiday.com/2019/09/21/puppeteer-run-js/','https://im.qq.com/','http://192.168.0.225/vul/frame221.html','https://en.mail.qq.com/','https://www.1688.com/',
      'https://github.com/','https://www.baidu.com/','http://www.guimaizi.com/','https://fanyi.baidu.com/?aldtype=16047#auto/zh','http://es6.ruanyifeng.com/#docs/promise',
  'https://security.alibaba.com/','https://security.alipay.com/home.htm','https://security.pingan.com/','https://bugbounty.huawei.com/hbp/#/home','https://www.youtube.com/watch?v=N0bypyMIt6w']
driver.get('http://www.qq.com')
for url in list_url:
    js="window.open('%s')"%url
    driver.execute_script(js)
windows = driver.window_handles
for num in windows:
    try:
        driver.switch_to.window(num)
        driver.set_script_timeout(5)
        #driver.set_page_load_timeout(5)
        print(driver.title)
    except Exception as e:
        print(e)
    finally:
        driver.close()
driver.quit()

• #CloudScanner
• #Crawler