selenium 如何抓取请求信息 |
您所在的位置：网站首页 › Selenium3监听请求 › selenium 如何抓取请求信息 |
|
页面性能监控
很多公司都会做页面性能的检测，做的方法也非常多。其实比较简单的是利用js去做，js可以很方便的调取浏览器的api，获取network里的相关信息，这个资料还是比较多的。唯一的难点在于如何注入js脚本，业内的方法的普遍有两种 利用selenium的execute_script，注入js脚本 在nginx层加入subfilter配置，注入js脚本这个具体的实现，可以去调研，暂时没有实践过。 不注入js脚本的前提下，如何获取页面数据组内对于以上两种方法，认为注入js对代码的侵入性太大，不是很客观，所以一直在搜索其他的实现方法，目前能想到的也就只有通过selenium和chromedriver，看能否从里面提取一些页面请求信息。本文只是记录一下调研的过程。 在stackoverflow上搜到文章 Browser performance tests through selenium，里面给了一个例子，是可以获取到相关日志，日志的量非常庞大 import json from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities #caps = DesiredCapabilities.CHROME #caps['loggingPrefs'] = {'performance': 'ALL'} # 78版本的chrome需要加这个，https://stackoverflow.com/questions/56812190/protractor-log-type-performance-not-found-error caps = { 'browserName': 'chrome', 'loggingPrefs': { 'browser': 'ALL', 'driver': 'ALL', 'performance': 'ALL', }, 'goog:chromeOptions': { 'perfLoggingPrefs': { 'enableNetwork': True, }, 'w3c': False, }, } driver = webdriver.Chrome(desired_capabilities=caps) driver = webdriver.Chrome(desired_capabilities=caps) driver.get('https://www.baidu.com') logs = [json.loads(log['message'])['message'] for log in driver.get_log('performance')] with open('devtools.json', 'wb') as f: json.dump(logs, f) driver.close()试了一下，可以拉起pc端的chrome，但我需要在安卓机上跑，所以需要更改下chrome的配置，代码如下 https://sites.google.com/a/chromium.org/chromedriver/capabilities https://github.com/webdriverio/webdriverio/issues/476 from selenium import webdriver from pprint import pprint from selenium.webdriver.common.desired_capabilities import DesiredCapabilities caps = DesiredCapabilities.CHROME caps['loggingPrefs'] = {'performance': 'ALL'} options = webdriver.ChromeOptions() options.add_experimental_option('androidPackage', 'com.android.chrome') options.add_experimental_option('perfLoggingPrefs',{'enableNetwork': True}) driver = webdriver.Chrome(chrome_options=options,desired_capabilities=caps) driver.get('https://www.baidu.com') log=driver.get_log('performance') pprint(log)拿到log后，接下来就可以分析这些日志了，日志非常多，格式化后有三千多行。 首先最外层是个大数组，包含若干字典格式的数据，如下： [ {'level': 'INFO', 'message': '{"message":...}', 'timestamp': 1515665118083}, {'level': 'INFO', 'message': '{"message":...}', 'timestamp': 1515665118083} ]在这一层没提取到什么信息，最主要的信息都是在message字段里，是一个非常长的json字符串，message字段里又包含message和webview字段，格式化后如下： {'message': {'method': 'Network.requestWillBeSent', 'params': {'documentURL': 'https://www.baidu.com/', 'frameId': '9616.1', 'initiator': {'type': 'other'}, 'loaderId': '9616.4', 'request': {'headers': {'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 ' '(Linux; Android ' '4.4.4; SM-A7000 ' 'Build/KTU84P) ' 'AppleWebKit/537.36 ' '(KHTML, like ' 'Gecko) ' 'Chrome/58.0.3029.83 ' 'Mobile ' 'Safari/537.36'}, 'initialPriority': 'VeryHigh', 'method': 'GET', 'mixedContentType': 'none', 'referrerPolicy': 'no-referrer-when-downgrade', 'url': 'https://www.baidu.com/'}, 'requestId': '9616.88', 'timestamp': 6117.672547, 'type': 'Document', 'wallTime': 1515665221.86887}}, 'webview': '0'}我们再拆解message['message']字段，里面一般会包含method和params两项。methods主要有以下这些： Network.requestWillBeSent Network.responseReceived Network.dataReceived # 触发多次 Network.loadingFinished各个阶段的返回信息如下： Page.frameStartedLoading Network.requestWillBeSent Network.responseReceived Network.dataReceived Page.frameNavigated Network.requestServedFromCache Network.loadingFinished Network.resourceChangedPriority Page.domContentEventFired Network.loadingFailed Page.loadEventFired Page.frameStoppedLoading其中一个正常的请求大致需要经历以下过程： {'method': 'Network.requestWillBeSent', 'params': {'documentURL': 'https://www.baidu.com/', 'frameId': '9616.1', 'initiator': {'type': 'other'}, 'loaderId': '9616.4', 'request': {'headers': {'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Linux; Android ' '4.4.4; SM-A7000 ' 'Build/KTU84P) ' 'AppleWebKit/537.36 (KHTML, ' 'like Gecko) ' 'Chrome/58.0.3029.83 Mobile ' 'Safari/537.36'}, 'initialPriority': 'VeryHigh', 'method': 'GET', 'mixedContentType': 'none', 'referrerPolicy': 'no-referrer-when-downgrade', 'url': 'https://www.baidu.com/'}, 'requestId': '9616.88', 'timestamp': 6117.672547, 'type': 'Document', 'wallTime': 1515665221.86887}} {'method': 'Network.responseReceived', 'params': {'frameId': '9616.1', 'loaderId': '9616.4', 'requestId': '9616.88', 'response': {'connectionId': 707, 'connectionReused': False, 'encodedDataLength': 875, 'fromDiskCache': False, 'fromServiceWorker': False, 'headers': {'Cache-Control': 'no-cache', 'Connection': 'Keep-Alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html;', 'Coremonitorno': '0', 'Date': 'Thu, 11 Jan 2018 10:07:03 GMT', 'Server': 'apache', 'Set-Cookie': 'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758; ' 'path=/; domain=.baidu.com\n' 'bd_traffictrace=111752_111807\n' 'BDSVRTM=278; path=/\n' 'eqid=deleted; path=/; ' 'domain=.baidu.com; ' 'expires=Thu, 01 Jan 1970 ' '00:00:00 GMT', 'Strict-Transport-Security': 'max-age=172800', 'Tracecode': '04233185700634468874011118\n' '04230518060407801866011118', 'Traceid': '151566522306294144108392574322183443433', 'Transfer-Encoding': 'chunked', 'Vary': 'Accept-Encoding'}, 'headersText': 'HTTP/1.1 200 OK\r\n' 'Cache-Control: no-cache\r\n' 'Connection: Keep-Alive\r\n' 'Content-Encoding: gzip\r\n' 'Content-Type: ' 'text/html;charset=utf-8\r\n' 'Coremonitorno: 0\r\n' 'Date: Thu, 11 Jan 2018 10:07:03 ' 'GMT\r\n' 'Server: apache\r\n' 'Set-Cookie: ' 'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758; ' 'path=/; domain=.baidu.com\r\n' 'Set-Cookie: ' 'bd_traffictrace=111752_111807\r\n' 'Set-Cookie: BDSVRTM=278; path=/\r\n' 'Set-Cookie: eqid=deleted; path=/; ' 'domain=.baidu.com; expires=Thu, 01 ' 'Jan 1970 00:00:00 GMT\r\n' 'Strict-Transport-Security: ' 'max-age=172800\r\n' 'Tracecode: ' '04233185700634468874011118\r\n' 'Tracecode: ' '04230518060407801866011118\r\n' 'Traceid: ' '151566522306294144108392574322183443433\r\n' 'Vary: Accept-Encoding\r\n' 'Transfer-Encoding: chunked\r\n' '\r\n', 'mimeType': 'text/html', 'protocol': 'http/1.1', 'remoteIPAddress': '61.135.169.125', 'remotePort': 443, 'requestHeaders': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, ' 'sdch, br', 'Accept-Language': 'zh-CN,zh;q=0.8', 'Cache-Control': 'max-age=0', 'Connection': 'keep-alive', 'Cookie': 'BAIDUID=21B2249D2761E4A9CE64A356CFA0A21D:FG=1; ' 'bd_traffictrace=111752; ' 'BDSVRTM=22; ' 'plus_lsv=bddd969db4a4e207; ' 'BDORZ=AE84CDB3A529C0F8A2B9DCDD1D18B695; ' 'plus_cv=1::m:21732389; ' 'Hm_lvt_12423ecbc0e2ca965d84259063d35238=1515664352; ' 'Hm_lpvt_12423ecbc0e2ca965d84259063d35238=1515664352; ' 'H_WISE_SIDS=102206_116791_102629_121253_120157_118886_118865_118852_118825_118802_107316_121254_121535_121214_117333_117431_121666_120590_121563_121044_121421_120944_121042_121363_121153_114819_120550_120852_119324_121325_116408_110085_118758', 'Host': 'www.baidu.com', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Linux; ' 'Android 4.4.4; ' 'SM-A7000 ' 'Build/KTU84P) ' 'AppleWebKit/537.36 ' '(KHTML, like Gecko) ' 'Chrome/58.0.3029.83 ' 'Mobile ' 'Safari/537.36'}, 'securityState': 'secure', 'status': 200, 'statusText': 'OK', 'timing': {'connectEnd': 352.51100000005, 'connectStart': 0, 'dnsEnd': 0, 'dnsStart': 0, 'proxyEnd': -1, 'proxyStart': -1, 'pushEnd': 0, 'pushStart': 0, 'receiveHeadersEnd': 1468.39399999953, 'requestTime': 6117.724403, 'sendEnd': 1062.45399999989, 'sendStart': 1061.98099999983, 'sslEnd': 352.458999999726, 'sslStart': 107.189999999719, 'workerReady': -1, 'workerStart': -1}, 'url': 'https://www.baidu.com/'}, 'timestamp': 6119.205715, 'type': 'Document'}} {'method': 'Network.dataReceived', 'params': {'dataLength': 7150, 'encodedDataLength': 0, 'requestId': '9616.88', 'timestamp': 6119.209894}} {'method': 'Network.dataReceived', 'params': {'dataLength': 32768, 'encodedDataLength': 3221, 'requestId': '9616.88', 'timestamp': 6119.256851}} {'method': 'Network.loadingFinished', 'params': {'encodedDataLength': 42585, 'requestId': '9616.88', 'timestamp': 6119.430418}}其中我们所需要的信息都在Network.responseReceived里面，有url、timing、remoteIPAddress、status状态码。其中这个encodedDataLength在response里面仅有875，感觉不太对，这个如何获取请求的大小比较复杂，不同的请求，获取方法不太一样。这个还没研究出如何获取。 如何获取网页使用的内存信息 参考文献：https://sites.google.com/a/chromium.org/chromedriver/logging/performance-log https://chromedevtools.github.io/devtools-protocol/ https://stackoverflow.com/questions/27596229/browser-performance-tests-through-selenium/ https://github.com/webdriverio/webdriverio/issues/476 https://stackoverflow.com/questions/27657655/unable-to-get-performance-logs https://groups.google.com/forum/#!topic/google-chrome-developer-tools/FCCV2J7BaIY https://chromedevtools.github.io/devtools-protocol/tot/Network/ |
今日新闻 |
推荐新闻 |
| CopyRight 2018-2019 办公设备维修网 版权所有 豫ICP备15022753号-3 |