基于Python的反反爬虫技术分析与应用

余豪士　匡芳君

摘 要：爬虫软件是现今互联网环境下，高效准确地获取数据的重要方式之一。针对传统的初级爬虫技术易于被目标网站拦截访问的问题，简述爬虫的工作原理和方式，讨论爬虫、反爬虫与反反爬虫之间的相互关系。分析应对目标网站的反反爬虫机制，包括伪装用户代理，设置IP地址代理、使用自动化测试工具调用浏览器等技术要点，并分析了基于Python语言中Requests库，构建了对网页的多种请求方式和数据获取方法的解决方案。结合反反爬虫机制与数据分析技术，以哔哩哔哩视频网为案例，分析其网页基本结构与调用的应用程序接口，使用Python与Requests库抓取网站所有视频的相关数据。数据清洗后分析播放量最高视频的相关信息，并将结论以数据可视化的方式呈现，实现对数据的获取、挖掘与分析。

关键词：网络爬虫； 反爬虫； 反反爬虫； 大数据； 数据分析

Abstract： Crawler software is one of the most important ways to obtain data effectively and accurately in the current Internet environment. In view of the traditional crawler technology which is prone to be intercepted by target website， the paper explains how the crawler appears to work， discusses about the relationship between crawler， anti-crawler and anti-anti-crawler， and analyzes the mechanism of anti-anti-crawler for the target website， including fake user agents， setting IP proxy address， calling browser using automated testing tools. Furthermore， multiple requests and data acquisition methods for web pages are built based on Requests Library in Python language and its solution is analyzed. Combined with the mechanism of anti-anti crawler and data analysis technology， the paper takes the Bilibili website as a case， analyzing its basic structure， as well as its API called. On the one side， all relevant data of video on the Bilibili website is captured using Python and Requests Library and the related information of the video， in which the highest click rate is analyzed after data cleaning. On the other side， the conclusion is presented in the way of data visualization， and the data acquisition， mining and analysis are also realized.

Key words： Web crawler； anti-crawler； anti-anti-crawler technology； big data； data analysis

引言

大數据时代下的数据来源和获取尤为重要[1]，爬虫技术作为一项获取数据的工具而被广泛应用。已超过60%的互联网流量来自爬虫（Spider），各大搜索引擎门户网站以及新闻网站的文章都与爬虫息息相关。爬虫技术已成为当今的研究热点，目标网站对爬虫软件所做的各方面防范，给出了不同的拦截方式[2]。开发者与开发者之间通过爬虫、反爬虫、反反爬虫技术进行较量，一方面开发者想通过爬虫脚本获取数据，另一方面开发者又想拦截爬虫，防止爬虫脚本妨碍本网站的正常运营，对正常用户的访问造成了负面影响。……