前言

今天来教大家如何爬取知乎网站，最近到底有哪些有意思的事情发生呢？

不过爬取网站首先会面临的问题，当然是登陆了。

准备

使用requests包来爬取。首先尝试用用户名密码自动登陆，如果失败，则需要采用cookie登陆。

配置文件config.ini，其中包括用户名密码信息，如果有验证码情况，需要手动登陆一次网站获取cookie信息。

判断登陆成功与否，看生成的html文件中有没有用户信息。

代码实现

# -*- coding: utf-8 -*-import requestsimport ConfigParserdef create_session():cf = ConfigParser.ConfigParser()cf.read('config.ini')cookies = cf.items('cookies')cookies = dict(cookies)from pprint import pprintpprint(cookies)email = cf.get('info', 'email')password = cf.get('info', 'password')session = requests.session()login_data = {'email': email, 'password': password}header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36','Host': '','Referer': '/'}r = session.post('/login/email', data=login_data, headers=header)if r.json()['r'] == 1:print 'Login Failed, reason is:',for m in r.json()['data']:print r.json()['data'][m]print 'So we use cookies to login in...'has_cookies = Falsefor key in cookies:if key != '__name__' and cookies[key] != '':has_cookies = Truebreakif has_cookies is False:raise ValueError('请填写config.ini文件中的cookies项.')else:# r = requests.get('/login/email', cookies=cookies) # 实现验证码登陆r = session.get('/login/email', cookies=cookies) # 实现验证码登陆with open('login.html', 'w') as fp:fp.write(r.content)return session, cookiesif __name__ == '__main__':requests_session, requests_cookies = create_session()# url = '/login/email'url = '/topic/19552832'# content = requests_session.get(url).content # 未登陆# content = requests.get(url, cookies=requests_cookies).content # 已登陆content = requests_session.get(url, cookies=requests_cookies).content # 已登陆with open('url.html', 'w') as fp:fp.write(content)

最后

这就先分享到这里，感觉有意思的，可以关注一下，更多有趣的文章持续更新中.......