前言
今天来教大家如何爬取知乎网站,最近到底有哪些有意思的事情发生呢?
不过爬取网站首先会面临的问题,当然是登陆了。
准备
使用requests包来爬取。首先尝试用用户名密码自动登陆,如果失败,则需要采用cookie登陆。
配置文件config.ini,其中包括用户名密码信息,如果有验证码情况,需要手动登陆一次网站获取cookie信息。
判断登陆成功与否,看生成的html文件中有没有用户信息。
代码实现
# -*- coding: utf-8 -*-import requestsimport ConfigParserdef create_session():cf = ConfigParser.ConfigParser()cf.read('config.ini')cookies = cf.items('cookies')cookies = dict(cookies)from pprint import pprintpprint(cookies)email = cf.get('info', 'email')password = cf.get('info', 'password')session = requests.session()login_data = {'email': email, 'password': password}header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36','Host': '','Referer': '/'}r = session.post('/login/email', data=login_data, headers=header)if r.json()['r'] == 1:print 'Login Failed, reason is:',for m in r.json()['data']:print r.json()['data'][m]print 'So we use cookies to login in...'has_cookies = Falsefor key in cookies:if key != '__name__' and cookies[key] != '':has_cookies = Truebreakif has_cookies is False:raise ValueError('请填写config.ini文件中的cookies项.')else:# r = requests.get('/login/email', cookies=cookies) # 实现验证码登陆r = session.get('/login/email', cookies=cookies) # 实现验证码登陆with open('login.html', 'w') as fp:fp.write(r.content)return session, cookiesif __name__ == '__main__':requests_session, requests_cookies = create_session()# url = '/login/email'url = '/topic/19552832'# content = requests_session.get(url).content # 未登陆# content = requests.get(url, cookies=requests_cookies).content # 已登陆content = requests_session.get(url, cookies=requests_cookies).content # 已登陆with open('url.html', 'w') as fp:fp.write(content)
最后
这就先分享到这里,感觉有意思的,可以关注一下,更多有趣的文章持续更新中.......