最近在研究爬虫,需要在前面部署IP代理池,于是在开源中国找到代理池。可以自动抓取国内几个免费IP代理网站的IP,并实时校验IP的可用性、数据库为SSDB。
IP代理池网站:
http://www.data5u.com/
http://www.data5u.com/free/
http://www.data5u.com/free/gngn/index.shtml
http://www.data5u.com/free/gnpt/index.shtml
http://www.66ip.cn/
http://www.ip181.com/
http://www.xicidaili.com/nn
http://www.xicidaili.com/nt http://www.xdaili.cn/ipagent/freeip/getFreeIps?page=1&rows=10 http://www.goubanjia.com/free/gngn/index.shtml
2 yum - y安装git
3 yum安装wget - y
4 yum安装curl-devel expat-devel gettext-devel openssl-devel zlib-devel
5 yum安装gcc perl-ExtUtils-MakeMaker epel-release gcc-c + +
8 cd/usr/src/
16 wget https://www.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz安装克隆工具git
17焦油-xzf git-2.9.5.tar。广州
18 cd git-2.9.5
19使prefix=/usr/地方/git
20使prefix=/usr/地方/git安装
21回声“出口路径=$路径:/usr/地方/git/bin"在比;/etc/bashrc
22源/etc/bashrc
23 cd . .
24 git克隆https://github.com/jhao104/proxy_pool.git克隆proxy_pool
25 cd proxy_pool/python - v
26 #查看python版本2.7.5
27 yum - y安装python34 #安装python 3.4
28 wget - no-check-certificate https://bootstrap.pypa.io/get-pip。py
29 python3 get-pip。py #安装pip
30 pip安装- r的要求。txt # proxy_pool的安装依赖包
32 cd/usr/local/
33 git克隆https://github.com/ideawu/ssdb.git克隆SSDB
34 cd SSDB
35 yum - y安装autoconf
37 cd deps snappy-1.1.0/#编译时髦
38 ./configure
39使
40 cd/usr/local/ssdb
41让#安装SSDB
42 make install
43 ln科幻/usr/local/ssdb/ssdb-server/usr/local/bin/ssdb-server
44 ln科幻/usr/local/ssdb/tools/ssdb-cli/usr/local/bin/ssdb-cli
45 ln科幻/usr/local/ssdb/tools/ssdb-dump/usr/local/bin/ssdb-dump
46 ln科幻/usr/local/ssdb/tools/ssdb-repair/usr/local/bin/ssdb-repair
47 ln科幻/usr/local/ssdb/tools/ssdb.sh/etc/rc.d/init.d/ssdb
48 chkconfig——添加ssdb
49 chkconfig ssdb alt=" centos7生产环境IP代理池(python) ">
具体使用请参考https://github.com/jhao104/proxy_pool
本文提供cenots7下的部署详情,在此感谢贡献者和j_hao104的无私奉献!