代码放在网盘里了,其中fetcher.py是主要模块:
http://pan.baidu.com/s/1pJMFpxDfetcher.py中distribute函数用于派发任务,也就是往协程池里添加协程,代码如下:
for url in task.urls:
proxy = self.getProxy()
print '[pid %d time %f] conPool size: %d distributing url: %s' %(os.getpid(), time.time(), len(self.pool), url)
self.pool.apply_async(self.fetch, args=(url, 'static', proxy))
理论上for循环能很快派发完任务,但是结果却不是这样的,1秒钟内只派发了几个任务:
[pid 5896 time 1429543816.844000] conPool size: 14 distributing url:
http://hao123.com/[pid 5896 time 1429543816.940000] conPool size: 15 distributing url:
http://zol.com.cn/[pid 5896 time 1429543817.035000] conPool size: 16 distributing url:
http://tudou.com/[pid 5896 time 1429543817.144000] conPool size: 17 distributing url:
http://cntv.cn/[pid 5896 time 1429543817.239000] conPool size: 18 distributing url:
http://china.com.cn/[pid 5896 time 1429543817.298000] conPool size: 19 distributing url:
http://southcn.com/[pid 5896 time 1429543817.304000] conPool size: 20 distributing url:
http://taobao.com/[pid 5896 time 1429543817.308000] conPool size: 21 distributing url:
http://douban.com/[pid 5896 time 1429543817.311000] conPool size: 22 distributing url:
http://58.com/[pid 5896 time 1429543817.315000] conPool size: 23 distributing url:
http://tianya.cn/[pid 5896 time 1429543817.361000] conPool size: 24 distributing url:
http://ganji.com/[pid 5896 time 1429543817.487000] conPool size: 25 distributing url:
http://pconline.com.cn/[pid 5896 time 1429543817.592000] conPool size: 26 distributing url:
http://eastmoney.com/[pid 5896 time 1429543817.646000] conPool size: 27 distributing url:
http://bitauto.com/[pid 5896 time 1429543818.759000] conPool size: 28 distributing url:
http://autohome.com.cn/[pid 5896 time 1429543818.778000] conPool size: 29 distributing url:
http://39.net/[pid 5896 time 1429543818.858000] conPool size: 30 distributing url:
http://iqiyi.com/[pid 5896 time 1429543818.949000] conPool size: 31 distributing url:
http://ctrip.com/用wing ide调试了一下,发现代码有时碰到print语句都会切换协程,很不理解。本人新手一个,代码写的很渣,求轻拍。