5.1 多进程、多线程
- 优点:可以为相关阻塞的操作单独开启线程或者进程,阻塞操作可以异步执行。
- 缺点:无法无限制的开启多线程或者多进程。
5.2 进程池、线程池
-
优点:可以降低系统对进程或者线程创建和销毁的频率,从而很好的降低系统的开销。
-
缺点:池中线程或进程的数量是有上限。
from multiprocessing.dummy import Pool def get_video_data(dic): url = dic['url'] name = dic['name'] data = requests.get(url=url, headers=headers).content with open(name, 'wb') as fp: fp.write(data) dic = { 'name': name, 'url': video_url } urls.append(dic) # 使用线程池对视频数据进行请求(较为耗时的阻塞操作) pool = Pool(4) pool.map(get_video_data, urls) pool.close() pool.join()
from concurrent.futures import ThreadPoolExecutor def get_video_data(dic): url = dic['url'] name = dic['name'] data = requests.get(url=url, headers=headers).content with open(name, 'wb') as fp: fp.write(data) dic = { 'name': name, 'url': video_url } urls.append(dic) # 实例化一个线程池对象 pool = ThreadPoolExecutor(4) for dic in urls: pool.submit(get_video_data, dic) # 等所有任务完成,再关闭线程池 pool.shutdown(wait=True)
5.3 单线程+异步协程
- 详情见:六、异步协程