五、高性能异步


返回

5.1 多进程、多线程

  • 优点:可以为相关阻塞的操作单独开启线程或者进程,阻塞操作可以异步执行。
  • 缺点:无法无限制的开启多线程或者多进程。

5.2 进程池、线程池

  • 优点:可以降低系统对进程或者线程创建和销毁的频率,从而很好的降低系统的开销。

  • 缺点:池中线程或进程的数量是有上限。

    from multiprocessing.dummy import Pool
    
    
    def get_video_data(dic):
        url = dic['url']
        name = dic['name']
        data = requests.get(url=url, headers=headers).content
        with open(name, 'wb') as fp:
            fp.write(data)
        
    
    dic = {
        'name': name,
        'url': video_url
    }
    urls.append(dic)
    
    # 使用线程池对视频数据进行请求(较为耗时的阻塞操作)
    pool = Pool(4)
    pool.map(get_video_data, urls)
    
    pool.close()
    pool.join()
    
    
    from concurrent.futures import ThreadPoolExecutor
    
    
    def get_video_data(dic):
        url = dic['url']
        name = dic['name']
        data = requests.get(url=url, headers=headers).content
        with open(name, 'wb') as fp:
            fp.write(data)
        
    
    dic = {
        'name': name,
        'url': video_url
    }
    urls.append(dic)
    
    # 实例化一个线程池对象
    pool = ThreadPoolExecutor(4)  
    for dic in urls:
    		pool.submit(get_video_data, dic) 
        
    # 等所有任务完成,再关闭线程池
    pool.shutdown(wait=True)
    
    

5.3 单线程+异步协程

返回