Paper:https://arxiv.org/abs/2010.15821
GitHub (NNI):https://github.com/microsoft/nni/blob/master/docs/en_US/NAS/Cream.rst
GitHub:https://github.com/microsoft/Cream
Cream基本原理
-
One-shot NAS:
- 搜索过程中,设置Prioritized Board,收集精度与速度满足要求的最佳Sub-networks,并按末尾淘汰制更新Board;
- 隔一定训练周期,在Sub-network更新之后,基于Validation-set、在线更新Meta-network;
- 搜索阶段,每次随机采样一个Sub-network,并通过Meta-network选择最佳匹配的Prioritized network、作为Teacher,实现在线蒸馏;
-
原理框图:
![](https://img-blog.csdnimg.cn/20210226164157160.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25hdHVyZTU1Mzg2Mw==,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20210226164355243.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25hdHVyZTU1Mzg2Mw==,size_16,color_FFFFFF,t_70)
- 随机采样Sub-network,并匹配Prioritized network:
![](https://img-blog.csdnimg.cn/20210226164636943.png)
- 训练更新Sub-network:
![](https://img-blog.csdnimg.cn/20210226164657982.png)
- 末尾淘汰制更新Board:
![](https://img-blog.csdnimg.cn/20210226164729612.png)
- 更新Meta-network:
![](https://img-blog.csdnimg.cn/20210226164749523.png)
实验结果
![](https://img-blog.csdnimg.cn/20210226164944756.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25hdHVyZTU1Mzg2Mw==,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20210226165025524.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25hdHVyZTU1Mzg2Mw==,size_16,color_FFFFFF,t_70)
NNI API
-
NNI作为相对成熟的AutoML、模型压缩框架,具备成熟的标准组件、与框架范式:
-
基本结构:
![](https://img-blog.csdnimg.cn/20210226165206743.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L25hdHVyZTU1Mzg2Mw==,size_16,color_FFFFFF,t_70)
-
NAS基本组件:
-
Mutable:基本搜索单元(LayerChoice、InputChoice),用以构建Diff/One-shot NAS的Super-network,实现可微分或离散搜索;
-
Tunner:AutoML tunner,为下一次trial生成优化配置;
-
Mutator:子网络采样器,负责随机采样、DARTS采样、RL采样等;