Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation
Knowledge Distillation is an effective method to transfer the learning across deep neural networks. Typically, the dataset originally used for training the Teacher model is chosen as the "Transfer Set" to conduct the knowledge transfer to the Student. However, this original training data may not alw...
Guardat en:
| Autors principals: | , , |
|---|---|
| Format: | Article |
| Publicat: |
2020
|
| Matèries: | |
| Accés en línia: | https://arxiv.org/abs/2011.09113 http://arxiv.org/abs/2011.09113 |
| Etiquetes: |
Sense etiquetes, Sigues el primer a etiquetar aquest registre!
|
| _version_ | 1879172635731951617 |
|---|---|
| author | Nayak, Gaurav Kumar Mopuri, Konda Reddy Chakraborty, Anirban |
| author_browse | Chakraborty, Anirban Mopuri, Konda Reddy Nayak, Gaurav Kumar |
| author_facet | Nayak, Gaurav Kumar Mopuri, Konda Reddy Chakraborty, Anirban |
| author_sort | Nayak, Gaurav Kumar |
| building | arXiv |
| collection | Acceso abierto |
| description | Knowledge Distillation is an effective method to transfer the learning across deep neural networks. Typically, the dataset originally used for training the Teacher model is chosen as the "Transfer Set" to conduct the knowledge transfer to the Student. However, this original training data may not always be freely available due to privacy or sensitivity concerns. In such scenarios, existing approaches either iteratively compose a synthetic set representative of the original training dataset, one sample at a time or learn a generative model to compose such a transfer set. However, both these approaches involve complex optimization (GAN training or several backpropagation steps to synthesize one sample) and are often computationally expensive. In this paper, as a simple alternative, we investigate the effectiveness of "arbitrary transfer sets" such as random noise, publicly available synthetic, and natural datasets, all of which are completely unrelated to the original training dataset in terms of their visual or semantic contents. Through extensive experiments on multiple benchmark datasets such as MNIST, FMNIST, CIFAR-10 and CIFAR-100, we discover and validate surprising effectiveness of using arbitrary data to conduct knowledge distillation when this dataset is "target-class balanced". We believe that this important observation can potentially lead to designing baselines for the data-free knowledge distillation task. |
| first_indexed | 2026-10-15T23:44:29Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-2011.09113</identifier><datestamp>2020-11-19</datestamp>
<dc:title>Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation</dc:title>
<dc:creator>Nayak, Gaurav Kumar</dc:creator>
<dc:creator>Mopuri, Konda Reddy</dc:creator>
<dc:creator>Chakraborty, Anirban</dc:creator>
<dc:subject>Machine Learning</dc:subject>
<dc:subject>Computer Vision and Pattern Recognition</dc:subject>
<dc:description>Knowledge Distillation is an effective method to transfer the learning across deep neural networks. Typically, the dataset originally used for training the Teacher model is chosen as the "Transfer Set" to conduct the knowledge transfer to the Student. However, this original training data may not always be freely available due to privacy or sensitivity concerns. In such scenarios, existing approaches either iteratively compose a synthetic set representative of the original training dataset, one sample at a time or learn a generative model to compose such a transfer set. However, both these approaches involve complex optimization (GAN training or several backpropagation steps to synthesize one sample) and are often computationally expensive. In this paper, as a simple alternative, we investigate the effectiveness of "arbitrary transfer sets" such as random noise, publicly available synthetic, and natural datasets, all of which are completely unrelated to the original training dataset in terms of their visual or semantic contents. Through extensive experiments on multiple benchmark datasets such as MNIST, FMNIST, CIFAR-10 and CIFAR-100, we discover and validate surprising effectiveness of using arbitrary data to conduct knowledge distillation when this dataset is "target-class balanced". We believe that this important observation can potentially lead to designing baselines for the data-free knowledge distillation task. </dc:description>
<dc:description>Accepted in WACV 2021</dc:description>
<dc:date>2020-11-18</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/2011.09113</dc:identifier>
<dc:identifier>http://arxiv.org/abs/2011.09113</dc:identifier>
</oai_dc:dc> |
| id | arxiv-2011.09113 |
| institution | arXiv |
| last_indexed | 2026-10-15T23:44:29Z |
| publishDate | 2020 |
| publishDateRange | 2020 |
| publishDateSort | 2020 |
| record_format | oai_dc |
| spelling | arxiv-2011.091132020-11-19 Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation Nayak, Gaurav Kumar Mopuri, Konda Reddy Chakraborty, Anirban Machine Learning Computer Vision and Pattern Recognition Knowledge Distillation is an effective method to transfer the learning across deep neural networks. Typically, the dataset originally used for training the Teacher model is chosen as the "Transfer Set" to conduct the knowledge transfer to the Student. However, this original training data may not always be freely available due to privacy or sensitivity concerns. In such scenarios, existing approaches either iteratively compose a synthetic set representative of the original training dataset, one sample at a time or learn a generative model to compose such a transfer set. However, both these approaches involve complex optimization (GAN training or several backpropagation steps to synthesize one sample) and are often computationally expensive. In this paper, as a simple alternative, we investigate the effectiveness of "arbitrary transfer sets" such as random noise, publicly available synthetic, and natural datasets, all of which are completely unrelated to the original training dataset in terms of their visual or semantic contents. Through extensive experiments on multiple benchmark datasets such as MNIST, FMNIST, CIFAR-10 and CIFAR-100, we discover and validate surprising effectiveness of using arbitrary data to conduct knowledge distillation when this dataset is "target-class balanced". We believe that this important observation can potentially lead to designing baselines for the data-free knowledge distillation task. Accepted in WACV 2021 2020-11-18 text https://arxiv.org/abs/2011.09113 http://arxiv.org/abs/2011.09113 |
| spellingShingle | Machine Learning Computer Vision and Pattern Recognition Nayak, Gaurav Kumar Mopuri, Konda Reddy Chakraborty, Anirban Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title | Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title_full | Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title_fullStr | Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title_full_unstemmed | Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title_short | Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation |
| title_sort | effectiveness of arbitrary transfer sets for data free knowledge distillation |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2011.09113 http://arxiv.org/abs/2011.09113 |
| work_keys_str_mv | AT nayakgauravkumar effectivenessofarbitrarytransfersetsfordatafreeknowledgedistillation AT mopurikondareddy effectivenessofarbitrarytransfersetsfordatafreeknowledgedistillation AT chakrabortyanirban effectivenessofarbitrarytransfersetsfordatafreeknowledgedistillation |