[R] Compressing vision-language and unimodal Transformers via structured pruning


🚀 Code: https://github.com/sdc17/UPop

📑 Paper: https://proceedings.mlr.press/v202/shi23e/shi23e.pdf

🧐 A Quick Look

  • What is it: UPop is the first structured pruning framework for vision-language Transformers. It enables effective structured pruning on various multi-modal & uni-modal tasks (including Visual Reasoning, Image Captioning, Visual Question Answer, Image-Text Retrieval, Text-Image Retrieval, Image Classification and Image Segmentation), datasets (including NLVR2, COCO Caption, VQAv2, COCO, Flickr30K, ImageNet and ADE20K), and model architectures (including BLIP, CLIP, DeiT and Segmenter).


  • What challenge does it tackle: The above figure demonstrates that Unified Search adopted by UPop rescues us from the burden of repeated experiments (e.g., doing grid search) for searching optimal compression ratios among different modalities and structures. Furthermore, Progressive Pruning adopted by UPop eliminates the weight gap between the searched model and the pruned subnet to be retrained, therefore gaining better convergence and performance, especially at high compression ratios.
  • How about the performance: On multimodal tasks, for example, UPop can achieve 2x compression with only 1.2% and 2.0% accuracy loss on the VQAv2 dataset for Visual Question Answer and the NLVR2 dataset for Visual Reasoning, respectively. On unimodal tasks, for example, UPop can achieve 1.5x and 1.2x compression without any loss of accuracy on the ImageNet dataset for Image Classification and the ADE20K dataset for Image Segmentation, respectively. Some examples of vector-level structured granularity are as follows.


submitted by /u/Salty-Situation2606

Source link


Please enter your comment!
Please enter your name here

Share post:




More like this

 PAW Patrol: The Mighty Movie – Watch in Dubai

PAW Patrol: The Mighty Movie is a 2023 American...

Watch Kannur Squad in Dubai – A Gripping Saga

Kannur Squad is a 2023 Malayalam-language action thriller film...

Ohio State vs Notre Dame – Excitement is in the Air

In one of the most anticipated games of the...

Reliable Postal and Courier Services Dubai – Fast & Secure Delivery

When it comes to sending and receiving packages and...