๐Ÿ“–
Asymmetrical Scaling Layers for Stable Network Pruning
Pierre Wolinski, Guillaume Charpiat, Yann Ollivier

Asymmetrical Scaling Layers for Stable Network Pruning

Science & Mathematics Published 2020 Verification Pending

Overview & Synopsis

We propose a new training setup, called ScaLa, and a new pruning algorithm based on it, called ScaLP. The new training setup ScaLa is designed to make standard Stochastic Gradient Descent resilient to layer width changes. It consists in adding a fixed well-chosen scaling layer before each linear or convolutional layer. This results in an overall learning behavior that is more independent of the layer widths, especially with respect to optimal learning rates, which stay close to 1. Beyond the usual choice of scaling each input by the factor 1/ โˆš #fan-in, we also propose a family of asymmetric scaling factors: this promotes learning some neurons faster than others. The pruning algorithm ScaLP is a combination of ScaLa with asymmetrical scaling, and weight penalties. With ScaLP, the nal pruned architecture is roughly independent of the layer widths in the initial network.

Indexed from repository: HAL Open Science (hal.science)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title Asymmetrical Scaling Layers for Stable Network Pruning
Author / Organization Pierre Wolinski, Guillaume Charpiat, Yann Ollivier
Subject & Discipline Multidisciplinary Academic Open Research
Categories Science & Mathematics
Publication Year 2020
License / Rights Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website HAL Open Science (hal.science)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-242689
Original Source Link Visit Source Page on hal.science »