On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks - Archive ouverte HAL Access content directly
Journal Articles (Review Article) Neural Networks Year : 2023

On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks

(1) , (1) , (2) , (1)
1
2

Abstract

The paper proposes a new class of nonlinear operators and a dual learning paradigm where optimization jointly concerns both linear convolutional weights and the parameters of these nonlinear operators. The nonlinear class proposed to perform a rich functional representation is composed by functions called rectified parametric sigmoid units. This class is constructed to benefit from the advantages of both sigmoid and rectified linear unit functions, while rejecting their respective drawbacks. Moreover, the analytic form of this new neural class involves scale, shift and shape parameters to obtain a wide range of activation shapes, including the standard rectified linear unit as a limit case. Parameters of this neural transfer class are considered as learnable for the sake of discovering the complex shapes that can contribute to solving machine learning issues. Performance achieved by the joint learning of convolutional and rectified parametric sigmoid learnable parameters are shown to be outstanding in both shallow and deep learning frameworks. This class opens new prospects with respect to machine learning in the sense that main learnable parameters are attached not only to linear transformations, but also to a wide range of nonlinear operators.
Fichier principal
Vignette du fichier
DeepSigmoidCNN_V13_Final_AfterProofs.pdf (3.14 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03115681 , version 1 (22-01-2021)
hal-03115681 , version 2 (02-02-2021)
hal-03115681 , version 3 (12-01-2023)

Identifiers

Cite

Abdourrahmane Mahamane Atto, Sylvie Galichet, Dominique Pastor, Nicolas Méger. On Joint Parameterizations of Linear and Nonlinear Functionals in Neural Networks. Neural Networks, 2023, 160, pp.12-21. ⟨10.1016/j.neunet.2022.12.019⟩. ⟨hal-03115681v3⟩
61 View
24 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More