Loading page…
EGD transforms a layer gradient G as (GG^T)^(-1/2)G (using pseudoinverses when rank deficient), preserving its singular vectors while setting every retained singular value to 1; the paper describes this as a whitened form of natural gradient descent. · CiteArk