Exact-SVD EGD is extremely unstable in the p=97 transformer experiment. Replacing exact SVD with RSVD rank 36 makes training stable and, according to the paper, more stable than either Grokfast or exact-SVD EGD. · CiteArk
Exact-SVD EGD is extremely unstable in the p=97 transformer experiment. Replacing exact SVD with RSVD rank 36 makes training stable and, according to the paper, more stable than either Grokfast or exact-SVD EGD.
来源：paper:PDF pp. 23-24, text after Table 7 and Figures 14-15
