Exact-SVD EGD is extremely unstable in the p=97 transformer experiment. Replacing exact SVD with RSVD rank 36 makes training stable and, according to the paper, more stable than either Grokfast or exact-SVD EGD. · CiteArk
Not assessedNo independent reproduction scheduledLimitationc17-transformer-training-instability
Exact-SVD EGD is extremely unstable in the p=97 transformer experiment. Replacing exact SVD with RSVD rank 36 makes training stable and, according to the paper, more stable than either Grokfast or exact-SVD EGD.
Source: paper:PDF pp. 23-24, text after Table 7 and Figures 14-15
Reported and observed measurements
No structured measurement is attached to this Claim.
Assessments (0)
No immutable Assessment has been published for this Claim yet.
