Paper summary · Aug 2026
Gradient Surgery for Multi-Task Learning
The main contribution here - a model-agnostic form of gradient surgery that project a task’s gradient onto the normal plane of the gradient of any other task that has a conflicting gradient in multi-task setting. The authors show that in multi-task learning enbironments the gradients from the different tasks can often be 1) conflicting, 2)with high positive curvature and 3) have large differences in gradient magnitudes. This triad nullifies the optimizer’s ability of reaching the joint valley and optimizing both. The authors propose that when two gradients are conflicting, one should alter them by projecting each onto the normal plane of the other, preventing intergering components. This is referred to projecting conflicting gradients or PCGrad.
PCGrad
Definitions
Definition 1. We define as the angle between two task gradients and . We define the gradients as conflicting when .
Definition 2. We define the gradient magnitude similarity between two gradients and as
When the magnitude of two gradients is the same, this value is equal to 1. As the gradient magnitudes become increasingly different, this value goes to zero.
Definition 3. We define multi-task curvature as
*which is the averaged curvature of between and in the direction of the multi-task gradient .
The Algorithm
Require: Model parameters , task minibatch
1: 2: 3: for do 4: for in random order do 5: if then 6: // Subtract the projection of onto 7: Set 8: return update