Either the pivot ball inside the gearstick tower (white/cream plastic bit in the car) or the plastic bushes at the pivot under the car. Yanoo sells both separately. Stiff shift for in the tower and brass bush upgrade for the pivot.
I have a full fast road shifter iirc from yanoo, longer gearstick 30mm, stiff shift and harder pivot ball, quick shift pin, brass bushes. That's his biggestl kit that he does and replaces everything except the the linkage rod.
You have a quick shift pin by the sounds of it, the piece that's attached to the g/box.