I actually hadn't considered that movement too much, so thank you for making a dark video so that I could see the light!
On a more serious note. So in that vid, only the link mount is flexing? Are the rod ends rotating on the rod ball at all causing some movement?
I think I may have a solution that doesn't involve links though. How about doubling up the tower, even just the lower portion with the link mount. It'll take some dremeling and 4 longer screw at the base of the tower, but I don't see why it can't be done. With the link mount doubled, a Maxx RPM arm (stock .15 used in pics) could be used with a small spacer to adjust the caster.
Only 1/2 the arm would "mount" to the link, but you'd do away with the rod end's movement, still have some caster adjustability and mate up to the pillow ball nicely. Using a stock "wide" arm instead of RPM would mean more spacers (more caster adjustability though) and less strength when you cut away the unused rear portion of the arm for chassis clearance.
I have no idea whether or not the shock or driveshaft will clear the arm since I don't have your setup, but maybe you could work with this or it'll give you another idea.