Ok. Two people, I will start work on this now.
If a maintainer says its not useful, I can point them here.
For the algorithm I was thinking a simple blur but that distorts towards the average which is not what we want really. So a real relaxation kind of thing, where each displacement point moves towards its neighbours or so? Not sure yet I need to experiment with it to see what I expect from it in every situation.
edit:
I tested blur (tested = wrote down numbers on paper and calculated what “blur” would do) and it sucks as expected. One important point about this whole thing is that we don’t really want the “relaxing region/brush” to move things on average. And yea, blur does that, but it also creates this blackhole effect. If my x coordinates of my points are for example 20,30,40, then the average is 30. In the most extreme case blur will approach exactly that. But we don’t want all the coordinates to get pulled towards 30! That would act a bit like the “scale: shrink” mode of liquify, and that’s not what we want at all.
So we actually want some sort of “neighbour comparative thing”. In fact, what we want is to preserve affine global transformations (as far as we can recognize them), and just undo any higher frequency mess.
Example:
Draw rectangle, enter liquify, in move mode with a huge brush move the rectnagle somewhere else (thanks to large brush size it should be barely distorted, just moved). Then change brush mode to “rotate”, again with a huge brush size rotate entire rectangle by 90deg or so. Now finally make some edits with any liquify deform mode that use a small brush size but strong effect. When I now apply relax to this, I would expect the overall large form not to change, and all the small details getting slowly lost / blended away as I keep applying the relax brush…
What I do not want is that any points try to move back to their original position, or points overall trying to rotate back to their original global rotation.
At least that is what I think about when I think “relax mode in liquify tool”. If anyone else has any other definitions of what relax means to them let me know. I’ll do some research on how to infer any existing affine transforms from points; could be pretty simple (just global average?) but maybe there is something I haven’t thought about…