Just a thought but could you calculate the distance between the button and the pointer then increase the a scaling speed the further away you go. Perhaps not linear scale but something that accelerates faster the further you are from the button allowing fine grained control and quick large scale changes.

Possibly a bad idea this is what happens when I think right before going to bed.