I recently found that I was lacking this feature in my Xubuntu 12.04. I remember once having this on by default in KDE, so I wanted to find out how to do this in Xfce.
So this is what you need to do to enable background sccrolling in Xfce (in a terminal):
Another way is to open Xfce Setting editor and navigate to
xfwm4 -> general -> raise_with_any_button -> false