Casey answered in one of Q&A about reason not to use references. Don't remember which one.
If I remember correctly, the references doesn't bring any advantage over pointers. They just limit you (no way to pass NULL). And if you later decide you want to allow pass NULL, you'll need to change all the . to -> and that is annoying.
As for not using references in "operator *" there are multiple good reasons.
One, as you wrote yourself, is that method is inline - compiler will inline it and there won't need to take address of B at all.
Second one is that you should pass small objects (let's say size < 16) by value, not by reference. By passing them by reference you are making caller to spill their value on stack so you can take address of it. Because often value of variable will be available only in registers that could lead to to performance penalty. For some architectures (x86_64, arm, fastcall on x86) first few arguments are passed directly in registers. So if your value is in regster and function accepts it in register, there is no need to put it on stack and pass an address.
Another one is that by using references you are making compiler optimizer life harder. It can optimize value semantics much easier. By references you are restricting what optimizations it can do. For more information on this topic read up these two presentations by Chandler Carruth:
* C++Now 2013 keynote - "
Optimizing the Emergent Structures of C++"
slides,
video, first ~39minutes goes very deep into LLVM optimizer, but easier to understand stuff starts at ~39minute and goes till end - watch it all
* from LLVM devmeeting - "
Zero-Cost Abstractions and Future Directions for Modern Optimizing Compilers":
slides,
video