There are three sources of memory:
Static memory is allocated for you by the program loader -- all of your global variables and static data live here. Stack memory is allocated when a function is called, and then freed when that function returns. Finally, heap memory is allocated by asking the OS to give you access to additional memory.
All of these are *just memory* -- the performance of accessing memory is dictated by whether it's in the cache, not by where the memory was allocated.
Now, as to your questions:
1. Stack allocations are only valid within that function and the functions it calls. The next function call can (and will) overwrite anything left on the stack from a function that has returned. When you return a value from a function, that value gets copied onto the stack of the calling function. If it's a large value, that's a large copy. If I pass that directly as a parameter to another function, there's another copy.
// Copy result into foo
BigStruct foo = GetFoo();
// Copy into parameter list for ProcessFoo
The compiler can optimize this away if it can reason about it, but it can be surprisingly easy to confuse a compiler.
If instead we allocate the struct on the heap, all that needs to be copied around is a pointer:
BigStruct * foo = GetFoo(arena);
There's no magic here, by the way, the stack works the same whether the return value is a pointer or a struct, but by allocating the struct on the heap the copies are much smaller.
2. This is a very context-sensitive question. There are two reasons to avoid stack allocations: stack overflows, and return value / parameter copies.
To avoid stack overflow, consider how deep your callstack is going to get. Is this function recursive (directly or indirectly)? Is it mostly called from functions high up the callstack or those lower down (you can't always determine this)?
For optimizing copies, consider the tradeoff between copying the value around versus a potentially cold pointer dereference. How big is your value? Does it fit in one or two registers? Test it. What code does the compiler produce for particular struct sizes?
Most egregious errors here are common sense. Allocating a kilobyte buffer on the stack in a recursive function is probably a bad idea. Passing 80 byte structures by value is probably not good. Single integers and floats fit just fine on the stack, etc.