The "I'm afraid" is obvious. It is short for "I am afraid you will be annoyed/disappointed/angry that I have to say no."
The house on fire one is similar to "I'm on fire!".
I think Roger is just making moutains out of teacups.
Mince, I realise what "I'm afraid" is short for. What I'm questioning is the USE of the word "afraid". To talk about "fear" when you're politely turning down an invitation strikes me as both odd and extreme---but it has become part of our language. When asked if you're going to the pub and you reply "I'm afraid not" is, on analysis, a strange response.
And, regardless of what you say, I can see nothing positive in equating "a house on fire" with a healthy relationship. I'm pretty sure that, if you had coined that expression today, it would be assumed that a previously stable relationship was about to crash to the ground.
So there.