The trick to this kind of anecdote is that they could have been there for two hours, explained that the grandfather had lived there before, and eventually someone said “oh, so he’s back?” and everyone laughed.

Change it just a little bit and there you go.