Shouldn't be here? Shouldn't be at all? Should leave them alone? So ambiguous.

If you'd rather I left you alone, that's fine.

It's not like he can tell they put their phone away. He can wait. Other conversations are happening--he's no longer forcing himself to be completely cut off from the world.