Coulson thinks it is fair since he himself was tricked into babysitting after he (in a moment of weakness he should never have made) demonstrated his competence at child-rearing by charming young Stark into keeping quiet while Daddy and Mommy were off fighting evil.