I'd say if one isn't too excited about shooting a buck that passes by on the first day they get out to hunt, then it shouldn't bother them later in the season .That's just me. It wouldn't bother me because I like to see bucks have a chance to grow. If a small buck walks by my stand during the season and I give him a pass, there is a decent chance he'll be in the same area again next year, only bigger. If someone else takes him, good for them! That's hunting and unless you own 1000 acres you have no control over that part. If someone wants the meat and takes the deer I pass, that is fine with me.
Bottom line : shoot what you want and enjoy it.