There's a moment in a lot of standups that nobody names out loud. The team is underwater, everyone knows it, and when planning poker comes around a story that's honestly a three gets called a five. Nobody argues. Not because anyone's lying. Because the number has quietly stopped being an estimate and turned into a shield.
I've watched that happen on teams I was supposed to be protecting. The velocity chart looked great the entire time. That's the part that should bother you.