Most of my dress shirts right now are BR Grant slim fit medium size.
I'm 5'10, 175-180lbs. 40 chest, 33 waist.

It fits me pretty well, but I noticed that my chest area pulls a little bit when I stand/sit straight, just enough for me to notice.
Buttons mostly don't move, but the shirt skims my chest, and I see a few small-ish pulls around my chest.

The level of pull is something like this picture, but a bit more on chest, a bit less on stomach.

I think I'm in a decent shape, and happy about it, but I also don't want be perceived as one of those guys who show it off, especially in a professional setting.

Personally, I really like BR products, and would like to stay with their dress shirts. I have recently tried on their Camden standard fit on medium, and there is way too much fabric on the waist. Grant slim fit on large is also a bit too big overall.
I've been wanting to drop some weight to 165-170 range, that could solve this problem for all I know, but then I've also been doing more weight exercise, so who knows?

What do you guys think is a reasonable pulling without it looking a bit silly/unprofessional?

Thank you!