I agree that you'd be better off with a sports coat and no tie than with a tie and no sports coat.
Also, you can get a a sports coat so cheap at Macy's or Men's Wearhouse right now (see links below) that it's not a waste to get it for just one use--$30 to $50. They aren't the best, but better than nothing. You will feel good wearing it.
And you can occasionally wear it on a date or for a work meeting.
As far as a shirt, you mentioned that you have button down collar shirts? Wear one of those in white or blue with the gray slacks and you'll be fine.
https://www.macys.com/shop/mens-clot...O_HIGH?id=9559
https://www.menswearhouse.com/sport-...rn,brand,price