I'm not against wearing a tie under a cardigan, but i would opt for something more casual and textured (such as wool tie or knit) vs a sleeker silk tie.