That’s a tough question. I’d guess more than 20 pounds loss before it starts getting obvious, but that depends on how everything fits. If they’re loose now, you have less to lose.
I’m 6’ even and at 50 pounds down, my pants are two sizes (4") smaller, from 42 to 38. My shirts went from XXL to XL for the comfort. I like my shirts a bit loose. I have some just marked Large that were are looser than others and feel fine to me.
Then there’s the thing that some companies just put different sizes on their clothes. I’m wearing 38 cargo shorts right now, but the Bass Pro shops fishing shorts are 36 and fall off if I’m not constantly pulling them up.