You’re allowed to make choices for yourself.
Be firm, but kind and just tell them you’ve cut out all sugar and refined grains from your diet for health reasons. If you’re in this for the long term, you’ll have to tell them some day. May as well get it over with now. There will likely be some kick back. Eating bad food brings with it a herd mentality. We justify bad food choices because “everyone is doing it”. They want you to eat it to justify their choices.
It gets easier over time. You can do it. You HAVE to do it.
Your wellbeing > other peoples feelings.