Quick answer: You should avoid an excess of them if pregnant, breastfeeding, an infant or elderly.
Personally I tend to avoid them as our family leans toward an organic diet.
Nitrates/Nitrites tend to be more a problem with drinking water from farm fertilizer run off than from cured meats. The EPA has contamination levels set at 10mg/L for nitrates and 1mg/L for nitrites. Back when I participated in a south central PA town, we had to relegate one of the towns springs to fire department use only because of its high levels.