Wearing a flag (or even a flag-printed shirt/pants) is not patriotism, it’s nationalism and jingoism. It’s also disrespectful of the flag per the US Flag Code (4 US Code § 8). If you don’t like being criticized for being disrespectful, you have it in your control to stop wearing flag clothing.