First off, the people who use that kind of counter argument are using (maybe not consciously) an old debate strategy called diversion. Or what is popularly known as Whataboutism.
For instance I would say, there are 29 million cows killed each year for food.
and they would say, Well, what about the 6 million refugees from Syria.
Its a very effective strategy in debate. although in my oppion it is sort of cheating. It's also called a logical fallacy.
I'm not sure there is a good defense for it. I mean if you attack it head on you end up taking about whatever it is they wanted to talk about. Although it might be worth trying a, "well what about it?"
Maybe a shrug or eye roll followed by the rest of your argument can work too.
As far as choosing ethical apparel, its certainly a good idea to look for apparel that has the FreeTrade sticker.
You can also google "ethical apparel" and/or "sustainable clothing".
Veganism is all about compassion - its part of the vegan psych to avoid or minimize cruelty to living things - including humans.
It can be a little complicated and time consuming to get into this but there are websites that can help. For me it becomes sort of supporting the "good" brands and boycotting the "worst" brands".
Use our guide to find Fair Trade Certified™ factory-made clothing, so you can make an ethical fashion statement from head to toe.
www.fairtradecertified.org
BTW, My two favorite food groups, chocolate and coffee have some of the worst practices in regards to human rights. Look for FairTrade stickers on coffee and chocolate, too.