This is a common diversionary tactic in meat eaters who would like to throw red herrings at you instead of have a rational conversation like an adult. The type usually also offers such gems as "but what about all of the rodents being killed when you eat soy or grains" and "there is no way to be alive and not kill anything!"
Well, no ****, Sherlock.
Veganism proposes to do the least harm. One time this semester during an outdoor lab on ruderal species, I accidentally murdered a snail, and I felt terrible, I just stepped on the poor thing. One of my friends/classmates even commented "wow the vegan is the one who killed the snail." He meant it as a gentle joke, but I really felt TERRIBLE.
I'm one of those people who allows skeeter eaters to crawl on me when I'm sunning myself on the grass on the campus lawn, and who tries to capture insects and worms and put them outdoors (unless they're spiders near my bed, then I'm sorry, it's over, guys).
People who talk about wasps in fig trees or about animals incidentally harmed in avocado or almond production don't even want to begin to calculate the mass death involved in doing nothing more than buying the cheapest animal products at a chain grocery store, nor do they want to think about how much more water in California is diverted to cattle than to almond production. In fact when someone is still talking about almonds, I literally have to grit my teeth.
Are there legitimate concerns? Oh sure. Legitimate concerns include things like if your palm oil was sustainably sourced (this is a real problem with deforestation and is also affecting primates not rodents or insects) and if you're recycling your packaging as a vegan. I'm honestly troubled by vegans continuing to drink single-use bottled waters instead of tap water (Brita filters work just fine in places like LA or the Bay Area, though I can drink straight from the tap in NorCal) because it's one of those things that is overlooked. I'm not one to be giving up my Tofurky any time soon (it's coming from Oregon anyway I'm not really disturbed by that distance) but on the other hand I'm not really comfortable with drinking wine from Australia as I once was.
The grad student selected to be my "peer mentor" next semester is actually looking at the sustainability of wine in California. It's a real eye-opener to me as someone who enjoys wine. I principally try to drink California wines across the board (not difficult, many of the cheapest table reds are CA wines) but didn't know how much damage and water use is affecting places such as Napa. Does this have anything to do with veganism? Why yes it does. Wildlife are animals too, and I've slowly but surely become more and more aware of water.
We can do what we can. Usually the people who will call you out with things like figs are people still eating cow flesh sandwiches from mega-corp fast food chains. They don't deserve your energy.