I feel like I take damage too often from failed checks

This happened a few times now and it's frankly annoying. Why would a bunch of small stones (as clearly described by the DM) destabilizing cause me to take any damage whatsoever. It makes no sense. Making a character lose HP is the cheapest way to cause consequence, and it’s made even worse by the lack of logic behind the decision.