A diamond can be more vulnerable based on where you wear it. A ring-set diamond, for example, is bound to take a more severe beating than one on an earring. Moreover, the setting can influence the degree to which your diamond is exposed to knocks, with tension settings being the most unwise choice from that perspective. Bezels offer the best protection for the girdle and any pointy parts the stone may have based on its cut.
If you do get your diamond damaged somehow, there is the option of re-cutting it to offset the damage, usually without substantial weight loss. Some ring insurers would even replace the damaged diamond for you, although the chipped diamond would have to be given up.