Be kind to your ears, listen quietly

Ear buds (left) are very different than in-ear headphones (right). Steve Guttenberg/CNET

Give it a try, turn the volume down a little, and once you get used to listening quietly, turn it down a little more. Granted, quiet listening works best in quiet places; in noisy environments stick with in-ear, closed-back, or noise-canceling headphones. Avoid ear buds and open-back headphones, they don't hush external noise so you have to play music a lot louder than you might realize.
If you do the bulk of your listening in noisy places, continuing with ear buds (the type that come with phones) may eventually lead to hearing loss from continued exposure over a long period of time to excessively loud sound. I covered how ear buds, in-ear, and closed- and open-back headphones work and how they differ on previous blogs.
If you have to listen in noisy places or while commuting, consider buying in-ear or closed-back full-size headphones to seal out noise. When you reduce the background noise level competing with the music, you can turn the music's volume way down, and the difference can be very significant. Even inexpensive closed-back or in-ear headphones will help you listen more quietly.
I find with the better-sounding in-ear and closed-back headphones I can listen at a much lower volume and still not feel like I'm losing detail or the music's energy. Quiet listening draws me in more, so I listen more attentively. Once you get used to listening quietly it will become the new norm, and your ears will suffer less listening fatigue.
Noise-canceling headphones block more noise than any other type of headphone, so you can turn the music down even more, but most noise-canceling models don't sound as good playing music as equivalently priced closed-back headphones.
Custom-molded ear canal headphones may not seal out noise as well as the noise-cancelers, but they sound a lot better. Custom in-ear headphones are more expensive than noise-canceling models, and before you buy a custom-molded headphone, you need to visit an audiologist to have them take "impressions" of your ear canals. It's a quick procedure and not very expensive.
Loud sound from your home speakers is another matter, but again, with better ones you won't feel like you're missing detail when you turn the volume dow