I find it difficult to do as well. I think part of it is poor range of motion, but I don't sweat it too much. Typically I follow the rule that if it hurts to do that.... then stop doing that.
If it's a flexibility thing, you could try increasing your range of motion by doing that exercise with a broomstick or very light dumbbells and bring it down all the way and see if after several weeks you start to feel more comfortable.