I think besides Genetics which is key, is because they are so commonly utilized (walking), etc..they are constantly adapting to very rigorous use and are harder to grow..likewise any muscle that is over used, it often smaller, etc..so you have to figure out a stimulus for them that will make them grow based on your genetic predisposition.
One trick I actually learned on the forums years back is this:
I actually found a that stretching them, or calf stretches work. Essentially you can lie in bed even and point your toes forward as far as you can and as hard as you can, keeping constant tension on the muscle (this requires the mind muscle connection) then slowly PAUSE for 3-5 seconds, and then, bring your toes back towards yourself, or the direction of your face you could say, and PAUSE there for 3-5 seconds. Do these for sets of 100 on top of your regular routine, 3-5 times per week and I promise you will see new growth