Does caffeine affect the fertility of women and men? The fertility of women and men depends on many factors. Caffeine consumption is also a factor limiting the ability of your husband and wife to conceive.