Researchers said animal studies showed abnormalities caused by environmental toxins were caused by genetic changes that were passed on through generations.
Smoking, drinking alcohol and cocaine were found to cause chemical changes to semen
Dr Cynthia Daniels of Rutgers University, said: "If I were a young man I would not drink heavily and I would not be smoking two packs of cigarettes per day while trying to conceive a child."
"Studies have shown significant associations between male toxic exposures and increased rates of infertility, miscarriage, stillbirth and childhood health problems."
"We need to open our eyes and look at the evidence. Sixty per cent of all birth defects have unknown sources. Why are we not examining such an obvious source of harm?"
Dr Matthew Anway, of the University of Idaho, gave pregnant female rats daily injections of the pesticide vinclozolin during the period when the sex of embryos is determined.
Male offspring had abnormalities including prostate and sperm development problems, and genetic changes that the researchers found were passed on through four generations when the males were mated with healthy females.
Presenting his findings at the American Association for the Advancement of Science (ASSS) conference in Boston, he said: "In addition to the spermatogenic and prostate abnormalities, trans-generational effects on numerous disease states were observed, including tumour development and kidney disease."
Dr Anway said the doses used in the experiments far exceeded the levels that humans could expect to be exposed to in the environment, but that the study was designed to demonstrate how toxins could lead to inheritable abnormalities.
Dr Gladys Friedler, of the Boston University School of Medicine, said: "Both animal and epidemiological studies demonstrate that paternal exposure to a variety of potential toxins can adversely impact fetal development, produce a wide spectrum of deficits in offspring and be expressed in subsequent generations."