A promising strategy for direct detection of sub-MeV dark matter is to look for phonon excitations in crystals. The crystal targets used in such experiments are typically not completely pure, and have impurities or defects. The point defects of our interest include the Frenkel defects, hydrogen and oxygen impurities, and so on. These defects can diffuse and recombine to emit energy in the form of phonons, and can potentially create a background for direct detection experiments. We estimate the defect densities produced through thermal excitations as well as radiogenic nuclear recoils. For various defect configurations, we quantify the diffusion and recombination rates for both thermal and quantum tunneling mechanisms. We find that the thermally generated Frenkel defects are effectively frozen at cryogenic temperatures and cannot diffuse to recombine with each other. The hydrogen impurity is the unique point defect that has non-negligible diffusion rate at cryogenic temperatures, and possibly contributes to the eV-scale events. The radiogenic defects produced on the surface can be annealed effectively at room temperature for typical defect configurations, but defects produced through radiogenic nuclear recoils in a shielded environment at cryogenic temperatures during the run-time of the experiment can recombine to produce eV-scale events. We provide estimates of these background event rates and give some remarks on subtleties we need to take care of in this kind of experimental setups.