Refractory caulk, also known as fireplace mortar, is suitable for minor repairs only. If your fireplace has broken or crumbling bricks, large gaps in the mortar, or missing mortar, have a professional mason or fireplace contractor inspect it. Significant damage in a firebox can be a serious fire hazard and should not be repaired with refractory caulk.
Source: The Spruce