We investigate a doubly resonant optical cavity containing a Kerr nonlinear medium that couples two modes by a cross phase modulation. One of these modes is driven by a single photon pulsed field, and the other mode is driven by a coherent state. We find an intrinsic phase noise mechanism for the cross phase shift on the coherent beam which can be attributed to the random emission times of single photons from the cavity. An application to a weak nonlinearity phase gate is discussed.