Recent works have shown that deep neural networks can be applied in the Fourier phase retrieval for images in real time. In particular, a structure named CDINN successfully conducted real-time Fourier phase retrieval for convex polygon images. However, few-shot learning has not been studied in this field. We trained this structure on different image datasets and found the model’s performance is poor when there is a limited number of training samples. In this work, we present a new model called CDIDANN, with higher converge rate and better performance compared to the original CDINN structure. Further, we enhance the model’s performance when there is only a small number of training sample available by combining the modified CDINN structure with a domain discriminator from the domain-adversarial neural network (DANN) model. We demonstrated the improvement of the new model by performing Fourier phase retrieval on MNIST and Fashion MNIST datasets.