We analyze properties of a map f sending a unitary matrix U of size N into a doubly stochastic matrix B = f(U) defined by Bi,j = der="0">Ui,jder="0">2. For any U we define its defect, determined by the dimension of the image SBHFKC-1&_mathId=mml1&_user=1067359&_cdi=5653&_rdoc=4&_acct=C000050221&_version=1&_userid=10&md5=aafd2f9bb4b7b74f3d19d45f9413008e">sbottom" border="0" height=17 width="69"/> of the space sbottom" border="0" height=16 width="36"/> tangent to the manifold of unitary matrices de9ecd8b6eecfe35cb713">sbottom" border="0" height=13 width="15"/> at U under the tangent map Df corresponding to f. The defect of U equal to zero for a generic unitary matrix, gives an upper bound for the dimension of a smooth orbit (a manifold) stemming from U of inequivalent unitary matrices mapped into the same doubly stochastic matrix B = f(U). We demonstrate several properties of the defect and prove an explicit formula for the defect of the Fourier matrix FN of size N. In this way we obtain an upper bound for the dimension of a smooth orbit of inequivalent unitary complex Hadamard matrices stemming from FN. It is equal to zero iff N is prime and coincides with the dimension of the known orbits if N is a power of a prime. Two constructions of these orbits are presented at the end of this work.