There exist several solutions. One way is to use the rotation method which is discussed in the book %%HREF%%Geometry Technique%%Home/35-books/94-book-geometry-techniques%%. The hint is that a triangle formed by sides $1$, $3$ and $\sqrt{7}$ has a special interior angle because by law of cosines we have $$\cos \alpha=\frac{1^2+3^2-(\sqrt{7})^2}{2\cdot 1\cdot 3}=\frac{1}{2}\implies \alpha = 60^\circ$$ Here, we present an equation based solution. Establish a coordinate system such that $A$ is the origin, $AB$ and $AD$ are the $x$-axis and $y$-axis, respectively. Because $|AP|=1$, we can assume its coordinates is $(\cos\theta, \sin\theta)$. Let the square's side length be $a$, then the coordinates of $B$ and $D$ will be $(a, 0)$ and $(0, a)$, respectively. \begin{align} |BP|=3 &\implies (\cos\theta -a )^2 + (\sin\theta - 0)^2 = 9\\ &\implies 2a\cos\theta =a^2 -8 \end{align} \begin{align} |DP|=\sqrt{7} &\implies (\cos\theta -0 )^2 + (\sin\theta - a)^2 = 7\\ &\implies 2a\sin\theta =a^2 - 6 \end{align} Squaring both equations and then adding them together give: \begin{align} &4a^2 = (a^2 - 8)^2 + (a^2 - 6)^2\\ &a^4 -16a^2 + 50 = 0\\ \therefore\qquad & a^2=\frac{16\pm\sqrt{16^2 - 4\times 50}}{2}\\ & a^2= 8\pm \sqrt{14} \end{align} Clearly, $\sqrt{2}\cdot a > PB \implies a^2 > 4.5$. Therefore $a^2 = \boxed{8+\sqrt{14}}$ which is the area of this square.