Clearly, any square number ending with $0000$ qualifies. We are going to show no other solution exists.

First, we filter by square number's MOD 4 property: $n^2 \equiv 0, 1\pmod{4}$.

If $n$ ends with $1111$, $3333$, $5555$, $7777$, or $9999$, it must be a square of an odd number which means $n$ must be a multiple of $4$ plus $1$. Therefore, among these candidates only $3333$ and $7777$ may be possible. However, neither of them is possible because no square number can end with $3$ or $7$.

If $n$ ends with $2222$, $4444$, $6666$, or $8888$, it must be a square of an even number which means $n$ must be a multiple of $4$. Therefore, among these candidates, only $4444$ and $8888$ are possible. However, $8888$ is impossible because no square number can end with $8$.

Next, we can apply the MOD $16$ property, $n^2\equiv 0, 1, 4, 9\pmod{16}$ to eliminate $4444$.

Therefore, we conclude the last four digits can only be $0000$.