Sam spends his days walking around the following $2\times 2$ grid of squares. $$\begin{bmatrix} 1& 2 \\ 4&3 \end{bmatrix}$$

Say that two squares are adjacent if they share a side. He starts at the square labeled $1$ and every second walks to an adjacent square. How many paths can Sam take so that the sum of the numbers on every square he visits in his path is equal to $20$ (not counting the square he started on)?