On Halloween Casper ate $\frac{1}{3}$ of his candies and then gave $2$ candies to his brother. The next day he ate $\frac{1}{3}$ of his remaining candies and then gave $4$ candies to his sister. On the third day he ate his final $8$ candies. How many candies did Casper have at the beginning?