Multiplying both the numerator and denominator by $(3+\sqrt{5})$ yields

$$\lim_{x\to 4}\frac{(3-\sqrt{x+5})(3+\sqrt{x+5})}{(x-4)(3+\sqrt{x+5}}=\lim_{x\to 4}\frac{9-(x+5)}{(x-4)(3+\sqrt{x+5})}=\boxed{-\frac{1}{6}}$$