This dissertation splits into two distinct halves. The first half is devoted to the study of the holography of higher spin gauge theory in AdS$_3$. We present a conjecture that the holographic dual of $W_N$ minimal model in a 't Hooft-like large $N$ limit is an unusual semi-local" higher spin gauge theory on AdS$_3times $S$ 1$. At each point on the S$ 1$ lives a copy of three-dimensional Vasiliev theory, that contains an infinite tower of higher spin gauge fields coupled to a single massive complex scalar propagating in AdS$_3$. The Vasiliev theories at different points on the S$ 1$ are correlated only through the AdS$_3$ boundary conditions on the massive scalars. All but one single tower of higher spin symmetries are broken by the boundary conditions. This conjecture is checked by comparing tree-level two- and three-point functions, and also one-loop partition functions on both side of the duality. The second half focuses on the holography of higher spin gauge theory in AdS$_4$. We demonstrate that a supersymmetric and parity violating version of Vasiliev's higher spin gauge theory in AdS$_4$ admits boundary conditions that preserve ${cal N}=0,1,2,3,4$ or $6$ supersymmetries. In particular, we argue that the Vasiliev theory with $U(M)$ Chan-Paton and ${cal N}=6$ boundary condition is holographically dual to the 2+1 dimensional $U(N)_ktimes U(M)_{-k}$ ABJ theory in the limit of large $N,k$ and finite $M$. In this system all bulk higher spin fields transform in the adjoint of the $U(M)$ gauge group, whose bulk t'Hooft coupling is $frac{M}{N}$. Our picture suggests that the supersymmetric Vasiliev theory can be obtained as a limit of type IIA string theory in AdS$_4times mathbb{CP} 3$, and that the non-Abelian Vasiliev theory at strong bulk 't Hooft coupling smoothly turn into a string field theory. The fundamental string is a singlet bound state of Vasiliev's higher spin particles held together by $U(M)$ gauge interactions.