Apache rewrite rule to rewrite to sub dir

One one particular site, I wanted to rewrite all requests to a sub dir. It took me over an hour, because of some obscure problem I haven’t been able to identify.

# The main site located in the /site dir. The .* after the ^ is weird, but without it, it wouldn't work.
RewriteCond %{REQUEST_URI} !^.*/site/.*
RewriteRule ^(.*)$ site$1 [L,R=permanent]

My guess is it has to do with:

AddHandler php-cgi-script .php
Action php-cgi-script /php5/php5-cgi