I used blastwave csw stuff for compilation:

export LDFLAGS='-L/opt/csw/lib -R/opt/csw/lib'
export CFLAGS='-I/opt/csw/include'
export CC='/opt/csw/gcc4/bin/gcc'
export PKG_CONFIG_PATH='/opt/csw/lib/pkgconfig'


for x264-snapshot-20111004-2245 i did:

./configure --prefix=/opt/csw --enable-shared --enable-pic


then modified config.mak:

LD=/opt/csw/gcc4/bin/gcc -shared -o
LDFLAGS=-L/opt/csw/lib -R/opt/csw/lib -lm -lpthread -mimpure-text -lrt


then shared lib successfully got compiled by issuing /opt/csw/bin/gmake

After this gmake install fails you have to copy manually the compiled objects to the right place:

cp libx264.so.118 /opt/csw/lib
cp libx264.1 /opt/csw/lib
cp x264.h /opt/csw/include
cp x264_config.h /opt/csw/include
cp x264.pc /opt/csw/lib/pkgconfig


also modify /opt/csw/lib/libx264.so symlink to point to the new version!