Increasing the memory allocated to the service console in ESX 4X from 300mb to 800mb requires editing of two files. The files are esx.conf in the /etc/vmware directory and grub.conf in the /boot/grub directory. 800mb is the maximum allowed. The swap partition should be changed as well from 600mb (twice the ram allocated to the service console by default) to 1600mb maximum.
# vi /etc/vmware/esx.conf
# vi /boot/grub/grub.conf