When removing an existing iSCSIvSwitch created using the MEM module setup, you will need to remove the stagnant VMkernel bindings with the existing physical NICs.
This is done by SSH-ing into the ESX host and running the following command:
esxcli swiscsi nic remove –n <vmk_interface> –d <vmhba_device>
You can determine which vmk_interface is associated with which physical NIC and associated vmbha, by running the following command:
esxcli swiscsi nic list –d <vmhba_device>
Once this has been done, you can follow the process of creating the iSCSIvSwitch using the MEM Module.
Simple process that held me up for a while, reference link: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1030674