Rache, if you feel unsafe, tell your hubbie when he gets home. Work out an agreement about visitors and what you expect from him regarding preotecting you.
Don't answer the door as everyone has said. I don't know the whole story but if he planned on doing something bad, would he have asked your hubbie what your schedule was? But women's intuition is often right. If he's creepy, stay away from him!
Please give yourself a hug and I hope your therapist helps you out today!