I have a rehomed Grey, and went through the same thing. In fact, I'm impressed he warmed up to you so fast. It was at least 3 months before we could handle Rosie. About that time she started picking up words, although she knew some from her old owners. Now she wont shut up
But like I said, it took months. Be patient and keep trying.
-Chris
Rosie (African Grey)
D'Artagnan--Dart for short (Jenday conure)
Lyra (Baby Budgie-hatched April 1st, 2012)