Yeah, I think Jack's, "there's just a handful of us now" comment was meant to communicate to the Doctor that he knew the dangers of something like Torchwood becoming too large.

Jack's timeline is definitely headachy. The reason I was struggling with the he-wasn't-part-of-Torchwood idea was I couldn't fathom what he was doing for the two months of the ghost shift. Jack doesn't seem the type to just let that happen. Unless Jack knew about Canary Wharf and knew the ghost shift and resulting battle had to happen for him to step in and rebuild Torchwood (which would tie in nicely to the fanon that Torchwood becomes the Time Agency.) But that seems unlikely if the Doctor had never heard of Torchwood. Ugh, timeline headache.