I am also 34 and a mother but of only one 2 1/2 year old. I really admire you for taking care of yourself by reaching goals as a runner. I can't begin tell you how much running and racing has helped me as a wife and a mother.
I have one friend who runs quite a bit faster than myself and the rest of my running friends are at about my pace or slower. Honestly, when it comes to running, I really don't think a lot about them. I hope that doesn't sound narcissistic or rude but I just don't have any type of feelings about my friends pace in relation to my own. I have run slow half marathons with friends who were slower than myself really only because it was so much fun I didn't see the point in going out on my own. On the other hand, my friend and I had a planned 10K together and we both knew that we would not run together only because the results were that she ran eight minutes faster than myself. It is all about what you want to accomplish and if someone else wants to stay with you or if you want to stay with them, that's great too. Your co-worker sounds like he has become competitive with you. I would shrug it off and do your own thing whether it is staying with him the whole way or not.