I love him, so does this mean I should let him go?

Meaning let nature take it's course. Should I just back down and watch things happen? I have been with this guy for about three years, he's 23 and I'm 24 and it's getting clearer to me that he really is interested in another girl. But he can't have her yet, because she is 17, not 18 yet, and she's taken right now. And now i'm pregnant with his child which I also think might be stalling it. But he always needs to be at our friends house when she is there, always asks if the "kids" including her, will be visiting there. She's really skinny, and blonde, and pretty with big blue eyes, has known him longer and has had a crush on him before him and I got together. He told me before that he's not interested in her, but his actions keep proving otherwise. So... should I just give up? just watch things happen? I mean, they're not even really friends, they hardly talk, but he's also really shy, and maybe doesn't yet have the self esteem to fully pursue her. I can't really leave yet, I'm pregnant with his child and I live with him, my family being in another state. Is there any suggestions for me? I know I will be hurt if they do choose to be together, but why be with someone who wants to be with someone else?