What was it about O'brien and the girl caused Smith to noticed them?winston realizes that he felt necessity to begin writing the diary because of...

Speaking of Julia, Winston narrates:

He disliked nearly all women, and especially the young and pretty ones. It was always the women, and above all the young ones, who were the most bigoted adherents of the Party, the swallowers of slogans, the amateur spies and nosers-out of unorthodoxy. But this particular girl gave himthe impression of being more dangerous than most. Once when they passed in the corridor she gave him a quick sidelong glance which seemed to pierce right into him and for a moment had filled him with black terror.

And of O'Brien, he says:

Winston had seen O’Brien perhaps a dozen times in almost as many years. He felt deeply drawn to him, and not solely because he was intrigued by the contrast between O’Brien’s urbane manner and his prize-fighter’s physique. Much more it was because of a secretly held belief — or perhaps not even a belief, merely a hope — that O’Brien’s political orthodoxy was not perfect.

Later, during the Two Minutes Hate, Winston notices Julia; she throws a copy of the Newspeak Dictionary at the screen, at Goldstein.  Then, regarding O'Brien, Winston says this:

Momentarily he caught O’Brien’s eye. O’Brien had stood up. He had taken off his spectacles and was in the act of resettling them on his nose with his characteristic gesture. But there was a fraction of a second when their eyes met, and for as long as it took to happen Winston knew — yes, he knew! —that O’Brien was thinking the same thing as himself. An unmistakable message had passed. It was as though their two minds had opened and the thoughts were flowing from one into the other through their eyes. ’I am with you,’ O’Brien seemed to be saying to him. ’I know precisely what you are feeling. I know all about your contempt, your hatred, your disgust. But don’t worry, I am on your side!’ And then the flash of intelligence was gone, and O’Brien’s face was asinscrutable as everybody else’s.

So, Winston is attracted to their eyes: at Julia's "sideways glance" and at O'Brien's "spectacles" and "eyes."  As you know, eyes are symbols of knowledge, gateways to the soul. 

But, if you've read Oedipus, the eyes can blind oneself to spiritual self-knowledge.  This is the case with Winston.  He mistakenly thinks Julia is a double agent and that O'Brien is not.  Since the members of the Party are unable to speak or write to each other, they must conjecture (through physical appearances only) about one's allegiances.  Throughout the novel, Winston will place more importance on love of O'Brien than of Julia: that O'Brien's abilility to see truth is more important than Julia's ability to give and receive love.  He will mistakenly follow the Father(land) instead of the mother.