You're right that Anna wasn't interested in Brian, but she wasn't interested in Jake at first either, until she became aware that he was available and interested in her. I feel that Brian's priesthood kept him "safe" and off Anna's radar and that if it weren't for that safety net, Brian would have been a "threat" in some way - either Anna could have fallen for him too or he would have had to have been disposed of in some way (usually by making him a bad guy or by dying) because you can't have a heterosexual man who is in love with "your woman" hanging around. But if he's a priest, he's automatically trustworthy and safe.

In fact, the whole movie could have happened just fine without Brian in the picture at all, because there was enough plot and conflict just with the whole different-religions problem between Anna and Jake. Mismatched backgrounds and star-crossed lovers has been the plot of romance stories for centuries, so Brian's character really didn't have anything to do there at all.

It would have made much more sense if Anna actually had returned his feelings, or if we had followed the whole story from Brian's point of view like in My Best Friend's Wedding. Otherwise, he's just extraneous and the friend-zoning (under the blanket of his priesthood) was how they wrote him off as a potential love interest in order to keep him as part of the group but not have non-monogamy or make him a "bad guy".