Adoption crisis and needing an answer regarding what to do...?

my 23 year old boyfriend was adopted by two people who he claimed didn't understand him, after I got to know them after 6 years of us being together I noticed that they werent mean or vindictive, he was a very troubled youngster and they did all they could to help him yet I can see how much they love him, they did everything true sacrificing parents would for their kids, seeing how he blame a lot of his troubles on being adopted, as his girlfriend I seeked out he's birth father with the help of no one and found him in 2007, they got to know each other but it wasn't anything fatherly and strong besides he was grown and his father had his own family to look after, in the mean time my boyfriend and I had a beautiful baby boy together everything seem ok when I noticed he still wondered about his birth mother and that pained him, so with my searching skills I look for her on facebook went in her school alumni profile and found one of her friends who had her contact me that way I was able to find her. I can say that as soon as they talked and met that i've never seen my boyfriend this happy so I thought finally everything was looking out not only for him but for our son and we could everything p us and now he now know like he said his true identity. the birth mother has her own family and was very welcoming to him and our family, on the day that my son was turning 2 the adoptive mother came over to drop my son's present off to tell me that her son (my boyfriend) has abandoned her and their family and it hurt her a lot and she cried. I guess in my hurry to try to give the father of my son peace and comfort I forgot what it would do to his adoptive parents, and now I feel that God is punishing me for it. Now he moved out of the state and left his only son to move in and live with his biological mother because he never had 2 real parents to raise him like I have he stated to me once and that he is making a sacrifice and that someday our son will understand. I am so confuse... My mother explained to me that it's just an honeymoon stage and if he abandoned the woman who cared and loved him for 22 years he will do the same to anyone and that he is an ungreatful person who only cares about himself and soon the husband of the birth mother won't be so ok with this, I feel like I have destroy 3 families, the adoptive parents, the biological parents family and all the same I have destroyed mine.it's been almost 6 months since he lives there and He barely sees his son, I'm a strong mother and I'm there for my son but the other day he called me from his biological mother's house in Idaho to let me know that he's going to take our 2yr old son to california on a 16 hours drive to see his birth father's family in Sandiego and will be back in 4 days and this will be a custom for the years to come, I really feel this is carma eating at me, I have physical custody of my son and although I would love him to have a great special time with his dad which he hasn't had in 6 long months but I don't know the biological fathers' family and California is so far away and I would never see my child and be there to protect him when he needs me most. I'm angry and outraged that he dare think just because he can leave the parents who cared for him for 22 years that he can take my little boy away...any advice pleaaaase?