In 1744, he purchased a slave named Jeanne in Saint-Dominique and brought her back to New Orleans. They grew quite close and he asked her to become his "placée bride" and live with him in his mansion. In 1747, Jeanne gave birth to their daughter Aveline. Philippe was so happy that he granted Jeanne and Aveline their freedom.
In 1752, Philippe married Madeleine de L'Isle in order to further his business. This arrangement strained his relationship with Jeanne, who vanished without a trace in 1757. Although he was devastated over the loss, he was pleased when Madeleine began to raise Aveline as though she were her own.