At some point between 1497 and 1503, Patrizio was in a relationship with Lucrezia Borgia, daughter of Pope Alexander VI. When the Pope died on the August 18th 1503, Lucrezia left Rome to live with her third husband, Alfonso I d'Este – the Duke of Ferrara – in his city.
In 1506, Patrizio was visited in his palazzo's courtyard by Ercole Massimo, leader of the Hermeticists, to discuss the purchase of the portrait. At first, Patrizio seemed willing to be rid of the painting, however, as they discussed the portrait's history, Patrizio became angered at the memory of how Lucrezia had seduced him before abandoning him, and decided that it did not deserve to belong to anyone.
As he threw the painting on the ground and prepared to destroy it, Ercole intervened and had one of his men stab Patrizio in the back to stop him. The noble died quickly, and the portrait was taken from him.