Brinley was born to English missionary parents, who endeavored to convert Indians and restore peace to the new colonies. In 1848, his father, Franklin Ellsworth, tasked Brinley with keeping watch over a young Duleep Singh. The two became fast friends, but were seperated when Duleep was taken to England.
Brinley stayed in India, where he eventually became an apprentice to a government official of the British Indies Company. His new duties consisted of keeping the peace by preventing another rebellion and ensuring Britain stayed in control of its colony. As a result, Brinley became involved in a plot to end Singh's attempts at restoring Punjab independence from the British Empire. He failed to do so as the plot was uncovered and foiled by the Frye twins Jacob Frye and Evie Frye. As Evie prepared to assassinate Ellsworth, Duleep stopped her and allowed him to leave, as he understood that if they were to kill him, it would not make him any better than the oppressive British Indies Company.