The construction of the First Government House — now the site of the Museum of Sydney — began in May 1788. From there it grew organically and often unsatisfactorily, and by the 1830s it was decided to replace it with the magnificent Gothic Revival building which now sits at the top of the Royal Botanic Garden.

Government houses were always seen as the ultimate symbolic embodiment of the power of Empire in the colonies — and so the historic part this Government House had played in the colonisation of Australia was well recognised. Nearly every view of Sydney Cove was focused on it. Governor Gipps, the first governor to occupy the new Government House, commissioned this house portrait for his own collection, in 1842.