Joseph Hyde Potts, who worked for the Bank of New South Wales, is the man behind the name of Potts Point. He bought 6.5 acres of waterfront property in what was then Woolloomooloo Hill, renaming it Potts Point.

A large portion of what is now Potts Point and the Elizabeth Bay district were initially granted to Alexander Macleay, the New South Wales Colonial Secretary from 1826 to 1837, for whom Macleay Street is called. Notable Australian poet Henry Lawson mentions Potts Point in his poem William Street. John Wylde, the man for whom Wylde Street is named, was a 19th-century public servant who owned land in the region.