The building was originally built as a private mansion by Aaron Cardozo in 1819. He was a prosperous merchant of Jewish Portuguese descent who had settled in Gibraltar, and the building became his private home. It was the grandest private mansion ever seen in Gibraltar, with the three-storey house dominating John Mackintosh Square.
The site had previously been the old hospital and chapel of La Santa Misericordia (English: The Holy Mercy), and later it had been a prison. As a non-Protestant, Cardozo was not legally allowed to own property in Gibraltar. However, as he had been a close friend of Lord Nelson and had supplied his fleet, he was eventually granted a site to build a house in the Alameda on the condition that it be "an ornament" to the square. Its cost was about £40,000.
After his death in 1834, the mansion was leased to John Ansaldo as a hotel, the Club House Hotel. It was bought in 1874 by Pablo Antonio Larios. He was a Gibraltarian-born wealthy businessman and banker, and a member of the Spanish Larios family. The family completely refurbished the building.
His son Pablo Larios, Marquis of Marzales (Master of the Royal Calpe Hunt for 45 years), sold the building to the Gibraltar colonial authorities in 1922, the intention being to turn it into a post office. However, it eventually became the seat of the newly formed Gibraltar City Council.
In 1926 the Gibraltar telephone service was operated by the City Council, and an automatic exchange serving the territory was installed in the then top floor of the building. The building was later extended to include a new storey and a new body to the North, modifying its original symmetry. Today, this historic building houses the Mayor's Parlour and the National Art Gallery.