Many visitors to The Cove do not realise that it is not located in New South Wales but in Jervis Bay Territory.
Jervis Bay Territory is a non self governing external territory of the Commonwealth of Australia, similar for example to Christmas Island in the Indian Ocean.
At Federation, with the establishment of the Australian Capital Territory as the seat of government for the Commonwealth of Australia it was determined that the Nation's capital should have its own seaport. Accordingly in September 1909 the New South Wales Parliament surrendered the area known at Jervis Bay to the Commonwealth for this purpose and by the Jervis Bay Acceptance Act 1915, Jervis Bay Territory was created.
The seaport never proceeded although Jervis Bay Territory has been home to the Royal Australian Naval College at HMAS Creswell for many years.
And if you walk out onto the beach past the high water mark you are back in New South Wales!