Little is known about the early history of the castle, from either physical remains or documentary sources. The builder was presumably William de Londres (d 1231), as his son Maurice de Londres is recorded as lord of Oystermouth in 1151 when he granted the advowson of the church to Ewenny Priory. RCAHMW considers it likely that its bailey is represented by ‘the embanked outwork set on the lower saddle to the north’, although it is thought that this is more likely to be a post-medieval quarry. It has been suggested that the earthwork castle is more likely to have been a ringwork, given the rocky nature of the site, and that such a ringwork might be represented by ‘the scarp above the courtyard’.
The castle itself does not appear in the record until 1215 when it was burnt as part of the campaign carried out against the Normans in Gower by Maelgwn ap Rhys and Rhys Ieuanc. The last member of the de Londres family, Thomas Maurice, is believed to have died with no heir at around that date, so the castle passed back to the lords of Gower, the de Braose, when it was returned by the Welsh. The earliest surviving stonework, in the keep, was probably already have been in existence by then: it is considered by RCAHMW to be of 12th century date, although it does not contain any characteristic features. Oystermouth was held by the de Braoses up to Alina, the last of the line, and then by her son John Mowbray and his descendants, apart from a hiatus in the second half of the 14th century when the Beauchamp Earls of Warwick were lords of Glamorgan. The de Braoses regarded their Gower base as one of their principal seats, and there is documentary evidence that they resided for at least some of the time at Oystermouth rather than Swansea. They would have been responsible for most of the rest of the surviving masonry structures. Alina is traditionally credited with building the chapel block, and the Decorated style architectural details of its architecture would certainly accord with her tenure of the lordship (1327-31). All the Mowbrays and Beachamps were absentee lords, as is reflected in the minimal additions to the fabric after the construction of the chapel block.
In the post-medieval period the castle was held, still as part of the demesne of the lords of Gower, by the Earls of Worcester (later Dukes of Beaufort), apart from the period of the Commonwealth when it was held by Oliver Cromwell. There is no evidence, however, that it saw any military action and there is no clear evidence either for any building work of post-medieval date.
The Dukes of Beaufort continued to hold it until 1927 when it was bought by Swansea Borough Council. Clearance and restoration work was undertaken in the 1840s and 1870s by the antiquary George Grant Francis, including a careful restoration of the tracery in the chapel window. Further consolidation and restoration works were undertaken by the council after it acquired the castle, of which the current programme is only the most recent.