Early history of the castle is not clear although Scott determined from the similarity of features with other fortalices, that circa 1455 was a likely period of construction. A.H.Millar (1895) described Tayport Castle as one of the most massive examples of the Z plan in Scotland. ”The castle was about 40 feet square, with two round towers at the south-east and north-west angles each about 26 feet in diameter. The walls varied from about 7 feet in thickness to 12 feet 6 inches. The entrance doorway, which projected a few inches, was on the east side, and gave access to the two vaults in the main building, and to the north-west tower.” He also advises that in several state documents, dated 1548, references are made to a fortalice built by certain Fife noblemen and gentlemen to guard the ferry, as Broughty was then in possession of the English.  Scott advises that late in 1547, after the battle of Pinkie, part of Somerset’s fleet sailed into the Tay and took possession of Broughty Castle. They pillaged the surrounding countryside and attacked the Abbey of Balmerino.

Simon Taylor (2010) refers to state papers recording Admiral Thomas Wyndham to Protector Somerset “ On the 22nd I went aland on Fife side with 200 men and burned Scotysh Crage …. The country gathered 500 horse and skirmished with us, dismounting and trying to get between our boats and us, but their hearts failed them”  Scott recounted a more favourable outcome for the Fife Noblemen in which he quoted Lindsay of Pitscottie “Fra that time forth, they desired not to land again in Fife.”