When Henry de Lacy died in 1311 he had no male heirs to succeed him, his only child being his daughter Alice. Alice was married to Thomas Earl of Lancaster.

Thomas, Earl of Lancaster was the son of Henry III's second son Edmund Crouchback and Lord of nearby Pickering Castle. He was also King Edward II 's cousin and through the inheritance of both the Earldom of Lancaster and the Honour of Pontefract was a very powerful man.

Thomas, Earl of Lancaster was initially one of Edward II's chief advisors but later opposed him, hating the King's favourite, Piers Gaveston. In 1312 Thomas, Earl of Lancaster was one of the barons who besieged Piers Gaveston in Scarborough Castle.

His marriage to Alice de Lacy was not a success. In 1317, John de Warenne of nearby Conisbrough and Sandal Castles allowed his squire to help Alice de Lacy escape from Thomas, Earl of Lancaster. Thomas, Earl of Lancaster in revenge divorced Alice and then besieged Sandal and Conisbrough castles. Although Edward II ordered Thomas to end this private war Thomas captured and retained Conisbrough and Sandal Castle until his death.

In 1321 Thomas, Earl of Lancaster rose in open opposition