Aberconwy House, 14th Century