Even the miserable lady in the gift shop shouldn't put you off. The guys on the Duck and other staff are super friendly. They were let down by the miserable ride lady in the Castle gift shop.
That said Elizabeth Castle is a great day out. It shouldn't be missed. I highly recommend you take a picnic and allow yourself several hours to enjoy.