Have not driven from amed to Balian. Have ridden to Balian from Kuta and also from Kuta to Amed - both fairly long drives in themselves so I'm thinking 4/5 hours may be a little conservative estimate, but they may have some short cuts which can cut the time down. I just checked Google maps and it said 3hrs 46 mins so your driver in Amed is possibly a slow driver.
Or the driver is allowing for ceremonies, which can hold things up.
it is roughly 5 to 6 hours drives depends on how much busy the traffic is.
The route from Padang Bai all the way up to Balian beach is a busy street which all trucks, buses are driving through therefore you take long hours driving.
should not be in rush then you will enjoy the journey.