One of the tankers was completely destroyed in the explosion, according to Mid and West Wales fire and rescue service.
Chevron confirmed that the four people who died were contractors and said another worker was receiving treatment in hospital for serious injuries.
Fire crews from around the area were called to the site of the explosion along with paramedics.
Local people reported hearing the explosion and seeing a plume of black smoke at around 6.20pm.
Earlier this year the refinery, which is one of the largest in western Europe and employs 1,400 people, was sold. Valero agreed to buy the refinery site for $730m dollars (£458m) and another $1bn (£611m) for assets including Chevron's petrol stations in the UK and Ireland.