The train, traveling to Taitung, came off the rails in a tunnel just north of Hualien causing some carriages to hit the wall of the tunnel, the fire department said in a statement.
At least four people are believed dead, three people with serious injuries have been sent to hospital and around 20 with light injuries are waiting to go to hospital, it said.
The train was carrying around 350 people, and rescue efforts are ongoing, the department said.
Between 80 to 100 people have been evacuated from the first four carriages of the train, while carriages five to eight have “deformed” and are hard to gain access to, it added.
“Is everyone out in carriage four?” a woman is heard shouting from inside the tunnel, in images provided by the fire department.
The accident occurred at the start of a long weekend for the traditional Tomb Sweeping Day hospital.
Taiwan’s mountainous east coast is a popular tourist destination.
In 2018, 18 people died and 175 were injured when a train derailed in northeastern Taiwan, in the island’s worst rail disaster in more than three decades.
This is a breaking story. Updates to follow.