Abe said Japan would not renew the mission after the current rotation returns in May. The 350-person team has focused on road construction.
The team, which arrived in South Sudan in November, was Japan's first to have an expanded mandate to use force if necessary to protect civilians and UN staff. The Japanese military's use of force is limited by its post-World War II constitution.
Abe said Japan would continue to assist South Sudan in other ways such as with food and humanitarian support, and will keep some personnel at the UN peacekeeping command office, according to AP.
The decision to withdraw comes amid concern about the safety of Japanese troops in South Sudan.
The South Sudan mission was the longest for Japanese troops, who also have served in Golan Heights, Cambodia and other areas for post-cease-fire assistance limited to noncombat roles.