Steve's barking at trying to understand the fear of the freed Taliban

I'm trying to understand the fear of the the five freed Taliban members in a 5 for 1 prisoner swap for (Sgt) Bergdahle.
Let me try to understand from what I do know.
Afghanistan was ruled by the Taliban in 2001.
The Taliban harbored Al Qaeda.
Al Qaeda was the terrorist group that attacked us on 9/11/2001.
To reiterate Al Qaeda (People of Mid Eastern Extraction, not Asian) attacked us.
To retaliate we (rightfully) invaded Afghanistan.
Is it the suggestion of those who are afraid of the freed Taliban that they will attack Americans on our soil?
( Our soil includes Embassies and military bases).
They haven't done so before, so they're going to start now?
Is it the suggestion that we're never leaving Afghanistan so they create a risk for our men and women over there? Is that the problem?