Why does ISIL remain so difficult to defeat and why does it appear to be a reoccurring phenomena?
It seems to me that Al Qaeda, the Taliban, ISIS and even Boko Haram are all manifestations of the same thing. They are a single ideology and not separate organizations as we tend to think of them. That is why the Chechen fighters who battled Russia seeking independence can move comfortably from there, to Afghanistan, to Iraq and now to Syria and believe they are fighting for the same goal. This is not too different from the State-Socialist Bolshevik ideology that fired up the wars for colonial independence 80 years or so ago. They usually achieved independence and quickly morphed into military dictatorships or dropped their strident socialism as soon as they got a taste of the beguiling delights of consumerism.
Contrary to prior policy in Afghanistan and Iraq, where we occupied the land, at least temporarily, while we installed a governing organization that we called democratic but many others referred to as a puppet regime, we are no longer seeking to either hold land or impose a specific regime. Also, the US now concentrates on targeting the insurgency’s leaders and not merely on killing its soldiers. That is a change from Viet Nam where we racked up the kills of soldiers and civilians to no avail.
ISIL and the others remain ideologies [Islam is mostly the cover] opposed to the organizations that are the dominant economic powers of the day and not an organization like those we are used to dealing with. Killing its leaders, like killing its followers will not work. New leaders and new followers are easy to find. Occupying the land and imposing a regime we know now does not work either.
So what works? I do not know, but we have had great success battling competing ideologies with massive economic development. Young men inflamed with the possibility of becoming rich usually do not have time to pick up a gun.