We have a responsibility as a society not only to our own residents but also to the poor elsewhere.  Free trade increases the size of the pie which means that all can have more.  The failure of our system is largely the failure to address distributional issues, but that is much better if addressed directly, through a progressive income tax and the equivalent of estate taxes, rather than addressed indirectly via measures such as restraining international trade.  When we restrict trade to favor jobs in the country - presumably to help the working class - many of the benefits are sucked away by the owners of the favored industry, and at the same time we hurt employment elsewhere in the world.

Policy tools should be employed which have the most direct effect on the desired outcome.  Otherwise one's good intentions can lead to surprisingly disappointing results.