If you owe enough in delinquent American federal taxes, the U.S. government can take your passport. The Internal Revenue Service certifies “seriously delinquent” tax debt to the U.S. State Department. This is an individual's unpaid, legally enforceable federal tax debt, including interest and penalties, that totals more than $55,000 (that amount is adjusted yearly for inflation). The How You Can Lose A Passport Due To Tax Debt