The most difficult countries to travel in?

I'd stump up Saudi Arabia as number one for a simple reason: it's the only country I know of which does not offer tourist visas, full stop. (They used to, with tight controls and for groups only, but apparently do not any more.) And unlike eg. Russia, you can't just ring up a hotel and get them to "invite" you into the country. Even getting a legitimate business visa can be a pain: I was invited by a well-connected, major company and I'm informed that it still took cough facilitation payments cough on the order of 500 euros to get the necessary Chamber of Commerce approval in a reasonable amount of time.

Once you're in Saudi Arabia, though, getting around is pretty painless, although some sites require special permits and the entire area around Mecca is off-limits unless you're a Muslim. On the other hand, physically crossing the Empty Quarter off-road would be a pretty epic challenge!

Update: In regard to mouviciel's comment, Hajj/Umrah visas aren't a realistic option for traveling to Saudi Arabia for three reasons.

1. They're only issued to Muslims, and converting to Islam is not that easy, especially if you prefer your dangly bits uncircumcised. (Women must, of course, be accompanied by a Muslim male guardian.)
2. Even if you are Muslim, you have to apply for a visa in the Hajj lottery, and the odds are pretty steep. Egypt has 80 million people, and gets a quota of ~80,000 visas a year: that's 0.1%. Umrah visas, however, are handed out a little more generously.
3. Last but not least, a Hajj/Umrah visa allows you to visit only Jeddah, Mecca and Medina, and that for a limited period of time; you're not allowed to leave and explore the rest of the country!

For establishing such a list, one needs to identify the possible barriers that would make travel difficult. I can think of:

• Natural barriers, like in Antartica or Sahara.
• Political barriers, like in North Korea or Tibet.
• Cultural/Religious barriers, like in Amish communities or in Mecca.
• Financial barriers, like in Bhutan or Switzerland.

Of course these barriers are different for different people, most of the time they are interleaved, and they change with time.

A definitive list seems an unreachable goal.

I think there is a (maybe too) simple index of countries to rank the "ease" of travelling. It's not very scientific but for what it's worth : Countries ranked by number of tourists

Its main problem is that it gives a very vague overview. I mean the fact that a lot of people go to one country doesn't mean every tourist can, see @jpatokal answer stating that a basic condition to travel as a tourist to Saudi Arabia is to be Muslim. So this index shows how accessible a country is based on a very vague overview. It is maybe too synthetic.

Another problem is about distinguishing the passive and active ease. This ranking actually mostly represents the active efforts of a country to attract people : if facilities are made for tourism, people speak English, it is affordable, ... But it doesn't mean the fifth will be way harder to visit than the first.

On the other hand, in countries at the top of the index, everything is made for tourists not to be lost (well at least speaking English and a lot of signs, information, documentation about the most random monument).

And overall, all the criteria @mouviciel listed are totally represented in this index. For example, when Tunisia was politically unstable, the tourists stopped going, it was a pretty big deal in a country that needs tourism for its economy. And if you take the top 10, all the countries are politically stable (not for parts of China but I suppose few tourists go there), and all other barriers are not a big deal in these top countries.

What could be interesting is the bottom of the index, but unfortunately they did not compute it.

Well of course the question is Difficult with respect to what? Having said that, the source I personally use is the website of the french diplomaty.

The French embassy provides very detailed information for travellers on every country (in the French version only). It is updated daily and contains information about security, transportation, leaving/entering the country, Healthcare and even cultural issues - See their website.

Of course it does not "rank" countries, but usually provides, on the first page, a general description of few lines describing the important facts and difficulties you can get to.

Your country probably provide something similar. I encourage you to take a look at your Diplomaty/Embassy/Foreign minister website.

I'd just like to add from personal experience that regarding countries in South East Asia, Myanmar (Burma) can be a very difficult country to travel in due to the fact that English just isn't that widely spoken. Regardless, the people are generally very kind so you're never without help, but the communication breakdown can make it frustrating at times.