I tried to find the most difficult sites and came to the conclusion that visit ranking is misleading - for example Timbuktu is rather high on the list (#761 with 29 visits) but, given security situation, it may be one of the most difficult sites now. Palmyra is almost exactly in the middle of the ranking (#528), but I would not consider it easier to visit than, say, Mtskheta (less visited according to the list):-)
I also think that cost factor should be included - there are some sites that are not difficult, but require a couple of thousand $ and at least one week to visit - e.g. Tubbataha. Basically, almost all of our connection "Takes more than 5 days to visit"
http://www.worldheritagesite.org/tags/tag.php?id=1214 should be treated as most difficult (except Ogasawara, much cheaper than the others) and some more (e.g. Putorana, Malpelo, Salonga).
Oh, and for me - Most Visited Unvisited - Versailles (#10), Least Popular Visited - Virgin Komi Forests (#962).