Bringing this up from the backlog discussion. How should a country rating be calculated?
https://www.worldheritagesite.org/ranking/country+ranking

Current approach is to handle all votes for all sites linked to the country as a vote for the country and then apply the normal score. This results in the top countries being mostly those with very few stellar sites. The Top 3 is Zambia (1), Vatican (2), and Palau (1).

Meanwhile, Italy ranks on #63, China is #48. Explanation is that their lists contain several average or below average sites.

Question is how could we do this better? Looking forward to suggestions.

Unless the aim is to just have single ranking system, we could look into a different approach.

In addition to a total list of country rankings, we can have a separate ranking where rankings are categorized per tier based on number of UNESCO WHS, like ranking for those with 1 to 5, then 6 to 15, then 16+ (these are arbitrary numbers). This will somehow put some level equity and fairplay both for bigger countries with more sites and small countries that cannot nominate more due to their size. We can base the range of tiers on number of countries per tier.

I always feel systems shouldnt grow to complicated. It's one number per country. And I am open to downweighing small countries.

We can lessen the complexity by using the same statistical method/algorithm in determining scores/ratings but the rankings are presented or organized per tier/category to reflect country size. This way, none is penalized because of their size and each country is compared with their "peers",

Otherwise, I agree with down-weighing small countries but again, it will turn the bias against small countries. Just my 2 cents.

Thanks nfmungard for moving this theme in the community.

I share with you my proposal in the backlog thread just to have it in the correct thread:
My idea is for each country, count the sites rated with 4.5 or higher, 4.0 or higher, 3.5... etc. After that, order by number of 4.5, 4.0, 3.5...
With that, we will have the countries with those sites best rated in the top sites. In addition, those countries less represented, will still be in the top if they have top rated sites (for example, Cambodia would be in a high position for having Angkor, and it doesn't matter it has only 3 sites).
I commented too, instead of having blocks of 0.5, we could have smaller one to give more important value to the quality of the sites, for example 0.3 (sites with 4.7, 4.4, 4.1... etc)

Thanks!!

I think it would be better to think what a country rating should look like. before we start talking methodology.

Looking at the current list, I feel the small countries are the issue. But only looking at countries with at least 5 sites you get:

1) Congo
2) Tanzania
3) Zimbabwe
4) Egypt
5) Turkey
6) Peru

Turkey would probably be in the lead overall. And those countries have short lists, which generally means they have better sites. Italy decided on submitting Piedmont and Prosecco and those simply aren't great sites.

Alternative would be to sort by classes as proposed by you. What would the result be?

It could also help to take into account a minimum of ratings given. That will not "solve" Zambia, but some of the others.
Including DRC, few people have been there and though all its parks are important noone can say that it is the greatest WHC country.
Egypt, Turkey, Peru, Cambodia to some extent would be more logical.
But for New Zealand, Sudan and even Palau - though low on WHS - I could defend their high country ranking.

That already happens via the Wilson score being applied (25% downshift). We could weigh this more heavily, but Virunga is always on my mind ever since you wrote your review...


That's what I thought, too. Meroe looks great. New Zealand was great. Cambodia also has stellar site..

Maybe weigh the Wilson score more heavily?

DRC has 5 WHS, but only 2 of those have been rated (and are indeed very good). So the country average shown actually is for 2 sites only and not for all 5 (because noone has been there).

Oh. Okay. Then it shouldn't show where it does. I thought it was 5* Virunga.

I do not agree. Instead, small countries are who have shown the issue.

Let me show you two very easy examples of how the list should look like (in my opinion) and how the countries with a low number of sites are showing the problem. I will use Zambia for both examples:
- Zambia vs Zimbabwe: Zambia has a very high position (the top) because it has only one site but a very good site: Victoria Falls. However, Zimbabwe has the same site plus 4 others sites. Basically we can say Zimbabwe is Zambia plus 4 sites. For this reason is very easy to see the list should have Zimbabwe in a higher position than Zambia, because Victoria Falls + 4 is better than Victoria Falls + 0.
- Zambia vs Cambodia*: Zambia has the 10th best rated site + 0 more sites. Cambodia has the 1st + 2 more sites. Again, a very easy one: Cambodia should be higher than Zambia. The 1st + 2 sites is better than the 10th + 0 sites.

The problem is not the small countries. The problem is the low rated sites instead of add value to the countries they are substracting value (for the use of average).
Zimbabwe would have a higher position than Zambia if its 4 extra sites where adding value instead of substracting value.
Cambodia* would have a highier position than Zambia if its 2 extra sites where adding value instead of substracting value.
*Cambodia, Ecuador, Chile, Jordan... choose the country you want with a site in the top 10 of the ranking and you will see the same.

Wilson score is a great idea to rate sites because they are individual sites. But for countries (or hotspots if you do a ranking), as they are group of sites, the use of average will still create strange positions (with or without Wilson).

I hope this help.

@AJRC:
For Congo (and the like), we have a bug, as only the steller site gets rated, but the unrated sites are ignored. That was pointed out by Els and I agree that this is wrong. Unrated sites should downweigh the rating.

For the rest, though, I disagree. Readers get the information how many sites are covered (#) in the score. So, if NZ has only 3 sites, that's there choice. And if Italy continuously decides to push through sub standard sites (e.g. Prosecco, Piedmont, Ivrea) it deservedly weighs them down comparatively. Irony of Italy is that there are several sites where the community agrees that they are stellar, they did not yet submit.

nfmungard

Unrated sites are those which the community don't know if they are good or bad. In my opinion you should take it as it dosn't exist. Why is an unknown site a bad site? Why is it worst an unknown site than a site the community agree have a low quality? I think a neutral value is more fair until it has its first rate.

I think you want to punish those countries with sites too low rated. It may be an option. Why not? It can be considered.
Anyway, please take into account the two examples a did before, and I would like you think about them:
- Choose one: (Victorial Falls) or (Victoria Falls + 4 more sites).
- Choose one: (10th rated site) or (1st rated site + 2 more sites).

I do not see any option to disagree in the chosen options. I'm pretty sure everyone would choose the options in the right side. If you are able to find a way where those "must" are considered, we would have improved the list. After that, if you want to decrease the value for those countries with sites rated below X, it may be an option.

I would not choose to decrease the value, because for me it is like the money: There are sites like a 100$ bill and others like 5c coin, but every site, even the worst rated, has a value. It's true Italy has a lot of coins, but it has some 100$ bills too.

Thanks!

I thought the country pairings were very interesting. The countries with about the same number of WHS and nearly identical rating scores.

Score: 3.32 - 3.35
Iran (24 WHS)
Japan (23 WHS)

Score: 3.60 - 3.67
Australia (20 WHS)
Canada (20 WHS)

Score: 2.85 - 2.86
Republic of Korea (14 WHS)
Belgium (13 WHS)

Score: 3.10 - 3.11
Austria (10 WHS)
South Africa (10 WHS)

Score: 3.38 - 3.41
Morocco (9 WHS)
Sri Lanka (8 WHS)
Romania (8 WHS)

Score: 3.06 - 3.09
Norway (8 WHS)
Tunisia (8 WHS)

Score: 2.96 - 2.98
Ukraine (7 WHS)
Senegal (7 WHS)

Score: 3.41 - 3.42
Jordan (5 WHS)
Panama (5 WHS)

Score: 2.88 - 2.92
Serbia (5 WHS)
Saudi Arabia (5 WHS)

Just going to chime in briefly on this one, as market research data analysis was my career before we started travelling. My inclination for this would simply be to require a minimum number of sites before a country average can be calculated. It's really not worth comparing the average of Vatican (base=1) against China (base=54), because in my opinion they aren't comparable anyway.

My suggestion would be to require a minimum of five sites in a country before calculating that country's rank. This way we're at least working with a reasonable amount of data, and more valid comparisons can be made.

If anyone's interested, my quick and dirty calculation is that the country with 5+ sites and the highest average ranking is Tanzania with an average average of 3.803, narrowly beating Zimbabwe on 3.802. Next up is Egypt, Turkey, Peru and Uzbekistan. By far the lowest ranked country is Oman, with an average score of 2.290.

Note that precise calculations might change, since I've excluded sites across 4+ countries like Struve and Le Corbusier (simply because they show up separately in my data and I didn't feel like taking the time to fix it 😂).

So yes there's still a slight bias towards smaller countries, but I think that's inevitable. That said, I think it still highlights the better countries to visit for World Heritage Sites, and doesn't require creating complicated mathematical models to solve a problem that may not really exist in the first place.