Ref Review 2012: Are we biased because Arsenal is included in the numbers?

Every now and then we have received some comments from people about our numbers. I don’t refer to the just abusing comments but people who tried to make a point. And one of the points was and I try to write it down as it has been said a few times:

“As this is an Arsenal site I find it difficult to believe your numbers because the numbers include Arsenal games and therefore the bias from you referee reviewers is shown in the numbers”.

And this was not only said by supporters from other teams but also by Arsenal supporters. So I can say that it was some kind of concern about our numbers. A concern I can understand completely. Of course I know that I and my fellow referee reviewers have done each game with the will to be as impartial as can be. The fact that our reviews are open to see and to consult adds to that of course. Nobody wants to look like a fool so we will not write down a bunch on nonsense because at some point in time it would backfire on us.

Of course there can be disagreement on some decisions we made. I remember Arsenal fans not agreeing with us when we didn’t gave a call to Arsenal because with the rules in hand we disagreed with the “general opinion”. And we have had supporters from other teams who disagreed with a decisions left or right about a game in which their team was involved.

Of course part of this concern could have been lifted by referees who support other teams if they would have joined us. Despite a few appeals, nobody really came to us and offered to help us.

So how can we show you that in fact the Arsenal games were not conclusive in this report?  Well this is what I will try to do from now on in this article.

First of all, we have based our final report on 155 games. Now if we would completely erase all the Arsenal games we would have done 117 games. And that would mean that we would have done 117 games on a total of 342 games (38-38 Arsenal games) and that makes a total of 34.21% of all the PL games. Not as much as the 40% we did in total but still more than 1 game in 3.

We will now give you two tables. An un-weighted and a weighted table. And in this table we will compare the % of correct decisions. The first number you will see is the one with all the games including the Arsenal games. The second number is the one with all the games but with the Arsenal games erased from the database. Next to that we have made a difference between both numbers and included the numbers of games on which this is based. Better see for yourself now and we start with the un-weighted % of correct decisions.

% correct decisions	Included Arsenal	Not included Arsenal	Difference	Nr Games
SWANSEA	81,730	81,930	0,200	6
AVI	79,110	74,680	-4,430	4
WBA	76,170	74,44	-1,730	4
BLACKBURN	76,080	74,830	-1,250	8
BOLTON	75,430	77,130	1,700	8
WIGAN	75,280	76,890	1,610	6
MC	74,750	74,900	0,150	30
LIVERPOOL	74,020	75,320	1,300	18
CHELSEA	73,320	73,170	-0,150	30
TOTTENHAM	72,960	72,820	-0,140	19
MU	72,800	72,580	-0,220	30
EVERTON	72,460	74,220	1,760	10
NEWCASTLE	72,290	72,410	0,120	14
NORWICH	72,100	72,470	0,370	9
SUNDERLAND	70,930	71,510	0,580	12
ARS	69,800	–
QPR	68,550	71,270	2,720	4
WOLVES	68,240	70,290	2,050	6
FULHAM	65,450	71,790	6,340	8
STOKE	64,480	66,300	1,820	6
Average difference on 19 teams			0,674

And if you now look at the teams you will see that in most cases there is a difference. And of course it is only normal that there is a difference if you remove part of the database.  But 8 of the differences are less than 1%. Then we have another 7 teams with a difference between 1% and 2%. So in total we have 15 teams (out of 19 as Arsenal doesn’t count) who have a difference in the numbers of less than 2%. That is almost 79% of the teams falling in to this category.

4 Teams have a bigger difference. But of course, as we have shown before, when a ref puts bias in a game it is obvious that there can be big changes if we remove that game. The biggest swing is found in Fulham but in one of those games we had ref Probert who got the lowest score of any ref in the season if I remember correctly. So it is obvious that this has a great impact on the numbers from Fulham. For Aston Villa we see a reverse situation as in the Arsenal game the ref did a great job and by removing this the number for Aston Villa goes down.

But the most important line is the last line we put in bold and I will repeat it for you.

Average difference on 19 teams: 0,674%

Because with or without the Arsenal games there is only a difference of 0.674% in total on the final outcome of the % of correct decisions.

Yes there is a difference but it is nothing more than that. In general there is less than a difference of 1%.

But for those who have been following us over ref and whistle in the past months or years you know that when there are un-weighted numbers we also have weighted numbers. So wait a second and find out those numbers also and the table has the same build up as the previous one.

SWANSEA	81,630	80,13	-1,500	6
AVILLA	79,810	75,14	-4,670	4
WBA	75,910	73,73	-2,180	4
BOLTON	75,180	81,12	5,940	8
WIGAN	75,100	71,98	-3,120	6
BLACKBURN	74,670	73,21	-1,460	8
Man C	73,720	74,21	0,490	30
Man U	72,580	70,42	-2,160	30
NEWCASTLE	72,290	72,86	0,570	14
CHELSEA	71,700	71,27	-0,430	30
TOTTENHAM	71,510	71,05	-0,460	19
LIVERPOOL	71,500	73,18	1,680	18
EVERTON	71,350	73,64	2,290	10
NORWICH	71,250	72,18	0,930	9
SUNDERLAND	69,980	70,54	0,560	12
ARSENAL	68,640	–
WOLVES	66,970	69,25	2,280	6
QPR	66,300	72,73	6,430	4
FULHAM	63,640	68,52	4,880	8
STOKE	61,660	61,14	-0,520	6
Average difference on 19 teams			0,503

As usual the fluctuations when we put weight on the decisions is bigger. This is of course because in the first table each decision only counts for a 0 or a 1. But in the weighted table it can be 0, 1, 2 or 3. Hence the possible bigger difference in some numbers.

We now have 7 teams with a difference less than 1%. And another 3 teams with a difference lower than 2%. Again the explanation that has been given after the un-weighted table can be applied here.  When a ref had a rubbish game this even gets a bigger effect in these numbers when we delete those games.

And yet again I want to draw your eyes to the most important number in this table:

Average difference on 19 teams : 0,503%.

So even when we put weight in the decisions we only see a difference in total between the table that includes the Arsenal games and the one without the Arsenal games of just 0.5%.

So hiding yourselves behind the fact that as this is an Arsenal site and that it is down to our bias that has given the results we have found looks a bit easy.

And if you really want to see that number of not even 1% as proof of us having some kind of bias, be my guest. You can take each number from now one and look back at them and add 1% to it. Or take 1% away from it. If you really want to do this you can take the numbers that way.  It will slightly change the final number. But it doesn’t change the final outcome of our reviews.

What was wrong, stays wrong. What was good, stays good.

And the one things that comes out of this comparison is that if there was bias it was such a small bias that we can be proud of the way our referee reviewers have done their job.

I dare even say categorically that this slight difference means that even in the Arsenal games our referee reviewers have done all what is humanly possible to keep their bias away from the reviews.

And I just want to ask you to give this thought some consideration:  just imagine that our referee reviewers did their job in an almost non-human perfect way and that our numbers FOR ALL TEAMS were correct. And if you still are not convinced about the job our referee reviewers did, I can only challenge you to come up with your own reviews. I can only tell you to challenge our reviews as they are out there in the open.

And if you don’t want to do this, or if you are not able to do this, then the only option you have is to accept that what we have done is something that nobody has done before. Like it or hate it. But that is a fact.

Next in our series we will give you all the different league tables with all the teams in it. In this articles you will be able to see who done good and who seen worse by the refs in one blink of an eye.

Footnote: As this article goes live, we are working on a scheme that will bring in referees who support other clubs to join us, so that we can have more refs and review more games. The discussions involve bringing in another organisation, and at this point we’ve no idea if we can make this work or how much it will cost us, but we are trying.  We’ll keep you informed.