Forum:Rules Proposal and Vote

Forums: Index > Rating Committee > Rules Proposal and Vote

Voting ended. The proposal is approved.

Votes (Voting ends October 19, 2009 at 23:00 UTC)

To vote, please insert your username and timestamp at the bottom of the appropriate list. Unsigned or anonymous votes will be reverted.

Yea

1. Yea. --DanielDraco 23:10, October 5, 2009 (UTC)
2. Yea. --Andrew Arnott (talk, email) 23:28, October 5, 2009 (UTC)
3. Yea. Surgo 00:03, October 6, 2009 (UTC)
4. Yea. --Jota 00:11, October 6, 2009 (UTC)
5. Yea. --Ganteka Future 00:16, October 6, 2009 (UTC)
6. Yea. - TarkisFlux 00:25, October 6, 2009 (UTC)
7. Yea. --MisterSinister 05:50, October 6, 2009 (UTC)
8. Yea. --Sulacu 17:30, October 6, 2009 (UTC)
9. Yea. → Rith ^(talk) 20:02, October 6, 2009 (UTC)
10. Yea. --Deranged. 20:37, October 6, 2009 (UTC)
11. Yea. --Eiji Hyrule 17:45, October 7, 2009 (UTC)
12. Yea. Summerscythe 20:52, October 7, 2009 (UTC)
13. Yea. --ThirdEmperor 05:50, October 9, 2009 (UTC)
14. Yea. --  Sam Kay   talk    contribs    email   14:05, October 11, 2009 (UTC)

Nay

1. No. --Jay Freedman 19:21, October 6, 2009 (UTC)

Proposal: Daniel Draco model

I propose that the system by which an article's level of favor (no distinction, bronze star, silver star, gold star, or featured article) is determined be defined as follows:

1. Each Rating Committee member (hereafter referred to as a "Rater") gives an article a rating of "good" or "excellent" based on his/her own subjective opinion.
2. A "good" article is loosely defined as being of noteworthy quality (by whatever criteria the Rater sees fit, e.g. formatting, power, and clarity), but not fitting the criteria of an "excellent" rating.
3. An "excellent" article is loosely defined as being of outstanding quality (by whatever criteria the Rater sees fit), and being in the highest tier of the Rater's personal standards (which will likely vary in breadth from Rater to Rater).
4. Counting a "good" rating as 1 and an "excellent" rating as 2, the overall favor of the article is determined by the sum of the ratings; 0 to 2 grants no favor, 3 to 6 grants a bronze star, 7 to 10 grants a silver star, 11 to 14 grants a gold star, and 15 or higher grants the status of being a featured article.
5. Raters are strongly discouraged from allowing the ratings of others to impact their own ratings. For example, suppose an article has a total rating of 10 (silver star). A Rater who has not yet rated the article thinks the article is "good", but does not think it is worthy of a gold star. This Rater should still grant their favor of "good", and therefore increase the overall favor of the article to a gold star. This is because an individual rater should only be concerned with whether an article is worthy of "good", "excellent", or no favor at all; the possibility of bronze, silver, gold, or featured status should not factor into any Rater's rating.
6. A Rater is allowed to change his/her rating at any time, e.g. after further thought or after a change to the article. For the same reason as Point 5 (above), Raters are discouraged from changing their ratings due to the ratings of others.
7. If a rated article is changed, it is the responsibility of the one who made the change to inform the Ratings Committee of the change via this forum so that the ratings can be reviewed and, if deemed necessary, revised.

Reasoning

Ganteka Future's interpretation of Surgo's system appears at first to be an effective way to handle RC ratings. However, after further thought, I realized that it brings us back to one of the same issues that we had with the rating system on the paleowiki. Under the Surgo-Ganteka model, a Rater only gives his/her favor if he/she wants the article to move on to the next level; this requires him/her to judge the value of no-distinction vs. bronze vs. silver vs. gold vs. favored, and give or not give a favor based on that distinction. Essentially, he/she must decide a rating from 0 to 4, which is not much better than the 0 to 5 system of the paleowiki. My system seeks to resolve the issue of the original Surgo model (wherein a good, but not great article can become featured with seven mild favors), while avoiding the issues of the Surgo-Ganteka model. I chose the numbers for the following reasons:

1. No single person can give an article a star.
2. An article cannot be gold without at least one "excellent" rating.
3. Any level can be achieved with only 8 of 10 committee members involved, which is especially important when the article in question is that of a Rater and there are therefore only 9 Raters who can rate it.

Discussion

Comments, questions, etc. go here. --DanielDraco 23:10, October 5, 2009 (UTC)

I have my doubts that this will resolve the issues you're concerned about DD (or that the issues will actually come up for that matter), but I don't generally turn down greater control and granularity. - TarkisFlux 00:25, October 6, 2009 (UTC)

Oh, I'll also be passing on rating anything in the list until this vote is settled. I don't really want to go back and track down all of my old ratings and convert them. - TarkisFlux 05:39, October 6, 2009 (UTC)

This is a nicely devised system and will serve the purposes of the wiki well, as long as every rater can be sufficiently objective to the material being rated (i.e. give honest ratings that are not predicated on who made the material in question, or by the established balance point chosen). --Sulacu 17:30, October 6, 2009 (UTC)

I vote No until a score of "0" or "playable" is included. I believe a rater can take the time to rate an article and come to the conclusion that it is "not good" however still technically usable. I also believe a recorded rating of "0" is not the same as a "needs improvement template". --Jay Freedman 19:21, October 6, 2009 (UTC)

Not rating at all is analogous to a rating of 0. Note that, even though this is a passive way of expressing disapproval, it is designed so that this tacit disapproval limits the favor that the article can achieve. For example, let's say three Raters think Article X is crap, and so they don't give it any favor. Because the article is missing out on these ratings, it can never achieve the status of Featured Article, and is very unlikely to reach Gold status. However, there's no reason we can't have a template for an official No Favor (a.k.a. "0") decision by a Rater, for sake of completeness. --DanielDraco 19:44, October 6, 2009 (UTC)

Edit conflict, so this may be superfluous. Anyway, if something is nominated, reviewed, and decided not to worthy of any favors (RC members are supposed to at least mention that they have viewed something so people know it hasn't been neglected) but not in need of assistance either, it is my understanding that would constitute a 'zero' in technicality, if not in fact under the rules of the proposed system. Whether or not the system actually needs to stipulate what a 'zero' is in said case is a matter for debate, but I think this limbo you speak of already does exist, just not in any official capacity. -- Jota 19:45, October 6, 2009 (UTC)

The balance system with Monks and Fighters and such is a bit unclear to me, but as this seems to be unconnected to the rating committee I am pro. I do think it's necessary to include a 0 category, just so the author knows that his/her article has been rated and deemed unworthy (as opposed to unrated). Deranged. 20:36, October 6, 2009 (UTC)

Looked it over, looks good man. This plan makes FA out of the more globally liked stuff and doesn't afraid of anything. Gentlemong, I approve. -- Eiji Hyrule 17:45, October 7, 2009 (UTC)

I saw that it says that a rater should ignore what star their rating will push the article to, but I don't think that'd work very well; let's say an article needs at least 11 to get a gold star and it currently has 10 points. If I believe that the article is well worth a silver star but doesn't reach the level of gold, I'm probably not going to post anything despite the fact that I think it's a good article. Perhaps another suggestion might be averaging the rating of the raters, and needing a certain number to reach different levels?

For example, we could have three ratings; "fair", "good", "excellent", and "outstanding". An article could reach a bronze star once at least one person has marked it "good", a silver star once the average of ratings is good and at least 3 people have rated it to this average, a gold star once the average of ratings is excellent and at least 5 people have rated it to this average, a platinum star once at the rating average is outstanding and at least 7 people have voted, and the status of favored article if the average is still at outstanding and 8 or 9 people have voted on it.

What do you guys think? --Ghostwheel 17:47, October 7, 2009 (UTC)

If I believe that the article is well worth a silver star but doesn't reach the level of gold, I'm probably not going to post anything despite the fact that I think it's a good article.

If this was something for the general population, I'd agree with you -- but it's not, it's for a small, carefully-picked committee. Put simply, if you can't be trusted to act in accordance with the RC guidelines (which say ignore the rest of the rates when rating yourself), then you shouldn't be on the committee. Surgo 17:54, October 7, 2009 (UTC)

Instead of going straight for my throat, why don't you think about my proposal for a moment? In the above proposal, if lots of people like an article and there are a few who don't, it can easily be pushed to gold star status quite easily. On the other hand, if an article is of excellent quality but not many people come around to rating it, it can languish at a low status and looks bad just because few people posted on it.

On the other hand, with the above suggestion you can have the average that people have rated so far and note that the reason an article doesn't have a higher star status is because not enough people have rated it. Furthermore, the minority have a little more pull and can rate things at "fair" rather than being denied any rating whatsoever, as well as being able to skew the average slightly, though obviously the majority will have the largest say.

If you don't like the suggestion, say why. But making comments that can easily be construed as threats is something that's best left on the sucky wiki. Let's not have another round of that? >_> --Ghostwheel 17:59, October 7, 2009 (UTC)

→Reverted indentation to one colon

That's getting to be a bit much Ghost, even for me, and it still doesn't resolve the issue you've raised. If you look on the star as a direct measure of it's quality instead of as an indirect measure of it's average rating (which it is under any proposed system) you leave yourself open to raters intentionally sabotaging their rating because it's not 'worth' the next star. If I didn't think an article was 'worth' a gold in your setup, I'd just give it a fair or good rating instead of what I actually thought it was worth. My vote could be overridden by others, but that already happens in all of the other proposed systems.

You're going out of your way to do what all of the other systems already did (though perhaps not as explicitly) in that you want to make Silver mean both 'this article has had some really good ratings, but hasn't been rated by a majority of the RC yet' and 'this article was rated good by the majority of the RC'... but that's what already happens in DD's system. It happened in GF's too, though it was more vulnerable to sabotage.

We need to stop looking on the actual stars as a measure of quality, and we need to babble about the actual quality of the article in it's talk page when we rate it and then let the star be the indirect measure of quality that it winds up being. Anything more direct than that and we may as well just vote things stars directly and stick the average on article. - TarkisFlux 18:09, October 7, 2009 (UTC)

That's two edit conflicts now...so here's what I said in reply to ghost:

I don't see why you took it as a threat, especially when I wasn't even talking about, specifically, you -- it's a pretty simple fact that if a guy who is selected to be on a committee can't even abide by the rules of said committee, they shouldn't be on the committee. How is that controversial? It's not a matter of not liking your suggestion (though I don't like it, and I will explain why in the next paragraph), it's a matter of the committee is picked because A) they have complete and somewhat different viewpoints on articles and B) they're able to be objective in their following of the rules.

Averaging doesn't work all that well. With 4 ratings, you have to translate them into numbers -- so let's say the obvious: 1, 2, 3, 4. If something wants excellent status and 5 people have to have rated it, 4 could rate it excellent and 1 could rate it fair and then it'd be only "good" -- and it would never actually reach "excellent" status, no matter how many people actually rated it excellent just because one guy rated it as fair. Same thing happens with one guy rating it as "good" and everybody else rating it excellent. Surgo 18:11, October 7, 2009 (UTC)

@Tarkis: If one doesn't take the rating to mean of what quality the article is, then that might make sense--however, the average user isn't going to look at the talk page. All they're going to see is the star on the article, and will assume everything from just that. Furthermore, the currently proposed method of rating gives a lot of emphasis to some raters, and very little to others. A person who's raising an article from 5 to 7 (silver star) isn't going to have nearly as much of an effect as someone who raises it from 10 points to 12 points (silver to gold star). That said, if we don't take an article's star rating to mean its quality, then it doesn't matter too much.

@Surgo: Just the switch from the singular "you" to the plural "you" wasn't very explicit so it appeared to be more personal than you intended it to be. Also, let's say from the above fair = 0, good = 1, excellent = 2, and outstanding is 3. If 4 people rate outstanding, 2 rate excellent, and 2 rate fair then an article is going to end up excellent--so even if some people sabotage their vote it wouldn't matter all that much as long as a few extra people post too. However, on the IRC channel you mentioned instead of averages doing a mode of the ratings--I think that's a good idea. Perhaps with the ratings, Fair, Bronze, Silver, and Gold? --Ghostwheel 18:29, October 7, 2009 (UTC)

Well, the average user isn't going to know what the star means at all unless they go and look at the RC page where we can define it as whatever we want and show cases where an article would have achieved that rating... so the fact that they probably won't look at the talk page to see specific criticism isn't really meaningful. Your second objection that the current system gives extra weight to particular votes also falls flat, given that you're using an averaging system that does the same thing in a different way. If you want to avoid that sort of vote weighting, you need to replace 'average of' with plain old 'at least'. So it's bronze if at least two people rate it good or above, silver if at least 4 people rate it good or at least 3 people rate it excellent or above, etc. No more vote averaging, just plain old tallying so no one can bring down the average and thus the article. I'm not particularly in favor of a system like that, but if you wanted to avoid the issues you think we have you need to do something like it. - TarkisFlux 18:52, October 7, 2009 (UTC)

I like the tally idea. Not the average idea. (Sorry Ghostwheel.) But I still want a rating of "0". So (like Ghostwheel commented), if an article has "10 points" and you rate "0", then you have neither hurt nor helped. And you can be honest about. If you really thought the article was "good" and it already has "10 points", maybe it does deserve a brighter star? For consistency. Tts not a big deal anyway. DM's will always check any new stuff regardless of "star power?".--Jay Freedman 19:49, October 7, 2009 (UTC)

Wow, a lot of things were discussed since I last checked. Alright, refuting a couple things. "If I believe that the article is well worth a silver star but doesn't reach the level of gold, I'm probably not going to post anything despite the fact that I think it's a good article." As Surgo said, that's the responsibility of a Rater. If a Rater cannot express their opinions as they are supposed to (with no rating, 1, or 2) instead of aiming for a specific star level, then they should not be a Rater. "On the other hand, if an article is of excellent quality but not many people come around to rating it, it can languish at a low status and looks bad just because few people posted on it." This is also the responsibility of the Rater. If a Rater cannot or will not keep up a reasonably high level of rating-activity, then they should not be a Rater. Lastly, I'd like to point out that we should be very careful to limit the number of possible ratings a Rater can give. I tried to keep it as low as possible while avoiding the issues of the 1-or-0 system. --DanielDraco 20:45, October 7, 2009 (UTC)

"A person who's raising an article from 5 to 7 (silver star) isn't going to have nearly as much of an effect as someone who raises it from 10 points to 12 points (silver to gold star)" Uh, yes he is, the guy raising it from silver to gold is only doing so because the first person gave it two points. I also think a rating of 0 is unnecessary, why not just have each of the raters stick their names in a box or something to show they looked at the article? By the way could somebody recalculate the percentages? I have no idea how to do so.--ThirdEmperor 08:40, October 9, 2009 (UTC)

Exactly right, ThirdEmporer. Sequence has nothing to do with it; no matter what order they rate in, every Rater plays an equal role. --DanielDraco 14:16, October 9, 2009 (UTC)

So, is this gonna get finalized or what? → Rith ^(talk) 18:52, October 14, 2009 (UTC)

I arbitrarily set voting to end after two weeks, on the 19th. At this rate, though, I don't see it failing. --DanielDraco 00:55, October 15, 2009 (UTC)

(Oh, must've missed that) Well, awesome, only five days left. I'll go ahead and try to make edits to my favor template. → Rith ^(talk) 01:52, October 15, 2009 (UTC)

→Reverted indentation to one colon

If I remember correctly from the RC meeting, we decided that all Raters will use the same templates. So it may be rather fruitless to edit your personal templates, when we're just going to switch to standardized one anyway. --DanielDraco 14:49, October 15, 2009 (UTC)

Missed that part of the meeting I guess... Still, that's fantastic, less work to do. - TarkisFlux 16:24, October 15, 2009 (UTC)

Since the vote has passed, I have updated the author template to account or the new favor levels. - TarkisFlux 19:25, October 20, 2009 (UTC)