Google Summer of Code Application Evaluation

Originally based on the NESCent Phyloinformatics Summer of Code procedure by Hilmar Lapp.

Overview

There are 3 rounds of scoring.

Round 1

Mentors review the applications for their project idea and score them (see below for specifics).

Round 2

All mentors look at (at least) the top-scoring applications for project ideas other than their own and score them too so that we get a poll on consensus or divergence of opinions.

Round 3

Where necessary, organization admins weigh in to ensure optimal alignment with OBFs projects and objectives.

Keep in mind that typically fewer than 40% of project ideas are funded. Some project ideas may not receive any strong applications, so the decision will be easier in those cases; however, we do expect that we'll have to make some very tough calls.

With that in mind, here are suggested guidelines for scoring in the first round.

Round 1 Scoring Guidelines

1. Review the applications that you would be primary or secondary mentor for (this should be apparent from the title). Point out any missing information, lack of detail, and any aspects of a proposed plan that you disagree with in the comment text box, check the box next to 'Public review', which makes it visible to the student, and click 'Submit'. The student should respond to the these and can presumably edit his/her application to respond to your request for information or criticism (at least they could last year).
2. For scoring, leave the 'Public review' box unchecked (i.e., the student will not see it, but other mentors will), change the Score drop-down to where you want it, add explanatory comment for your score, and hit 'Submit'. Scores are cumulative, so you can revert a score by subtracting what you previously added. Scores without explanatory comment are hard to weigh in the case of differing opinions, so please do add a brief (!) statement as why you are giving this score.
3. If you are a secondary mentor, to avoid confusion I suggest you propose your score as a private comment (i.e., as above, except write the score in the comment and leave the drop-down at zero) for the primary mentor to take into consideration, or email the primary mentor.
4. Give the top 2 or 3 applications for your idea that may warrant funding (see below) a graduated positive score to establish your preference (+4 for the best, +3 second, +2 third). Add +1 to the best one (for a total of +5) if your comments only point out refinements and in principle you're happy with the application as is.
5. Only give a positive score if you have confidence that the student is already or would after requested improvements be worth accepting, and you spending your mentoring time. There may be fewer than 3 of those for your idea!
6. If you are a primary mentor, request mentorship by selecting 'Wish to mentor' on the left hand column 'Proposal Actions' box for those you have scored positively. This should be no more than 3.
7. Keep in mind for any of the above that total scores you assign aren't final. By adding or subtracting you can shift an application from one category to another, and you can withdraw your mentorship request later.

This should leave us with those applications that a slot is being requested for at score +4 or +5. It would be great if we can get to this point by Tuesday night. (Yes, you can change your mind later.)

Round 2 Scoring Guidelines

In the second round of scoring, all mentors will be invited to look at all applications scored positively in the first round (except those they'd mentor themselves), and add scores corresponding to the degree they agree an application should receive a slot. We'll wait with this until Tuesday and the first round is hopefully complete.

Interviews

Finally, the primary (or secondary) mentors should arrange an interview (if one has not already been done) with at least their top applicant, or possibly the top two applicants if the decision is close. This lets the mentor better assess the student's communication abilities and whether personalities (yours and theirs) match well enough. The interview should take place *during the review period*, the earlier the better, and you should summarize your conclusions from it a private comment attached to the student's proposal. The interview may be by email, chat, Skype, or phone, depending on what you feel lets you best make the assessment.

Things to keep in mind

1. Your top student may also have applied to another org and may stand to be accepted there too. Students can only be accepted for one project, and hence one org; if in the end that student goes to the other org, you lose the slot if you don't have a second one of comparable quality "wait-listed", which is the reason you want to keep working with more than just the best one (though doing so with more than 3 is a waste of time).
2. When ranking student applications against each other, weigh the complete picture, a significant part of which is the long-term prospect of the student continuing to enrich or advancing your open- source project or bioinformatics as a field through what you will teach him or her over the summer. If an application seems more solid technically than another but the student's career and personal interests and past activities suggest that he/she would likely disappear immediately after Summer of Code ends, would you still prefer to invest your mentoring time in that student? Our goal in participating in GSoC is not just to get things done, but to train and recruit future bioinformatics open-source software contributors.