Like most faculty I have to evaluate the scientific "performance" and "potential" of applicants for jobs, promotion, prizes, and grants. This is a difficult task because we are often asked to evaluate people who we don't know, are unfamiliar with their work in our (somewhat related) field of expertise, or are working in completely different fields we know nothing about. For example, I have been on a committee where I had the ridiculous task of evaluating people in fields such as veterinary medicine, geography, and agriculture! This is one reason why metrics are so seductive and deceptive.
Here I want to focus on evaluating applicants who work in an area that is close enough to my own expertise, according to the following criteria. I can read one of their papers and make a reasonably informed assessment of its value, significance, and validity. This is important because it is easy to loose sight of the fact that there is only ONE measure that really matters: the ability of a person to produce valuable scientific knowledge. All the metrics, invited talks, grants, hype, slickly presented grant applications, enthralling presentations, .... are not what really matters. They are not research accomplishments. This measure can only be assessed from the actual content of papers.
I am embarrassed to admit that I am finally trying to work harder at ``practising what I preach.'' When I need to assess a credible application I try to identify just one paper that the applicant identifies as significant or for a grant application a paper that is central to the proposed project. I then look at this paper and then go back to the (copious) paperwork of the submitted application. You might think that this takes more time. But, actually it may save time because it can be so definitive to my view (positive, neutral, or negative) that I have to read and agonise less about my assessment.
I have always done this before when assessing applicants for postdocs to work directly with me, but much more irregularly for other situations.
Ideally, this should not be necessary. Rather, a good applicant would be someone whose papers we have already read because we wanted to. However, that is not the world we live in.
Is this a reasonable approach? Do you do something like this? Any other recommended approaches?