Category: I/O Psychology

Over on my other, sometimes updated blog I made a nifty update about using Microsoft Excel to do test item analysis. At least I think it’s nifty. It’s got a spreadsheet! You can read it here.

Now that I’m actually doing this kind of thing at work again, I hope to have more stuff like this to talk about. Hopefully selectionmatter.com will be updated more than once a month.

I don’t want to go so far as to say that my other blog, which focuses on the wonderful world of employment selection, is “back” per se, since every time I make that claim I end up coughing out a handful of posts before letting the site lay fallow for another six months. But I have posted a few things there lately:

• The “name letter effect” in job Attraction
• The use of matchmaking websites to help practitioners and researchers collaborate
• A proposal for a research program to study organizational stakeholders’ reactions to selection systems
• A primer on testing issues

Happy clicking, if you are so inclined. You can even subscribe to the site via the RSS feed.

Looks like the July 2007 edition of The Industrial-Organizational Psychologist (a.k.a., TIP, SIOP’s quarterly magazine) is online, including my column that I co-author with Marcus Dickson. The column is “Good Science Good Practice” and in each issue we try to highlight research that bridges the gap between scientists and practitioners. Preferably without resorting to cage matches. This time we talk about things we saw at the 2007 SIOP conference in New York. Here’s a smidge:

Later in the conference a panel of other experts on employment testing gathered and discussed how certain scientific and methodological advances in test validation were faring in the field. Specifically, the symposium, entitled “Validity Generalization in the Workplace,” discussed alternatives to traditional test validation strategies that are widely accepted as useful and acceptable by researchers and other experts in the testing industry, but which are sometimes regarded as inscrutable or untested (pardon the pun) by others. Examples included job component validity, validity transportability, and meta analysis. The panelists, all of whom use these validation tactics in their everyday work, explained that they are most often useful and necessary when traditional approaches like criterion-related validation are impossible due to time constraints or the lack of enough incumbents to achieve adequate statistical power for the required procedures.

The incumbents were also forthcoming with many of the sometimes irritating realities of this kind of research, including the fact that one must be able to replicate job analysis procedures for transportability studies and that there was still a certain amount of legal risk involved in these processes given that the courts have yet to build up a strong history of neither support nor opposition for these approaches despite their widespread acceptance among academics and other science-minded practitioners. The presenters also sheepishly provided a somewhat unsatisfying answer to the question of “how close is close enough” when it comes to comparing the components or requirements of two jobs for purposes of transporting validity: “That’s up to you to decide.” It seems there is still a place for professional judgment in the brave new world of alternative validation approaches.

Also of interest for those you who get the print version of TIP, they’ve changed the cover format to include photographs on the covers. A welcome change, and I since photography has become a hobby of mine I’m hereby making it my life-long goal to get a photograph on the cover. That way I’ll be both inside and outside the magazine, which will pretty much make me unstoppable.

After almost a year off, I’ve decided to revive my other blog, SelectionMatters.com. It’s less of a personal journal and more of a professional one, with an emphasis on I/O Psychology, employment testing, recruitment, and selection. I had a few people at the SIOP convention last weekend either tell me they recognized my name from there or that they used to read it and were disappointed that it went away. This was kind of cool, since one of the reasons I made that site was to build my network and give prospective employers something job-related to uncover if they went and did a Google search on me.

So if you’re so inclined, check it out again. I’ll be updating every Friday.

Just noticed that the January 2007 issue of The Industrial Psychologist (TIP), the quarterly publication of the Society for Industrial-Organizational Psychology, is online, even though I haven’t gotten my print copy yet. As usual, my column on Good Science Good Practice is in there.

This was an interesting one to write. Most of it centers on a series of articles in the Journal of Occupational and Organizational Psychology (JOOP) that ask if the journal should be more inclusive of less scientifically rigorous (and, let’s face it, less boring) articles so that it can serve a wider audience. Not everyone starts to dance from foot to foot in glee when presented with a giant table of statistics or a Methods section that takes days to read.

Of course, nobody was suggesting that JOOP should become, as I say in the column, “the I-O equivalent of People Magazine or Highlights.” But it’s still a sticky issue. How far do you bend to accommodate people without the advanced education or experience required to consume (and, perhaps more importantly, to make use of) traditionally erudite scientific research? At what point does such accommodation start to undermine the science and the mission of your publication?

Read the column for the full discussion.

Apparently my column in the October 2006 issue of The Industrial Psychologist (or “TIP” to the cool I/O Psychologists) is out. I haven’t gotten my print copy yet, but it’s also available for virtual perusal by clicking right here. God, they still have that horrible picture of me where my head looks really tiny. I need to fix that.

Anyway, I’m sure this is of interest to exactly one regular reader of this site (hi, David!), but in this issue my colleague and I once again discuss research and other articles that exemplify the much sought after Scientist Practitioner. Here’s a taste:

Finally, Roth, Bobko, and Switzer recently published an article in Journal of Applied Psychology that illustrates how practices can sometimes drive research instead of the other way around. The authors model the behavior of the “4/5ths Rule” for determining the presence of adverse impact in a selection system, but they do so using a variety of computer simulations in both hypothetical and realistic situations. For those of you in need of a primer, the 4/5ths rule, whose origin it turns out is more indeterminable than you might guess, is a relatively simple rule of thumb that says that a selection system creates adverse impact if a protected class’s selection ratio is less than 80% (i.e., four fifths) of the selection ratio for the most often selected class. This procedure is unfettered by complex statistical significance tests and thus preferred by courts and government agencies who don’t want to require such specialized knowledge of key decision makers when it comes to evaluating adverse impact claims.

Riveting, right? I could talk more about Samantha’s potty training instead if you’d like.

It’s no secret that a lot of companies are looking to cut costs and increase effeciencies in their hiring processes. While pre-employment testing has demonstrated benefits in terms of getting better people in the job, it can be costly. Assessments for higher-level managers or executives can costs several thousand dollars per person, and even simpler tests of basic aptitudes like computational ability or language skills can rack up costs pretty quickly.

Fortunately, I recently found the answer to this problem while strolling through the “Everything’s A Dollar” store in Tulsa, Oklahoma. There on Aisle 3 between off-brand dishwashing liquid and various mass-produced knicknacks sat a rack full of these:

There were also booklets for math and spelling skills, and for simpler jobs like “Consultant” there were books that test your knowledge of shapes and colors.

Brilliant. I mean, you can’t beat that for a buck, and for an added bonus adverse impact will work in girls’ favor for once!

A couple of weeks ago I found myself wandering through one of those stores that sells all kinds of novelty tee shirts with pithy, snarky, rude, and otherwise clever sayings. Silly stuff like “I’m with Stupid” or “Bikini Inspector.” And I got to thinking that there’s a market here that’s not being served. Where are the novelty tee shirts for Industrial/Organizational Psychologists? It’s a wonder we bother to get dressed at all!

To remedy this, I came up with a list of slogans suitable for tee shirts or maybe the occasional bumper sticker.

Shirts for Everyone

• I’m with low g →
• I neither agree nor disagree
• Subject matter expert
• Meta analytic evidence suggests that you should just shut up
• Low emotional stability
• You’ve got a criterion problem
• More research is required
• BFOQ SME ADA CRA EEOC WTF?
• Mom says I’m a Type II Error
• I cause adverse impact

Snarky Shirts for Single Women Out at Bars

• Your confidence interval is too wide
• You have a skills gap
• “Stupid” seems to be one of your core competencies
• Sorry, that position has been filled and I’m not recruiting at this time

Snarky Shirts for Single Men Out at Bars

• Beer: increasing measurement error since 1935
• Rejected like H₀
• Unit weighted
• I have high reliability

Shirts for Babies

• p > .05 …MUCH greater
• I’m significant at alpha = .05
• Corrected for shrinkage

Now, if you laughed at any of those, congratulations –you’re a dork! Join the club and feel free to add your own suggestions in the comments section.

Selection matters, yes it does. So I decided to create a new weblog for it: Selection Matters dot com:

“Selection” in this context deals with employment selection and a constellation of related sub-topics: employment testing, interviewing, recruiting, employment law, I/O psychology, and more. There are a lot of Human Resources blog out there and there are a lot of employment law blogs out there, but none of them focussed narrowly on employee selection, which is my area of expertise. So I thought it would be fun to create one. And, honestly, I also wanted to network in the I/O community and give future prospective employers something relevant to find when they did a Google search on me.

I waited a few weeks before mentioning the site here or trying to promote it elsewhere, just to see if it was really viable. Apparently it is, as I’ve had enough material to make posts nearly every day. And not one of them has a picture of Samantha in it. Yet.

I think that the site is also the best one I’ve made from a code point of view, though the real pros out there would probably still call it armature. It’s totally CSS-driven, without a single HTML table or even image tag outside of the blog posts. All the layout images are handled through CSS, as is the positioning. It also makes way more use of Movable Type than jmadigan.net, with essentially every single page being handled by MT modules, indexes, or inserts. The only thing I use FTP for is to transfer images for individual blog posts, and I’m working on setting MT up to handle that, too.

So anyway, check out the site and if you like it, bookmark it or subscribe to a RSS feed. And let me know that you’re visiting!

I’m not sure why I haven’t mentioned this before, but in the last few months I’ve been fortunate to be part of a small group of people who decided to create a professional association of industrial/organizational psychologists working or studying in the San Diego Area. Starting an organization like this was both weird and banal at the same time. The idea had actually been bouncing around between myself and a few people at work for over a year, and we eventually worked up to just picking a date, calling a meeting, and contacting everyone we knew. The response was pretty astounding –over 100 I/O students and professionals showed up. We chatted a bit about what we should do and then chatted about how often we should meet, and then –presto!- We had ourselves a professional organization.

The only thing we really had to argue about was the title, which turned out to be “San Diego Industrial/Organizational Professionals” or “SDIOP” for short. There was some rancor over the “P” standing for “Professionals” or “Psychologists” as the vast majority of us (maybe even all of us) were not technically licensed to call ourselves psychologists. Having had all interest in the name game ground out of me in the course of naming products at GameSpy, I didn’t care as long as we didn’t come up with anything that abbreviated as “NAMBLA.”

Over the course of the next few weeks we organized our first meeting and I got myself elected as an officer. Not president, though –that went to the very capable Ben Schneider, who is about as an experienced and noteworthy a guy as we could have hoped for to be in charge of the thing. I ended up as Secretary/Webmaster, which worked out pretty well as one of the first things I did was bang out our website:

Kind of fruity, I know, but I like it. It was fun to design a website for once that wasn’t a blog. I actually would have linked to it earlier here but it was kind of …fundamentally broken in one spot and I didn’t figure out how to fix it until yesterday. Sorry. Nobody complained in the meantime, though, so maybe it wasn’t as bad as I thought.

At any rate, if you’re an industrial/organizational psychology professional or student in the San Diego area, we’re meeting again in August. The specific place and date are still being worked out, but should be posted on the website if anybody can get that lazy webmaster to do any work.

Speaking of Harry Potter, this kind of made my mind boggle. TalentSmart, a company specializing in leadership and employee assessment, wrote a white paper about the different displays of Emotional Intelligence in the Harry Potter books. It describes one dramatic exchange between Harry, Ron, and Hermione, then shares the following wisdom:

Let’s stop here a moment and notice that Ron is highly disconcerted by Harry’s suggestion and Hermione feels anxious. Harry’s friends know him well enough to realize that something major is upsetting him. Harry jumps to the conclusion that their respected schoolmaster has doubts about him. In response, Ron focuses on Harry with “don’t be stupid,” while Hermione focuses away from Harry on to Dumbledore: “Of course he doesn’t think that!”

Those familiar with the story know what happens next: Harry becomes intensely bitter and shouts at his classmates. They are left to watch his frustration “pouring out of him, his frustration and lack of news, the hurt that they had all been together without him.” Once Harry’s feelings burst out this way, there is nothing to hold them back. A painful summer cumulates in an emotional explosion directed at his schoolmates.”

I still can’t figure out if this is silly or a brilliant move to bring one part of I/O psychology closer to the masses (and market your product in the meantime). But at any rate, wonder if examining emotional intelligence in the context of a pissed off teenage wizard is the kind of thing that will rack up the credibility for a construct that’s already kind in a lot of people’s “iffy” column.

Then again, this is a company that offers a pop culture fueled training film called BRAINS! apparently for use in teaching zombies about the wonders of emotional intelligence. I’m so confused.

Here’s an interesting story that relates, in a way, to employment and selection. It’s about two girls who did a bit of a social experiment for a high school class. Both girls looked pretty similar to start with: tall, thin, blonde, and attractive. The hook is that one girl dressed up in preppy clothes that presented a clean-cut and generally “all American” look. Her friend went goth, wearing heavy eyeliner, black clothes, black hair dye, and a bared midriff. See the picture to the right if you need a visual aide.

Then both girls went to apply for jobs as register monkeys at Abercrombie & Fitch, an almost overbearingly trendy and preppy clothing store. I think you can probably see where this is going.

The A&F manager practically stumbled over himself trying to hire Ms. Preppy, despite the fact that the girl said she had no previous retailing experience and no references. I think she may have even said she was mildly retarded and was always being blamed for stealing stuff. Ms. Goth, on the other hand, was treated like a pariah by the (presumably) same A&F manager, despite the fact that this girl said she had worked two retail jobs before and had great references. The girls then repeated the experiment at Hot Topic, a much sluttier vendor of midriffs and miniskirts. The results were reversed, though not quite as drastic.

Of course, this is all utterly unscientific. It’s two girls doing clumsy manipulations on an ill-defined variable and running only two uncontrolled trials. So you aren’t going to see their stunt in the next issue of Journal of Applied Psychology (much to their dispair, I’m sure). Thing is, it doesn’t need to be scientific. There’s already plenty of scientific research showing that you’re more likely to get interviewed or hired the taller you are, the thinner you are, and the more professionally dressed you are. Same for the “like me” effect that makes an interviewer like an interviewee the more similar they are in appearance. So saying that looks really do matter shouldn’t really elicit much more than a resounding “Duh!” from the audience.

But while that may be true, it’s still incredibly easy (and a lot safer) to instead focus on a handful of simple measures to find the occasional diamond in the goth, even for low-level jobs like this. Previous work experience is a no-brainer, and I’d like to slap the A&F manager for completely overlooking this. I don’t put any stock in references (research shows there’s hardly any variance and they have almost no predictive validity), but a few simple interview questions could screen out obvious misfits. If you want to do even better you could add biodata. Even better, cognitive ability and personality tests could be used.

Sure, the retail managers in this story may think that people who look like Ms. Preppy work out better than Ms. Goth, just because that’s the way it is, and they may even have some examples to back this up. But that’s a clumsy hiring practice –measure what you need to measure and nothing else. It’s so easy to do so much better.

In fact, given all that, the funniest part of this story is the “insta-poll” the reporting website was running:

So, after illustrating the dumb generalizations made by retail managers about skin-deep (heck, not even that; clothes deep) features, which dumb generalization would YOU make, dear reader? Heh.

At any rate, I deal with this kind of “I just KNOW a good employee when I see one” fallacy all the time in the professional world, and it’s alarming to see that it has seeped into our shopping malls. Won’t someone think of the children!