A CykoMetrix Spotlight Production

Every week, the Spotlight shines on an amazing professional with a story to tell and lessons to teach. Welcome to the CykoMetrix Spotlight.

The following is an adapted transcript of the exchange between Sylvain Rochon, CMO at CykoMetrix as host, and Dr. Ellen Julian, Psychometrician & Testing Consultant.

Sylvain Rochon: Hello and welcome to CykoMetrix Spotlight. My name is Sylvain Rochon, the Chief Marketing Officer at CykoMetrix, a SaaS-based platform that allows companies especially training companies, consulting companies, and staffing companies do psychometric assessments and track their clients’ moods, psychometric values, cognitive abilities and all sorts of other stuff over time so that they can apply training and development in the continuous matter to help them grow and become more effective, derive ROI, and all that cool stuff.

Today, I have a very interesting guest, Dr. Ellen Julian. She is in Maryland. I was very excited to connect with her specifically because of her job. She can correct me if I am inaccurate in the description. Her entire job is to assess psychometric assessments in different ways. We’ll get into that and the details because that is the topic of the talk. Dr. Julian has been fascinated with tests her whole life, and honestly, so have I. I love tests. I like to take tests but not evaluate them. She made a career out of it and has a doctorate that is related to that and more studies. She’s directed the testing programs for the Medical College [Admission Test] Administrations or what we commonly call the MCAT, and also ultrasound certifications. 

She is now a Founder and Principal Consultant at Julian Psychometric Consulting which offers strategic decision support, and problem-solving analysis for certification and other assessment programs. So, she’s all about testing assessments and how they are classified, and the certifications around that. She already educated me a while back when we were talking about the differences between certificates, certification, and accreditation. We thought it would be a great idea to expose the differences and to talk about the different applications of these in the market. So, welcome to the Spotlight. Ellen, if I can call you Ellen instead of Doctor. So informal. Welcome to the Spotlight.

Dr. Ellen Julian: Thank you. I’m delighted to be here.

Sylvain: So, right into the topic. There is a lot of talk in the industries, in plural, about getting a certificate versus getting a certification in something, and accreditation programs. In particular, certificates and certifications, even I, you corrected me, are different even though they sound extremely similar. The roots of the words are the same. So, help us, just at the base level, understand the differences between the three.

Dr. Ellen: Certainly. A certificate is a recognition that you completed a course of study and met all qualifications required, which may often end with an exam. So, there is often an exam involved, but a certificate is “You took our course. You did everything we asked you to do, and we want to tell the world that you did this. So, here’s your certificate.” Your Bachelor’s, your Ph.D. is really just a certificate. You did everything you needed to do. When you finish that, they put their stamp of approval on what you did and gave you a certificate. It is yours forever. It belongs to you. They cannot take it away; whereas certification is something you get for a given time span, like 3 years or 5 years, and you have to renew it usually, at least with continuing education or re-exam. It is a testimonial of an impartial industry group saying, “This person knows these things. They can do these things, and as long as they hold this credential, this certification, and it’s valid, we put our name behind the fact that they can do this thing.” which is very different than a bachelor’s saying you actually remember your American history or whatever. So, the certification is offered by a particular industry. Now it may have an organization sponsoring it, but it is supposed to reflect an entire profession – the knowledge, skills, and abilities you need to do in order to practice that profession. So, it is more skill-based than many certificates, but some certificates are also that.

So, that’s certification. If you want to know if a person’s any good, you look to see if they have a certification, but how do you know if that certification is any good? You look to see whether it is accredited. So, the accreditation is for the certifications or the certificates, not for people. Accreditations are for programs or organizations, whereas certifications and certificates are for people. So, the accreditation, the one I work with, is formerly known as ANSI, American National Standards Institute, implements ISO standards that are international. There is a set of standards for certification exams that needs to be impartial. It needs to be fair. It needs to be valid. It needs to be reliable, all of these good things. And then my job as the psychometric assessor for the accreditation program is to go out and lift the hood on certifications and look at how they’re actually built. Are they involving the right kinds of subject matter experts to make sure that all perspectives are included in the test? You don’t want to have just a big industry people and no representatives of the small self-employed version of whatever your profession is.

You’ve got to have a balanced representation. You need to have processes and procedures in place and evidence that you actually follow them showing that your exam is kept up-to-date, current, and correct. How are you checking to make sure it’s current? How are you checking to make sure that people aren’t cheating? We check all of that. If the certification program is doing all the right things, if they have the right promises and they can demonstrate that they’re executing on those promises, then they get accredited. That really is a big accomplishment for a certification exam.

Sylvain: Okay. So, I’m going to give you an example. Tell me if I got it right about the three programs. I think I’m not getting all three but guide me through this. For example, I have a bachelor’s degree in education. That’s a certificate.

Dr. Ellen: Certificate. Right.

Sylvain: I got all the credits, all the courses, and passed the exams. That’s the university that says, “These are what you need to do to get the certificate.” and I’ve accomplished that. It’s on my wall. It’s in my files. Done. No need to do anything more to maintain that. So, that’s a trait of a certificate. To teach, however, at least in my area, and I think it’s true throughout North America and in States, you need to go through a certification process. In my province, I know there’s the Federation of Educators in Ontario. I forgot the specific name but there is an entity that checks, “Okay. Well, does he have a certificate from the university, a bachelor’s degree and also, possibly other things?” I need to apply and pay a fee every year to maintain that certification. So, so far so good? Is that a good analogy?

Dr. Ellen: You probably need to do some continuing education every year too. Maybe.

Sylvain: Now, you need to maintain your degree and you need to not become a criminal. You know, you have to prove certain things that you’re still up for the job but I don’t think there’s any continuing education that is required. In many cases like for engineers, for example, or doctors, you do need to keep up-to-date and things like that. So, I’m aware of that. Now, for the teachers’ stuff, I don’t know if there is accreditation. The accreditation would be for the Federation of Educators that is issuing the certificates. Is that an accredited organization? Am I correct? I’m not sure if it is a thing in this particular case, but there may be.

Dr. Ellen: I don’t know if they’ve been through the accreditation process. If they don’t have any competition, they’re not as motivated to go through accreditation. It is a particular exam certification program that is accredited, not the whole organization. So, if they offer a number of different credentials, and certifications, each one would have to be accredited separately because they might do a really good job on one and not so great on the other.

Sylvain: Well, I’m going to pull another example where I know there is such a thing and you would know more than I. You’ll be able to explain more. In the trades, any kind of trade industry, if you’re doing construction, plumbing, electrical things, I know there are certificates and certifications and there’s definitely accreditation involved in those areas. Can you talk about that, maybe using that as an example for the people watching?

Dr. Ellen: Certainly. Working as an accreditor has exposed me to a much greater variety of certifications than I had experienced as an employee because most of my employee roles were in the health professions. Accreditation, as you say, is pretty industry-heavy. And so, I’ve worked with the crane operators. You want the people driving those big 300-foot cranes to know what they’re doing because if they mess up, you know, it takes out a city block. So, sure, you want them to be certified, but you also want to know that that certification is really doing a good job at telling which ones are safe. So, those certifications apply for accreditation, and then they can say, “We are an accredited certification.”, so you know that this crane operator is, in fact, safe. Work with the EMTs, and non-destructive testing.

There’s a wide variety of industries out there that have certifications. Yeah, it is tricky. They have certifications and have gotten accredited. You know, one industry that has gone most for accreditation is cybersecurity. There are a lot of certifications out there, but to get on,…there’s a Special Department of Defense List. (This is the United States.) If you get one of the certifications that’s on their list, you get a big raise from the government. There are not a lot of ways to get a big raise from the government. So, getting your certification on that list of Department of Defense-recognized certifications is a great way to get a lot of volume really fast because all those people will want to get your certification. So, all the cybersecurity certifications are really eager to get accredited and are often surprised at what an uphill battle that is because it really is a pretty high standard. A lot of people going into building certification are in it because they’re passionate about the content, but don’t really have any exposure to psychometrics, the testing industry, and how to build a good test until they’re often surprised at how much work it actually is.

Sylvain: Would you say that the areas that have the most rigor surrounding certifications and accreditations would be where outcomes are either dangerous or have high social impacts like construction and things like that or are they pretty much universal? Are you seeing variations by industry?

Dr. Ellen: I think it is the important outcome, whether it’s the safety of the public or the whole health-professions thing. It really kind of started in the health professions. You’ve heard about the physician boards, the nurse boards, and all of those forever. You want to know that your physician knows what they’re doing.

Sylvain: Yes. For me, like I know healthcare is full of that. In nursing, they have constant training and they need to be certified to use machines and different things. Same with engineering, the trades, as I mentioned that I’m aware of from afar because I’m not a tradesman, but I’m aware of that. Because it’s dangerous work for the workers sometimes, but also the population, like you said, like the crane operator, for example, while you can take out the building next is high impact. And you know, in my current business, which is psychometrics, typically, what we deal with is kind of the opposite. It may be a high impact but not a high danger, life and death type of things that we work typically with Human Resources Departments at like a cubicle company. Basically, we’re just like office workers. There, they want to do training and development. So, it’s not a higher-risk thing, but psychometric tests are, you know, they are judged by their accuracy, foundations, and stuff like that. So, what I’ve noticed is a bit the opposite in that particular space. A lot of consultants in other companies use psychometric tests A, B, and C. The tests are sometimes conjured out of thin air, or they are standard but there’s no certification around them or no training around the use or interpretation of them. I mean, certificates do exist in the training. They do exist.

Dr. Ellen: The scoring. Yes, I want to talk about the scoring of these tests. That’s my passion.

Sylvain: But the point is because there’s no physical risk, there’s less rigor and validation of all these processes in how they’re used. So, what do you think about that? I know that must make your hair stand on end sometimes.

Dr. Ellen: It does. The physician board exam when I started working with the National Board of Medical Examiners, (now they’re called STEP in the United States at least), was a two-day exam. To get the whole thing at the end is a series of exams throughout your whole educational career. That initial one was two full days of multiple-choice questions to get one score. So, a big part of psychometrics — and I spell psychometrics with P-S-Y-C-H-O, that’s I actually identify my profession as I’m a psychometrician, I was a little intrigued when I saw the name of your company. “Wait a minute, they misspelled it.” —

Sylvain: That’s okay. Go on.

Dr. Ellen: A fundamental principle of psychometrics is that you sample a whole lot of things covering, you know, if this is the whole domain that you’re trying to cover, you need to make sure you get something from all different corners of it and then get enough of them that you can tell idiosyncratic responses from general themes. I am not aware of any certification exams that have been accredited in the high bar that are less than, like, 2 hours long for 1 score, for 1 pass-fail. So, when I see the discussion of “Was it really a 20-minute exam and 7 scores?” I would like to see how it’s done. What practically comes out of that is that they probably are not terribly reliable, and I want to make a distinction between ‘valid’ and ‘reliable.’

So, reliable means if we tested you again tomorrow, or when you were in a bad mood versus a good mood, or right after a fight with your spouse, you would get about the same score. That’s what you get from repeated measures and deep sampling. But, valid is talking about “Are you measuring? Do the scores mean what you think they mean?” is really the best definition of validity. Let’s just talk about my Myers-Briggs type. It was a pretty short version of those tests, and I’ve taken the long extensive ones many times over my life, and I always get the same outcome whether it’s the short test or the long test. The first time I took it was when I was a sophomore in college in 1970, and when I take it now, I get the same result. That’s a very reliable test. Now, is it valid? Well, you have to say valid for what? It’s a valid representation of my, I think they say learning style. I think of it as a kind of my way of interacting with the world, people, and how I like to do things but that’s not a valid measure of my potential as a manager. So, it could be a very reliable test.

Sylvain: What exactly are you measuring here?

Dr. Ellen: What are we measuring? What do these scores actually mean? So, I make it about the scores, not about the test because you can have a great test and then score it poorly, or the items are all now on the internet. There are a lot of things that can go wrong between the test and the score. So, it’s whether the score means what you think it means.

Sylvain: Right. So, in psychometrics, as you mentioned, Myers-Briggs for example, and there are others that are commonly used, or at least the brands are popular.  Myers-Briggs is one of them, DISC, and there are a few others that are used a little bit like off the shelf by multiple business service providers because they are useful to measure certain things. I would suspect most of them, if you take them from good sources, are valid because they’ve been studied for such a long time.

Dr. Ellen: … for some uses. Valid for some uses.

Sylvain: For what they should be used for, like off the ticket, right? But because they are used by anyone, as far as I can tell, there’s no certification process. Like, you must have this certification in order to use such a test. There’s no such thing.

Dr. Ellen: I think, the classic intelligence tests like the Stanford-Binet, you have to be a Ph.D. psychologist and probably go through their classes or something. So, the really high stakes ones. I think there is a certification for them but for the HR-type ones, probably not. I don’t know.

Sylvain: I’m not familiar with it, or they may be but it’s not something that would be recognized internationally that everybody knows like the MCAT. We know about it, and we know what it’s used for and it is a very regulated area because it’s healthcare. The follow-up question is, do you think that it should be, or is that a method to get more valuable psychometrics and a little bit less confusion in this particular market? Do you think there is any value in establishing some kind of certification process or body that would do that? Or should it just be left to the wild west? Now, people say, “Let’s do this. Check it out. This is cool.” Then they apply some training. This is what it is like now. So, what are your thoughts on that?

Dr. Ellen: Interesting question. I want to promote more training, rigor, and certifications. No, not all certifications are necessary. But, okay… I have a little bias here. I think of these HR tests as a little, forgive me, lighter-weight and lower-stakes. As you said, it seems high stakes if it means you get the job or not, that’s pretty high stakes, actually. But if it’s HR training, “Am I going to take this course or that course?” Or “Am I going to tell my boss I’m truly an introvert or an extrovert?” I don’t think people would go through the certification to interpret those. They just find a different test they didn’t have to get certified for. Truthfully. So, you don’t want to make the hurdle so high that people just go somewhere else. But I do think that training on what the scores actually mean, and what interpretations are reasonable and valid based on the data that were collected about the exam, is very useful. I bet a lot of people who are interpreting those scores have not read that material, or maybe a little certificate saying you read our training material.

Sylvain: That’s where we’re coming from because this is a bit, of course, a discussion for us being a platform. We integrated what we think, as we have tested, is a reliable test and we can argue that we evaluate people.  Other people can integrate their own tests there. That’s one part. Then we have to determine as a company, “Okay. Well, what is the wording around this particular test in the system to make sure that a person that’s using it has the right idea of how it should be used and interpret the data?” So, that’s one piece and the other part is our clients, our consultants, and training companies that are using the data. They have communications with the end-users and the people that should take the test, not us. We don’t know what they’re saying and how their interpretation matches what the assessments say. So, we have set up a certificate program. This test, at least ours, is what this thing means. Then we have a little test getting full access after which where we can say, “You get it. You understand.” and we may, yet at some point, develop a CykoMetrix, with the C and the X, certification program that would be validated or renewable to make sure that whatever is on our system is interpreted correctly in the real world. These are things that we’re thinking about. That’s why I asked you a broader question. We’re thinking about this, great, but in the industry at large, in a lot of cases, there is no appetite to do that because training and consulting companies are more focused on “How can I get the new workshop sold? And so, the psychometric test is a bit of a lead to get the work, to get the contract in. You see what I mean?

Dr. Ellen: That would mean they don’t focus on it for its own sake. They don’t love the test. I love the test.

Sylvain: Yes. I like the tests. I want those, I’m thinking, you use the test to get valuable training, the right kind of training that the person actually needs, but the market is a bit wild. We don’t control that we’re just the test makers and the platform makers. So, that’s why I asked you about the value of broadly speaking habits like I’m a general certification process and your answer, I think, is as good as saying, “Well, maybe there is some appetite for that in some cases, but for in not a lot of cases, maybe not at all.”

Dr. Ellen: I think, its certificates that are probably about the right level of rigor for that. You might also want to have some fallback if you think somebody’s misusing it. The right to pull it back and go, “No, guys. You need to go take the certificate course again because you’re not doing this right.” I don’t know how you…

Sylvain: You’re pulling the certification if I use the right terms, “No, you’re not certified anymore.”

Dr. Ellen: Remediate, somehow.

Sylvain: Remediate, yes, in some ways. Well, that’s really interesting. Let’s go a bit beyond this. What do you see, in your mind, as the future of psychometrics? I don’t know if you have ever thought of where we are going with this because the market is indeed changing constantly. So, what are you seeing as the next few things in psychometrics, and/or also the certification process that may surround it?

Dr. Ellen: My mind goes first to statistical techniques. I suspect that’s not entirely what your question was. But one of the places where we go through motions without a lot of results because we don’t have the skill set yet it is looking for bias in test questions and in a test as a whole. Now, in a test as a whole, you use the test score to predict some future behavior and see whether that prediction is different for different groups. That might be biased in the test or it might be that different things happen to those different groups after they take the test to impact the validity of the prediction for these different groups. Because it just works for one group and it doesn’t work for another, it doesn’t mean the test is biased, but you need to know this. We don’t have good standardized techniques for figuring that out. For test questions, individual test questions, which is where the focus often is, “these were just questions were biased against me because I didn’t know the answers.” It’s really what it often boils down to, but the concern is absolutely legitimate.

There are hidden ways that bias can creep into questions by tapping irrelevant knowledge like that classic example of an old SAT item that was about, I can’t even come up with a word right now, a regatta. Let’s say a regatta. Some people are more likely to know that word than others and it may not necessarily reflect your vocabulary. The only way we have now besides training experts to read questions very carefully and then getting a wide variety of those experts to talk about it is to do what we call a Differential Item Functioning, a DIF analysis, D-I-F. It requires 500 people in the minority group and, typically, a much larger number in the majority group before you can get any good statistics about whether that item is functioning differently for different groups. Very few tests have the luxury of 500 people in the minority group doing that. We need a better way of looking at item bias. So, that’s a direction I would love to see psychometrics go in. I think there’s going to be more and more performance tests where you actually have to do something and then get scored on it with machine learning and artificial intelligence.

The computer is helping more and more, for instance, anything with natural language. Natural language processing is now making it easier for the computer to help with the scoring but there are going to be some things where it will always be an expert judge. Oral translations or interpretation. You hear one thing, and you have to have judges. So, the scoring of these things, how do you sample? How are you making sure you cover all the corners of the domain and yet have something that’s repeatable is a challenge there because each behavior sample now takes 10 or 15 minutes instead of 1 minute we typically allow for a multiple choice question. So, you can only get a few samples on a performance test and to make it really reliable, it would be most reliable if you sample the same spot over and over. You get a really reliable exam because you’re measuring the same thing over and over but you haven’t represented the rest of the domain then. So, you have this conflict between sampling spread and depth, and you want both, which is the reason you end up with two-day exams. So, we’ve got to get more efficient at that combination, figuring out how to get performance exams recorded so that they can be validated or challenged, and you can actually see what they did, and then how to get them scored in a fair, unbiased, and transparent way. I think the profession is working on that. The physicians actually started it with the standardized patients where they have live humans who are trained to display signs and symptoms of particular problems. They go in and help educate physicians but then, eventually, that’s part of the exam they take at the end. You have to go in and do a physical exam, an abdominal exam, I’m poking my stomach you can’t really tell. A live patient will score you or a physician sitting in the room scores you on how well you actually poked all four corners of the abdomen and did you do it right? I don’t think there’s any way we can do that remotely with the mouse and a computer.

Sylvain: Yes. There are challenges around that. What you seem to say essentially, is that to improve on the whole industry, we just need more data. More volume data that is categorized correctly represent all the samples. We also need help from AI, for example, which is great at crunching a whole bunch of data and seeing patterns including natural language processing. I know we were talking about plugging a psychometric system through an email system and analyzing email language. So, you gather more data that way, right? You get a natural language processing and correlations between the use of words and things like that. Again, getting the volume of data on the same people without them sitting down to a test. So, you take the tests for data but you also test the content of the data. So, more data to get more value, more precision, and so on and so forth. These are all things, I think, as technologists and psychometrics experts, we’re all kind of working on getting more bang for the bucks.  It’s kind of moving and interesting spaces. So, we’ll see what, in different years the tools that will come out. We’re certainly part of that game as a technology company and you’re certainly part of that game too because you’re looking at the whole thing, seeing what was valuable, analyzing and testing these things. You’re measuring your own metrics, basically, for your clients throughout this whole process of innovation.

Dr. Ellen: We’re measuring the metrics and evaluating the metrics, actually. May I add one more future direction?

Sylvain: Absolutely.

Dr. Ellen: Working with teams. There is some psychometric work being done now on the functioning of teams and each member’s contribution to the team’s work. I think that’s just fascinating. It’s requiring some pretty sophisticated statistical techniques to pull out from the language that the team is using as they talk to each other, determining how many times people are agreeing with somebody else or offering a new idea, or expanding on someone else’s comments. Natural language processing is helping to codify that. So, it’s not such a tedious process. Then there’s a lot of deep research going on to determine which of these behaviors actually ends up helping a team. The hope is that eventually a measure will come out for the team as well as for the individual’s contribution, and I think that is going to be a game-changer.

Sylvain: Absolutely. That’s actually something that we have worked on in our initial test. We have outputs that are team-centric, team effectiveness, and team climate from data from relatively recent research like you said on team dynamics, and what correlations or what different parameters or values actually output with high alpha reliability for certain team characteristics. So, it has been worked on for the last 20 years and I think more of it because they are complicated matrices and because dealing with one individual is complex enough. Then, you’re dealing with a combination of visuals and behaviors that are complicated. So, you know, computational power, analytics, AI, and all tools that allow us to predict patterns in psychometrics.  We’re in the game.

As a company, we’re definitely into that because it’s new and interesting, and we like measurements as well. Well, thanks a bunch, Dr. Julian. This has been enlightening, I hope people have learned. One of the topics was what is the difference between certificate, certification, and accreditation. Also, what is the value of each one of those, and how they can be applied in different ways. We talked about sometimes in the HR environment, there’s no desire to have those levels of test or scrutiny. So, we need to be aware of it. We need to, perhaps, be innovators and build more where there are missing bits. So, I’m hoping people that are watching this have learned and will be innovating, developing, improving, and doing lots of research. I hope for human behavior and teams because it has become…

Dr. Ellen: And publishing that so the rest of the world can make use of it too…

Sylvain: …well, yes, in gathering lots of data, right? So that, we, researchers can go into that data and derive correlations and all sorts of interesting models. It’s all part of this exciting research. So, for those of you who want to contact Dr. Julian, we have her website down below in the video and also in the blog if you’re watching this from the blog. Please contact her, she is an absolutely wonderful, interesting person and probably one of the best-positioned individual and professional to evaluate assessments. Get her to help you out with your needs regarding anything surrounding tests, she’s the gal. You want to go on and contact her. So, thank you so much, Doctor for participating in this and educating us.

Dr. Ellen: My pleasure. Thank you for having me.

About Dr. Ellen Julian – julianconsulting.com

Dr. Ellen Julian’s early fascination with tests led to a PhD in Measurement and Testing from Florida State University. Her psychometrician roles with the physician and nurse licensure examinations focused on developing innovative assessments and moving their paper exams to computer. She directed the testing programs for medical college admissions and ultrasound certifications, and is now the founder and principal consultant at Julian Psychometric Consulting, offering strategic-decision support and problem-solving analyses for certification and other assessment programs. She serves as a psychometric assessor for the accreditation of certification programs. Her mission to make high-quality measurement understandable and accessible has generated decades of presentations at NCME, ATP, ICE, and CLEAR, peer-reviewed publications, book chapters, and now service as a Public Member on a certification’s Examination Committee.

About CykoMetrix – www.CykoMetrix.com

CykoMetrix is a leading edge combinatorial psychometric and human data analytics company that brings the employee assessment industry to the cloud, with instant assessments, in-depth analysis, trait measurements, and team-based reporting features that simplify informed decision-making around recruiting, training, and managing today’s modern workplace.