Overskilled and Underused? What PIAAC Reveals About the Canadian Workforce

Alex Usher: Before our show starts today, I just wanna take a minute to note the passing of Professor Claire Callender, OBE. For the last two and a half decades, she's been one of the most important figures in UK higher education studies, in particular with respect to student loans and student finance. Holder of a joint professorship at UCL Institute of Education and Birkbeck University of London, she was also instrumental in setting up the ESRC Centre for Global Higher Education, of which she later became deputy director. I just want to quote the short obituary that her colleague Simon Marginson wrote for her last week after her passing from lung cancer. He said, "What we'll remember about Claire is the way she focused her formidable capacity for rational thought on matters to which she was committed, her gravitas that held the room when speaking, and the warmth that she evoked without fail in old and new acquaintances.
My thoughts and condolences to her partner Annette, and to her children. We'll all miss Claire.
Hi there. I'm Alex Usher and this is the World of Higher Education Podcast.
I suspect most of you are familiar with the OECD's Program for International Student Assessment, or PISA. That's a triannual test of 15 year olds around the world. It tries to compare how teenagers fare in real world tests of literacy and numeracy. But you might not be as familiar with PISA's cousin, the Program for International Assessment of Adult Competencies or PIAAC. To simplify enormously, it's PISA, but for adults, and it only comes out once a decade with the latest edition having appeared on December 10th of last year. Now, if you're like most people, you're probably asking yourself, what does PIAAC measure exactly?
PISA pretty clearly is telling us something about school systems. Adults, the subject of the PIAAC test, they've been out of school for a long time. What do test results mean for people who've been out of school for, in some cases, decades? And what kinds of meaningful policies might be made on the basis of this data?
Today my guest is the CEO of Canada's Future Skills Centre, Noel Baldwin. Over the past decade, both in his roles at FSC, his previous ones at the Council Minister of Education Canada, he's arguably been one of the country's most dedicated users of PIAAC data. As part of Canada's delegation to the OECD committee in charge of PIAAC, he also had a front row seat to the development of these tests and the machinery behind these big international surveys.
Over the course of the next 20 or so minutes, you'll hear Noel and I, both fellow members of the Canada Millennium Scholarship Foundation Mafia, discuss such issues as how the wording of international surveys gets negotiated, why we seem to be witnessing planet wide declines in adult literacy, what research questions PIAAC is best suited to answer, and maybe most intriguingly what PIAAC 3 might look like a decade from now.
I really enjoyed this conversation and I hope you do too. Anyway, over to Noel.
Uh, Noel, I, you know, some of our listeners may be familiar with big international testing programs like uh, PISA, the Program for International Student Assessment. But what is the program for International Assessment of Adult Competencies? What does it aim to measure and why?
Noel Baldwin: Yeah, so it is somewhat analogous to, to PISA but primarily focused on, on working age adults. So again it's one of these large scale international assessments that's organized by OECD in this case, by both the education and labour secretariats at OECD um, and then administered by national statistical agencies or government agencies on the ground in participating countries, and PIAAC's primarily focused on measuring skills like literacy and numeracy. Although over time, the OECD has started to add other measures, other skill sets that are relevant to that intersection between education and, labour and labour markets. So things like digital skills or technology use, problem solving, and social emotional skills. And in addition to the assessments themselves, there's also this large battery of background questions that collect a lot of demographic information about respondent's work life and other factors like health and wellbeing so that you can start to try to draw some correlation between the core skills that are getting measured and how they're getting used or, or what kind of impact they're having on people.
Alex Usher: And, And how do they know that, what's being measured? Because it's asking you questions about, you know, the literacy section is reading comprehension, and the math is sort of, you know, if two trains move towards each other, one from Chicago and one from Pittsburgh. I mean, it's those, I mean, they're a little bit more sophisticated than that, but that's the kind of thing, how do they know that that actually measures anything useful about workplace competencies?
Noel Baldwin: Yeah, that's a, that's a good question. So, one thing to start with, right, is, that the questions build from quite easy and, and simple tasks or questions to answer to much more complicated ones. And it allows the OECD at the end to try to create these scales. And they talk a lot about levels, you know, level one to up to five and even below level one in some cases for people with some of the weakest proficiencies. And the, although PIAAC itself is new the, the assessment of the competencies is not, it actually goes back into the early 1990s. And so there has been quite a bit of research done both by OECD and then by psychometricians and other researchers about some of the connections between these things. And the, I think the, the main, kind of core thing to understand is that, you know, over time the links between stronger literacy and numeracy skills and a variety of life outcomes, including labour market outcomes are pretty strong, right? Like you're much more likely to have it's a bit like education attainment and, and to some extent, you know, these things become proxies for each other a bit, but the stronger your skills, the more likely you are to be employed, to have higher wages, to avoid unemployment and be adaptable and resilient. But it's other things too. Life satisfaction actually interesting findings coming out of this one about democratic participation and your kind of sense of how your society is doing. But there are pretty strong correlations between advanced skills and those outcomes.
Alex Usher: But I can imagine, you know, the nature of an economy, you know, whether it's more manufacturing based or service based, that might affect things and so different countries might actually wanna measure slightly different things. How do you get 50 or I'm not sure how many countries, dozens of countries to agree on what skills to assess and how to measure them.
Noel Baldwin: Yeah, I mean the, the point at which OECD countries agreed that they wanted to focus on literacy and numeracy sort of predates me, right? But it also predates a lot of the current focus on other sets of skills that are much more digitally oriented. Right? There was a much more analog world when this started, and so literacy and numeracy, I think made a lot of sense when people were consuming almost all of the information that they were getting in some form of media where reading comprehension, the ability to navigate text, and then the ability to do fairly rudimentary to advanced problem solving with numbers made a lot of sense, right. So I suspect that at the time, kind of as this was being developed through the eighties and into the early nineties, that there was a high degree of consensus on, on those things.
The development of the instruments themselves is also an international effort, so it's led by the OECD, but they work with experts in a number of different countries to test and validate the items that they use for the assessment. And so, you know, you know, educational testing services in the US, they're quite involved. There are experts in Australia and in Canada. In fact, Canada was at the start of this process, both at Statistics Canada and other experts here were quite involved in developing some of the early instruments, especially for uh, measuring literacy, I believe, right? So the consensus building process includes not only what you're gonna measure and how you're gonna administer it, but the actual items and whether they're effective. And they do field testing in advance of putting them out into the main assessment to try to get the validity as high as they can.
Alex Usher: And so once the results are in and published, what's next? How do governments typically use this information to change policy?
Noel Baldwin: Yeah, that's a great question. And, And I'm gonna confess, despite having been on the inside of some of this, it sometimes is a, is quite a black box. And increasingly, and I think we'll get into this a little bit through the course of the conversation, increasingly a bit of a black box. But one thing I can say for sure is, you know, in different countries and even in different provinces and, and territories in Canada, you see different ways the information gets used. So it certainly is integrated into all kinds of internal briefing. And, you know, I spent some time as you know, at the Council of Ministers of Education and, and we could see that both in our work and in the work of the officials we worked with across provinces and territories. After the last cycle of PIAAC, Quebec produced some fairly detailed reports analyzing the performance of Quebecers on the PIAAC scales and the, the proficiencies comparing them to other provinces and other countries and, and using that to uh, certainly to drive some conversation about the implications of that. New Brunswick, shortly after the last PIAAC cycle launched a literacy strategy. So you can see examples of this, but it's certainly fair to say like a lot of it becomes internal information to drive conversations inside of governments and even since the most recent cycle of PIAAC uh, was released in December, I've seen examples of that but there's not as much in the public domain as you might expect and as there used to be, I think that's also fair to say.
Alex Usher: We're gonna take a short break. We'll be right back.
Advertisement: Student success at scale with a proven ROI of 4. 4 times return for universities and colleges. Studiosity's AI for Learning is designed specifically for post secondary students to develop real skills, critical thinking, agency, belonging, and retention, to empower educators with reporting and insights into learning progress, and to build human connection, made possible with Studiosity's strictly help not answers Formative feedback and integrated humans in the loop design In partnership with post secondary institutions in Canada and around the world, Studiosity is for future ready graduates and future ready institutions. With over 200 post secondary partners and more than 2 million student users, Studiosity is grounded in an academically driven evidence base, leading the way to increasing life chances for students everywhere. Learn more about the world's leading ethical AI student support at studiosity.com.
Alex Usher: Welcome back, Noel, some of the findings in this latest PIAAC cycle, I mean the, the sort of the headline that got the most uh, traction, I guess was the fact that we're seeing declines in literacy and numeracy scores across much of the OECD. I mean, there are a few countries Canada wasn't much of was a little bit of a decline, but you had some parts of Northern Europe I think that did okay, but everybody else was down. What are the possible explanations for this trend and should we be concerned?
Noel Baldwin: I think we should be really aware. The part about being concerned, I'm always sort of hesitant to, to declare a crisis. You know, there, there's a lot of work that has to be done to unpack what's going on with this cycle in PIAAC And, and one thing I would point to is just you know, most of the responses from people were collected at a time of just like ongoing global turmoil. There, it was done during the pandemic, right? It was still 2022. We were still in the middle of this. And so getting the sample collected was a major challenge. Much bigger I think. With that caveat, I mean the OECD has started to postulate a little bit on what may be going on, especially on the literacy side, and part of what they're pointing to is just the way that people consume information has changed so radically in the 10 years since the last time the study was done. People are consuming much shorter bites of text that is delivered to them in a a much wider variety of ways. Like there's still items in the literacy assessment that would sort of be akin to reading a paragraph in a printed newspaper article, right? Which is not how most people are engaging with information Um. that they're reading. So we're also getting much more of it through content that's delivered by video and audio than we used to. So I mean, I think that some of that is at play, but you know, until we see more analysis, there are little signals of things, you know, performance on gender and stuff in different countries that I think we've gotta unpack. And until we do that, we're not gonna have a, a great sense of why the outcomes may look the way that they did.
Alex Usher: Wanna focus in on Canada for a second here. You know, as in most international education comparisons, we come in the top, we come at the bottom of the top third, basically. It doesn't seem to matter what we do or when it happens, right? We're always in the top group, but never right at the top. When you look at global trends, do you think Canada stands out in any way positively or negatively, or are there things we're doing right, things we're doing wrong?
Noel Baldwin: Well, you know, I would say we continue to see and the OECD notes this almost every time we do one of these things that the difference between our top performers and our lowest performers is not as big as it is in some countries, and that's often pointed to as a signal of equity. And so, I would say that's certainly a good news story.
You know, in the global comparison, we held pretty much steady on literacy and lots of countries didn't, I think, you know, holding steady when everyone else is declining, is not a, is not a bad news story and we actually perform better in numeracy. I think the distribution across the provinces is also a good news story. It was, quite a bit different the last time. And so there was much more variety and there were more provinces below the OECD average below the Canadian average. So, but, you know we're, we're pretty tightly packed. And I think that's something that we need to pay attention to. I also think that on the performance when you start to disaggregate a little bit, for example, and look at uh, OEC doesn't have kind of true measure of immigration, but they do, they're able to identify people who were born outside of a country, or whose parents were born outside of a country. And given how different our demographic profile looks from almost every other participating country in the OCD, but especially the ones in Northern Europe like I think that we do pretty well by that. And certainly, especially given some of the discussion over the last couple of years about immigration policy and the impact on all kinds of aspects of our society, I think it's a pretty good news story that we see good performance in that regard as well.
Alex Usher: I know we'll disagree about this next question. So I, I my impression is that in Canada, the way that PIAAC gets used has changed quite a bit in the last decade or so. Like the, the first Piac results got a lot of attention, right? StatsCan, the Council Ministers of Education they both publish quite lengthy analysis of the results. And maybe, you know, crickets might be too strong a word to describe the reaction this time, but it's quieter, that's for sure. And my impression is just that governments just don't care anymore. Right? Like when they talk about skills, like the general narrative about skills is they only care about nursing and they care about skilled trades, 'cause those are seen as, you know, a bottleneck on the social side and a bottleneck on the private sector side. And they have very little interest in improving transversal skills and even less knowledge or idea about how to do so. Make me less cynical.
Noel Baldwin: Well, it's funny um, because the, this question is sort of what got us started in a conversation that leads up to the podcast. And I'll confess that you've had me thinking about it now for several weeks. So, I mean, one thing I want to draw a distinction between is sort of caring about the skills versus the way that the data is being put out into the public and used publicly.
And I, you know, there's no denying and I think, there's no denying that there's less coming out publicly from the governments that paid for the study. Like that's just, that's just true, it's undeniable. And I, I'm not sure that's gonna change. And I think that that's a reflection partly of you know, the kind of changed environment we're in fiscally and what people want to pay for. But it's also, I mean, I think it's a real reflection of the kind of information environment that we're in today versus 2013. I don't know. Like I, as I've been thinking about this for the last few weeks, getting ready to talk to you. I mean, part of what I've reflected on is I wonder if in 2012 and 2013, we were actually at the, the tail end of the era of evidence-based policymaking and we're now into the, the vibes policymaking period. And so to, to the extent that, that may be true, you know, why would you write up a bunch of reports about this thing? You're kind of gonna work from your gut. On the skill side, I think there's still an interesting question there and, I think I was a bit more firm on this a couple of weeks ago. It, but I still would say like, I don't think that it's a, that they don't care about these skills. But I do think that part of what's happened with the conversation about skills is it started to change a little bit. And our understanding of the way that skills ladder on top of each other from foundational literacy and numeracy up to problem solving up to some of the digital skills may be missing.
I mean, I think about the kind of, craze, maybe it's too strong a word, but the moment we're in around AI and the idea that, you know, people are gonna become really quite good at prompt engineering without pursuing much formal education, Mark Cuban had a, I don't know what you call the thing on Blue Sky. It's not a tweet, but he made a point about like. Skeet. That's terrible. Um, But you know, he said something to the effect of, you know, you're not gonna need formal education with generative AI. If you become really good at getting answers out of the large language models, you're gonna do better than people with advanced degrees. And I don't think that reflects how much you've gotta be able to interpret information in order to ask good questions and then understand whether what you get back is worth paying any attention to. We may start to see some of that shift again. You're right though, that there has been a move much more towards thinking about workforce rather than thinking about skills in the last few years. And that may be part of what's going on too.
Alex Usher: What do you think is the most interesting or under-explored question that PIAAC data could help answer, but that we haven't yet fully investigated? You know, what's the, I mean, this data allows you to do a lot of interesting stuff, so, what, where would you, if you had to, could wave a magic wand and get some top researchers on the data, where would you want them to look? Either a Canadian question or a, an international question.
Noel Baldwin: Yeah. So first I'm gonna just make a, a small plug, which is that we've been working away on, on what we think can be a PIAAC research agenda, obviously that responds to some of the things we care about at Future Skills Centre, but that we hope to be able to advance in the coming weeks and months. So we are thinking about this.
I mean, I think that there's a bunch of them, right? And, And we've seen the reemergence of a conversation about productivity in Canada. And I think that PIAAC can shed some light on the role that skills play in that, you know, Conference Board of Canada did a piece ago that, that looked at how much of the gap between Canada and the US was uh, skill or labour based. And it's not that big, but I think we can use PIAAC to continue to investigate that kind of thing.
One of the things that the OECD often talks about with Canada that I think is really interesting is the degree to which workers are overqualified or overkilled for the jobs that they're in. And again, this is another discourse that's been around for a while. One of the ways in which I think it would be very interesting to try to link it using the PIAAC data is at the same time we're having this conversation about, how we build our labour supply, the role of immigration. And then, you know, some people have hypothesized that one of the reasons we lag on things like integration of technology or capital investment is because we've been substituting skilled labour for that. I think that with PIAAC you might actually be able to look at this question of over qualification or over-skilling and examine whether some of that is as a result of the way that we've that we've managed some of our immigration system over the, over the last couple of decades. So there's a, a, a few things there that I think are relevant and then lots of international comparison. And, And you're right, we should be looking for examples of international success that we could emulate or borrow from or get inspired by.
Alex Usher: You know, I, I suppose, I don't know if either of us wants to be doing this, still working in 10 years time, but if we had this conversation again, a decade from now, what do you think or hope will have changed? What do you, what will the impact of PIAAC 2 have been over the long term and how do you think PIAAC 3 is gonna be different?
Noel Baldwin: Well, I mean, I, I think I need to, I think I need to say this out loud. I, I am worried that there won't be a PIAAC 3. And we're, you know, we're recording this in a pretty turbulent moment early in, in 2025 when lots of changes happening around the world. One of the things that seems clear is that the new US administration is not interested in Department of Education, which I suspect means that they won't be interested in having a National Center for Education Statistics. And like with lots of things on the international stage, the US is a driver of initiatives like PIAAC and valuing initiatives like PIAAC. And so I, I worry about whether we'll have a PIAAC 3 So I hope in 10 years we're talking about PIAAC 3, and I hope it's robust and, you know, with lots of participation across OECD countries.
You know, the other thing I think that I hope is that uh, there will continue to be some investment in using PIAAC data to answer research questions. It's, It's one tool, right? But, it is a big one. We talk a lot about it being the only direct assessment of adult skills that we get where someone has actually assessed a particular set of competencies that we're looking for. And so it's really valuable in that sense and for our organization that is focused on the skills of adults when they're in the workforce you know, I think it's incumbent on us to, to push to get some answers to some of these research questions and I hope that what we and others can, undersurface will make its way into the kind of policy discussions we're gonna have about how our workforce and skill needs and economy are gonna be changing over the next 10 years. It seems it would be a, an unfortunate wasted opportunity if it wasn't.
Alex Usher: Noel Baldwin, thanks so much for being with us today.
Noel Baldwin: Thanks Alex.
Alex Usher: And it just remains for me to thank our excellent producers, Sam Pufek, Tiffany MacLennan, and you, our readers, listeners, and viewers for tuning in. If you have any questions or comments about today's episode, please don't hesitate to get in touch with us at podcast@higheredstrategy.com. Quick request, run don't walk to YouTube and subscribe to our YouTube channel. Never miss an episode of the World of Higher Education. And next week our guest will be Shai Reshef. He is the Founder and President of the University of the People. That's a free global online university that gives 150,000 students in over 200 countries an education, and he'll be with us to answer the key question, how do you run a free university with no government support?
Bye for now.