EP 159: Human Powered Technical Assessments with Woven’s Founder and CEO Wes Winler.

He's joining us today to tell us all about his product . Wes , thanks for joining us .

And then I made my fourth hire and it was not great and it was my fault . I hired someone who was trying really hard . I put them in a seat . They were not going to be successful .

It was really bad for our biggest customer , for my team , and that was my wake-up call that I don't have a gut and actually you need to be good at this thing and it's hard . And read Thinking Fast and Slow , and talked to a lot of other engineering managers and read IO Psych Research like what does science say ?

And my insight was if you're going to hire dancers , you should probably watch them dance . Doing the job predicts doing the job , but that was really hard to do . And then when I sold that startup , I founded Woven to make it easier to assess folks in a real-world manner in an engineering context for tech , because it's not easy .

Right , it's not just about best practices will get you a really long way , but it's also understanding the nuance of the employer right , like the specific requirements they have , their environment , their strengths , their weaknesses .

A lot goes into it , right , definitely pretty complex process , but I think what's great is the products coming out these days seem to hopefully be handling , absorbing some of the complexity through some of the products that are being built .

So I would love to learn more about your product and figure out how you're solving , exactly what problem you're solving , and maybe we can go ahead and get a layer deeper as well .

Can you write code that passes some automated test ? Can you handle a messy real world situation ? Here's this pull request . How do you prioritize it ? Here's this system that's broken . Here's this email from a colleague where they're not even clear what they're asking you but you're supposed to make a technical business decision . Respond back to them .

Because we're human powered , we can evaluate that messier work which creates candidates like it more , because they like doing stuff . That's like the job . That's why they have that job and that's why they stick it out .

And then it also creates more signal on who's going to pass , and especially for those folks who don't quite have the prestigious resume like that's . What got me fired up about this is there's hiring managers have opinions right , and some of them are informed by the real world . Some of them are just opinions about what they like .

And if an assessment can change a hiring manager's opinion about what resume and background really matter . Take a candidate from probably not to yeah , that's where I feel like we can make a big impact , like someone who just needed a shot .

And this assessment gives someone a way to get a read on their actual skills without having to commit to a one-hour interview with them . Because y'all have seen it , you can only do so many speculative interviews with hiring managers before they start to be like , eh , maybe I don't trust this person Judgment . So that's what we do .

That's our core business and we've started a just for . We have a small RPO arm for some of our customers on the smaller end who don't have recruiting . We have a small RPO arm for some of our customers on the smaller end who don't have recruiting , have intermittent hiring , and we do a best described as like a resume AI powered resume matching engine .

I don't like the word matching , but that's what the market calls it . But generally like how can you find if this candidate meets your requirements in a consistent way when you got a thousand applicants in your ATS and you are one person trying to read through those ?

Can we just get a little bit more detailed into that part ?

That's the same . We , you pick assessments . What is different is we are able to offer different types of assessments , so things that are more free form . And then on the backend , where the humans come in , is we actually have two engineers that are blind , evaluating that candidate's work .

So it's not just some automated thing , it's two engineers looking through that analysis , blind to anything about the candidate , blind to each other , scoring independently and then creating feedback for that candidate . So every candidate that goes through gets feedback on things they did well and things they could have done better .

So the folks that you're advancing , they're going to get a feedback email , like within a day after completing the assessment . That's praising them for the things they did . The folks that maybe you're not going to advance this time are at least going to get something useful out of that assessment .

They're going to get an area for improvement that they can level up their career . That's what you get when you use humans versus getting automated tests . You can't really do that .

The typical spot is a recruiter has done a resume screen , probably a first screening call , then woven as an assessment before the hiring manager or technical interview , and that allows the recruiter to take more shots on maybes and saves hiring manager time while giving them some more signal .

It depends because if you're hiring a more entry-level role , sometimes you can skip that recruiter screen and just have a more rigorous application . Sometimes you might be hiring for a VP of engineering and actually it's worth doing an extra call before you send the assessment . So it moves around a little bit , but typically after the recruiter screen .

And if you can skip any of those other things , you can do more other valuable things . So if you can skip the screening step , like you said , have an engaged candidate . We're seeing more and more customers do like video recordings . That first five minutes of a call you just repeat over and over your mission , your vision , why your founder is awesome .

You record that in a loom and send that to any candidate that passes , your knockout questions or your early screening , and you can get a lot of candidates that are very engaged without needing to schedule that screening call which slows things down . Most folks have jobs . It's hard to schedule it during the day .

So I think there's a lot of exciting things happening in that candidate engagement that aren't jump on another 30-minute screening call where you smile at somebody .

Because you mentioned something about screening questions , so I just want to double down on that .

Just I like to see how , what's the state of the art in RPO and staffing . So every once in a while I'll make a hire with a staffing agency just to see how the process was . Recently went through one with a company everyone has heard of .

They make they have $20 billion in revenue every year in staffing and I was looking for a front end engineer for a one-off project . I was looking for a React experience and they rejected a bunch of candidates for me that were like senior React engineers because they didn't list CSS on their resume .

And if you're not a technical recruiter , everyone who does React does CSS . You cannot do React without CSS , but because that got into the requirements list , that becomes candidates that get rejected for no good reason .

So we have a tool to get the requirements list solidified and that means being and this actually we take a lot longer on this than other times because no one likes to do this part . They like to see candidate resumes but saying is CSS a nice to have or a must have is really important ? Because no one asked me that question .

It was obviously a nice to have , but it got into the must have lists and job descriptions are crap . So we we essentially turn those requirements into evals for an LLM yeah , so yes , no questions .

And then we create a hierarchy , those evals , and then we can run those evals against a resume and application plus knockout questions that we generate , and then that allows us to sort candidates into qualified and unqualified buckets .

But we all know those other 950 folks are not going to get looked at the same way .

I am not super stoked this is maybe a little controversial but I'm not super stoked about the EEOC decision . When is it a hiring decision ? And if they have applied , it's a hiring decision . If you're doing outbound , it's not . So that's the thing we're all skating by on .

But like ranking when the other people are not looked at , I feel like that's , it's like this cloth . So I feel like , as vendors , we need to be building systems that pass scrutiny . So Workday is getting sued right now . Have y'all seen that lawsuit that pass scrutiny ? So Workday is getting sued right now . Have y'all seen that lawsuit ?

And then Coinbase stepped forward and said we're going to be regulated and we're going to lean into it .

We're going to ask for regulation , we're going to meet the standard of goodness and lack of shadiness In this case it would be lack of bias and that's a system we're building that can be run automated because you put the human effort at front at the requirements Does this house need a basement or not ? If you decide later on , you want to change that .

It's very expensive . But if you spend the time up front saying , okay , this human signed off , doesn't need a basement , then you've created the paper trail that the EEOC needs .

And then all the LLMs all the AI is doing is like data entry , it's just doing dumb things like matching companies versus criteria or looking for not keywords but like skills , because LLMs are already better than recruiters at most of the skill matching , like everyone's still using examples from the keyword things in that resume .

Oh , it's Kubernetes , and someone said K8S . So of course these robots are done . They are already better than most , better than me , better than most tech recruiters Like I . One of our testing data . We're looking for a engineer with TypeScript and someone came through with Nextjs experience .

I didn't know that Nextjs is a platform only written in TypeScript , but the robot knew , so they marked that candidate as a pass . Like the bots are already better at data entry , we should let them do data entry .

Could you help explain that a little bit more , like how you think these products and tools are going to protect against that and like what the best practice is going to be ? Is there any more you could share there ?

Yes , there's some stuff you should do , but it's obvious stuff . Same thing with EOC . The stuff you should do is , it's obvious . It's not like they're not asking for crazy stuff . They're just asking for documentation that a human made this decision they were not using .

They made it for good reasons , they can justify it and you have a trail that you were then following that criteria and you could do this manually . You build a big resume rubric or application rubric with 17 rows , you weight the criteria , you fill out zeros and ones for every application .

It takes nine minutes per candidate and no human would do that because you're immediately like , oh , I can't do this , I'll just use my deep learning network that's between my ears to make a judgment and there's a carve out . For human made a judgment , they're probably not biased .

Can you prove they were biased , whereas for the robots you have to prove that they weren't biased , but there's already a way to do that . The thing about robots is they don't get bored . They will fill out that 17 item rubric and they will do it better than a human if you build the tech right . Like , hallucinations are one thing .

When you're asking someone to look up something , when you're asking someone to search a small document for a very specific criteria or enrich a document with LinkedIn data , hallucination is not the problem . That's just a lot of plumbing . So for the EEOC , you need to say like this requirement is job related , it's bona fide .

There was a person that made that decision and here's how that requirement chased through and here's why we rejected that person based on the requirement . There's nothing about knockout questions in any of the EEOC . Everyone uses knockout questions , but there's not like a carve out for the exception for automated knockout questions .

There's just is this a job requirement ?

The more wide in scope the AI is , more data the AI is evaluating , then it could be a little bit harder and somebody could have had short tenure because they had a baby or something like that . Then there could be something that you wouldn't think could be related , could be discrimination or considered discrimination could slip or there could be .

It could be looked at in a different context . And that was an interesting counterpoint . He's just like , yeah , you got to be careful because sometimes there might be things that are introducing bias or discrimination or whatever else into the process . You just have no idea . It's just hard to catch everything . But the more limited , I feel like , the scope is .

Like just looking at a resume , I feel like , but again , there's the tenure , there's stuff , there's always things . I think the more limited the scope , the more we can prevent against that at first , steve is a very smart guy .

From my perspective , we're not comparing to some perfect recruiter who can spend five minutes per resume and notice that was a gap but then notice there's oh , this is a woman , so that was probably that . And that's not what happens . Like you look at data recruiters get , depending on which study you use , between 15 and 30 seconds per resume .

That is not enough time to take all of that in there . Nobody is that good as recruiters . We get good at it , we feel good at it , we can do it easily . That doesn't mean we're actually effective at it . That doesn't mean the result is suitable for purpose . That's a different . Expertise needs a feedback loop . It doesn't mean just feels easy .

It feels easy for us because we do it a lot Doesn't mean you're good at it when this is studied . So interviewingio is the best study in this . They did 10 years ago and they just did a new one . They asked tech recruiters working at tech companies . Hey , here's some resumes . Categorize them based on their likelihood to pass a technical interview . Easy right .

These are tech recruiters . That's what they do all day , that's like their main job . They were slightly better than a coin flip slightly better . And when they did some post-h hoc analysis on what predicted a recruiter picking a resume , it was underrepresented status .

Recruiters really do care about diversity and it was prestige of previous employer and specifically name recognition of previous employer , not was this employer selective ? It's have I heard of this employer . That's what mattered .

The hiring managers want people who are good and they would like to live in a world where they don't have to confront the reality that there are a thousand resumes and a lot of them look good . A recruiter has to pick , and so I have to pick on something .

And what they tend to pick on , regardless of whether they admit it , if you look in this study and others , they look at brand name recognitions Like , oh , you worked at Airbnb , cool . The problem with that ? It actually is pretty effective . There are some . A lot of name brands are selective institutions .

The folks you pick from there really are more likely to pass your interview . It's not wrong . It feels icky . The problem is it's incomplete because there are selective institutions .

A startup that you have never heard of , a scale up that you have never heard of , because now we're all recruiting remote , and there's all these companies we've never heard of , all across the country in the world who is better than the one you've heard of ? Who is more selective than Airbnb .

So there's this prestigious resume that super predicts passing your tech screen because they have a harder tech screen than you do , but you never heard of that company . So you as a recruiter , in a hurry , you have to pass on them because you just don't recognize it .

The robot can build a list of what are selective schools , what are selective employers , and match it against the list . So you're doing the same thing , you're just doing it better , and we can talk about how to fight against just prestige bias that's in their topic .

But at the start , if we're going to do the same thing humans are doing , let's just do a better job at it .

Let's match the prestige list to one that is more complete and hits the people who haven't worked at Google but have worked at the most selective startup , in a fintech startup in New York that recruiters have never heard of but is incredible at selecting developers .

Uh , all right , like from a looking for , like a technical perspective , not like engineering technical , but like looking at it from an analytical perspective of what the actual environment looks like and matching that is much more critical .

I go through this all the time with my customers that are in the startup and growth stage phase and it's just , I'll see . I got a customer in HR tech right and they are a startup or growth stage company , probably around 50 employees , 400 plus customers and their primary they were looking at . Okay , we need to hire salespeople .

Oh , let's get people from LinkedIn . And I'm of the opinion you don't really sell LinkedIn . Sorry , it's a monopoly business . People come inbound , you're shuffling papers around . It is what it is right .

A lot of the times you don't have to develop very strong sales skills and it's a totally different motion than working for a startup or growth stage company that nobody's ever heard of , that doesn't have every resource available under the sun , that isn't heavily automated , has every point possible technical stack thing in place , the motions , the consultative , strategic

motions of everything , and knowing what it's like of working for a startup , being spread thin , the work ethic , everything that goes into servicing customers . It's just way different . So , yeah , I want the no-name startup that's growing fast . I want people that come from the same environment . I want people that I don't think could do the job .

I want people that have done the job . I want people that have done the job and I can get references from previous direct managers . I don't really care where you worked from , as long as the environment fits . So I think it's just like if you're enterprise and you're going to another enterprise , it's yeah , you could do that .

But if you're enterprise going to a startup , I don't care where you come from , I see that as a riskier hire , like it's just riskier . Like even an engineer working at a big company . Now , there are situations where this is the nuance right , they were working at a bigger company . It was on a smaller team . Was it like in a subsidiary ?

Was it like in a new kind of project ? Did they have fewer resources than like the parent company ? Or like they were off doing their own thing over here ?

So they were doing a lot more Sometimes , like you'll see , even on an engineering point , as we're a startup engineer , if you're first like one of the first 10 or first 20 , your scope of what you might be doing is a lot wider right . The technologies you might be working on are a lot like more recent .

Am I going to skip the Google resume Because they'll be excited about that Google resume ?

If they're not like doubled at our comp range , then yeah , maybe we should consider them .

Realistically , they should put their effort at the very front , defining the requirements and getting really uncomfortably specific . So prestige is something nobody likes to talk about unless you're doing outbound , like all the outbound tools . They have that filter .

They have that prestige top 1% filter , top 20% filter , but no one's built that on the inbound yet because it feels gross . I don't see a lot of scorecards that are like must be from a top 20% institution anymore . But the reality is your recruiters often are having to make decisions and they're using prestige .

So why not make it an explicit requirement and get to decide ? Is it a must have ? Is it a nice have ? Right now ? Let's put that thinking upfront . Have the hard conversations and don't let it get into the squishiness of the recruiter with 30 seconds trying to figure out if this person gets to have a screen or not .

Does that make sense ?

I prefer having another document that is not the job ad , that has the actual requirements and rules To me job description . I think companies who see that as an advertisement perform much better than companies who see it as a job description . I think that's a distinction that I make and everyone does .

But at least you have to show that , whatever the thing is , whether it's a another document or job ad , you are tying your actions to that , and then the EOC tends to be , and that means you need documentation . So that's the key is write something down somewhere that you can send to a lawyer .

Not only is that good for the lawyer , but that's good for us to stop to not lie to ourselves that actually looked at this .

Should they be helping companies create the job descriptions so the company can type in your share role requirements and then it can refine JDs ? Because a lot of these products are doing that too .

A lot of products right now are actually , it seems like , almost helping shape role requirements , and so is that something where maybe people should be staying away from having AI craft requirements , and it's more of like giving AI like very clear requirements and then like Personally , I worry more about AI helping with .

I worry more about that than I do about Gen AI ranking resumes , frankly , because if you do the second thing , the first thing is where , if you get that wrong because you just cargo culted someone else , everything else is going to be wrong yeah , just , it's like the yeah , the foundation to everything you're doing .

struggle a little bit and it gives you like more things to manage . If the job description is essentially like the core requirements and then I don't know , maybe AI is going to create a job advertisement that's just like a few bullet points and is more of like marketing marketing , I guess that's fine .

But yeah , I've tried to manage both and I think it can be a huge challenge to try to maintain multiple documents . And then maybe there's risk , right , when those get out of alignment .

A requirement change on the job description nobody updated the job advertisement , and then how does that work with the scorecard , right , that's been created and then any of the questions , right , that are trying to pull certain responses or examples to fill out the .

I just , yeah , I think there's a lot of inconsistency with if there's a job description , a job advertisement , a scorecard , questions that are aligned with the scorecard . Those rarely seem to all line up in this like beautiful , consistent way .

If you're going to have a job description somewhere else usually it's Google Docs or like a Word file , but then you have to go remember to update the ATS , because the ATS is where the scorecards are housed , which is going to be what the recruiters and the hiring teams are using to actually evaluate the candidates .

If , to your point Wes like , if there is becomes this distinction on who's defining requirements , as like people that are writing requirements and reviewing requirements , then I could see these , the product roadmaps building out this distinction of here's the JD and here's the job ad . But then you're like Elijah , I think you touched on this too .

I wonder how it's going to be viewed . Think about , like transparency laws , right Around compensation . Are there also going to be ? Like how much of the requirements need to be publicly facing too , and having two different documents . That's a I don't know , it'll just be . It's weird .

I think we have to keep like product roadmaps a little bit loose , like you try to guess . I think you're thinking about it Like it really . It seems it makes it seems very logical to me .

We already have four things we're keeping in sync and the other thing adds an additional one which is hard . It's hard to keep those things in sync . I'm excited about vendors like Poetry that seem to be targeting this problem of reusability reusable things , maybe you can use something to generate another thing .

I don't think they have this yet , but I would love for them to build it . And Ashby has , as far as ATS vendors to complement , they have a version of creating requirements that are separate from your job description . They're suggested based on the job description and then you have those as a separate thing that you can add or remove .

I think that's a good advancement and you can use LLMs to score them . You're on your own for prompt engineering the stuff .

It feels very you know beta , but I love that they're doing it and putting it out there and letting folks get the power of technology , because the lms are at least as good as a very busy person , in my opinion yeah , I know there's a lot of like concern around llms involved in the hiring process , but I I get it .

Yeah , it seems very obvious to me and it did at first , before I was really diving into AI and LLMs , before I really knew and I don't think really any of us really knew a whole lot about the technology that came out a couple of years ago .

Like maybe you did , but I , a lot of us , were like , well , how the hell does this really work and what's , what are the ? But now that I've learned like a fair amount , like it just becomes more and more clear to me . I understand the fear . It's a priori . Their CEO , elijah , was . He came on the show and we were talking with they do .

It's another AI kind of product , but what they were essentially talking about is like the analogy to autonomous cars , and so it's like statistically it's safer , but people are still scared of it , like that concept . I think it's like the same with AI . Here .

It's statistically , we should be able to create this tech like in the near term to be significantly less biased . Like bias is a huge problem in the United States . It's massive . Okay , so this is an opportunity to make it significantly better . So it's you know , I think it was like all this like fear , and I get it , but I don't know .

I think this is like . I think the more that people become a little bit more comfortable with it , it's pretty clear . It's pretty clear that this is going to be so much better , so much better as people .

But I think anyone's looking closely at the advancement and saying we're not going to give it this A to Q problem . We're going to give it B and C and E and F and Z . Wait , that's outside . You give them the pieces that it's going to be really good at and I think the drudge work of like kind of data entry . Does this application match this criteria ?

That's a really good use case right now .

That's where I say like wider in scope , where it's , if it has access to demographic data and stuff like that , like you're , of course , the likelihood of bias like creeps significantly .

But if it's very dialed into looking for very specific information and also like the , I think , the prompt engineering and what's happening with the like AI and all these types of things , people are writing in things already to prevent bias .

It's an ongoing thing where there's already a lot happening to train these systems to not be biased , and it's going to be a lot more effective than your employee taking a compliance class once a year . This is like mentally checked out . It's the last thing they want to do after working 40 hours a week .

It's just I think hopefully there's going to be mistakes made . But it's the same thing with the autonomous cars , like just because if one crashes like we shouldn't just say , oh , it's not safe , if statistically it's safer , and then we need to keep refining it , and there's never an okay amount of bias or discrimination ever .

Anything above zero is bad , but if we can move in the right direction and it has significantly less , yeah , let's do that and I think it will .

Every time I've done this , when I can get the hiring manager to like really partner with me and be specific , the whole search goes better . So you basically get . I think it's three to five . What are the top three to five things that need to be accomplished within the first , let's say , 12 months to determine whether or not this was a successful hire ?

So you're tying together , almost like the performance evaluation at day 365 after their start date , to figure out how are they going to be evaluated and how are we going to know that we made a successful hire a year in . And then you're determining those well needs .

To close , let's say it's I don't know $800,000 in new business If it's a sales role needs to , and you go through these three to five key performance objectives . Then you use that to build the job description and the requirements and anything else .

And then the scorecard right in the evaluation is also what , like the key performance objectives are on the scorecard . So you're basically trying to figure out like , can this person do what we need done ? And , as James alluded to earlier , have they done ? Do they have examples of doing that before in similar contexts to this ?

I'm a big fan of that performance-based hiring . When you can get those performance objectives , everything else is more consistent and clear throughout the whole rest of the process , including their first one-year performance eval .

Yeah , the tech industry . So , like one of the things , revenue , collective revenue yeah , the revenue collective . Uh , yeah , so they're , it's a cool group . But west I don't know if you're they have a ceo group . That might be interesting for you guys .

But anyways , yeah , like , one of the things he always talks about is head count is not scale , right , like scale is unit economics , your revenue , your margin , scaling at a rate where you're making more money because of certain investment decisions . And he often says a company's mistakes scale with just growing teams , which is really just a cost burden often

. And so it's getting into that mindset of ROI surrounding hires and there isn't a clear way to track back how this hire is helping the company achieve the North star metric . Like , why is that hiring being made ? And you're right , that's how we should be thinking about everybody . We should be thinking about putting together job descriptions .

There should be a very real ROI that's tangible and not just it doesn't even have . It doesn't have to be a sales role , it should be any role within the company .

And for me , one time I went through that exercise and realized I wanted to hire . I was like I'm going to hire a VP of marketing . I was like no , I don't need a VP of marketing to do these things . I need an entry-level person . It'll be way cheaper , they will actually like their job , versus if I get a VP to try to update AdWords .

They're going to hate me and it saved me $100,000 and a lot of pain just because- .

What's also really interesting is that when I'm evaluating talent like one of the things that I also look for if I'm hiring for my team is does the person I'm hiring understand how their role directly impacts North Star Metrics ? Do they understand the correlation to what their activity , why it actually matters and how it's driving the business forward ?

And are they aware of their environment in terms of how they might be doing that ?

Or do they have ideas on maybe more efficient ways to do that , how they think the team could operate and I do this for ICs Even if they're not in a strategic role I want to know if they have that self-awareness , business acumen to some extent and if they really understand the impact that they need to have right Like , versus just tactically operating day to

day .

They should do the thing and tell you they're doing it while they're doing it , so you can correct it , but they just go and do it . So it shows that they know the next level it , but they just go and do it . So it shows that they know the next level . And one of the things in the book is asking do people know what they're connected to ?

And I was like , yeah , I'm crushing this . And then I went and asked I asked that and everyone I want to have for the next two weeks , and I was not crushing it .

Yeah , I also like one of my go-to questions when I was scaling out either my team aggressively in 2022 , we were just hiring like recruiters every multiple month and I would ask like recruiters , like two years of experience and be like hey , so if you were made CEO of your current employer tomorrow , what are the , what would be your top three initiatives and why

? What would you double down on , what would you change , what would you discontinue ? And I feel like I got so much value from that , just again like feeding into their awareness , like their understanding of their role within the company , other people's roles within the company , like to me , it makes a huge difference . So it goes both ways .

It's like the hiring team needs to understand people's point of impact . How's it ? Impacting with stars , Like best candidates are going to understand people's point of impact .

at it on resumes .

Look , this has been a really fun episode . I definitely learned a lot . I really enjoyed the EOC conversation and you definitely had me thinking about some different problems and challenges and opportunities in a new way today . So I'm sure I'm not going to be the only one , as people are tuning in here .

It's definitely a lot of value that you've shared with us today . Wes , thank you very much for taking the time to educate us and our audience on everything that you're working on and knowing . It's definitely really impressive , and we're really thankful that you've come on the show today to talk to our community and help us out . Pleasure is mine . Thanks , guys .