In our most recent episode of the Certified Skills in AI and Tech podcast, we had the privilege of speaking with Jeremy Kirshbaum, an esteemed venture builder and strategist specializing in generative AI. As the founder of Handshake, a leading generative AI consultancy, Jeremy has worked closely with Fortune 500 executive teams, governments, and NGOs, providing invaluable insights and expertise in navigating the complexities of AI integration.

Real World Application of Generative AI

What is generative AI and what are you doing in this space?

Generative AI is any kind of machine learning algorithm that produces data as well as analyzes it or describes it. Any kind of algorithm where the purpose is to produce something like an image, a piece of text, or a video. So, under the hood, it leverages a lot of the same things that you’d use in classic machine learning. But ultimately the effect and what you’re trying to do with it is much different, and we’re focused on that.

We’ve been focused on that since about starting in 2019-2020 when GPT 2 and Bert were coming out, we were doing more general innovation consulting and came across these things. The feeling I would say that we had at that time was kind of like being a farmer with a shovel, being shown a tractor for the first time because our job as innovation consultants was to come up with ideas. And even at that time, these things were quite good at that. And so we realized that if we didn’t want to be driving someone else’s tractor so to speak, we had to get good at this.

At this point, we are very good at it. We build custom generative AI systems for clients as well as do consulting for folks on how to invest into, build with, or integrate generative AI into their organizations and teach others how to integrate it into their work and their teams with courses and training.

Demystifying Generative AI for Companies

How do you begin the conversation about Generative AI and how do companies decide what works for them and what doesn’t?

Well, the conversation is already happening in companies. We certainly don’t have to begin the conversation, and I think you need to help people boil the ocean. People oscillate between hearing that generative AI is just stochastic parrots and useless statistics, it can’t do anything and always lies, and [alternately] that it’s magic and will fix everything in our organization. And of course, neither of those extremes is true. Depending on where an organization is at, it can be anything from helping them automate something isolated, like sorting their text messages better, helping them write contracts faster, or helping them bring their reports and research to life more dynamically, to more general things about helping them think about what their users and clients will need in terms of use cases around this stuff. It can be quite unintuitive at times, right?

I don’t think there was any science fiction futurist writer or thinker before 2018 would have ever guessed that AI was going to be good at poetry but bad at arithmetic. But that’s the situation that we’re in. And so, there aren’t a lot of easy maps to previous things that you can use to understand it. When you get down into the details, how to implement it is quite unintuitive.

In your case of certification, there’s also a whole suite of skills that people don’t know to hire for yet and are still trying to figure out what is a new skill and what’s not. We hear a lot about prompt engineering. Do I need to hire prompt engineers now? How do I test if someone is a good prompt engineer? How do I certify someone is a good prompt engineer? Is there a difference between someone who does prompt engineering as part of their normal workflow at their desk versus someone who’s building prompts that go deep in the level of the application, deep down in the stack? How do I test for these differently? Do I give them my technical review to see if they’re good at Python, or do I give them my qualitative interview to see if they’re a good culture fit or both or neither? And so all these things are evolving and they’re right now very particular to different organizations in terms of how they’re expressed and we help people figure that out.

The Evolving Landscape of AI Skills

Are you working on building any courses or training for your client? How are you guiding them toward answering some of these questions when you start looking at prompt engineering?

We have online classes that teach from a more general lens, but there are basic principles that we can teach people that will apply across many different use cases. However, we also work inside of organizations to do research and strategy around what might apply in a certain use case and then build out processes and structures that will apply to that use case that is more customized to a specific setting.

So, you’re right that there are big differences and how the specific processes you’d use for code versus marketing for instance. However, some overall principles apply widely. Things like few-shot learning, things like chain of thought. Things like prompt engineering can get people a lot farther, and faster, which is like a few principles to push them.

What would your advice be to people who are looking to upskill in this area?

The one thing I would say is that no matter what course you’re taking, it needs to be hands-on. So, our course doesn’t involve coding, but we do get people. Even if you’re the partner to a hedge fund, we get you in no-code tools building things because there is no way to just communicate this through concepts right now. There are no best practices right now. These have only existed since 2019-2020, and they’ve only been what you could call good or close to good since like 2021-2022 probably is safer. And so anyone who comes along and says I’m an expert, this is how you do it, is either oversimplifying it for you or lying because no one’s been an expert with something that hasn’t been around for more than a few years.

Kryterion’s Approach to AI-integrated Test Development and Delivery 

At Kryterion, we clearly see the opportunities and disruption that AI is bringing to our industry. Our top priority is to provide clients with the best mix of security, innovation, service, and value in our tools for test development and delivery. AI is quickly becoming an integral technology in our products and direction. The path we are on with AI encompasses the immediate benefits it provides while maintaining a sharp awareness of its evolution allowing us to ensure our products continue to meet or exceed the needs of our clients.  

Please contact us to set up a meeting so we can further discuss how important AI is to our future and our clients. 

Want to keep up with insights on AI in the testing industry? Listen to the Kryterion podcast