Jen Wolke, Cooking Up Quality with Chef Claude

[0:00] I apologize that you may not even be able to see me over the stand and the computer here. All right, let's get started. Welcome to Codegarden Cooking Institute. My name is Jen Wolke, as was mentioned, and I'm a senior developer at ProWorks. I'm in the US, and I've been working with Umbraco since version 8. Oh, I guess I could expand this. And I do enjoy cooking, but I really hate cleaning up the mess, and I can make a really big mess when I'm cooking. I'm also a huge proponent of QA. I don't know where to stand so that you can actually see me, because I'm going to need to adjust the slides here. I believe that testing is vital to the development process, and that well-executed QA makes the whole team look great. However, that doesn't necessarily mean I'm particularly skilled or efficient at it. I'm just a lowly developer.

[0:51] So today I want to share some simple ways that I've found to use AI to improve my testing practices. So whether you're a seasoned QA professional, just getting started with QA, or a QA-challenged developer like me, I hope you'll find what Chef Claude and I have cooked up today interesting. If you want to access anything from today, you can go to this QR code, and it'll bring you to this website that has all the information. So, speaking of our head chef, Chef Claude, let's talk a little bit about him. He's a multifaceted guy, with three different modes you can work with: Cowork, Chat and Code. These are accessible via the desktop or the web, and we'll look at the desktop in a little bit. To oversimplify a little, Cowork is the most basic mode, while Claude Code is the most advanced and requires the most technical understanding, and Chat falls in between.

[1:48] This is not to say that Cowork and Chat can't produce some really complex results. They both can. They're just a little easier to work with, and don't require as much technical expertise. Today we're going to be focusing on Claude Chat on the desktop exclusively. We'll also be using a Pro plan, but everything that I show today can be done on just a standard plan. So, in addition to the three ways to interact with Claude, there are also three main language models that you can work with: Haiku, Sonnet and Opus, again in order of increasing complexity, Opus being the most advanced. Models are kind of like flavors: some are stronger than others, which also makes them more expensive. So Opus is obviously the more expensive one. We're going to use Sonnet, right in the middle, again today, and Sonnet 4.6 specifically, because that's the latest version.

[2:43] Next up, there are four tools in Claude's repertoire that we're going to see today. There are a few others, but these are the four that we're going to focus on: instructions, projects, files and skills. And combining all of these tools with a well-written prompt creates a recipe for success. So, now that you know a little bit about Chef Claude and myself, let's move on to our menu. We've got a nice salad as an appetizer, some hearty main courses, a little bit of dessert, and let's also take a little tour around our kitchen to see where everything is. So, let's head over to the Claude desktop.

[3:30] So, first of all, you've got your basic prompt window, where you're going to interact with Chef Claude. Many of you have probably seen this, so nothing new here. And if you take a look right here, you can change your language model and shift it up or down, if you want to do that for just a single session, or if you want to change it permanently. Again, we're going to stick with Sonnet. Next, if we go up here, we've got our projects. And our projects are kind of like our mixing bowl. They hold everything. If we take a look at this project, we've got our instructions up here, and our files, and our main prompt. And then we'll talk about where the skills are in just a minute. Our instructions are kind of like our cookbook, or cooking manual, for everything in the project. Not just a single recipe: they govern how everything in the whole project is prepared.

[4:30] Next we have our files over here, which are kind of like our ingredients. So what have we got here? We've got some eggs, shell and all. What else do we have? We've got some butter. We've got some sugar, and some flour. I don't really know what we're making, but there's butter and sugar in it, so it's going to have to be amazing. It's also going to be really tiny, since this is the bowl that fit in my suitcase. And finally, we have our skills, which are over here under the Customize section. And these are kind of like our cooking techniques: dicing, chopping. In this case, whisking, because I decided the knife on the plane probably wasn't going to go very well. And I really hope we only need a whisk today, because that's all I brought. So, we've got that.

[5:25] So now that we know where everything is, let's go ahead and get started on cooking. So first up, let's put together our skill creator salad, if the internet cooperates with me. So, this offering of mixed greens will walk us through creating a new skill with the help of Chef Claude. A few notes about skills before we get started. As we saw, they live outside of any project. They're reusable actions that you can use across your projects. The easiest way to create a skill is to actually use another skill, called the skill creator skill, which comes already installed with Claude. You can import them, download them and export them, so you can not only share them across your projects in your desktop, but you can share them with other people. You could create a skills repository and have them checked in, so that you can share them across your team. And lastly, there are a few different ways that you can invoke those skills. You can invoke them by name, using a prompt or your instructions. You can invoke them using plain language. And you can also invoke them implicitly, by using some other action that requires the skill, and Claude will figure out that the skill is needed and invoke it for you.

[6:35] So first, since we're going to create one of these, we don't need any instructions. We're going to use the skill creator skill, as we talked about, no files, and we're going to use this prompt. And I'm going to copy this, because we're going to attempt to do this in real time. We'll see how it goes. It doesn't take very long to put together, and it doesn't take very long to cook up. So, let's go over to Claude and... sorry, I was already in the skills section. Let's go back there, and we're going to create a new one right here. And we're going to say Create skill, Create with Claude, and it's going to start the prompt for us. And I'm going to paste in the rest, which is basically to create a skill that says "Thank you for attending, bon appétit" any time the user says a farewell. Obviously this is a little bit contrived and a little bit trivial, but I want to show just creating a basic skill using the skill creator. So I'm going to let this run.

[7:42] While it does that, I'm going to jump over here real quick and show this bottom section of the page. So each of these recipes has a section at the bottom which will show you the output in Claude. So if you want to come back and reference this later, it's all here, since obviously you're not going to have access to my Claude desktop. So you can see what the chat looks like when it's run. So, let's see how it's doing. Oh, it's already finished. So it says, "Great, I have everything I need. Here's your skill." And if I got really lucky, it named it what I expected it to, so the rest of this demo works. And then it also tested it. That was super helpful. That was nice. Oh, and it's going to ask me some questions. So, it wants to know... I'm going to tell it I only want it to say the exact phrase, and on any farewell.

[8:52] So now it's going to tighten up the skill. All right. And now it's going to package it up and create all the files that I need. The nice thing about using the skill creator is it scaffolds out the structure for you, so you don't have to remember what goes where. So now we've got the skill right here, and I'm going to say Save skill. And I'm going to have to adjust Claude's hat, because I can't see. I'm going to have to just move that. And it saved it. So now if I go over to the skills section, I have a new skill called Codegarden Farewell. So that is creating a skill. Like I said, that one's obviously a little bit trivial, but we'll see how to use it in just a minute.

[9:46] All right, so next up we have "Does the author field exist?" pizza, with a side of backoffice QA. So, this deeply satisfying dish verifies the presence of an author field on a form, and validates the available options. Sorry for the really bad pizza pun there. So this one's going to be really basic. We're not going to use any instructions, skills or files. We're just going to use a prompt. And this one I'm not going to try to run live, because I'm saving my... not tokens, but goodwill, for a little later. So, Claude is going to be an experienced test professional: "Write and execute a test plan to confirm the presence of an author field on this form. The field should be a drop-down with the following options." Again, pretty straightforward. So let's take a look at what that looks like.

[10:54] So this one I iterated on a little bit. I did the first prompt, which, I don't know if you can see it, says, "I would like to verify that the form on this web page has an author field," and that's all I gave it. It did a little bit of testing and spit out some results. And I just said, ah, that's nice, but maybe I can make that a little more specific and a little more involved. So that's when I said it should be a drop-down with these values. So then Claude goes in and does all the testing for that, continues on, and gives me a nice test report, and a nice table with all my passing criteria. And then it gave me a summary, and said 10 of 10 passed, which is awesome.

[11:41] So next, let's test the backoffice, which is pretty much the same prompt, just a different form: the page in the backoffice that has this on the document type. So we're going to run pretty much the same thing. And for this one I'm cheating a little bit, because I've already logged into the backoffice in the browser. And one thing I didn't show is that, in order for all of this to work, you have to install the Claude browser extension, right here. And this is what allows the desktop to interact with the browser, and scroll around, click the buttons, open the drop-downs, all those things. So you do need to install that. And if you don't have it, the first time you run Claude, it will tell you that you need to install it, and kind of give you instructions. So, it's super helpful.

[12:33] So, like I said, I'm already logged in, and I'll show you in a minute how you can make this a little more autonomous using a skill. I have a login skill that can be used so that, if it's not already logged in, it will log in for you. So, because I'm already logged into the backoffice, it's going to go to that URL and essentially do the same thing that the other test did. But here's where the human intervention comes in, because for this one... Where'd it go? I can't find my inconclusive test. Claude is quite chatty. He likes to tell you everything he's doing, which is helpful most of the time, except when you're trying to find something.

[13:21] Ah, so here it came back with an inconclusive test, and the key finding, if you can see this, in the second paragraph, says this is likely a data API loading issue. And in fact, it's not. It's actually a bug. So Claude has determined that it's just a timing issue with the drop-down not loading, but in reality it's not actually loading in the backoffice, and it's a bug that we had to fix. So again, there's that human element with anything with AI. You've got to verify your results. You've got to make sure that it stays on track. You can't just set it and forget it. All right. So that is our pizza with a side of backoffice testing.

[14:07] So next up we have our accessibility audit soufflé. So for this light and airy dish, we'll use some instructions, combined with a skill, to go with our prompt, to do an accessibility assessment of this website here, just the homepage. It's nice and small and contained. So, in my instructions, I'm going to tell it to explicitly call that skill that I've already created. And then we're going to use two skills in this one: the greeting, and then the farewell that we created earlier, which I'm going to invoke in the prompt using plain language. And I'm not going to use any files for this. We're going to keep this nice and simple, to do a quick example. And we're going to try one more live. We'll see how this goes. It has not been cooperating well in my practice runs, but I do have a backup plan.

[15:07] So, I'm going to create a new chat in the project, the QA project I have for this website. So, I'm going to paste in my prompt and let it go. It's going to analyze the accessibility of the homepage of this website and report on the top 10 issues which need to be addressed, in case you can't see that; it is pretty small. So it ran my greeting and said "Yes, chef!", because that's what I had the skill do. And the first thing it says is, I read the Codegarden greeting skill. So while that runs, we'll give it a few seconds. Okay. So, now it's going to go through and do some testing while that runs. Yep. I think we're going to have to.

[16:08] And this will be the last one that I attempt to do live. Everything else is going to be pre-prepared and reheated, because I don't have good luck with live demos, as we're seeing right now. Oh, there we go: only one violation. And then it's going to, hopefully, spit out some... Come on, you can do it. Nope. All right, we're going to stop that. You can just hit the stop button in the bottom-right corner, and I'm going to show you one that I've run previously, just so we can see the output. So basically, it says it has enough data, and it did an audit, and it listed the top 10 issues. And these were pretty much in line with what I expected, based on how I developed the page: all the things that I already knew were wrong. So, just a quick check through here. This looks pretty good, so I'm pretty happy with what Claude came up with.

[17:04] And then here it used our goodbye skill, or farewell skill, that we created earlier, and said "Thank you for attending Codegarden Institute, bon appétit," just as we expected. So, like I said, that will be the last attempt at a live demo, because I've exhausted my favor with the demo gods. So next up we have... oh, and this is worth showing. So while it was running that, it had launched the page for me, and was attempting to go through the page until I cut it off. And up in this right corner, I don't know if you can see this, but it says Claude MCP. So it had actually launched the browser and was attempting to interact with the page, and you'll see this with anything that it needs to do on the web. I'm going to go ahead and close that, so I don't accidentally switch to that tab.

[18:05] All right, so next up we have fish and regression testing chips. So, for this classic meal, we're going to compare a critical subset of pages between a QA environment and production, to ensure that no functionality was lost during development. And in this one, we're going to give it some instructions, and we're going to start to tighten up the instructions and give it a little more direction. So before, I was just saying test this, run it however you want. But now I want to try to be a little more consistent in the output. So, I've told it that all the test results should always be in a tabular format with these columns, and that they should always include a defect log and a test summary, because I want to get that consistency from test to test if I'm doing QA on a project. So, we're not going to use any skills or any files, but we are going to use several prompts for this one, because this is a bigger test.

[19:02] I've got about 20 pages. I think there are 18 that I want to test, and I didn't want to just turn it loose on all of them right at the beginning. I wanted to see what it was going to output. So I did the homepage and an interior page before I said, okay, test them all. So this really just breaks things out into chunks. And this one obviously took a little more time to put together, and it is going to take quite a while to run, because there are 18 pages. But this is, in my opinion, a fraction of the time it would take me to manually take 18 pages, pull them up side by side in my browser, click everything, and make sure all the functionality is going to work. Because it's going to look at not just the visual but also the functionality of the pages: any of the animations or interactions or anything like that.

[19:58] Okay. So let's go over to Claude. I won't go through all of this, because it is pretty extensive. But the first one was basically: I want to do a comparison of these two domains on these pages. The first one is the homepage. It's going to navigate to both pages and compare them side by side. There's some stuff in the middle here where the browser plugin didn't have access to the domain, so I had to grant it access. And it tells me how to do that, and walks me through it step by step. Apparently I did it wrong the first time, and I needed to do it again. And then eventually we get to a very nice table, with 18 passes and 11 failures.

[20:53] That's a lot. And because I'm comparing two different environments, a lot of those could be content-related, but I don't necessarily know that. And this is again where that human element comes in. I've got to go through and make sure that those are legitimate, or maybe they don't need to be fixed right now. Maybe they're pre-existing. So I'm going to have to go through and apply what I know about the project, and what I know about QA, to analyzing those and figuring out if they're real or not. So then, if we keep going, I got a defect log, which is quite long. And then I got a test summary, just like I asked for. And the next thing is a single interior page. It's going to do the same thing. I have a slightly shorter list, not by much.

[21:41] And then I said, okay, great, this all looks good. Again, I have my defect log. All looks great. And then I say, you know what? Let's do them all. Everything looks good. So now I say, here are the remaining pages, and I give it 16 pages. And it goes through, and eventually now I have 42 failures, which is a little intimidating. It's a little overwhelming, but still, that took way less time than it would have taken me to find those 42 issues, because I was going to have to figure out whether they were real or not, whether I ran this using Claude or did it myself. So that part hasn't gone away, but it has made this a lot faster.

[22:31] So, let me scroll down here, because the next thing is that this sounds like a great skill, because I'm probably going to do this more than once. I'm probably going to want to do this every time I do a release. So I basically just said, "Hey, Claude, turn this into a skill for me." And it says okay, and it turns it into a skill. I haven't rerun this skill yet to see if it needs any tweaking, but it packaged it up and created a skill over in the skills section, called MD QA Comparison. One thing I will note is it's kind of a pain to rename these. The only way I found to easily rename these is to have Claude rename them: just type in there and say, rename this skill for me. Otherwise, you're kind of digging through the files to find all the places you have to update. But you can just tell Claude to do it, and it's done. So that is our regression testing fish and chips.

[23:33] Let's see what we've got next, because I can't remember. Ah, required fields validation ratatouille. So with this robust dish, we're going to ensure that the required fields on a change password form are appropriately denoted and validated. And for this one, we're actually going to use a few different tools: a skill and a file. So now, I'm again tightening up the output. I've said, you know what, I told you this much, and you're doing a good job of that, but now I want you to use this file that I'm going to attach as an example of what I want. And the file is a Markdown file. It's just my personal preference for what I wanted. So, if you know Markdown, or if you don't, this top section is a table, and then I've got my defect log here. So I'm basically telling Claude, this is what I want my output to look like.

[24:25] So, I've added that file to my project, and I'll show you that in just a second. And then I have a prompt to say, write and execute a test plan to verify the fields which are required are marked, and I gave it the URL of the form. Now this is where the login skill comes in, because this form is actually behind login. This is not a forgot password form, but an actual change password form, so I've got to actually log in. So I've previously created a skill that will log in, which I'm not going to show, because it has the password in it. But that's already been set up. So, with that already done, and the file already created, the prep time to write up that prompt was pretty quick, and it won't take long to cook that up.

[25:13] So, let's go back to here. Actually, first, let's look at the project, and look at the instructions for this project. So, we've got our instructions here, which are what we just saw, with the file. And then we've got the sample file attached here. And then I ran the prompt in here, but I don't see it, so I'm going to just use my pinned chats to find it.

[25:59] And for some reason, it doesn't retain where you were, so you have to scroll back up to the top. So, I'm going to go ahead and run that prompt. And it says, I'm going to go find the login form. Okay, great. I'm going to fill in the credentials. And it says, wait, there's a reCAPTCHA, which is true. So it asks me to check the reCAPTCHA for it, and tell it when I'm ready. So I did. I think the skill is still useful, because I don't have to remember the credentials; they're just programmed in for me. So it says, okay, now the reCAPTCHA's checked, I'm going to log in. And it's going to test the form, and it's going to find no errors, which is great. We did a good job developing that. But one of the things that's interesting about this is that, because I gave it an input file, it says, oh, you must want an output file. Which I didn't necessarily know I wanted, but now that I have it, this is really great, because I can share this.

[26:50] So if there had been defects, I could share that with the team. I could put it in a bug tracker. I can distribute it out. The other thing that I can do is... you'll notice this is starting to get really cluttered. You know, I've come up with kind of a naming convention for my projects, to see them in my recents. I pinned things for ease of finding for this demo, but there's really not a great way to organize the chats inside the desktop. So I can basically save these off, and create a repository of all the tests that I've run. The other thing that I can do, and that we'll see in just a minute, is that I can take this and feed it back into Claude to retest. Let's see if I missed anything in my notes here. All right.

[27:48] I think I covered everything. So let's go on to Lighthouse performance testing linguini. And this is going to be a short one. But I want to show that, okay, I've got all these instructions, but what if I need a different output just as a one-off? Well, in this dish, we're going to ask Claude to ignore previously provided instructions in order to test the performance of a website, because I want a slightly different output format. So, I'm going to have Claude ignore my previous instructions and files for that same project we just looked at. And then I want to run a Lighthouse-style report on this website. And this is in the prompt, where I tell it, please ignore everything that I already told you. And hopefully Claude is going to do it. This is where the stereotypical temperamental chef sometimes shows up. There were a few times where Claude said, no, you gave me instructions, I'm going to do it. So I had to ask nicely. I guess I had to put please in there; I don't know. But eventually I landed on a prompt that actually worked. And as you'll see, I got a top five list of things to do instead of what I got before, and I did not get a file as an output. So I was able to kind of override what I had previously told Claude.

[29:07] So, next up we have retesting required fields burger and fries. So in this fan favorite, we run a previously executed test to determine if defects have been resolved. So this is where we're going to take one of those files and pass it back in. So we have all of the same instructions we had before for that same project, same files. And now we're going to add a file on our prompt, and use that as input to the testing. And we're going to use the same skills. And this is all I'm going to tell it: rerun the tests represented in the attached file, to determine if defects have been resolved. That's it. I'm not going to tell it anything about what the test was. I'm just going to give it the file and say, retest this.

[29:55] So if we go back over here, you'll see the file here. And the way you attach that is, if I go to my project and go here, when I start a new chat, there's this plus sign right here, and I can add files right there. So if we go back over here... How am I doing on time? 15 minutes. All right, should be great. And this one pretty easily runs through, and says your bugs have been resolved, which is great. So, thumbs up there. But I basically just passed the file back in. Oh, and it used the skill, logged in, got stuck on the reCAPTCHA again, and then reran the tests. I didn't have to tell it what the tests were. I just said they're in this file, go do it. And then, because I didn't tell it to ignore the instructions, I got a file as output, which is useful. Sorry. Okay, what else do we have? I think that's it.

[31:01] All right. So that is all of our main dishes. Let's move on to a little bit of dessert. We've got some tips and tricks à la mode. Most of these I've talked about, so I won't talk about them in depth. You can also come back here with that QR code. I'll show it again in a minute, and you can take a look at these. But one thing I do want to call out: there's a lot of talk right now about the price of tokens going up. So the more specific and directed you can make your prompt, the more efficient it's going to be, and it's going to save you tokens, or ultimately money, because there's going to be less chatting back and forth.

[31:46] The more you chat back and forth, the more tokens you're using. It's kind of a fun thing to do, that kind of vibe coding where you're going back and forth, but it does eat up your tokens really quickly. So the more guidance you can give Claude, the more efficient Claude is going to be. And speaking of the costs, if you're not using the Pro plan, you should be able to use the /cost skill in your chat, to get it to tell you how many tokens it's used and then convert that into currency. I don't know if that still works. It worked a couple of weeks ago when I tried it. I haven't tried it recently, but if you're on the Pro version, you can go to your settings down here, and your usage, and you can see when your tokens are going to reset, and what percentage of your tokens you've used. So you can see this at any time. So that's my tips and tricks à la mode.

[32:49] And next up, QA is better with AI surprise. So, this is a surprise dessert, because I didn't know what we were going to be getting. Thanks to Irina and Stephie for organizing: we have some Danish butter cookies over here, so feel free to come and get one afterwards if you'd like a mid-morning snack. It's all right. You can still have dessert even though we haven't had lunch yet. We're adults; we can make that choice. So how do we make QA better with AI? A little bit of patience, some creativity, some critical thinking, but mostly an understanding that it's not going to be a silver bullet. Again, we've talked about how there's going to have to be that human element to verify and validate things, and to make sure Claude stays on track. As with any use of AI, it needs some oversight.

[33:35] But we have seen that there are a lot of types of testing, such as accessibility testing, regression testing and performance testing, that can be time-intensive, that your team might not have time to do, and that you can use AI to do for you. We've also seen that even some of those really targeted scenarios can be faster with AI, like that required fields testing. Even though it took a few minutes, it was still more efficient than me writing out all those tables, clicking everything, and then recording the results. You know, what took it five minutes might take me 30 minutes to an hour. That's still a performance saving, or a time saving.

[34:23] You can also use this to run a lot of different types of testing all in one place. So I could go use Lighthouse and do the testing, but I can also centralize it in here, and keep all of my results together. And as we also saw, AI found some things that I might have missed, that I might not have thought about. There were a few tests where it tested some things and I was like, oh, I never would have considered that. Maybe a real QA professional would have, since I'm just a developer. But if you have a team where you're light on QA, or maybe you don't have a dedicated QA team at all, this can really help speed up and improve your QA processes.

[35:11] But again, this is not to say that AI can replace a good QA team, or good QA practices. It can't. There's still, as we saw, the need for that human intervention, to evaluate the results and make sure things are staying on track. So, that's what I've got. And I'm excited to see what you might cook up. And thank you for joining me today. Feel free to take a snack on the way out, and come talk to me around Codegarden. The QR code is there, and also a few references that I used that were useful in the talk. And enjoy the rest of Codegarden. Thank you! [Applause]