CI/CD, We Salute You with Brenden Duncombe

Parker Dillmann
We have an update from Macrofab. Misha and Brendan of Macrofab are doing a webinar on January 30th at 12 PM CST. The topic is enhancing operational safety through cyber resilient approaches for physically secure PCB designs. Participants will learn about the typical vulnerabilities and electronic hardware and discover advanced design and production strategies, prioritizing security, including material and layout considerations. There is also a local event happening at Macrofab HQ.

Parker Dillmann
Become a part of the Bright Minds Brighter Future event on February 1st from 11 AM to 4:30 PM CST. Join us for a day of innovation and networking. Step into our state of the art factory where the future of electronics manufacturing is being shaped. Please pause the episode and head on over to macfab.com/events and register right now before you forget. There is also a link in our show notes if you want to go there as well.

Parker Dillmann
We will be right back here waiting for you, our listener, when you return. Welcome to circuit break from MacroFrad, a weekly show about all things engineering, DOI projects, manufacturing, engine news, and continuous integration and deployment for hardware problems. We're your hosts, electrical engineers, Parker Dolan. And Steve and Craig. This is episode 415.

Parker Dillmann
Circuit breaker from Macrofab.

Stephen Kraig
This week, we're happy to welcome director of customer engineering at Macrofab, Brendan Duncombe, to the show.

Parker Dillmann
Brendan's extensive experience in electrical and systems engineering and his leadership in steering engineering teams from concept to manufacturing makes him an essential voice at MacroFab, and we're happy to have him on the show. Thanks so

Stephen Kraig
much for coming on, Brandon.

Brenden Duncombe
Hey. How's it going, guys?

Parker Dillmann
So before we dive into this idea of continuous integration and deployment, let's learn a bit more about you, Brendon. Because this is the first time you've been on the podcast. I've been trying to get you on the podcast for, I want to say, almost a year now. So

Brenden Duncombe
Yeah. Well, thank you for having me. Before I was in manufacturing, I did a lot of product development. So I've been kind of hopping around different startups. My initial start out of college was in kind of like contract product development.

Brenden Duncombe
So I got an engineering shop and did tons of different projects. We did, you know, 3 d print supercars, some consumer electronics, big industrial things, kind of just a big mix of different electrical and, you know, embedded systems projects. And then lately, I had taken more of a turn into kind of manufacturing, and then finally ended up at Macrofab where I'm doing, you know, actual hardcore electrical manufacturing.

Stephen Kraig
So what is a director of customer engineering? Because, I have not heard that before.

Brenden Duncombe
Yeah. Yeah. It's a title for me as it turns out. But basically it means that I'm more focused on I mean, like I said, my background is in product development. So a lot of my focus is on the customers, like, specific design requirements rather than on, like, how do I necessarily improve efficiency on SMT lines?

Brenden Duncombe
It's more about, you know, helping customers get where they need to be when they're going to production. There's a lot of steps that happen between their initial prototypes and then what they wanna do when they go to, you know, mid and high volume production. And a lot of what I do is help them get there.

Stephen Kraig
So you're removing roadblocks.

Brenden Duncombe
Yeah. And just answering questions that, you know, it's not that often that you are doing that, like, v zero all the way to high volume production. There's only so many cycles people get in their career. And so we do it a lot. So just answering the, like, easy questions of, like, hey, you want to do x, y, and z?

Brenden Duncombe
This is very common. Like, you should start now instead of waiting until, like, right before you hit go on production.

Parker Dillmann
So, Brandon, do you want to jump into CICD, which is continuous I had actually looked this up when you said CICD. I'm like, I have no idea what that is.

Stephen Kraig
I've never heard of this either.

Brenden Duncombe
Oh, yeah. For a bit there, I was doing some software development, like strict software development. And CICD is basically table stakes for software projects these days. Right? By default, the frameworks you use, the tools you pick, they they come with the stuff built in.

Brenden Duncombe
They come with the hooks, like even GitHub now. Right? Like GitHub actions and stuff like that. It's by default designed to get you into doing CICD automated testing, automated deployment, automated build, all of that is kind of table stakes. And, you know, I had a background in electrical engineering.

Brenden Duncombe
And so for me, it was a little bit weird that embedded development, especially, just was not there at all. There's so many benefits to doing that. And there are the obvious roadblocks, right? There's definitely the problems with, you know, like, you have actual hardware in front of you, and it's not so easy as just, like, clicking a button and having everything happen, which, like, software development, CICD, typically is once you set it up. But the benefits, I think, far outweigh that.

Brenden Duncombe
And so like in the context of embedded development, CICD, I think, is like a fascinating thing to pursue. And there are some, I don't want to say like common issues, but there are definitely prevalent issues when it comes to people deploying firmware that they don't often test. Right? And so especially when it comes to industrial and critical where you don't have that many deployments, you know, it's a little bit easier to manually test the surface area of your code as opposed to trying to hit it the way like any random consumer would hit it. And so, yeah, there's a lot to be gained there.

Brenden Duncombe
And I think it's just, it's an interesting topic in software development, and I think, even more interesting when you apply it to firmware and electrical engineering.

Parker Dillmann
So how does, like, this continuous integration development, like, how does that actually you said, like, hitting more edge cases and that kind of stuff. How does that help you actually do that though?

Brenden Duncombe
Yeah. I mean, it it's not, like, the only way to do that, but when you think about like a typical QA cycle, right, even if you look at it from a pure software development standpoint, there's kind of like your automated unit tests. And then after that, you'll probably have some integration tests. And then a next step over that is like you might do some higher level, like some people do like UI tests where they're you know, actually, like, fake hitting inputs. And then you'll have, like, real QA testers who will you know, if you've got connected peripherals, they'll connect to them and then try out your software, or they'll try it from, like, a bunch of different browsers and things like that, or they'll try specific use cases.

Brenden Duncombe
You can see this all the time in, like, a range of industries, not just, like, you know, you can see it in like the gaming industry and not just like, you know, web applications and things like that. And so, but the benefit of doing some of your continuous integration and automated testing is that every build hits those, right? Like when you finally get to a human at the end testing it, right, the difference between what typically happens in embedded development and pure software development is you'll see that oftentimes way bigger stretches go between testing, and the only testing that they end up getting to is the the QA at the end. Right? And often, like, the most rapid testing they get is them sitting at their desk testing it themselves, right?

Brenden Duncombe
Like an engineer just like doing their benchtop testing. And so when you do automated testing, you save a lot of time. You can afford to put a lot more of your edge cases in there. You don't have to remember to do them all the time. They automatically happen every time you deploy, so you you never, like, run into this problem where it's now 3 prototypes later and, you know, 4 months later, and all of a sudden, like, everything breaks, and so you're not actually getting that much useful data.

Brenden Duncombe
It's better to just every time you make a code change, try and push, execute it, you know, stretch all of your features, make sure you're actually hitting everything you do as much as you can, and then pushing that.

Stephen Kraig
How is that actually managed though? Because the okay. So it seems like this is some kind of a progressive mentality where where you take small chunks and push them out and test them immediately. Right? But that seems like a nightmare to try to manage.

Brenden Duncombe
So, yeah, I mean, it it can definitely be a nightmare if you're trying to manually keep track of all these little changes. That's the whole point of kind of CICD and, like, this automation in general. So automation takes care of it. Right? You need to set it up.

Brenden Duncombe
And whenever you push your code, it will build, it will test, it will deploy. Like the whole point is you don't have to touch it, right? Once you set it up, every time you do any little thing, everything gets exercised, and you don't have to try and keep track of every little piece, right? That's usually where the difficulty comes in is if you're trying to manually track all this stuff, you know, you're trying to, like, make sure, like, okay. When I go to this revision, I need to test these new features, and I need to test the upgrade and the downgrade of the old firmware, and then test these different like use cases.

Brenden Duncombe
That is where you can run into problems. But when you just write all those tests, every time you push it, that's what happens. It automatically exercises all of that.

Parker Dillmann
So unit tests on software is you basically just run the run the, software with inputs, and then Mhmm. You have expected outputs for the test. How does that work for hardware, though?

Brenden Duncombe
Yeah. There's a lot of interesting approaches for this. There's kind of the what sometimes is referred to as host tests, where you just actually are simulating, like you're running real software unit tests, right? Like if you're let's say you're programming in C or something, like you, you know, you're using something like CMOC and these, and these other things that you can use to actually just do unit tests and simulate it. And so it just runs on your computer.

Brenden Duncombe
You don't need anything else. Then you can do on device tests where, like, if you need to exercise some of the peripherals or make sure some of your, like, actual hardware is there, You basically it builds like a a test version of your firmware, and and there's there's frameworks for this that you can use, and it will deploy to your device and then run the test and basically report back if it was successful or not on each of the unit tests that you run. You can also take it a step further, and this is where the, like, default frameworks start to fall apart and it gets a lot more custom is when you do, like, your own integration tests on your hardware or you have, like, an automated tester, you know, think of a bed of nails or something similar or, like, a black box that plugs into your main connector, things like that, that is actually exercising it in conjunction with a piece of test firmware.

Parker Dillmann
Yeah. It's it's it's really funny to think about it that way. Back when I was working on a lot of pinball stuff, we had kind of a same setup except that it was, like, the board was just attached to a, basically, a pinball machine. And we would make a code change, and, immediately, it would just go out to and program the machine, basically, and we could test it. Now we didn't have a automated way of playing the game, but that that would be, like, the next step.

Parker Dillmann
Right?

Brenden Duncombe
Yeah. No. That that's a very common first step. Right? Most people and and, like, this is why we talk about this is because sometimes even production, the, like, end of line test is just plug it into the actual thing and see if that works.

Brenden Duncombe
Press, like, a certain set of buttons and make sure those buttons work and then call it good. Right? So the first kind of the first foray into, like, at least we do a little bit of kind of this automated deployment is we're hooked up to the actual end product all the time. Like, we've got our hardware in the end product, and we're just, like, flashing it constantly or deploying to the device. You know, maybe it's over the air or something and just trying out the actual product, which which is, like, common because that's a good first step.

Brenden Duncombe
Most engineers already have that set up. Right? Like, if you're writing a bunch of code or, you know, like testing things, you've already got your computer hooked up to a device ready to program, usually with all of your debug stuff splayed out over your desk. Right? And so this is just the first step is just like clean that up a little bit, set it to the side, and keep it set up all the time so that you can just deploy constantly.

Stephen Kraig
I find this to be a little funny because at work, what what we're describing here is so opposite of what happens at my day job, mainly because we're designing equipment that takes a long time to build. And with our hardware, we'll design the hardware, and as we're getting it built, documents are being written on here's exactly how I'm going to test them. And and we're not talking about a document that takes an afternoon to write. We're talking about a document that might take 2 weeks of an engineer's time to write this. And it is this multi page document of here's how I'm gonna validate this.

Stephen Kraig
Here's how I'm gonna do this. And we go through these massive phases of I'm in design right now. I'm in document writing, or once the hardware actually hits I'm in full test mode. And it's interesting to kind of in a way what I'm hearing is there's like a smearing almost of all of those happening kind of continuously at the same time as opposed to these siloed actions that I'm used to.

Brenden Duncombe
Yeah. Yeah. No. I mean, it's very, like, agile, like this is a very agile thing, right? Like try and smear everything together, do these tight iterative loops.

Brenden Duncombe
I mean, and people give it all sorts of names. Right? Like, we're blurring a bunch of concepts here. We're talking about, like, CICD. We're talking about, like, unit testing.

Brenden Duncombe
And this is also kind of bleeding into, like, what is test driven development? And, like, why did that get really popular? When you talk about and test driven development, for people who don't know, is, like, you write the test first and and prove that it fails and then write code that makes it pass. Right? And that's kind of this, like, idea of, you know, then when you write your code, you already have your test cases done.

Brenden Duncombe
When you talk about test specs, I I think those are fascinating because there's actually, there's another concept called like behavior driven development, which there are some frameworks for too, that you write your test spec, like, with the behavior of the device, and it, like, auto generates the test cases, like, the code to actually test it. So you kind of write this human readable, like, when this happens, this should be the result, and then it runs that through there. But all of that work you put in right at the end to write this whole test spec, you know, which takes 2 weeks, like, that's effectively the English version of the code that you need to write to test everything. Right? It's a big bunch of cases of, like, under these conditions, with these inputs, you should get these results.

Brenden Duncombe
And one of my the main pieces of advice I give people, like, this is like the core of my rant about this, is like you put all this work in at the end. Right? At the very last step, you put all this work in, and the only people who ever get to benefit from it really are you when you validate the final prototype or the final, like, you know, go to production model and your manufacturer when they have some test that runs against that all the time. Right? Whereas the whole time you're building this thing, it would be great to be testing against that.

Brenden Duncombe
That every time you write a new firmware version, make sure it still passes that. Every time you make a new hardware revision, make sure it still passes that. You have to put in all this work, and you don't get the great benefit of it that you can have all the time. Now let's be clear. There are downsides of this.

Brenden Duncombe
Is it like, it takes a little bit more work front, so it does stretch the beginning. But overall, it speeds you up at the end. And at the end, you don't have this horrible crunch, which I'm I'm I don't want to put words in your mouth, but I'm sure you've experienced, where you have to rush to get the documentation out. You have to rush to build some sort of end of line test for production or, like, ads build some, like, tools that whoever, like, your QA testers are going to be in order to actually, like, interact with the thing. So you build all this stuff, and again, you just build it right at the end, and it's a mad rush to make sure it's good.

Brenden Duncombe
And my main like, the main topic that I like about CICD, like, test driven development, like this idea of, you know, kind of smearing it altogether is that you get the benefit of all that work over the entire part of your you know, the whole development life cycle.

Stephen Kraig
I just kind of foresee, at least with my experience, that I understand the benefits to it, but it seems like it would be pretty difficult to actually sell this in a way because there is that upfront overhead cost and being able to sell that to the bean counters who are just looking at time and saying, it doesn't look like you're making any progress on this. I don't know, do you have any, anything to speak on that?

Brenden Duncombe
Yeah. No. It is hard to sell. Right? And and like, sometimes it takes like a a successful cycle of this to prove that it's there.

Brenden Duncombe
Now, again, it's become really the default in software development. And and to be clear, they have really good tools for it, and they get to kind of sidestep the hard part of hardware, but it still takes a bunch of upfront time. Like, you know, when people are setting up these, like, big Kubernetes clusters and, like, their automatic deployment thing, like, that doesn't come for free. Like, they they have to build all this automatic deployment infrastructure, like their unit tests infrastructure, and they have to pay that price of, like, every time they go to push code, they need a certain amount of of unit test coverage and things like that. Right?

Brenden Duncombe
Like, they are paying for this all the time during the development cycle. Now in the hardware space, you know, you have the same thing. It's a little bit more expensive at times because you have these kind of custom pieces to interface with your hardware. But one of the things I often do, like I'm doing now, is I point to the software development world and say, like, look how nice it is that you see all these applications that don't just, like, push and crash all the time. It's like big news when someone pushes a bad piece of code, and it brings down a huge website.

Brenden Duncombe
Right? And that's because of all of this, like, live service kind of expectation where everyone is under the same standard of we need to push working code. And in order to do that, we have to use this infrastructure and test all the time to make that happen. That's not necessarily like a here's how you sell it playbook, but it is generally the speech that I use to try and convince people to do it.

Stephen Kraig
Yeah. Like you said, going through one cycle of it and demonstrating it would probably be the best ammunition for that.

Brenden Duncombe
Yeah. And, I mean, like, when you guys had Teenage Engineering on, right, this is, like, another good example. They mentioned their automated test infrastructure that they have. Right? They have, like, the standard for the test equipment they use end of line and for testing their production tools.

Brenden Duncombe
And so yeah. Like, you know, they saw it once probably, and then they know they're releasing a bunch of products and investing in this kind of infrastructure is super useful. And so, yeah, it's easy once you have it. Also, it's really easy once you build, like, the base infrastructure for everyone to go and, like, be, like, super excited about it. Right?

Brenden Duncombe
And just like, Hey, we're going to keep doing this for everything we do now. But that first hurdle can be difficult. But, you know, again, when you're talking about this investment, it doesn't have to be like one product line, right? Once you get your infrastructure up and running and you make a habit of this, your teams are ready to do this, you know, for other product lines that you build as well.

Stephen Kraig
Yeah. I I think with the teenage engineering guys, they were mentioning, you know, developing their own test equipment and things especially for, like, testing the the the force feedback on buttons and something like that is not an overnight task. That's that's a huge ordeal. So you sure. Once once you have it there, it's there for the future, but that's a huge investment.

Brenden Duncombe
Yeah. The problem is those like, you know, if that's a thing you wanna control, the secret is you were gonna build that thing anyway when you got to production, and you wanted to make sure that the fit and finish of your finished product is up to your standards. Right? So, like, that's why I say, like, bring it early. Because a lot of times, you know, when you write these test specs, like, if you write them early, you know exactly what you need to write to and the type of test fixtures you need to develop.

Brenden Duncombe
And, again, like, if you wanna control these things, the you end up doing it end of line of production anyway. And so, like, if you're gonna invest all that time, like, maybe just do it earlier so you can take advantage of it. And the whole time you're doing this and prototyping different hardware, you can keep putting all of this equipment through the thing that tests button pressure and be like, wait. This hardware, not as good. Like, this method, not as good.

Brenden Duncombe
Right? We need to get back to what we're doing before. So I focused a little bit on firmware, but this happens with hardware too. Like, especially hardware regression testing, I think, is a big thing that people really struggle with. Right?

Brenden Duncombe
Like, how do I know that this board design is better without making any mistakes than the previous one? Right? You have the feature that you actually did to upgrade. Right? There was some reason you did a board spin.

Brenden Duncombe
So you can test that, but then, like, how much effort does it take to go back through and test all of the other features that you'd already built in? And because that takes a lot of time when you're doing it manually and it's hard to keep track of, the result is is people try and make less prototypes. Right? They try and make less revisions because, you know, they're like, oh, well, we already know that this is, like, validated hardware. If we need to, you know, change a little thing, like, now we have to revalidate the whole hardware.

Brenden Duncombe
How many times have you heard that anywhere? Right? Like, this is validated hardware. If we change something, we have to revalidate it. So that test fixture that you were gonna build end of line to test all your boards and flash your firmware and do your serialization, if you do that earlier in the cycle, you know, I'm not saying there'll be no overhead to your test fixtures when you switch revisions.

Brenden Duncombe
Right? Like, your test points may move around or the board size may change. But, generally, if you've got all that hardware sitting there, every time you do a new revision, your regression test is put it back in the fixture and run it. Right? And make sure that all of those variables we set are the same.

Parker Dillmann
And it's not too big of a of a lift too to have to switch. Let's say you have, like, a a your tester and stuff, but you really would just have to change the ICT part, the the pogo pin carriers. And that's not a big lift in terms of if you have to make that change.

Brenden Duncombe
No. And if you get really slick with it, you can, like, auto build that. Right? You have CAD that says where your test points are. You can, you know, kind of auto generate all the, like, laser cut pieces that you need to do or CNC piece.

Brenden Duncombe
You know, you can generate all those design files and kind of hit print at the same time if you want. Yeah. You can tell I've been, like, practicing this speech about like, hey, this is a thing I try and sell all the time because before when I was in product development, it was useful. But now what I see a lot is people get to us in manufacturing and they're like, okay, great. We're here.

Brenden Duncombe
We've got something about it. How do we test it? End of line. Or like, how should we do our serialization? And, you know, the answer is, like, hey, I want you to invest in all this infrastructure.

Brenden Duncombe
And, you know, trying to get the return on investment on that is, like, well, do it earlier and you'll have a lot more return on investment for it during your development.

Stephen Kraig
Yeah. I was I was about to say, coming to your contract manufacturer and asking them how to test your product, you're way too late if if if that's when you're asking that question.

Brenden Duncombe
It's so common, especially in the startup space. I'm not I'm not like trying to like paint with a broad brush on all startups here. But, you know, you, you oftentimes you're so focused on getting these validated prototypes and, you know, right product market fit and hardware in your beta tester group and like really showing that like your idea can be there, that at the end of it, you know, you've gotten some rough pricing just to validate your your cost, and then you go to you select a contract manufacturer, and you're like, okay, great. Now hit print and build me 10,000 of these. And you maybe got some good DFM feedback along the way, which is like a huge step, but there's all of these little details that a lot of people forget, especially if you haven't been through it before or even if you've been through it, if your organization hasn't been through it for before or not in this product category of, like, you know, all of a sudden you're doing connected device.

Brenden Duncombe
Okay. Well, like, how do we get keys to it, and, like, should it connect before like, to the should connect it to WiFi before we ship it out, or or should we connect it to cellular before we ship it out? And, like, what do we do with the batteries? You know, every time we test it, we just put the battery in, we ship it to people, but, like, what if it sits on a shelf for 3 months? You know, there's all of these, like, little details that come up when you get into manufacturing.

Brenden Duncombe
You know, even down to, like, like, how am I gonna get the number that's in my chip for my serial number onto the outside of the box so that I can skip right? There's just, you know, organizations who practice this a lot. Right? They're like, yeah, this is part of our thing. That's stage 6.

Brenden Duncombe
Right? We do that as part of the checklist. But organizations who don't do this a lot are in a different category, like, you know, they were doing analog stuff before, and now they've got things that, you know, connect to the cloud. It's easy, you know, to not think of this until someone pushes back about, like, what do you want me to do for this? Right?

Brenden Duncombe
How do you want me to validate your radio? Things like that.

Stephen Kraig
Yeah. I could say having worked at contract manufacturers, it's always refreshing when, a client approaches you and says, hey. I have x y z tester already done, and here's my test suite. How do I integrate it into your process? As opposed to, I need to test my thing.

Stephen Kraig
How do I do it?

Brenden Duncombe
Mhmm. And and the other response is, like, you have stuff that can test this. Right?

Stephen Kraig
Right. Right.

Brenden Duncombe
You know? Like like, you you guys have a default thing that you use, so you're just gonna you're gonna put it in there and test it.

Parker Dillmann
My favorite thing.

Brenden Duncombe
Yeah. Yeah.

Parker Dillmann
When they expect Well well, when they know that you know how their product works. That's my favorite thing.

Brenden Duncombe
Mhmm.

Stephen Kraig
I I sometimes your contract manufacturer knows your product better than you do.

Parker Dillmann
There is a couple at MacroFab that, yes, I do.

Brenden Duncombe
Well, especially if you're developing this stuff. Right? Like, if you're the one like, you know, let's say you really are asking a lot of your contract manufacturer and they're developing the test set for you. Yeah. I mean, there's a knowing the ins and out of our products is definitely when you're writing all of the test cases and understanding the behavior of that.

Brenden Duncombe
Like that, that gives you a pretty good map of what a product is like, Especially when you see things like, you know, you see this all the time when you go to a manufacturer who's been making a product for a long time, like they'll see an error on a test result thing, and they immediately know, like, oh, I need to do this, this, and this, right? Because they just have this experience with your product that you know, sometimes you don't get to see unless you, you know, visit the factory regularly or check it out because they just get in the groove of they're trying to push out product and, you know, they know your they know they know how to fix it when it goes wrong or or things to look for. And so, yeah, they definitely get to know it really well.

Parker Dillmann
So what would be the ideal hardware development cycle then?

Brenden Duncombe
Yeah. Great question. I think it depends a little bit on the type of products you have. But generally, I like to have the you know, my ideal is you have your hardware and your and kind of your bed of nails or whatever your simulator is, right? Like kind of your hardware in the loop situation, whatever that is, you build that early and you build your test set for that early.

Brenden Duncombe
Right? You have that set up through your whole product development life cycle. Like, that that's key because then when you get to the end, it's all easy. You just take that thing. You take it to your contract manufacturer.

Brenden Duncombe
You strip out some of the tests because it's way too exhaustive, and you don't need all that in your cycle time. And then then you go there. And on the firmware side, you are taking advantage of that same exact hardware setup to develop your firmware, right? Your firmware has unit tests. Your firmware has integration tests.

Brenden Duncombe
Your firmware gets flashed into your big hardware in the loop setup, and it gets exercised too. So that's definitely like in an ideal world where money and time are no object, right, like, this is the type of thing you would do. Is it realistic for every project? Probably not. Right?

Brenden Duncombe
But it can be, it can be kind of the ideal of what you wanna do. Yeah.

Parker Dillmann
Because I'm imagining this is would be super useful for when you kinda hand off the board design, so to speak, to like a firmware developer. And they program it, but they go and they have to hand it, like, let's say, back to you to make sure it's working right. You know, like, if the motor driver is outputting the sine wave correctly.

Brenden Duncombe
Mhmm. Mhmm.

Parker Dillmann
You know? If they just had a fixture that had the board in it that was hooked up to, an oscilloscope that you would it would just automatically tell you if it the sine wave was right or not. And so they could just keep tweaking their code to make that wave work better.

Brenden Duncombe
Yeah. If you've ever worked with a connected product, you've probably built that thing. Right? Especially, like, if you have, like, a big software development team. If it's, like, a connected app or, like, a web application that connects to a device, like, the first thing that'll start happening is all of a sudden, you know, the pure software developers, like, I need I need hardware to test with.

Brenden Duncombe
I need something to test my code against. Right? And, like, you end up building these little, like, black box simulators or some things that you can, like, fuzz some switches on to put it in different modes. You end up building that thing, but, you know, the real difference maker is you build that thing with intention versus you build that thing at the last minute to just meet some requirements that the software development team is looking for. Like, oh, I need to test these 2 cases.

Brenden Duncombe
But if you build that thing to be, like, extensible, you know, you have a huge advantage when it comes to, like, trying to find weird edge cases in your code or or moving faster on additional, like, hardware and software iterations.

Stephen Kraig
Well and one other thing that I know seems obvious, but in practice usually gets missed, build 2 of them, at least 2. Don't just build 1 and that be the one that you send to your contract manufacturer.

Brenden Duncombe
You wanna have one on your desk too. Yes. Right? So that when they say there's an issue with it, you can, like, try and reproduce it. There's also this idea of in agile, there's kind of this idea of, like, if it hurts to do it, you know, you need to be like, keep doing it until it's not as painful.

Brenden Duncombe
Right? Like, if you have a process that like you dread doing, like over there updates or something like that, you need to like, get good at that until it doesn't hurt. And you're not afraid of it, right? Like this is part of your tool the unit your tool belt now. And so this same idea, like, if you only build one of those testers or simulators or whatever you wanna call it, whatever this fixture is, if you wanna build one of those, you know, like, if anyone ever asks you for a second one or to make updates to it, like, that's gonna hurt.

Brenden Duncombe
You need to get you know, you have to practice at that and make that a muscle that you can exercise at any time.

Stephen Kraig
Well, and a little bit further, Build 2, document them, and build them the same. Don't make mods to one of them. Don't or or if you do have to make mods, make them at the same time to both of them and document what you just like I said, this sounds obvious, but I'm sure all of us have been in the situation where we're like, I'll just do this real quick. And then 6 months down in the road, what the hell did I do? You know, and there's one other pitfall that I've run into as well.

Stephen Kraig
It's at the beginning, dreaming up this amazing automated test rig that does everything and exercises every little piece of your product, and and it's just magical, and it ends up that if you really look at it, the test rig becomes more of the design project than the product itself, and you can easily paint yourself into a corner with that. So being realistic with what you actually need to test and what that tester needs to do, I think is critical.

Brenden Duncombe
Yeah. Yeah. Absolutely. I mean, you you wanted to you don't want to invest more in your tester than you do in your product. Right?

Brenden Duncombe
There's a fine balance there.

Parker Dillmann
Yeah. I've definitely been down that road where tester ended up actually taking longer to develop than the actual hardware did.

Brenden Duncombe
Oh, yeah. Yeah. Definitely. Definitely.

Parker Dillmann
I guess it depends on your use case though, because if you're designing a lot of products, like, let's go back to teenage engineering for the example. A lot of their products are kinda the same form factor. They have analog inputs, analog outputs, and they press a lot of buttons. So designing something they are. That's all yeah.

Parker Dillmann
Exactly. Designing a tester that can just handle that really well is probably, you know, worth their effort to do.

Brenden Duncombe
Oh, yeah. Definitely. Like, especially if you're in the same product category, you definitely can run into problems where like you have like complimentary products that are like across product categories and your testers just don't apply at all. That can definitely make it more difficult to want to build up this infrastructure when you can't really spread it across your product portfolio. But yeah, I mean, there's just like I said, in general, my recommendation is, you know, we talked about like this big end goal of all of this, right?

Brenden Duncombe
There's a lot to be gained from just like a little bit of this, right? I will settle for people just, hey, we use unit tests in our firmware now. Like that's a really big win, right? Like, we should start little steps, get good at the little things. You don't have to go swinging for the fences, build all this stuff.

Brenden Duncombe
It's great to get there, but a little bit of this will will give big rewards.

Parker Dillmann
I think one thing is to also look at, let's say, microcontrollers, for example, is look at if you're picking your next microcontroller, maybe look at companies that have built in the ability to do unit tests, like simulation, we were talking about earlier. Being able to run your code in your IDE and simulate what the inputs and outputs of the chip should be. So, like, you can, let's say, simulate the I square c bus. So you're like, oh, it actually outputs the addresses correctly and does all that stuff, DMAs, that kind of

Brenden Duncombe
Yeah. Yeah. I mean, and and not just that, but, like, when you look at what you're gonna run for your firmware, right, if you're gonna do, like, an RTOS or something or any other framework, it does not necessarily an RTOS, but if you're gonna pick a framework, maybe look for something that already has, like, a unit testing structure built in or some CICD hooks already built in, you know, like tool chains that can, like, run-in a Docker container, things like that. It can make your life a lot easier if you wanna go down that route.

Stephen Kraig
I could see a lot of hardware not being very compatible with this system. However, perhaps there's a kind of a, compromise. Give your firmware guys dev boards. And as you're doing the electrical engineering hardware design, they can be doing all of these unit tests on a dev board level such that a lot of the things are proven by the time that actual hardware hits their bench, and then you can go into the further the the deeper level stuff of the actual hardware testing. Don't think there's a lot to be had with parallel work there.

Brenden Duncombe
Yeah. Absolutely. I mean, yeah, you wanna be doing stuff on dev kits really early anyway. I I completely agree with that. You You should have a lot of dev kits.

Brenden Duncombe
You should have a lot of dev kits hooked up to breadboards. Right? You know, when you're trying to pick out what you need to do. Definitely agree. Firmware development should happen there.

Brenden Duncombe
A lot of times, you know, we were talking about this, like, on device test versus a host test. Right? A lot of times that is a dev kit. Right? If you don't need any, like, external peripherals from the microcontroller itself, like, you just need the peripherals in the microcontroller, dev kit's perfect for that.

Brenden Duncombe
Right? Like absolutely, that should be your like first stop of like, hey, we just part of our CICDs, we have like a computer over here that when it sees a new thing, it runs it on the dev kit and, like, you know, runs the test from around the dev kit and makes sure that it's still good.

Parker Dillmann
Make sure that heartbeat LED still blinks.

Brenden Duncombe
Yep. Or or, you know, like your watchdog timer, things like that. Right? Just like all the basic stuff that you wanna make sure doesn't break. It it can also help you do things like, you know, you wanna test counter rollovers, things like that.

Brenden Duncombe
Like, all of that stuff, you can do. You can actually turn the hardware on and, like, initialize it really close to the end of the counter and, like, see what happens and test that all the time. Right?

Stephen Kraig
Yeah. I see a lot of value in that because there's certainly the situation where, you know, an electrical engineer has designed their entire board, or a hardware engineer, sorry, and they're ready to pass it off to their firmware guy and they're ready to just get going with, let me get a probe on here and let's start, you know, changing outputs and doing all these things, and if you haven't already had all that bring up code and you haven't had time to test things like the watchdog timer and whatnot, there's a lot of downtime while firmware gets spun up with all of that other testing.

Brenden Duncombe
Yeah. And I I typically see the dev kit start as the first step anyway, because a lot of times you wanna, like, when you're trying to select a microcontroller, you wanna look at what you actually need. You know? Like, you're running you know, if you're running RTOS or something, maybe you're testing out a couple different versions of microcontroller, like, which peripherals really work for you. So sometimes I think that, you know, at that early stage, yeah, like, you're doing that bring up code and test code just to help you make your selection sometimes.

Parker Dillmann
One of the most important things with that is, let's say you're running a lot of external peripherals is you don't need the external peripherals a lot of times, but to make sure that your microcontroller has enough interrupts, has all this all the architecture that you actually need built in to do what it needs to do. Yeah. Older Arduino platforms are really bad at that because they don't have, like, a DMA setup, like, old Arduino Mega, basically. And you run into this a lot where you wanna start running, like, let's say, a lot of LEDs, addressable LEDs. And you're just, like, well, when it's putting out LED data, it can't use the serial bus.

Parker Dillmann
And it's fine when you, like, you

Brenden Duncombe
Yeah. That's that's a huge one. Right?

Parker Dillmann
That's like all the time

Brenden Duncombe
you run into it. It's like it has the peripherals, but, like, you didn't realize when you're reading the data sheet that it's like, if this peripheral is on, you don't get all these other ones. Right? That happens on all of them.

Parker Dillmann
And so writing all that all that code and then being able to test it and see if, hey, can we actually run what this product needs to do at the, controller level is, super super important. So, like, that's kinda like taking your specification doc and then writing a test doc on top of that. And then writing the code to make that work.

Brenden Duncombe
Exactly what we're just talking about. Right? Like, do your test stuff first. You know what you need. And all that code that was testing out those peripherals, right, like, hey.

Brenden Duncombe
That is now the basis for your test firmware where you need to make sure like, say, you've got, you know, like a CAN bus or something. You wanna make sure it's good. Like, you just run arbitrary data through it at, like, max speed, you know, as part of your underlying test or your regression testing just to make sure that when you do new hardware revisions, it's like that. You know, like, again, a lot of these things you can, like, build off of each other. And so keeping that around and also treating that early code as a first class citizen.

Brenden Duncombe
Right? Like, the it doesn't do the, like, hey, I just got it in one giant file problem. You know, you actually put it into GitHub, and you're you're actually, you know, tracking it because you're gonna use it later. Yeah. If that one giant file line hits too close to home for you out there, like, that's a good place to start.

Brenden Duncombe
Right? It's it's hard to do unit testing when it's in one big file. It's hard to do a lot of this stuff when it's not broken out there. So that also can be, like, a great place to get started.

Stephen Kraig
Lots of organization.

Parker Dillmann
Yeah. Say they were, like, setting up the automation for, like let's say you have, like, API keys and you're using GitHub, a lot of times you don't want to put those keys in there, especially if multiple people are using the GitHub or it's an open source project. Setting up the automation to make sure you don't accidentally publish those keys. That's a very good step.

Brenden Duncombe
Yeah. Or, like, the environment variables that you need to set up to grab them locally. Right? Like, actually actually making the config file that it grabs those from as part of, like, a programmable thing. That is yeah.

Brenden Duncombe
All of that is great stuff to practice at and helps you later too because now you have a great way of, like, you know how to add keys. Right? That that that you need to do that manufacturing. You gotta give each of your devices a new key. So how do you do that?

Brenden Duncombe
You've been practicing.

Parker Dillmann
So if we have, nothing else to talk about with CICD, you know, I kinda wanna talk about future stuff you're working on at, Brent, Brendan at MacroFab.

Brenden Duncombe
Yeah. Future stuff. There's a lot of interesting future projects that we're doing for sure. You know, in the thread of this kind of automation thing, I think there's a lot of automation stuff that I'm interested in looking at here at MacroFab. You know, the same way that we you know, there there's also this you know, we just talked about investing in these tool chains or, like, these fixtures that you use.

Brenden Duncombe
Well, we have maybe more reason to have standard fixtures because we do a lot more testing than any given customer does. So, like, what kind of tools can we build to help designing or making test pictures for our customers easier? Right? Because, like, you know, we may only build you may only build one product with us, but we build pen that can maybe leverage a similar set of tests. And so how do we kind of do that is a really interesting challenge.

Stephen Kraig
I could totally see Magofab taking that to the next level and saying, how do we leverage that as the product? In other words, how do you put that on the website as something that someone can manage and input? Yeah.

Brenden Duncombe
I don't know if we're quite at, like, make it part of the product, but we're definitely in the early stages of, like, hey. I don't wanna repeat this work a 1000000 times. Like, let's actually get something as a basis that we can use.

Stephen Kraig
I'm I'm sure there is some future in that, but that would be pretty cool if I can manage my bill of materials. I can manage my product. I can manage box build, and I can manage all the testing, including fixturing all in one location. Now I understand that's asking for a ton there. That is that is gigantic.

Brenden Duncombe
Yeah. Thinking about your fixturing that way is great. Right? Like, is your product really just your EBOM, your MBOM, like your assembly altogether and your PCB? Or is it, like, all the stuff that also goes into building your product?

Brenden Duncombe
Right? People see it all the time with, like, hey, we've got the versions we need for, like, our PCBA and our assemblies, but, like, less about, like, hey, what's the version of the test we're running? Right? And a lot comes into that when you start thinking about it that way. You start thinking like, hey, I maybe now when I record my test results, I need to say what version of my test firmware am I running, and what version of my test hardware am I running on, and, you know, which of the versions of the stuff that I'm building are compatible with each other?

Brenden Duncombe
And can I roll that over as I make upgrades to my product? Are they still compatible? And so if you think of, like, kind of your whole product as not just, you know, your big sort of tree of all of your assemblies and your bombs and the version subassemblies, but if you also start thinking about how that is the case for the fixtures you use, the tooling that you use to build those products. Right? And I don't even mean just, like, the ones that you see, but think about the ones that you use on the manufacturing floor too.

Brenden Duncombe
You've got, like, your carriers and your stencils, and you've got you know, when you're doing product assembly, you've got all of these, like, fixtures that help put it together. Right? Like, understanding and keeping that in your mind, like, hey, we'll be able to reuse those. Are we gonna incur costs when we make these changes? Like, you know, it is the collection of not just the thing, but also all the stuff to build the thing.

Brenden Duncombe
Right? And so keeping it kind of that way, if you if you kind of, like, treat some of your fixturing and tooling and your production as part of your product, it makes it easier to justify some of this infrastructure build that you need to do. Because the reality is, like, it is part of your product. Right? You know, a lot of things struggle when you get to production.

Brenden Duncombe
Right? Like, it's not uncommon to just get stuck. Like, you can you can build 5,000, and when you have everyone, like, hand touching and stuff, when you really gotta, like, speed up and scale, you you can run into these things. Right? And so it absolutely is part of your product.

Stephen Kraig
It can be difficult to, once again, to sell that though. Mhmm. Just because the the overhead cost of design time for something that doesn't appear to directly impact profit is, I don't know. Like, just being very directed with this from the beginning makes the most sense, like we've been saying the entire time, just so that you don't get to the very end of the project and then have to justify all of these costs.

Brenden Duncombe
Yeah. Absolutely. And like I said, this is a reason why this is kind of like a future product where, like, thing I'm working on at MacroFab is, like, again, it's a lot easier to justify this stuff if, you know, like, the base infrastructure of, like, you know I mean, it's always easy to, like, generate ICT fixtures and things like that. But, like, the base of, like, hey. We have a pretty good platform to build the functional stuff off of as well.

Brenden Duncombe
Like, it's a lot easier to joining us today on the, or, I guess, tonight. But we record at night.

Parker Dillmann
Joining us today on the, or I guess tonight.

Brenden Duncombe
We'll be

Parker Dillmann
record at night. I would say day, but it is night. Join us tonight on the podcast. Brendan, where can people hit you up and ask you questions about hardware as, you know, continually developing?

Brenden Duncombe
It may be kind of, I don't know, basic, but I respond to people on LinkedIn. Like, that's a good spot to find me. You'll see me around on some of the MacroFab stuff. And if you ever come to, like, one of our meetups or events, please don't hesitate to reach out and ask questions.

Stephen Kraig
Well, thank you so much for coming on, Brendan.

Brenden Duncombe
Yeah. Thank you guys for having me. It was a blast.

Parker Dillmann
So please visit macfab.com. We'll also have links to Brendan's LinkedIn and that kind of stuff in the show notes, and, thank you for listening to circuit break. We were your hosts, Parker Dohman and Steven Craig. Later, everyone. Take it easy.

Parker Dillmann
Thank you. Yes. You are listener for downloading our podcast. Tell your friends and coworkers about Circuit Break, the podcast from Macrofab. If you have a cool idea, project, or topic you want us to discuss, let us know or in the community.

Parker Dillmann
Our community is atform.macrad.com where you can find personal projects, discussions about the podcast, engineering topics, and maybe Brandon will be there. So, yeah, form.macrofab. Maybe.