What You Are Designed To Do - A Joe Grand Manufacturing Adventure

The Mack Feb injury podcast design contest sponsored by Mouser electronics is been extended the topic is useless machines. So we have extended the deadline from August 10 to August 31. And this is closing fast. We have cash prizes up to $1,000 for winners. More information can be found at macro fab.com/blog

Welcome to the macro fab engineering podcast. I am your guest Joe Grande. And we are

we are your hosts Steven. This is episode 185

which means it's been 112 episodes since I last interrupted your intro. Yes,

thank you Joe Grande.

So our guest this week is Joe grande also known as kingpin a computer engineer, hardware hacker, product designer, teacher,

advisor runner daddy, honorary doctor, TV host member of legendary hacker group, loft Heavy Industries and proprietor of grand idea studio. Yeah, he has been creating, exploring and manipulating electronic systems since the 1980s. Joe Grand has been on the previous McAfee web engineering podcast number 73.

And I have the t shirt to prove it.

I'll still got that. Awesome.

So thank you, Joe, for coming onto our podcast again. It's been two years I think at this

point. Crazy. You guys you guys have done a lot since then. So yeah, thanks for having me back. It's it's always fun. I'm still, you know, I'm still delirious from the 11 days in Vegas from DEF CON. So ignore my my, I guess excuse my bad puns and, you know, rambling sentences.

That's what people download this podcast for? Yeah, exactly. Right. They

need something to do during a long car ride.

Well, and I guess I guess we're not letting you off the hook yet. Because I guess some of the conversation we're gonna have is about DEF CON, your your involvement there?

Well, we might as well do that while it's still in my head.

Yes. So today, we'll have Joe Grand on the podcast. So we're going to discuss offshore vers onshore manufacturing, how it pertains to DEF CON badges, tariffs and Chinese customs. And just the DEF CON badge design.

Sure all the fun stuff.

Or painful stuff?

Yep. Yep, that's it. I guess it depends on which side you're on.

Yes, yes. So I think we should just jump right into the meat of it. onshore versus offshore manufacturing. So why heard was the DC 27 badge was made in America, at least the PCB was,

yeah, so actually, so the fabric assembly, I should know, the assembly and final assembly was done in a factory in the US about 30 minutes from my house in Hillsboro the PC board fab was still done in China, they were the only factory we could find that could manufacture these crazy, you know, four layer via in pad, ridiculously small feature size boards within the timeframe that we needed, which was basically 30,000 in less than six weeks, with 100%, electrical test and stuff. So we were kind of limited in our options for that, because the timeframes were short, but there was definitely a conscious choice to stay in the US as much as possible for this project, especially given how you know my previous time of working on DEF CON badges back in the day from 14 through 18. And with a lot of the other production manufacturing I've done is typically been in China. And you know, if anyone was around for those projects, or even some of the more recent ones, there's a lot of these unseen, you know, kind of hidden costs that nobody wishes happens, but always seemed to happen. So this was a choice when, when Dark Tangent was like let's let's just manufacture in the US to avoid the customs issues to avoid the communication issues. Especially with this Chinese trade war, we were still getting charged a lot of duties to bring components in because I was doing all of the parts sourcing myself so I could have control of supply chain. We paid a lot for that. But on the flip side for DEF CON China, which we should also talk about, that one was manufactured 100% In China, and that was mostly because the sponsors of DEF CON China, you know, wanted to use as much Chinese resource as possible, where the US doesn't have sponsors and DT sort of controls, you know, those decisions. So yeah, there's definitely some trade offs there.

So let's wind back a little bit is Dark Tangent and DT who is

that? Yeah, so who SO SO THE DARK TANGENT, also known as dt, also known as Jeff Moss is the founder of DEF CON. And also the founder of Blackhat, which is sort of the, you know, commercial InfoSec conference that happens right before DEF CON. And he was the he's been kind of the the guiding Late for a lot of stuff that I've done over the years, we've known each other for a really long time. Since I don't I mean way early days of black hat or DEF CON, and I hadn't even gone to those conferences when we first met. But it was his idea. You know, I've been giving a lot of talks at BlackHat, we might have talked a bit about this on the other podcast, where he had been giving a lot of talks about, you know, embedded security and hardware hacking back in the day. And he sort of saw that people were getting more and more interested in that. And he's like, Hey, why don't you put together a hardware hacking training for Blackhat, which was at the time he was running that and I'm like, Really, people want to have like a two day class on hacking hardware. And, and he was right, so I put this class together. And I'm still teaching it 14 years later, with some variations, but like, the classes of attacks haven't really changed. So he sort of had that vision. for that class, I had made a G shaped circuit board, which is in the shape of my logo. And we use that for soldering exercises, D soldering, reverse engineering, the board has a security mechanism in it. So the goal was to like break the security of the board. And that's kind of the final challenge for the class. So he saw that board and said, Hey, we should do something like that for DEF CON, just kind of out of the blue, you know, because he has these amazing ideas. And, again, he just sort of saw that people would be interested in that he knew. And so the next year, we made this simple badge for DEF CON. And that, you know, kind of kick things off as far as far as badge life. But this year, I think, because of the troubles that he had been having with badges over the past number of years, he just wanted to keep things closer in hand, you know, and like, I could drive 30 minutes and I went to the factory, bring it back to the roots. Yeah, well bring it back to being able to be in control of it. And I could drive 30 minutes, and I did you know, four or five times I met with the met with the factory, before we even started, I gave laid out the prototype and what I had in mind to do, and gotten their feedback as early as possible. Because that's something you know, a lot of times people will put together design and just send it out to be fab. Like you guys probably see this all the time, like you get you get a design that's already production worthy, in quotes, my air quotes on a on a podcast. So you know, the designer thinks it's finished, they send it to the FAB, that gets manufactured, assembled. But for something with up to 30,000, like we made 28,600. And I wanted to get the factory involved as early as possible. So they could say, well, we recommend, you know, we can handle up to this size component. And we recommend you do things this way. And they really needed to be part of that process. So we could figure out the best way that they could manufacture in the most efficient way. So we would have the best yields, and that they would actually get it done in time. So you know, I took their feedback right from the start. And they did circuit board review, and not really schematic review, but mostly circuit board review and in parts kind of supply chain review to make sure that the parts were coming in at the right times. And of course, they helped panelized the board. So I gave them a one up design of a single badge. And they penalize that based on their requirements for their machines, adding the fiducials outside of that they worked with the circuit board fab to deal with all that low level stuff, which sometimes I do, but I but I normally don't. For high volume stuff. I don't want to mess with that. Because it's sort of a waste if I do it.

Yeah, each one each fabricator has their own method of doing that's

right. Yeah. So, you know, there's no point in trying to do that, and then have to recreate the wheel. So I think, you know, getting involved with the, with the factory as early as possible is, is the way to go. And doing that in the US is just way easier, you know, to, to sit down and show them and say here's the stuff and to get a tour of their facility and say Hey, so, you know, this is a non standard process, I have the circuit board. And then I have a handcut quartz gem crystal that needs to be affixed to the to the to the board. So it's not their typical, you know, kind of box build, where they they assemble boards and put together into a box and they ship it. But they had a separate room available for sort of these weird requests. And I can see that the first day I met with them. And this is part of the reason I chose this factory versus other ones in the area where they were a smaller company but very nimble because they said oh, here's where we'll put the the double sided stickers onto the gemstones. And then here's what we'll do the final assembly of the gemstone onto the circuit board. And it was awesome. So it was really a great learning experience for me. The other thing too is I needed to make sure that they understood the importance of the project that it had to be done by DEF CON. You know, a lot of times, deadlines will slip and product release dates will get pushed back and everything but like we don't have that luxury with DEF CON. And which is part of the reason why I'm glad you're not doing video. I definitely got a lot of gray hair over the past few weeks of trying to push this things through. But they you know, they fully understood and they were fully committed and fully determined to get everything done. And we actually got badges a week before DEF CON, which is like mind boggling. That hasn't happened since DEF CON 14.

Was that was that on schedule.

That was actually a little bit ahead of schedule because I had told them you know Worst case, let's get it. Let's get the Inhumans in over the weekend before DEF CON, so the week before, you know, absolute worst case, things can arrive Wednesday, because real registration starts Thursday. And that would mean that the DEF CON goons, the organizers would have to be up all night long prepping things. And this factory just knew the importance of it and just cranked and they actually hired part of the the additional cost, we did go a little bit over budget, even though the budget a lot of times is just sort of this random number grabbed from somewhere that sounds reasonable. We went over budget because we had the, we had to pay quite a bit over six digits in expedite fees, to make sure that we could, you know, hit the schedule, and that's not insignificant, and it hurts, you know, to say that because it's like, I'm spending def cons money. And I want to be very careful with with how I use it. But on the other hand, you know, engineering is is what it is. And when you have to expedite to get stuff done, you have this kind of rolling ball. And if you if you don't get your first stage done, you can't do your second stage. And then everything gets put

in that first, you know, section, your Gantt Chart slides down a week, everything has to cascade,

right. And that's what happened especially because we were doing DEF CON China at the same time. And so there's a lot of overlap with with some of the work. But then eventually it was this mad rush of DEF CON us and the factory, which I thought was really cool is they sort of knew their capabilities, like like they're they're small, they have, you know, some main engineers, and they have some staff working under them. But they knew from what I sat down and said we need 20 28,000 of these things that they were going to hire, they needed to hire extra staff. So they hired about 20 Extra temp employees sort of on a as needed basis. So a few of them came in at the beginning when when things were slow, like while we were waiting for boards to come in and they would prep the gyms and things like that. And then as stuff started ramping up, they'd bring more and more people in to do the deep analyzing the testing. And it was actually kind of cool, because sort of a side effect of that, besides it costing money is we had a bunch of you know, high school and college temp workers who were interested in engineering, like they got to work at a factory to kind of learn this process and see a pretty wild, unique product being built. And I got to sit down and like talk with them while they're working or while they're on break. And just explain what they were doing. And like why this was cool. And why this was important. And some of them were like, Well, where do I Where do I even look to learn more about engineering, you know, so I gave out some some resources. And that was kind of cool. So it's almost like by by requiring more people for the project, it ended up inspiring more people, you know, for future like who knows what they're gonna end up doing. But it definitely was a crazy, crazy time. And I'm glad it's over.

Well, I think it's kind of, I think we should take one quick step back, because we've been talking about like courts and gyms and things like that you're probably sick of talking about this. But for for everyone who doesn't know, the DEF CON badge is a PCB that's affixed to a chunk of quartz, right?

Correct. Yeah. So so the circuit board. Actually, I'll even take it a step further back, the theme of this year, DEF CON has started having themes for their conferences, and you know, a lot of the the previous themes they've done for the past few years have been very dark, like, you know, big brother see, yeah, big brother's watching, which is sort of how technology is right? This very dark kind of big companies controlling your data and all this stuff. So the theme this year was called technologies promise, which is really like what, what if technology was just used to actually help you and make you a better person, not control you. And when I heard that, and when, when, when DT showed me his first kind of inspirational image that he found, which was this advertisement from the 70s or 80s, of a woman sitting on like a lotus flower holding a laptop. And there's like clouds and pastel colors. And like all this great stuff. I'm like, This is amazing, like this is what we need is like happy, fun technology. And at the time, I'd been going to like the New Age gem store with my kids and like picking different types of gems, just for fun. But that image, like to me seemed perfect of like, we need some soft element to this project, like some not just a hard circuit board, like everyone's been doing for the past however many years now, but some other element to it to just sort of soften it. And it was just sort of like, let's use a gym with the circuit. We didn't know what that would entail. And that whole process, which I talked about in my DEF CON, badge talk, which was the opening ceremony track that was recorded and will eventually be online as far as I know. And all the slides by the way and like design documents are on my website, which you can just search for DEF CON 27 badge and you'll see it but it was you know, I didn't know what that would entail but it just seemed like a good idea. So yeah, there's a 55 Actually let me measure it. It's been so long.

So let me get this straight like your you pick the gym on just almost on a whim. It feels like though Yeah, it wasn't like it wasn't like a a committee so to speak. You're like I want to Here's a gem and he went Cool.

Well, no, I mean, Jeff and I were sitting around at a coffee shop talking about this theme. And it just, I just really want to use the gym, I didn't know what I didn't even know what that meant, like, you know, all the gems you get in stores are either are either raw pieces of rough gem, or they're shaped into a heart or something like that. I was like, well, we could get get some gems and like, drill into them and put the circuitry in it. But we had no idea what that what that was. We eventually settled on on the 60 millimeter discs with five five millimeters thick, that would sit on top of the circuit board, and I base the gemstone size on what I thought would be a cool size for the circuit board. So there was some back and forth with me of building a paper prototype to see what would be a cool size and not an overbearing piece of jewelry that you're wearing. And it wasn't until like months later, where I actually went to the Tucson gem show and figured out, you know, found a vendor that I needed through meeting a bunch of people and then pointing me in the right directions. And I had 36 hours to go from knowing nothing to finding a person or a company that could support our project. And my mind blowing, but yeah, so we settled on Brazilian courts. And we have different colors that are dyed for the different badge types. Because you know, a lot of times conferences will have different colors for attendee types. DEF CON has 10 Different attendee types. And we dyed the gemstones to match the circuit boards, which match the attendee types for for five of them. So we have 10 Different colored circuit boards and five different color gemstones, because we couldn't dye the gemstones in all of those varieties of colors. But we have red, orange, no, red, yellow, orange, green, blue, purple, and then natural. Are the colors I think. So it was pretty crazy. Yeah, and, you know, just another element and, and it turned out fine. But it could have turned out horribly, you know, and that's sort of the risk a lot of times, and those also, by the way, were manufactured in China. The way I found this factory was just through a series of people that I met at the show, the Tucson gem show, and I had talked to people who own mines in different places. And they're like, yeah, we can get the material but we don't have the manpower, or woman power to create, to physically create the stuff from the, from the stone. And then one guy I talked to was like, You got to talk to this guy at this place. Tell him I sent you. And you have, you know, they close in half an hour at this venue, down, down down through the city. So I hop in the car, get over there, meet the people I'm supposed to meet. Tell them about this ridiculous project, which everybody I talked to in the gym, you know, world, they don't normally see electronics, and they don't see a guy like me running around trying to try to buy 30,000 of something, you know, it was like, just like, I was sort of stood out because of that, which helped because I said I'm building something for hacker conference. They're like, What are you even doing here? And so I found these, I found this company, and they ended up a few weeks later making samples for me. And that's when I started to kind of be able to coalesce, you know, the design and make it something that started to feel more real.

So I got a quick question, how much does 30,000 of these things weigh? Like, what's the tariff? Like? Report on that?

I'm trying to think of the why No, I have it in email. It's a shitload of weight. And I swear,

when it comes to something that's big, yes. Yeah,

it's an ass load. It's huge. Weight. Okay, well, let's see, we had 107 boxes. And I think each box was 35 or 40 pounds, because there's a limit. Like they didn't want to make the box too big. And they need to be small enough. There's some sort of one person can pick it up. Yeah, so I think let's use a manlift. So 140 107 boxes, say let's, let's say 40 pounds, just to be safe. So that's 4280 pounds. So two tonnes of handcut. Stone

rack, well, that's postprocess. Right? That's that process. That's not the raw or

no, the raw material was bigger. And actually, I went to visit the factory because I was in China for DEF CON China, and then went into Shenzhen to visit this factory. And they had huge amount, I think it was like they said 20 tons, but I'm not sure if the if the translation was correct. But I saw giant bags full of the cuts, you know, the extra pieces, and they're like, can you please ask your customer if they want to do anything with these leftover pieces, because they can't do anything with them. You know, it's like the corners and the pieces that broke and stuff like that. So in one way, you know, I sort of feel bad because we have all this leftover material, kind of that we you know, wasted. But then on the other hand, I think like the world is some giant percentage of quartz crystal, you know, so we're just taking it from one. We're just taking it from one, you know, area in a dark cave and just bringing it out into light. You know, so all goes into some other dark factory that goes in yeah, we're just we're just moving the material around, or some other dark conference right or some dark conference.

So All right, so you talked a little about DC, China. What was the difference between manufacturing the DC China badge?

Right? Okay, so the DEF CON China badge was actually a flexible circuit board. I don't know if you've seen it, I'll try to explain it while I'm showing you on the on this little video here. But it's a it's a flexible circuit board and a bunch of LEDs on it. And the it's actually there's a white cover lay on it. So it looks like white, you know, a white flexible board with some different color solder mask, depending on the attendee type as our as our colors in the shape kind of as like a digital tree. And there's 32 LEDs 16 of them are in the roots, and 16 of them are in the branches. And the concept sort of like DEFCON China or DEF CON us is to kind of build community because DEF CON China was definitely a newer community where you almost had to tell people, it's okay to hack on something. You know, I don't exactly know the politics there. But I'm pretty much going to assume that that's normally not an accepted practice, you normally want to follow the rules. So at DEF CON, we're like no, you can you can, you can actually like take your badge and plug it into your computer and do stuff with it and change it and modify the code. And then there's this was actually the first time I worked with an Arduino as a development process, not just like an Arduino Uno or something. So this is this is an Arduino platform, where I put all of the circuitry in one horizontal row. To prevent it, you make it a little harder to bend, like I put it right where the battery holder is. So it's this long strip of circuitry. And then the rest is more flexible where the LEDs are. And there's an FPC connector on their flexible printed circuit connector that plugs into these different circuit boards I made around the conference. So as you complete different tasks, the light the route lights turned on. And as you complete four of the routes, part of the branch is turned on. When you complete all the routes, all the branches turn on and sparkles and everyone's happy. And you know, celebrates. The reason I did this one, it's funny, because when Jeff and I were talking, he asked me to do the the DEF CON us show first? And I said yes, of course. And that would be fun. And not realizing how hard it was not remembering how hard it was. And then he's like, ah, what about you won't do the China One, two? I'm like, No, I probably shouldn't do that. Because it was only like a few months before the US show. He's like, Well, let's just think of something simple that we can do. And then it'll be fine. And I'm like, Well, I've always wanted to work with Flex. So let's just do something simple with Flex. But you know, nothing's ever simple, though. It seems simple in my head, right? But then like, you start building it, and it's like, Oh, my God, this is not simple. And I was hating life. For a long time. I did find a company in Chicago, am I allowed to mention company names? Sure. Okay, so I found a company called Electronic interconnect through one of my local reps here. And they do some manufacturing in in the US, but they also have some subcontractors and partners in China. So this was another one where we had we manufacture these in China. But that was sort of what we wanted to do anyway, because the request from the sponsors, but having not worked with Flex before, it was really nice to have a US connection, where they could sort of translate things into the terminology that the factory in China needed. Whereas, you know, with the US one, having an extra middleman sometimes made things harder, because I know how to manufacture a rigid board, I didn't know how to manufacture flex board, so having an extra person was helpful. So there was just a lot of emails. And you know, they dealt with me very nicely because I was sending a lot of mails. And really the important thing about this board, because specifying Flex is easy. You know, you do have to have a slightly different layout, depending on the orientation of your components and how you're flexing and you want to make sure you're not placing traces on stress points. And I learned a new command I use Altium Designer and learn a new command called teardrop and you could just add teardrops to every single pad and every single trace connection that thickens those areas and rounds them out so you don't have these right angles

which reduce like that through reduced like a stress riser. Copper. Yeah,

exactly. But really the the key of this board was the thickness because the datasheet for the FPC connector has a little bit of a stiffer stiffener requirements. So you have to have a stiffener on there. And then you have your like the stiffener is usually like a black piece on a flex and then you have your fingers on the on the other side. So everything was driven from that requirement of this particular thickness. And that needed a lot of back and forth because it was like even the thickness of the adhesive interlayer adhesive had to be considered the thickness of the cover lay the thickness of the stiffener, the thickness of the copper like all these things that normally a lot of times with with a typical, you know, two layer board for rigid you're just like the 62 mil and make the board and you want to layers into vias. So this was really really tricky. Luckily, we had time to make to two runs of prototype. So I could get one version back and I had a few different like they had black cover lay white cover lay and clear or just the polyamide you know orangish looking and I got to the aside on that got to make some changes to the board. It turns out that repairing and doing like kind of circuit board, repairs on a flex board is painful, really, really painful. So, because you think, Oh, I'm just going to cut this trace and reroute it like on a normal board, you can do that. Because there's a there's a thick fiberglass substrate and fr for substrate in there, on a flex board, your substrate is so thin that you end up cutting through the board. And I had to do a little rework on the board. And luckily, I didn't cut into too much stuff. And I was able to reroute and use super, super glue. But I would say like, if you're actually doing a flex board and have time, you would want to build your prototype on a rigid, make sure all your layout is right in then go to your like pre production prototype on a flex. That was the that was the key thing I learned. But the rest of it once we got the thickness down, we verified that made the changes, like the rest of it was pretty, pretty smooth sailing. And they came out pretty cool. And like I tried to we didn't really have many parts come off the factory Did you know they worked, they worked through the night to get these things done, we only made 3000 for the for the China show, which is still significant when you only have two boards per flex panel. So they had 1500 panels they had to make. What I also did for this one is the factory was responsible for testing and programming. Because there was no way I would have been able to get my firmware done in time to like get that done by a third party before sending it to China. So I did have the arduino bootloader loaded into the Atmel chips. So the factory was able to assemble them. And then they had a bootloader. So through the USB connection, which had an FTDI, FTD 31x on there, they could load the final firmware, do their testing, and you know, even load the configuration for the FTD 31x. So I made a little Raspberry Pi with some scripts that would program the 230 1x to set the USB configuration, and then load the Arduino code into the Atmel part and then do a little system test. So it was actually kind of cool. And I sent them a bunch of those. Because trying to get a factory 1000s of miles away to you know, load a piece of software on their own computers or something is just asking for trouble. So I thought I thought that if we could consolidate into a unit, all they have to do is plug it into a TV and give it power. And they're good like that, that saves a lot of trouble too. And it scales, right. Because if it's like you buy five raspberry pies, and if they need to double their throughput, you just buy another five and send it to them. So that that made things really cool.

So for the DC China batch, was there any tariffs that you had to worry about? Or was most of your stuff purchased and sourced in China? So you didn't have to worry about that?

Um, so great question. I ordered all of the parts locally, or sometimes had to order them from China pay tariffs, bring them here and then ship them to China? Because every project I do, I really liked it to maintain that control of what I'm ordering who I'm ordering from and verify training. Yeah. So they didn't order anything themselves. So yeah, I had to pay quite a bit of tariff on that project. And the thought was like, Oh, we can just buy stuff and ship it to China directly and avoid the tariff. But it doesn't work that way unless you're buying components in China and they stay in China. Right? So even right? Even like we were trying to save save money on tariffs. And I was like, well, can't we just since King Brite is in China, can't we just order them there and have them shipped to the factory there. But since their headquarters is in Hong Kong, you still have to order them and then you pay tariffs. So there's probably some tricks that you can get around not paying them. Put in your suitcase, yeah, put them in your suitcase and smuggle them. But you still have to get them You said to get there. You know, it's just too many, too many hoops. And we just did it on the up and up and got everything there on time. We did have some parts held in customs though. So when we were shipping the components into China. We had to pay additional duties actually, I think sometimes we actually double had to double dip, like we had to pay twice because we paid for parts to come to us and then paid for parts to go back in. Even though normally that doesn't happen. But if the paperwork is not right, you have you have somebody that is really scrutinizing things, but we had a few boxes that were quite a few days late. And we had all the invoices and everything but they just didn't trust us that things were coming from where they said or that the box was so big How could the components be so cheap? It just looking for any way to kind of mess with us? And that was right, during like the announcement of the tariffs increasing and the China trade war and all this stuff. And you know, all the engineers and all the all the workers of the factories like we all kind of pay the brunt of it. You know, nope, nobody except maybe the politicians are benefiting from this. Because then it just puts pressure on the factories when they get stuff late. They have to work harder to do it. And I doubt they're getting, you know, any increase in pay. And it's just a really weird kind of time for that stuff. And it was definitely I really enjoyed being able to keep it local to have just a Avoid those headaches, you know, just avoid the customs headaches and the duties headaches and things getting held even if you like we had all the right paperwork and stuff was still held, just for no reason. And it like you can't run a business that way. And try to get stuff delivered on time. So there, there were definitely some challenges and some nail biting moments. And actually at one point like are the Raspberry Pi's were held in customs, and with the file systems, like for the testing and the programming, which I had shipped it early enough to anticipate that, but because those were held, I went out and bought 10 More Raspberry Pi's and 10 more micro SD cards and image those and made 10 more programming stations, they all eventually got released in time. So the factory just had, you know, twice as many. So they could do it twice as fast. So it turned out, okay, it was just, you know, it could have been a lot worse.

Yeah, the I really understand, like, wanting to control your own supply line too, because I know there was I'm not gonna say names or anything. But there was a badge maker, who got all his stuff in for DEF CON, and the PCB assembly house in China, put counterfeit parts on his board. And he didn't have the time to get new parts and re solder, you know, all those components?

Well, that's right. And that's, that's a common thing. Like, I think, I think most of the time when factories are replacing parts, they're not doing it intentionally, maliciously. It's, oh, I have a relationship with this company, if I'm responsible for buying the bomb, I'm gonna buy it from my friend who I know can get a better deal than cheap as possible. Yeah. And it just happens that that friend has the counterfeit parts or gray market parts. And I mean, that happens at every level, it happened. It has happened to me, but you know, I was able to not on this project, but I was able to be like, wait a second, no, you can't start switching components on me like you have to use the ones that I exactly the ones that I recommended on the bomb. And that sometimes can be detected if you're doing pre production prototypes and testing. But again, it's one of those things that you don't really want to happen. And you could you could specify on your design documentation, you know, do not change parts. But you can't control that unless you're there, right, and you don't know their their own supply chains and their own relationships. And that's really why I like sourcing my own stuff and keeping control of that, and you end up paying more. But at least you know, if there's a counterfeit part, or if there's a faulty part, or you get them with weird date codes or something, you can trace those back yourself. And it makes things much easier than trying to jump through a bunch of hoops for a factory. That's that's all the way halfway around the world.

Yeah. So before we move on to next topic, is there any other like, what would you do different if you could redo DC 27

I would definitely get a pre production prototype. Absolutely. So we basically went from six prototypes, because we didn't have the time I got home. So I'll give you a quick, quick little timeline. The day the day before I left for DEF CON China, I had to send the prototype Dec 27 design to the fab to the assembly house so they could fab the boards assemble six prototypes. So when I got home from that trip, two and a half weeks later, I could get the prototype. And then that was June 10. And then have until June 17, to make sure the hardware worked, which meant writing all of the low level code to verify that all the hardware was working properly, make whatever changes I had needed to do to the board and then place an order for the 28,600 on June 17. So I had seven days, we went from six boards to 20,600. We didn't test the final changes that I made on the production units. And fr for Yeah, yeah. And fr for I mean, it was like not recommended, right. Because

if I remember specifically Parker was in the hotel room. And, and we were and I had watched your your talk, and I just said, Damn, that's ballsy.

Yeah. It wasn't like I was doing it to be like, yeah, ballsy.

Like, he was just like, necessity, right? Like, I

have to do this now. Or we're going to end up paying even more and expedite fees. And if something goes wrong, we're not going to have time like looking at the calendar. It's like, oh, DEFCON six weeks away. But that's not a lot of time, right? And I had to do it. And I really tried to push the factory, I'm like, is there any way we can get a pre production prototype there? Like that's going to take two more weeks, because the minimum time to make these complex, four layer boards was two weeks. We had we didn't we just didn't have the luxury to do it. So I wish we could do have done a pre production prototype. That's what I do with all my other projects. And that lets you catch these minut production errors. So to give an example, I didn't go into this in the DEF CON presentation. But the prototypes came back. They looked great. They worked great. The production units luckily all my changes, which but mostly we're just moving components out of the way and rerouting some stuff like luckily those have all worked, but you The factory had a different cam engineer working on the production version than the prototype. So he wasn't familiar with what we had done on the prototype. And like that you shouldn't change any traces, you shouldn't change any graphics. So he made some he made some unauthorized changes to the board. Now not out of intentional maliciousness, it was just because he wanted it to be as manufactured as best as possible, given their manufacturing constraints. But those weren't relayed to us, so we couldn't approve them. So we got boards back, and we're like, wait a second, what are these extra vias on here? And why were they connected and like, you just changed connections to components that could have affected signal integrity. And we made 28,000 of these, and you didn't even tell us. So we would have known if we had done a pre production prototype, we could have verified with X ray, which is how we discovered it. And we could have just made sure that things were kosher before we went all the way, and then let the faculty know, like, don't do that. Right. Like, that's not a good thing. And part of that was just communication issues, because we were again dealing with with ei in the US, and then they were dealing with their partner in China, is some of the importance of these issues weren't conveyed. Even though in the design documents, it says certain things, it doesn't say, don't change artwork, you know, it doesn't say don't change layout, which it probably should, like, everything needs to be specified. And that you know that some of the some more aesthetic things were changed slightly, that no one really is going to notice. But it was those same sorts of things, the factory doing it, either thinking it was the right thing to do, or thinking that was what it was supposed to look like, without checking with the customer. And those things, I think are just communication issues. That that can happen with a US fab also, but are more easily solved with us fab, I think because it's just much easier to say, hey, you know, if you make any changes, let me know, we're in, in this case, in China, there's long delays, and they were stressed, you know, to get this thing manufactured as best as possible and get out the door to us. So yeah, I think having a pre production prototype would have made things a lot easier, we could have ironed out a few of the things that drive that drove me nuts from just an aesthetic, you know, as just a really kind of anal engineer, that didn't affect the system. But I like to have things as perfect as I as I possibly can. And it would have made me sleep a little better, before placing this massive order without being able to test anything. Luckily, though, like everyone crossing my fingers, like everything, everything worked from an electrical perspective, and from an assembly perspective and a functional perspective. So we really, really got lucky. And it you know, I'm basically just complaining about these minor aesthetic things that I'm not even going to mention because I don't want people to go go looking for them and then think I'm crazy for even mentioning them. But little things minut changes that I just don't like people doing on my boards without verifying, you know, with the customer.

I know how people, I guess you can go on your website. It's all open source, right? Yeah,

everything's open source, except the Gerber's. We don't put the Gerber's up. Because you know, one of the things people like to do is make counterfeit badges and try to get their their make their black badge, which gives them lifetime entry into DEF CON. So we don't release the Gerber's. But all the other design documentation is up there, schematics and firmware and test procedure and all of that stuff. So you can definitely learn a lot from from looking at that stuff. This was also by the way, I didn't mention this. This was like my, my first time kind of working with BJ parts. I wanted to go like super, super miniscule, which maybe we're jumping ahead to other topics. But I really wanted to try something new, besides the gym of like, let's see if I can use the smallest parts possible.

I was going to get into that. Yeah. Can't you just keep getting more ballsy with this?

Well, no. And that was the thing is like, you know, like we always say like DEF CON is a great place for people to go and to learn and try something new. So I'm just eating my own dog food and doing it on a much bigger scale, and taking way more risks than I probably should. But it just seemed cool. Like if I wanted to have the small badge, so let's use the smallest possible parts. And the factory was totally on board with it. And like they could they could deal with it. But I didn't even know like could my 36 Ball footprint, my BGA footprint was that going to be good and we have chip scale packaging on here. Just crazy, ridiculous stuff. And debugging that stuff is hard. So the prototype luckily I put tiny super tiny test points on there that I could solder some wire wrap wire to I use some zero ohm some 1206 and some 8050 ohm resistors as jumpers so I can isolate different components to make sure power consumption and stuff like that. But on the production version, I took away all of that, which is you know, not recommended either. Like if I had to troubleshoot the production board, I would have had to do it just on the sides of components that that I could get to. But not all of those components are connected. Some connections go from the BGA of one part into an inner layer and an up underneath To the via and pad of another part. No way to test that unless you're cutting into the board and tapping down. So yeah, I made probably some people would say some pretty, some pretty stupid leaps to get this done. But, you know, part of it was just like the kind of confidence of knowing the prototype work. Part of it was the necessity. And part of it was probably a little bit of just naive, you know, let's just hope it works. And the factory seemed to back me up. They're like, Yeah, well, you didn't, you know, you didn't make any major changes, like this will probably work. And yeah, I'm just, I'm glad it did. And I just wouldn't recommend this this condensed of a process. I wouldn't even wish it on my worst enemy. Actually, no, that's not true. I probably would. I wouldn't wish it on my friends.

It just, it just kind of seems fitting with the whole Vegas roll of the dice kind of thing.

Yeah. And there, you know, it was a very controlled roll of the dice. I would say, you know, like, it wasn't, we weren't totally jumping, jumping off a bridge, but it was, it was a controlled gamble. And luckily, things came together enough where it worked. But it wasn't a traditional engineering process as much as I would have liked to have it be one

where Let's go. Let's go jump into that badge. So you said it's a four layer board. And it has it has via and pad that are kept Correct? Correct. And so, is there any blind and buried vias or is just cat vias?

So we have via in pad and then we also have some blind? No, not blind, we have some buried vias also. And the buried vias actually are what I think I had a few of them. But that's what the fab added to additional buried vias, because it was easier for their process for the inner layers. For some reason, move it from a via in pad blind to a buried or there was something where they change it and didn't think it would affect signal integrity. Luckily, it was just on the ground, vias going down. So it didn't affect anything. But if it was on, you know, if it happened to be on the antenna lines, going to our inductor for the actual magnetic communication, like that could have been a problem. And they didn't even ask, so it's just yeah, there there was the captive via and pad epoxy filled. And then,

and that's why the boards were would take so long to manufacture is because those those additional manufacturing steps and how much drop out they would have on their end.

That's right, and they were seeing they were seeing a little bit of failure on the prototypes, which is why they made this change in production. It just wasn't translated to us. But they they thought they would get a better yield by changing that a little bit. And yeah, I mean, it's, it's, it's a complex process to do. So I'm super proud of what they were able to pull off, even though they had these other authorized sort of changes, which is typical for factories, the fact that that they could do this, and if you look at a board, you'll see like the traces are super thin, and there's a lot going on, and there's even a lot going on in interlayers, which you're not gonna able to see unless you X ray it, I'll leave that as an exercise for the user,

you're not going to post any of those cool pictures.

No, in my slides, I actually have a few pictures of X ray. So you can see the X ray of the Calle 27, which is the microcontroller you can see the X ray of the HX 2261, which is our and FMI magnetic induction chip. And then I actually put a DEF CON logo on one of the inner layers too. And those are all in the slides just to sort of show like there's some there's some fun stuff in there.

I really liked the via stitching that that's going on, near the inductor on the left side. Yeah, it's, it's, it's really like, it's really anal I like

that was so that was recommended by NXP, luckily with this process with this project, and I mentioned this in the main Batch talk is NXP really stepped up and helped essentially a really small volume customer use one of their crazy parts. And they had a team a very small team in Belgium who was working on this radio circuitry they deal with like, you know, million volume customers. And they did a circuit board layout review for me and they gave me documentation and wrote some code for the radio chip. And they were really wanted they wanted to make sure that the radio layout was going to be correct. And they had said you know, do your ground plane but no ground plane underneath the antenna traces but then use via stitching and you're sort of creating and then your spacing has to be right between the between the two signals. And you're basically creating a ground sheath around the

like a PCB a coax almost yeah and it was really neat

to sort of to learn about that and see how see that how that's happening. And they recommended doing via stitching all along those traces and are the final boards the production boards ended up having a magnetic field communication distance of about a foot and a half which is pretty crazy because this FMI technology you know it's designed for hearing aids it's magnetic induction so it's essentially an air cooled in air core transformer that you're creating. And it's designed for like you know, ear to ear communication or body area network like like arm to, to chest communication like like low small distance. And when I was originally talking with the engineers, they were thinking like yeah, we'd maybe get six to eight inches. But we essentially doubled that probably because the circuit board layout was good and followed exactly what they recommended and there's no way I would have been able to know that on my own you know, it's not an RF radio wireless person.

What about the Chi from the courts?

The oh the Chi that too? Yeah, the you know, the crystal energy I'm sure like, boosted the ring. Yeah, boosted right. Sure. Yeah.

So let's go into the NF mi Sure. Why did you do NF mi verse like a normal like NFC or RFID type technology,

right. So I'm going to call NF mi infamy because it's way easier for me to say, and it sounds cooler. So infamy. Basically the way that came about is, you know, there's a lot of wireless modules out there we've seen ESP 32 and ESP DCS exists and all the all the Bluetooth, chipsets and modules, Wi Fi chipsets and modules. NFC stuff, like all the RF stuff, to me seemed pretty played out. Even though a lot of people still use it, which is fine. But I didn't want to just do something like that. So typically, what happens when I started a DEF CON project is I call friends of mine that work at various companies. And I called a friend of mine who was at NXP, and he actually helped support DEF CON 15 through 18. For me, because he was working there when when it was Freescale and DEF CON 14, I'd use microchips that was a different group of people. So I call up my friend and say, Hey, surprise, you know, nine years later, I'm doing another DEF CON badge. And he's like, wow, like, he never thought that would happen. And I just said, Hey, like, what cool technologies do you have that you think a DEF CON attendees would appreciate? Like, would think are cool. So he just listed out a bunch of stuff. And he's like, Yeah, you know, we have one group that's doing this infamy stuff. But you're gonna have to convince them to, for you to use it. Like, you know, they only deal with high volume customers, their their group is very small, I think there's eight of them. And, you know, you just have to convince them, maybe they'll do it, maybe not, I can't make any promises. But I'll make the introduction. So I wrote this email explaining DEF CON and how, you know, it's a cool audience that are willing to learn new things. And I think people would be excited because it's like this crazy technology that most people will never ever see. And it is also in some high end headsets, to have like Bluetooth to one side, and then that one side of the head to the other side of the head with infamy. But it was just like, something so cool. And I guess my writing was desperate enough where they're like, Okay, sure, let's Yeah, we can do that. And, and hooked up with those guys, and just started that process.

And you get an email from the king pin, you're gonna respond?

That's right. Yeah, if you got it from king pin. And it was, it was funny, because I was like, Yeah, you know, I'm a hacker, but I'm also a professional design engineer, if you give me the resources and the documentation, like, I'll try not to bug you too much. Because I understand that you're busy. Because I figured, you know, if it's in the docs, I can find it and figure it out and work with it. Which, to the most point was true, but eventually, you know, when it came to crunch time, it was like, Oh, this is hard, like, this is really the documentation is great. But it's still hard. So we had, you know, a bunch of back and forth emails of like, highly technical subject matter of like, you know, there is some of their code, some of their documentation didn't quite explain some of the packet, say, within i squared, C, and how I had to do different things to load the firmware in or I found a bug in the system at the last minute. Because the radio chip was was was responding or handling all I squared C traffic when it was really just should be looking at the ones that were addressed to it, which was a simple code change to to just physically disconnected the the switch matrix inside of the chip. But it was like all these things. And it's a very, very complicated side of the board. That subsystem is just hardcore. And they were great. I mean, they just, you know, really supported supported me, I will say to before I forget, like this project stretched every side of my engineering fabric like it, at some points made me question why I'm even an engineer, like, Why did I even choose this path, like engineering is horrible. But then it also, you know, once I solved the problems and things were working, it like really strengthened my my, it gave me more confidence as an engineer, because I was I was able to use BGA. And it worked. And I was able to troubleshoot this ridiculously complex board where I wanted it to look as simple as possible. But every time I got, you know, some sort of bug, I was like, alright, you know, instead of freaking out about it, I would sit down and set up my test equipment as I needed to, you know, write some code that I needed to do whatever it was set up my breakpoints and just find the root cause of that problem. And it really strengthened that debugging process and gave me that confidence of like, Alright, maybe I should be an engineer, even if I don't like it all the time. Like, this is probably what I'm designed to do or something like, what we know what just what my what my, what my path is supposed to be or whatever, whatever it is. So that was sort of cool at the same time. But now that I've done it's like, Alright, maybe I should just go like, you know, live in a cave in nature and not come out with a bunch of crystals. Yeah, with a bunch of crystals, because I'm like, so on the edge of burning out this whole year. And yeah, I mean totally, like just wrap myself in crystals, and like grow a long beard and become wise, you know, one with the world, which of course won't happen because I'll probably start working on stuff, you know, tomorrow. And then next year, we'll talk again and I'll be even more gray and have similar stories or something.

But it honestly just sounds like you need a vacation.

I do I need a vacation. And but you know, as being an engineer, like I already have ideas for future projects, whether or not it's going to be DEF CON badge, or something else. There's always something in my head, right? And as engineers, it's sort of like, it's our goal to get those from our head to reality and people might use them people might not but I just keep having ideas and I keep having thoughts in my head and I need to do something with them.

It's a curse. Yeah, it's a curse

and also the amazing thing about engineering.

It is right it's a blessing and a curse.

So what made you go with the KL 27 microcontroller because that is very interesting choice.

Sure. So. So the the main functionality of the board was the infamy chip, like everything started with the infamy chip. Once I worked with with NXP, their reference design for infamy was based on an LPC 1115 which is another general purpose microcontroller of some sort. That's like a Cortex M zero, I think, I think so, what I wanted to do is basically take their their references on their evaluation board and just shrink it down. But there was no available stock of this LPC 1115 Because I figured like, whatever the microcontroller is, I'll be able to learn how to use it right. It's usually like the development tools are good abstract enough for you, where you don't necessarily need to understand all these just change the header and hit Compile. Exactly, yeah. Yeah. Yeah, yeah, right, exactly. But at least it's a start, right. I mean, at some point, every micro is going to have peripherals that you have to configure a lot of gooeys now the IDE will have a nice way for you to set the different pin states and all these things. It's still not as comfortable as Arduino you know, as much as people make fun of Arduino.

The fact that you can go into a drop down and select a different board. And yes, about 99% of time it compiles still, yes, that's pretty

Amanda. And that like when you want to use you are you type of command, you know, start you are or whatever it is, or start I squared C, like all the low level, the bare metal stuff is done for you. And that is a huge, a huge benefit. Even though people make fun of it for being Color by Number of paint by number, whatever it is, or like the libraries are not good or whatever. Like it's still a great platform to develop with. And bare metal is no joke, like to even get your art working to get I squared C working to get interrupts working to get flash memory storage working. Everything was like a you know, 18 to 20 hour day 4am staying up till 4am Like trying to get stuff working. So it was just really painful.

But what am I always I always suggest when like you're starting with a new project or a new a new piece of hardware. I actually generally I used to be the prop parallax propeller, right? I would do a lot of my prototyping in that. But now I use Arduino. That's where

we met using a prop. Right that was it propeller Expo.

Yes. Are you excited for the prop two?

I'm excited, because they're gonna have another cuz they're gonna have another propeller Expo. Yeah, we can go and hang out like, I'm not, you know, I haven't really been keeping track of it. It's something. It might be useful for things I don't I don't know, if it's hardcore, you know, it's gonna be in a lot of products. I think it has some interesting capabilities. It has debug, which is what I'm excited about. Yeah, I'll probably use it and poke around with it. But yeah, I don't I don't know. Like, the stuff I usually designed is going to be lower power, lower cost, you know, real deep embedded stuff, like this badge where the P two, or an Arduino generally isn't going to work because of power consumption issues and stuff. But I I'm curious about it, but I'm most looking forward to their Expo whenever they have one so we can all go there again.

We had them on we had Ken and chip on the podcast. Oh, what? Four or five months ago

a few months ago.

Shout out. Shout out to Canada. Shout out to Ken and chip. Congrats on finally that

was a that was a great podcast and awesome. They invited us to the expo whenever and wherever

it happens. I just think I think it's great that those guys you know their brothers and their but they're so different in their mindset. And I just I love it. And you know, when their dad was around, he was another great addition to parallax and like those three really was this family business and they were doing and they are doing what they love to do and not following any particular direction. And like that's pretty. That's pretty hardcore.

And if I had to work. If I had to work with my brother, I'd go insane.

Yeah, no. I mean, they're great about it. And like I've, you know, I've worked with them for since like, 2003. They've been they've been selling some of my products and just just they're good, honest people. And there's not a lot of there's not a lot of people like that. Right. So I guess on that point, too, I'm looking forward to buying their part. So I can actually kind of kick back to what they're doing, because I think that's important. All right. So back to Yes, you were saying you were saying like you normally are using the Oh, yeah.

So I usually I used to use a parallax propeller philosophy. But now I've kind of switched over to the ad CMD, which is a it's actually it's what's on the Arduino m zero platform. Okay, so I can use Arduino to prototype new hardware, and then go okay, now I can switch over to Atmel Studio, do the low level stuff for the actual product.

Yeah, that's generally a good way to go as you start higher level and then you can narrow down. But so so it turns out that the LPC 1115 wasn't available, like there weren't 30,000 that we could buy, to start working with.

It's funny about that is there's usually not 30,000 of any

No, there's not like that's a lot of that's a lot of parts.

Like I'm actually suppose NXP had 30,000 of these. And for me, the internships.

Yes. So some of those were allocated to another customer. And they were able to pull some history away from them, they weren't they hack them away, they sort of snuck them away for us. And somehow it's really great

for the kind of lower volume thing that they were that they pulled strings for you.

It was it was great. It was really, I mean, it was it was amazing. And I but I think they saw the value in DEF CON in that new audience and like, NXP ended up sending an engineer an actually like a real life engineer, to DEF CON to hang out in the hardware hacking village and show people how to get the development tools set up and help them with their badges, the technical issues and everything. And like, it saved me a lot of trouble because I like creating things and then letting people use them. Like I don't like necessarily the customer support side, which is why I don't sell my own products.

I was gonna say you don't like support, like support. Those dang customers,

I'll just plain and simple I don't like support, I will do it of course. But if somebody else can do it, for me, that's better. And you know, running around DEF CON is hard. And and I feel like having having somebody from NXP, like they sent him there, you know, and like, that's, that's really awesome that they were able to do that, which let me focus more on just kind of seeing how people were doing and, and passing passing them over to this guy. So So yeah, that was cool. But yes, they really allocated some of the infamy chips for us. But in that then I had to find a microcontroller. And the reason I could sort of choose whatever micro I wanted is we had kind of separated this badge into two systems. One was the kind of UI user, you know, front end part that control the LEDs and all of that. And then there was the infamy side, the radio side, the infamy chip actually has its own ARM core in it. And most customers, you know, the high volume hearing aid customers and everything will write their code that goes inside the arm inside the infamy chip. But all of that, you know, the infamy side requires non disclosure agreements. And they don't want to release their code of the radio. So we sort of came up with a solution of like, alright, well, DEF CON is all about sharing an open source. But but we want to use your chip, how are we going to do that if you're not going to let us release the data for your chip. And they said, well, we'll write some code for the chip. And we sort of treat that as a black box. And we'll just send if there's a valid packet read, we'll send that data over UART interface. And then we'll use some IO pins as interrupts to wake up the host microcontroller. So it's sort of like, if you think of any other peripheral chip you buy, it's basically a black box. And that's what we did in this case is turn this like highly configurable radio into a black box. Which means if we're just using UART, and some some IO pins, and then I squared C to load the firmware into it at the beginning on powerup, we can pretty much use any any microbe we want. One of their other boards used, they had a board called the Freedom. It was a freedom kale 27. They have a bunch of low costs, different development boards that actually were Freescale boards, rebranded as NXP. They had a Freescale or sorry, a freedom kale 27 board that some of the guys in Belgium it said, Yeah, this, this ship should probably work. And we have probably, you know, we have stock of it. So we just sort of took it because that was available, there was a low cost part. It was available. We got it through future electronics who was our distributor for this. And actually, that was like the only part that was available because we had future look for other low low cost parts that had the memory size and the peripherals that we needed. That was the only part available. So we bought all the possible stock we could have of that part. And then it crammed them together. And magically it worked. And it turns out it's a great part super powerful like to be able to reconfigure the pins for the peripherals. As painful as it was to get everything working like the libraries part of the SDK was pretty good for low level stuff. It was a nice part I'll probably use it again, just because now I have this code base set up and I understand how to use the park. It was yeah, it was surprisingly nice. I should mention I originally, the kale 27 also has USB functionality built in. And I wanted to have the user interface via USB connections so that more people can just plug in a USB port cable to it and have an interactive menu or a way to mess around with the badge. But it turns out this was caught by one of the engineers at future when they were doing a schematic review for me of in order to run USB, the USB core has to be at three point or 3.0 or 3.3 volts and or greater. And I'm running at 1.8 volts. My battery is a three volt, you know CR 2032 coin cell. So it's at three volts, but it will drop down to two volts at end of life. So I couldn't run the USB off of that directly. So he caught that. And that was right before I started writing the board and sending it out to fab. Actually, I probably was already routing the board at that point for the prototype. So I had to switch from USB to UART. So the Calle 27 Has the USB functionality, I didn't use it. And now I just have a 1.8 volt UART on there, that was easy enough, we bought a bunch of headers, and people could solder them on and you get, you know, an interactive way to mess with the badge. And when you complete the badge quest, which is the concept for this for this badge, it unlocks additional functionality that you could then use kind of to mess with other people and do fun things like that.

DEF CON style things

DEFCON style, like Yeah, you know, you do this, you finish the the normal design, and then it's like, Alright, let's do a little bit of, you know, basic hardware hacking, which to, you know, 30,000 people 20,000 People like soldering a header is hardware hacking like that's, that's their entry point to hardware hacking. So it gives them something where they can visit physically interact with the board, make them feel like they've modified it in some way, and then discover some additional functionality. And a lot of people do that. And it was like, it was pretty cool. I was worried that people would be like, Oh, it's uh, you are lame. But actually, I think I think it was better because some people were like, well, how do I plug it into my computer, like, I don't see a USB connection, because that's what they're used to seeing, you know, on most complex badges, most complex devices, any sort of product. So this was a great teaching experience of like, Look, if you're hacking on a circuit board, you might not see a USB interface, but there's probably going to be a UART interface on there, that's going to give you debug output, or console access or administrator access, or diagnostic mode, or bootloader menu or root shell or whatever it is. So this is a great experience for people to learn, you know, the right pins to connect and find the UART and do something with it. So it's almost like it's the simplified board made it better for people to learn with because you're not forced feeding and feeding them everything that they need.

So I got a question. How, how did this design evolve? Knowing that 30,000 plus people were going to be wearing this around their neck? And it'd been hit around? And yeah, yeah. So basically been the living hell out of the hardware, right?

Yeah. And while I wanted to use like the most fragile tiny components possible, exactly. So which those two things don't go together. So what I really wanted to do is on this badge, we had a unique kind of mounting method, where we have these two giant, high current jumper bars that I'm using as lanyard straps. So if you see a lot of the pictures online, the lanyard slides through the straps, and holds the badge above like sitting on the lanyard, not on the lanyard clip. And the real intent of that was like, Well, you know, with all the badge life stuff, people always have a bunch of other badges that they're wearing that are clipped to the clip, like let's move up the stack, and bring our board a little higher up and have it be a little more stable up on the lanyard part. So that moves you out of the way from a lot of the banging that happens when you're walking around. I also tried to place the components as far as I could, away from where the lanyard goes. So were any of that interference or kind of where would happen. And the badge does, it bounces around a little bit but way way less than then if it's down lower, I guess physics or something. But we had three main points of failure. And this is something that we sort of anticipated. So we had extra components to let people fix their own. And the the main things were like the lanyard straps would sometimes get pulled off. If you didn't mount it through the through the the lanyard properly, like a lot of people would just clip on to the top lanyard strap and then their badge was dangling on the lanyard, which makes a lot of force back and forth. We do have via stitching on those lanyard pads. But sometimes the entire pad would still get pulled off the board. So I sort of attribute that to like user error or not, you know, following Twitter work because we mentioned and posted a picture of like the right way or like not go to the presentation. And over the time at DEF CON people started mounting it better. So we had a few actually not that many failures of that but we had extra straps that they could solve They're down and everything that was no big deal. The battery was the other one the battery clip. I learned this and I don't exactly know the science behind it, but apparently like these gold plated battery holders that you get, you know, coin cells and everything. gold plating is great for conductivity. Say for, you know, on the on the pads where the battery goes to it. But gold plated is not great for solder adhesion. And there's something called like, Is it gold and Bertelsmann, I think it's called where sounds correct where the you probably know this better than I do. But like the gold pad doesn't really adhere well to the solder. And that's just a fact of this gold plating. So we had just a handful of failures of the of the CR 2032 coin cell holder popping off. And that's no big deal, because it's another easy one to solder. And then the main component that would pop off was the antenna. Those antennas have a very small footprint, or I guess, no connection area of the part. Even if you had a massive footprint, it wouldn't matter because there's only a tiny area of the actual antenna that's making physical contact to that pad. So those those fell off a little bit. And that was probably the most common failure. And we had 1000s of extra of the of the antennas. And probably, you know, maybe 50 people at the conference went and fix theirs. So we didn't really anticipate for any, we knew people were going to mount these in different ways and bash on them and everything. But my sort of response was, Well, they're going to work when you get them. Everything else is up to you. If something breaks, this is DEF CON, like it's the perfect place to go learn like go to the hardware hacking village. teach yourself how to solder have somebody teach you how to solder and like try to put stuff back on. Because it empowers you. Right. And that was the thing I wasn't doing it to be a dick. I was doing it partially because I don't like doing support. But it's mostly to empower other people of like, this is this is your piece of electronics. It's a high tech, super fragile piece of jewelry.

But you know, to get drunk and then slam into Yeah, like if you

like, that's what you get for drinking. Yeah, you know, it's like if you do that, that's, that's, that's on you.

But if we didn't, we didn't mention that these badges are meant to be like clicked together in a way like,

Well, yeah. So that's, that's the other thing they did. They were meant to interact. But it's funny because the human nature, I don't know exactly how this happened. But people thought that you had to physically connect them or physically touch them. I think because when I said magnetic induction, they sort of thought magnet and with magnet, you need to touch them together. But the only similarity is like a magnet has a magnetic field. And this coil has a magnetic field, but you don't have to touch them. But people were actually like kissing their badges together. Which to me was it was making me nervous because you have two gemstones connecting, you know, and like, it's glass on glass. Like, that doesn't sound good to me and they could shatter. But people did it seemed fine. Communication actually works better when you're a little bit further apart. So you're in in the actual magnetic field, and not too close to the actual coil part. But yeah, you know, it's like, all these things that you can't anticipate, like, I didn't anticipate that somebody would clip onto the lanyard strap, or that they would touch their badges together, I thought about more of the use of like what they would do when they got it and realize, oh, there's an antenna, it probably has to communicate or something. But all the human use, like, it's a great I don't know, if it's like a sociology study or like a just a humanity study of like, you can never plan for what somebody's actually going to do with your product. Even if you know, it's a constrained environment, like a badge, you just still don't know what they're going to do. So it was really like a learning experience for me. But also at the same time, since I really wanted to do this particular build, I almost didn't care what was gonna happen, because I just wanted it to be this way. But yeah, there definitely were some people now we're still tweeting like, Oops, mine came off during traveling, I gotta fix the antenna. And the antenna is not something you can just buy on digikey unfortunately. So I know a lot of the manufacturers reps are going to get a lot of phone calls from hackers, you know, asking for replacements samplings one on one off, and hopefully, hopefully they'll be cool with that, because we did buy a whole lot of these, the hardware hacking village does have extras so if you wait a year, you can go back and you know and put it back on. On its own it doesn't you know, if you just have your badge sitting if you only have one, it doesn't really matter if your antennas there because there's nothing to communicate with. But if people have more than one or they want to build some project, it would be nice if they work. And you know, what I should have done is just kept like a real in my lab. So people you know, kind of covertly could be like, Hey, Joe, do you have an extra antenna? And I can send them one but like I didn't I didn't think that far ahead and that would require being like customer service, which for a project like this it's sort of like it's you know, this project is it's done and it's like Okay, time to hack your own make your own coil or find you know, social engineer one or whatever it is. Because that's just the hacker way you know, it's like another thing I noticed this a little bit of a rant but like DEFCON is growing a lot, and that means a lot of people that maybe aren't as technically inclined. Maybe they don't know how to solder. Maybe they weren't don't even know what an electronic badge is. So it really is, you know, it's something where you really want to push people to, to think creatively and get out of their comfort zone, get out of their comfort zone and try something and, and explore and learn. And, you know, I got a lot of questions about things. And my answer was like, try it and see what happens. Right? Because that's how you learn, not from not from me telling you the answer.

That's actually what I kind of liked about this year's badge was the, because I got a press badge for the podcast. And you became like, the most important people for someone to hunt out, because it was a lot of green and the badges suppressed became popular. Imagine Yeah, exactly. And only so I got to meet, I got to meet a lot of people. Because basically, someone say, Hey, can I bump your badge? Right? And I'm like, I gotta give you my elevator speech about macro fat first.

Well, that's so I guess we should explain just like the general concept of the bad, right? We didn't do that.

Did we know a little bit? But yeah, we can go more

into it. Okay. So Toby, just so people understand like, why why were they even going up to a press person in the first place. So the concept for the badge is similar to the DEF CON China one, where we wanted to create something where the attendees could kind of meet each other and, and kind of work towards this communal quest. So each badge has a badge quest, and there's a series of states, there are seven different states. The first one is what I call attract mode, where the lights just blink every once in a while in a certain pattern. That pattern spells out DEF CON in the, in the wit in the the the way the LEDs are turned on or off, sort of to mimic the shape of a D and an E and an F and A C and N one and n when you first communicate with any badge that brings you to the next state. From that point forward, you have five tasks that you have to complete at DEF CON. And the goal of this like I thought about when I created this and like, okay, DEF CON is massive, like if I was going to DEF CON for the first time, what would I What would I want to do? I probably want to see a talk, I want to go to a contest, I'd want to go to a village, I'd want to go to a party I'd want to go to see like a concert arts and entertainment. So the the quest, part of this quest is to is to go to each of those five tasks, and experience DEF CON, you know, do those things, and then find a goon in those areas. The goons are the volunteers that work at DEF CON. They had what I call magic tokens. And those were specific to the task. So speaker Goons had the speaker token, the village goons had the village Shogun, and you would go up to them and say, hey, you know, I just saw this talk, can I get a scan of your token? And that would bring you to the next state. So it was a way to get people to go through DEF CON, interact with people talk to the goons, you know, appreciate the goons, because a lot of times people working behind the scenes never get credit that they deserve. The conference sort of goes on. But you don't realize that there's hundreds of people actually making that happen. So it gave them a little bit of you know, street cred, but also kind of mob ism at the same time, because once people realized, Oh, I gotta find the goons with the magic tokens. All those goons like were prevented from doing their job because they were just scanning tokens all day. But it made it more fun. Like it was sort of this interactive thing. And then the final state, once you did your five tasks, which could be done in any order, the final state was to find one of each color gemstone press being one of them. And either scan them altogether, which I thought was more fun, I call that group chat. Or you find one individually, you scan each color gemstone, and that will bring you that will you'll win the game you get a little surprise song and unlock the extra functionality, a little animation on the on the display. But it was fun to see how when people you know, you could you could learn all of this by looking at the source code. But people didn't look at the source code. They were like running around all conference scanning everybody's badge, everything, not realizing that that was only helpful on that final stage, which was another sort of human nature side, which I just didn't anticipate of people, you know, running around scanning each other. But it was meant to be very simple and inclusive. And let everybody kind of get involved in it. And keep it simple. And I think what happened though, too, is a lot of people had expected you know, these very complex puzzle badges and things like that, that have been kind of the norm over the past few years. But I'm just not like that. Like I'm not a puzzle person I suck at puzzles. And my mindset is is is different. Like I have an engineering mindset is just very straight. No tricks you know for a clean cut no trick thing

but doing that tricked people well that

it did it tricked people because people were still coming up to me thinking I had a special badge or I had a special thing. And I was like no no no like, this is created for you to go out and explore like I didn't I didn't want to involve myself in be part of that puzzle. Because that felt weird. Like I'm just a designer of one one portion of this massive conference but People, you know, I think a lot of times were overthinking it. And sort of, they thought I was not telling the truth. They thought I was tricking them by saying I wasn't part of it when I really was or whatever. Like, it was pretty funny to see. And I had a great time, like it was, it was a lot of work. And you know, all day, every day, but it was just so fun to see people running around and playing with a badge and trying to scan my badge and like, asking for autographs and asking questions. Like, it was just cool. Like, it was really, it was like it, you know, a nerd engineers dream to be be appreciated, you know, for a project assigned my badge. Yeah, I mean, did I sign yours? No. Oh, yeah. I mean, like, you know, most of the time we're sitting in, in labs, right, or sitting in an office, and we work on a project and it goes out the door, and your name is never on it. Nobody knows the names of the people that worked on some of the most popular products out there, you know, like Apple, at least on the Mac, like people had their signatures injection molded inside the case. But engineers don't really get respect like that, except at DEF CON. And except that like, you know, hacker conferences and maker conferences, so it really was like, kind of a fun, a fun thing, even though that's not why I do it. Like I don't look for the for appreciation, but I appreciate the appreciation. You know, like, I think that's pretty cool that that my peers saw the the coolness of it, and like I can finally share that passion with other people. Because, you know, my family, they understand what I do, but like, they're not technical, and they don't really need to know every low level bit. When I'm like, Look, I did the via stitching on the thing to do the blah, blah, blah, like, they don't really they're like, great, you know, but at DEF CON, like, that's a cool thing. And so it felt really good. I don't know what's going to happen in future years. But, you know, this was this was a kind of a really, it was definitely a crash course into reminding me how hard this stuff is. And like, it's called badge life for a reason, right? Like it takes over your life, it consumes your life. For me, it was definitely not a mentally healthy year. It like literally drove me into the ground a bunch of times. And even at DEF CON, I was close to having like a massive meltdown of just the overwhelming nature of it, and, and all the effort that went into it. But it turned out, okay, and I think future years just needs a little more balance of like, doesn't have to be complex stuff. It just has to be something that people can enjoy and use, and then something that isn't going to consume my life or like shorten my life, you know, like, it's not worth, no project is worth shortening your life over. Right? Or like, not seeing

whatever project that eventually extends your life.

But if it shortened mine, and I wasn't able to use it, I don't know. I don't know if I would do it. But it's like, no, you

know, a quick suggestion is, the next time you have something huge on the horizon, go back and listen to this podcast and hear yourself say what you just said, Yeah, because like, the tendency with with engineering is just like, well, the next one is going to be bigger, right? So I'll just do more.

But it's also you can say the next one is going to be simpler, but it never is like it always gonna be some issue. But I think nothing is worth, you know, missing out on having dinner with your kids, or like saying goodnight to your kids, or seeing your family or whatever it is like the engineering, like it was such hardcore engineering that I was in my office for 18 or 20 hours a day to get it working. Because my goal, I agreed to this project like it had to be done. And that's what it required. And my wife even was like I was complaining to her about it. She was like you signed up for this, like, this is your world. And you said you could do it. So go do it. And you know, come back when you're done. And six months later, I said hello to her. And I was like I'm done.

That's some that's some good encouragement from the wife unit.

It was it was very Yeah, I mean, it was she was really understanding it. And I was at one point complaining like I'm missing. You know, I'm traveling or I'm here and I'm working on this. I'm missing, you know, what our kids are doing. And she's really like, well, you know, you're really not missing much like they're just kind of misbehaving and fighting or whatever. So, but it's still like, it would be nice to be around. And you definitely need to have a balance of engineering, and your family. And normally my balance is running, like I will go running. And then I will work. But this this project got me so into it that I even stopped running, which for people that know me is like whoa, like that's abnormal. It's a little bit weird. But my mind was so focused on this project, I could not like literally could not do anything else. So there I have to work on that, like figuring a balance. I tried to do some meditation. I think that's going to help if I can, like, actually do it and sit with it. But I'm always on I'm always doing something. So it's like I just got to settle down. And future projects need to have balance. And I think people also need to, you know, other people need to probably have that same sort of thing conversation of like, I have to balance it. But it was fascinating to see didn't use and like, I'm happy with how it turned out, I'm super happy with how people used it, and appreciated it and, and loved it and like guy got some great feedback and great comments and like it made me feel really good. And I think made other people feel really good. And that was the intent. You know, just build a community and let people have fun and explore and like, try new things. And it really worked for that.

So based on the talk you gave this year, and you did like release some information early about the badge, which is really unusual about DEF CON, right? Do you think because of that, do you think DEF CON is gonna change their policy regarding digital badges in the future? Because basically, you released this like, information or like how to, like, talk to the server like it was one

shorter like that. Yeah, like the tools you needed.

So when you did that, like everything on Amazon went out of stock with us, right.

So I would say that, I would say that it's there wasn't a policy, it was the choice of the designers. So if you look at besides DEF CON 14, which we did as a complete surprise 15 through 18, when I was working on the projects, I would post on the forums, here are the tools that we're using. We didn't have Twitter, like you know, it was a much smaller distribution channel. But the the tools you needed and the like the main processor, like that was never a secret, because we wanted people to be able to go and get the right the right tools and hack on it because I was also hosting a batch hacking contest where people, we wanted them to modify their badge and do new cool things and not have them waste half their con trying to find the right tools. So there was never a policy about that the policy really has been to keep it secret enough. So you don't spoil the fun for people that stand in line for two days. Not because they have to. Because registration is so smooth. Now. It's ridiculous how amazing it is. But because they want to know they have lion con and they sleep there and talk and have fun. But they liked the surprise of being the first to get the get the badge and look at it. So we don't want to spoil that by by releasing too much. But I always want to release as much as I possibly can. Or at least the tools or at least something to get people going. I even met with with with Leap bunny from oh shoot is that his handle from hardware hacking village, I only know his name. And I think it's leap bunny, that might be somebody else. But anyway, he runs hardware hacking village, sorry, Chris. He's probably gonna listen to this. He runs hardware hacking village and I actually went to go meet with him in person. And with Skyrim, sky sky Ria, his partner and show him the badge. So he was the first person outside of DEFCON to see it, because I wanted him to be prepared of the main functionality and how it worked and the tools you needed and all of that stuff too. So they could get prepared. And apparently, you know, he was like, this is the first time that we've ever known since hardware hacking has existed, hard rocking village of what we need to do to prepare, because they usually get thrown under the bus. And you get surprised at the last minute. But you know, to me, this is stuff that doesn't need to be a secret, it needs to be out there. So people can can prepare and have fun. Because that's the point is having fun, not making everything a secret. But for these for these other years that I wasn't involved, it was just you know, up to the designer of how of what they want to release. And if they want to be more more secret or less, or how hackable the badge was or not. And like, I have no say in that stuff. But as long as I'm involved in as long as there's a hardware component to it, like I'll just release what I feel is a fair enough amount of information to not spoil it. But let people at least be prepared.

And so was there a design reason to not include a should he add on connector?

Yes. So I originally wanted to put one on there. And as I started kind of mocking up the board and prototyping the board, I realized like it just didn't go with the aesthetic design of of of the product, like having an essay Oh connector it was it was going to be on the back of the board. And like that doesn't make any sense. Because the board's gonna you know, the badge is going to be mounted to your chest, the gemstone is going to be outward facing the Yeseo connector is going to be stabbing you. And then if you plug something into it, it would be no one would see it. So I love I love the SAL I love the bachelorette community. And there's, you know, 100 other projects, at least that people did that support SEO? I didn't feel like the this badge had to do that. Because, you know, technically it's not. It's not part of that ecosystem. It's like, you know, this is this is separate from that ecosystem. There is the UART there is the SWD connection, like people could have a way to do some SEO add on if they wanted, but it just didn't fit with with with the intent of the design. Right. And that's what that was the main thing.

And you had the lanyard set up so that you could easily add another badge down to the lanyard. Yeah, you

add another you add another badge either to lanyard or on the lanyard clip if there's other stuff. But I figured Yeah, I was sort of expecting to get a little bit of flack for that. But it just didn't. Like I'm not going to add stuff just because that's what's expected. That's just not my personality. And in this case, it was you know, I tried for it. It just did makes sense. So I took it off. And I was like, alright, you know, I don't want to kill the battery by having an essay on there. And I have these like super low power modes, and I spend a lot of time doing this power efficient stuff. And then you have this, this thing connected to it that's strong, 30 milliamps, and the battery dies in like two hours. You know, there's, it's just, there's so many other badge like badges that that support, it was like, I didn't need to, I didn't need to put it in. It wasn't gonna, you know, make anybody sad that it wasn't there. And it turns out, nobody, nobody complained, like it was totally fine. But I did I should mention on the China badge, that FPC connector also has a shitty add on the shitty out on pins, the power, whatever power ground and I squared C. So I made a little shitty add on adapter, like this little white board with an FPC connector on it that would plug into the side. And then it had the SEO break or the SEO female connector. So if somebody wanted to add a shitty add on to their China badge they could the problem is there was no badge life community in DEF CON China and no stos. So it was sort of like nobody used it. But the capability is there and DEF CON in the US ended up selling some of the extra DEF CON 20 boards. And there were some of the SEO connectors floating around and everything. So like it's it was there. But it turns out like there's there's a use for it. If if it's part of that ecosystem of badges and these just really, these just really weren't, you know, they were just separate to that. Yeah.

So what do you want to see DEF CON 28? Or is that going to soon now? Too soon?

Rest and relaxation?

I think I want to I want to see. Well, I don't want to spoil anything, because I've already had some conversations.

I want to see, I want to see, you know more simplicity and more community involvement. You know, it's like, it was just so cool to see. Especially first time DEF CON attendees. Start this quest and make friends at DEF CON like can you imagine how hard it would be going to a conference of 30,000 people and knowing zero, knowing nobody is bad enough? Like yeah, you know, it's like, yeah, Steven, it's like going to, it's like starting to, you know, go to school for the first time. Like nobody likes being new, right? The new guy or the new girl or whatever. So it gave it gave them something. I think that community involvement like, really helped. So I don't know what what I would really like to see, but I would like to be involved if it's not going to kill me. Okay, so I got I got partially joking. Partially serious. Yeah.

Yeah, I got I actually have two questions left. Sure. Who's a better engineer Matt Damon from the Martian or Matt Damon from Interstellar? If you've seen those hands,

so I haven't seen either one. But what about like, Matt Damon from Goodwill Hunting? Is that a valid answer?

He's a mathematician and good

mathematician. Like, okay, I don't know. Like he knew he knew his shit. And, and I'm from Boston, so I liked that movie.

I want to say that's a valid answer. Yeah. Okay,

good. Valid, at least that Matt Damon and I haven't seen those other two movies, or should I watch them? Like, is that about your shirts?

Watch them.

I would personally say watch The Martian. Maybe not an interstellar.

Okay. Yeah. All right, then. I'll do that this week while I'm relaxing.

Yes. And Joe, where can our listeners find out more about you? So

my main kind of public facing outpost is Twitter. So I'm at Joe grande on Twitter JOEG, r a n d, like $1,000. Not like a big burrito. I don't post often on Twitter. But it's basically like, Hey, here's your like, the documentation is released for a project or here's this or here's that. I don't really communicate much on it. at DEF CON, I did. Like that's the rare thing. Like I would respond to tweets. And I would post a lot more than I normally do. I really use it as like a low traffic way to share kind of what I'm working on, or what's been released. But that's definitely the best way. My website is grand ideas studio.com that has a list of like my currently scheduled events where I'm going to be in public. It has news articles, it has, of course, all my what I call portfolio items. So all the projects I've worked on the products, details of those if they're open source, there's going to be lots of information about the products, the DEF CON, badges are on there. There is a little search engine on there. So you could just search DEFCON China DEF CON 27, or go to the the miscellaneous portfolio section and look there. But I tried to put everything that I possibly can on that site. It's not the easiest to navigate. I'm actually working on an update to that. That eventually will come out to make it a little bit easier to kind of find the projects and find the documentation. But yeah, those are the two ways you know Twitter and website. There is a contact form on My website. So if you do want to contact me outside of Twitter is probably recommended. Like I don't have Twitter on my phone. So go to the contact page, send me an email through that. And then I'll respond. I do try to respond to everybody's email at least once. Sometimes they get a little out of hand and I stopped responding. But it's not because I don't like you. It's just because more emails come in. And it might take a month or two, like I have some emails that are like four months old, because they're not super urgent, I will eventually respond to everything. It just you know, life, and badgers get in the way.

So, thank you so much, Joe, for coming on our podcast and taking some time from your r&r to come talk to us.

Well, I appreciate I appreciate you having me on again. And thanks to everybody who's still listening almost what an hour and a half in. Yeah, you know, it's it's a great kind of community that we're in and people are doing a lot of really cool stuff and the resources are out there and like it's just such a fun time to be in this world. You know, and like to see what people are doing and, and pushing the limits of stuff like it's exciting not just from a badge creator point of view, but just from a connoisseur of this stuff, you know, and from an actual engineer like it's, it's pretty cool. So yeah, thank thanks to everyone and thanks to you guys.

And you want to sign us out Joe? Oh, yeah.

Okay, I'll sign you out. That was the macro fab engineering podcast. I am or I was your guest Joe Grande.

And we are your hosts Parker Dolman.

And Steven Craig.

Lady everyone take it easy. Bye Thank you. Yes, you our listener for downloading our show. If you have a cool idea, project or topic, let Steven and I know Tweet us at Mac fab at Longhorn engineer with no O's or at analog EMG or emails at podcasts at macro lab.com. Also, check out our Slack channel. If you're not subscribed to the podcast yet, click that subscribe button. That way you get the latest episode right when it releases and please review us wherever you listen. As it helps the show stay visible on helps new listeners find us