Espress-ify? Designing Products Around the ESP-32 Platform.

Welcome to the macro fab spooky engineering podcast. We your guests Hi Ron and zap and we are your hosts crab foam and Blitz.

This is episode 144.

So, you guys have probably heard of our guests before, but we'll do a recap. Zep is the chief bleeding officer at and not XOR. During his limited free time he writes vulnerable C code dabbles in key CAD and trolls Arduino bot accounts on Twitter. He has never been seen at the same time and places Batman, but then again, neither has Blitz. Heilbron is a button pusher it's easily replaced by 1000 monkeys with 1000 laptops, but manages to crank out firmware and hacker puzzles. If you're reading this statement, he is already routed your computer.

We have had zap and Chiron on the podcast before check out map episode 69 incognito mode and map episode 109 Arduino the gateway drug to hashtag badge life, which we also had like eight bitstream. And who else was on it? That stream I think it just bit stream. Yeah, stream it was only 333 of y'all. Yep.

I thought, Man that now this is getting all mixed up. I thought when this was the fourth time you guys are on this. Like it's only been three times it feels

like we're on here every week.

You get you guys are now tied for the most times on as a guest. So we'll have to have you again to earn the title.

Yeah, take that Joe Grand.

Oh, we should build we should build a title belt for the most times you've been on the podcast. Like, like a WWE belt, but it's like a big PCB.

The most uncomfortable belt? Yeah.

Do it to wear all three shirts at the same time now. Is that how this works?

Yeah, at the same time,

okay. We'll just make sure it happens in the summer. So it's incredibly uncomfortable

in Houston, of course.

So, at the beginning, the podcast, you heard Hieronymus Thurman that he built.

Oh, that so? Nothing amazing behind that. Other than, you know, sometimes you you have thought strike you and you go go to Google, and you're like, what is that spooky noise I hear in movies. And yeah, it turned into me going on Google starting to learn about what the hell is a theremin. And it was actually pretty cool when I looked into it. So if the ears or the sound people like blitz laugh at me, I had no idea how they worked before. And it was kind of cool to learn like, so if you ever seen those, it's a wooden box of two antennas coming out beside and people usually wave their hands next to each antenna to control the pitch and the volume. The way it's working is those are dielectrics, basically, and it doesn't have both plates. So basically, your hand in the air is making a full capacitor. So as you move your hand closer or further away from the antenna, it's affecting the capacitance of the system. So you control the pitch of the antenna. But it oscillates to Flass. So they heterodyne it with a different crystal in there to try to lower the frequency down. So it's kind of like you have to have capacitors on there to wave your hands back and forth. But what was more interesting to me was I'm looking this up and Leon Theramin, who made it, he called it an ether phone. But a cool hacker history. Especially because we we run in the DEF CON crowds and deal with a lot of security stuff. I did not know that that device that music instrument was used to make the first electronic bug. And he made this like back in the 20s. And he went back to Russia just right before world war two ended and the USSR kidnaps him, puts them in the gulag, and they force him to develop stuff. And they tell him they want spy gear. And so he takes a lot of the technology he invented doing the Theramin. And he takes like the small capacitive membrane connected to an antenna and they hide it inside this wooden engraving and they give it to the ambassador in Moscow and it just kind of sits behind their desk. But since it has this small membrane and like a quarter wavelength antenna that's kind of tuned they would illuminate it with a microwave frequency. So about 100 miles away, they could hit it with 330 megahertz and it basically is like a plated antenna with no dielectric so it takes the vibrations microwave alters the vibrations it reflects back and you have a passive microphone that's working from about 100 miles away. Which is crazy because that's like 19 46 And he essentially invented like the precursor to RFID. All from this weird electronic instrument that turned into a hacker bug, but they're beaming

it with a microwave, from how far away a mile away.

I know they set up to 100 miles

100 miles away. So is is either really focused or they're just basically cooking the guy in his office. Oh,

I think it was pretty focused. But it got discovered by British, like amateur radio hobbyists, because they were blasting it so much. They were on different wavelengths, I think 800 megahertz or 1200. And they were picking it up and hearing it over FM. So are over am so it wasn't like they were hiding it too hard. It's just some British hobbyists were like, I can hear some Americans talking about political secret things. And so you know, they contacted EMI five and whatnot. And they realized that their office had been bugged for like seven years. Hmm. So interesting, kind of cool, where it turned in from what's the spooky electronic sounding instrument to Holy shit, this guy was forced to make spy gear out of his electronic instrument. And it turned into bugs and spine for seven years. And I think they have a replica that at one of the NSA museums or cryptography or there's like the whole spy wing of the NSA museum, you can check that stuff out.

The NSA has it easy because we just carry microphones with us everywhere we go. Yeah. And zaps a description that he wrote himself today. What are the Arduino bot accounts?

Yeah, so one thing I figured out when I was getting frustrated the Arduino which I talked about the last podcast, when you bashed Arduino on Twitter, all these Arduino bots start retreating you. They're just set up based on keywords. And so you're just sitting there like how, you know bash on how bad the IDE is, or whatever, not using rod Arduino. It's a great starter toolkit, but then they just start retweeting me, so I used to troll them with purposely putting out bad tweets just to see how far they would amplify it.

It's so unnecessary. Oh, you know what? I got to give them credit that Theramin it's open theorem, and it's an Arduino hat. Oh, cool.

Well, where can you Where can you get it at?

WWE? Now I put the I put the link in your, in the show notes for you. But yeah, it's a lab that makes a open Theramin hat kind of the cheapest route you can go as opposed to buying a real one. Maybe I should tweet about it and see how many times I can get Arduino bots to retweet me.

There you go. I noticed that the are one of our podcast competitors, the amp power. They have they have a subreddit on on Reddit. And if you post a link there, the Twitter account for the amp power role just just scrapes it and puts it on Twitter. And I've used that a couple times to actually just like throwing up like our blog links for the podcast and see it. We'll see how long I'll take to Chris camel, like, figures out what I'm doing.

Probably not very long, I would guess.

Alright, cool. So let's do a quick recap of DEF CON 26 in the newest and not XOR badge before going on?

Yeah, so for for 26. He probably saw it all over Twitter. If you follow the DEF CON stuff. We came back with the western theme, and is a Western we doubled the number of LEDs. We put on our own scripting language called lols code. And that's all fully documented online. We also had a puzzle game that Hiren worked tirelessly on lots and lots of hours on that it was to teach people some basic hardware hacking skills people a lot of fun with that. Also brought back our botnet and

see what else we did anyone get control of your botnet. This time?

No. One of our major issues is here was for the the sheriff's are on benders hat that had to be a ground plane. And the ESP 32 is on the opposite side of that. So they actually cause a lot of interference with the RF. So the range we're getting was on the order of about 10 to 20 feet, the year before we were getting 100 200 feet in some cases. So it didn't really get taken over. I put a lot of time in encrypting the botnet as well. So the two of those two factors kept people out of it. Which was unfortunate because we actually enjoyed it when people took over the button and I know it's a weird

push and pull because before we left a little open, we got stressed out but you're right. It's a fun game to try to take control back and then we're like alright, we're going to secure this and see what happens and that's just boring.

Need to loosen it up a bit. You should you should heavily secure it but then just like The Encryption Password is this like, password or something.

That's, that's actually the kind of things we do. So funny story. And actually, macro fab would like this. So we built in a way to do over the air updates and from Raspberry Pi. So they get within range of Raspberry Pi detected. A, there's a new firmware update available for me, connect to it over Wi Fi, pull down the the firmware, and then flash itself. And then basically the flashing itself was reading from the SD card. Because we couldn't do the two, the two images in Flash sort of thing. Well, we encrypted it. The problem was when I did the encryption, originally, I used the key of 11223344, and so on. And I had that in our bootloader and then shipped it off to you guys when you did the production. So all the boot loaders, I set up that super secure private key, and then I changed the private key after the fact. And then when we started getting these things in, none of them were flashing with our firmware. And that was why so people could actually override with their own firmware if they knew the key. So it's kind of unfortunate, but little funny story.

So So I'm curious about the badge. You know, I have actually received one, gosh, a handful of months ago. And thank you for sending that to me guys. And I played around with it for a while actually I took it to work and gave it to a handful of other people and had them kind of run through it. And just kind of handling it for a few minutes. You realize that there's a lot of stuff on it and stuff might just be the right descriptor there. Just all kinds of stuff. That what I'm what I'm curious about is have you guys, or do you guys know if everything has been unlocked by at least one person? I mean, hey, are there still secrets? Lying in that bag that no one has found?

There? There is always secrets. We put things pretty deep in there. Reference references or secret things that mean something to us. We do you know that somebody was able to unlock all 16 unlocks a guy I think George callow. Yeah, he has a Curious George icon on Twitter. He's actually from San Diego. But yeah, he unlocked everything I asked him in person, did you do it with you know, after the source code was released? He said no. So I was pretty impressive, because one of them was modifying an intentionally broken lols code file. And he was able to solve that. We do know, one guy has also solved the puzzle, hardware puzzle before the source code was released as well.

So Steven, also in that badge, I don't know if you've played around with it too much. But there's a text adventure in that,

I don't think I don't think I got to and you know, I accidentally left it at work. I was gonna bring it and have you guys walk me through some stuff on the podcast, but I'm done. And I left it at work.

Well, I'll be going through some of that at Hackaday this weekend. But definitely plug it in. I mean, if if you open up mini comm or putty,

you log it? Well, yeah. And you'll

think of like Zork on our call, okay. It's, and by the way, when I showed it to Ben, we're back there. Ben. Hank was like, why don't you have NFC or W because I was having that type walk north walk south walkies walk west. So I put that in there, make sure I was compliant with the old school type games, I completely forgot. But you basically have like this console based RPG. But instead of solving or playing a game that's completely you know, text based or contextual. It's extended all the badge hardware. So in one area, you know, joking around, I am, by the way, I asked all them ahead of time, I could use their names in vain, like I have Joe Grand running on a treadmill to power something. And then you have to crack a PIN pad on there, which I got from a lesson Joe was giving on side channel analysis. So for someone to go through that they have to realize, when you type in a pin, if you type a wrong digit, it's actually going to complete the cycle faster, because it's going to break the loop or it's checking your password. So by doing that, a normal person won't be able to see it. But if you hook up a logic analyzer or no scope, you can actually look at the clock line or look at the data and see that as you get closer, the delay gets longer as it takes time for something to complete. And I guess the whole point with these puzzles, which is really awesome is I had dozens of people like I left my DMs open people are emailing me messaging me asking for hence but I had people tell me like I've never used a logic analyzer before, or I've never used an otoscope before I went to the hardware hacking village, someone taught me or even someone said, Hey, I went got a bus pirate and they still work as a really shitty logic analyzer, but they work. And they're like, Hey, I was able to like actually look at the I squared C line and see what's going on and it's not so scary. Thanks for you know, give me an excuse to learn something new. And I Basically, that's kind of what every puzzle is like we had a hardware hacking a wireless, a cryptography puzzle, packet capture stuff. But I think that's what we try to do where the puzzles aren't necessarily hard. But it is something where someone could learn a new skill set, and it may force them to go, you know, in the one of the villages, talk to someone new, make some friends and just kind of expand their their life toolbox of things they know how to do so. And hell, the more people that understand hardware, the better. Right,

exactly. That's what this podcast is all about?

Sure. So just for those who don't know, can you explain what the the villages are? Think of it like,

you know, you go to a conference, there's big talks going on. Villages are more focused areas on certain skill sets. So like, if you were to go to the hardware hacking village, everyone is going to be very focused on soldering and electronics. How would you dump firmware? How would you extract flash from an EEPROM, or serial flash, whereas like, if you were to go to the packet capture village, people are really into teaching folks how to use wireless and Wireshark, and how to do Packet Analysis and decrypt packets that you may have caught on the network. So very different skill sets and focused areas. But what happens at those kind of conferences is people stick to what they know. So you know, you'll go to DEF CON, and maybe never even stepped foot into another village because it's like, I don't know anything about wireless, or I don't know anything about hardware. So to complete the puzzle, I tried to make sure that there were four completely different distinct types of challenges, where if you didn't know what you were doing, you would have to go to these different areas to kind of get, you know, learned up on and I just encourage people, you know, bring a six pack of beer, give someone a beer, make a friend say teach me something, and I'll do what I can.

That's a really good philosophy for the hardware and software. puzzles,

like bringing a six pack of beer, that and

though and making them something that different genres, so to speak of engineering and tech philosophy, or what's a good word for it?

Just disciplines principle is that's what it's thick enough. Yeah.

Yeah. So yes, so we talked about it all the puzzles, what's the engineering process behind like, the hardware and software design?

Of? Yeah, so if you saw our Hackaday talk last year, or Cypher CON talk, we follow a basic system engineering process where we, we can't we come together as a group, usually over a couple months, this time, we actually started back in back in June, I think for for DEF CON 27. But just collecting all these ideas, and then taking those ideas and figuring out okay, so what are some commonality? What sort of hardware do they need? Going from that that hardware figuring out? Well, this idea, it's the only thing that needs a speaker, right? Is it really worth placing a speaker and all the associated components that go with it, and all of the peripherals are required on a microcontroller to do that one idea. And so we may go through and make the hard decision of No, we're not going to do it. And so that's kind of the process we're in now of, of necking down that to kind of a, a high level bill of materials, and then we can go into prototyping. So you take all that kind of, throw it into one big spreadsheet, figure out what your bill of materials is, is it realistic? You know, with respect to your budget? Do you have sponsors? Or do you think people are willing to pay for this? Does it fit in your design? How do you manufacture it, and those sorts of things at the same time, hiring or myself are in the background, also writing firmware and drivers and things and starting to kind of pull the whole system together. That's basically a talk that we keep at Hackaday in about two minutes.

But then there's like iterative testing, because we have a couple of people working on stuff, you know, we're gonna step on each other's toes and break things at times. So after we get past our first prototype, and we know what we're doing, and we have like a first type of board, not just like a hat that sits on a dev board. It's usually Hey, someone's working on something cool. I'm working on something, we merge our Git repository and then a quick regression test every week or two to figure out something then you go, Oh, shit, you broke something. And it's like, no, you broke something. And I'm like, Alright, one of us broke something and this doesn't work anymore. Now let's, you know, iteratively go through and make make stuff break it reengineer reintegrate it and keep doing that until we hit. I think for us about April May timeframe. Like kind of our operational test, we go to a conference called layer one and Pasadena or LA in general that's held every day. And we've tried to stop ourselves from adding new features anytime past May. And then pull out all the bug fixes for the next two months until we release.

So you stop you stop feature creep in in April. Yeah,

it's hard. But we usually create a spreadsheet. And that's where all the ideas start getting dumped in its work where it's like, oh, wouldn't it be cool? If all you know what, we'll throw it in the spreadsheet? We'll start working on it for the next year. Yep.

Well, and it sounds like you guys are starting early or an earlier for each year now. Right? Is there? Is there overlap between badges?

Yes, so the first year was about a nine month project nine to 10 months, maybe. The second year was a 11 to 12 month project, and this year is 13. So we were writing ideas, and researching, I was researching microcontrollers while we were delivering and fixing last year's badges. So I don't know if we can sustain this, you know, longer and longer cycle times. But

it's fun, though,

it is eventually going to start to badges at the same time.

We did take a quick break. We we we had a visit out to Derby con in Louisville, Kentucky about a month ago, beginning in October. And we made 50 little cockroach badges for Trevor the cockroach grifter and we just hand soldered them made 50 ones like a little at Tiny 817 on there. And that was kind of fun to have something with only like 8k of RAM.

No 500 bytes of RAM.

bytes of RAM. It's nice having that limitation. So like, alright, I can write some interrupts do a little embedded console and just hand them out to people saying hack this and you win it.

So one of the really interesting design things about badges is a lot of groups want to put a display on there. It's actually a huge driver of the overall bill of materials. Right? So if you want to put a large, like an IRA 9341, that's very popular, you can get them on eBay, China, whatever. For those are very large screens. They, it takes a lot of memory just to drive one of those effectively. And so that's going to drive all sorts of stuff in your design as far as how fast your spy bus is, how much RAM you need the microcontroller.

How much software you need to write for it,

how much software you'd write for it. Yeah, do how many Adafruit libraries are gonna go borrow to go interface with this thing?

You know, it's actually it's actually funny that you bring that up is it's the same thing in the pinball world where, when we were running, you know, a first year running like character displays. For the first like, 10 years. This is like digital pinball machines. And then you go and Adafruit pinball library, not maybe not yet. But then they were to 16 segments, alphanumeric displays. Oh, that's, that's fancy, we're going over seven segment, right. So you basically have to double the amount of RAM, you have to address basically the double amount of segments. And then dot matrix happen dot matrix at first was like four colors, or four shades, and 128 by 32 pixels. And now we're at HD. And it's not so much RAM as the problem or because because now like RAM, and storage is cheap, like, SD cards are really cheap. The problem is making the content before you just had text scroll on the screen. And then after that you had like, you know, low resolution pixel art that anyone can make, but now yet doing full HD graphics that you have to like, pay someone to draw. You can't just have an engineer do the art.

Well, you're gonna go on Fiverr.

Yeah, yeah, Fiverr.

We did just for perspective, or last year's bed was 220 by 176. And we use 16 bits per pixel. And that was about 77 kilobytes per frame. Right. So if I want to fully buffer that, that's a lot of RAM in the ESP 32. And we'll get into that later, I think, didn't have that much available for me. But our first year, we use an STM 32 F 103. That only 20k of RAM. So right away, I'm already more than three times my initial RAM budget for my first badge. So yeah, those displays all the way up and down the bill materials, they drive almost everything. You know,

you didn't bring it up earlier, or you didn't explicitly say it but when we said we're we go through like listing all our ideas and everything we want to do. In the spreadsheet, we're still pulling out technical specs and details from like all the potential chips and peripherals. Because we've had people come up to us say, you know, the hold out of the figuratively holding out a chip in their hand, say I want to make a badge or I want to make something and use this and they already have I want to use an ESP or I want to use, you know, an STM 32. And I look at that from an engineering perspective and go No, don't, don't do that, figure out what you want to do. And pick the right piece of hardware that's going to support it. You know, don't don't pre guess, or preset yourself on some MCU or some whatever. Because until you figure out all the different peripheral requirements, you could quickly shoehorn yourself and find out, I'm out of RAM, or BIM, like spy doesn't have enough, you know, bandwidth support where

you need more ADCs? Or you needed a DMA bus or something like that. Yeah. Well, but

but I think I think you're kind of hitting some on the head there that kind of separates different different groups of thoughts there where it's, yeah, I might, you know, someone might pick STM 32, because there is hordes of support behind that, as opposed to the wild wild west of having to read a datasheet and figure out, you know, something else, you know,

but learning is fun, right?

To an extent to an extent, well, yeah. So

for us, this is not really, I mean, this is a hobby. So it's not really a money making venture. So I I try to maximize non recurring engineering, that makes any sense, right. So I think Parker, in a previous podcast, you mentioned that there's a question about this in the Slack channel, that hey, do you use what's familiar with you? Or do you go and try to do something different? I'm always on to, hey, what can I find that's different. That's why I went with ESP 32. It was an arm, we done arm two years in a row, had some interesting experiences, but it worked. So that that's kind of our goal, because we're trying to learn just as much as we're trying to teach. And if it's not, if we're not learning something new, we're not going to keep doing this.

Now, that totally makes sense. And I actually like the idea of basically going, cuz I've actually never designed a product like this, which is it's kind of open ended at the beginning. It's like, it needs to be a badge and look pretty, right? And blink. Like that's the initial overall arch of the product, right? Whereas what else it does, it doesn't really matter. I guess, it's like it has, like, you're basically picking features, and then figuring out okay, like with a Venn diagram, basically in like, what's as the most amount of features, we can capture in the least amount of hardware, and go from there, which is a completely different interesting way of doing product development, right.

And it's fun, because then creativity and all those things where you're like, Man, I wish I had an excuse to do fill in the blank. It's a perfect project to do stuff like that, if we can, you know, reasonably tie everything together, where stuff and be like, are like last year, I don't know anything about Bluetooth, right? Let's figure out how Bluetooth works. I've never touched that before. And, and just, you know, repeat that kind of thought process over and over.

Yeah. So wouldn't it be cool if engineering,

right? Exactly. Yeah, cuz our day jobs, we don't really get a lot of choice in that sort of things. Right. It's the same thing, day to day to day, delivering products that, you know, have decade lifespans. So that's that's not quite as interesting as let me go, go mess with something that's brand new that I keep seeing on Hackaday. Or on Twitter. We go play with it and see what happens.

Yeah, it is kind of weird, too. Because like you bring up usually it's, we have customers, customers want this, go make this for your customers and what this project we're doing, it's like, no, we're doing this for ourselves. And if people want them, they can come get them. That's great. But initially, we're doing this for us to learn what we can learn. And so far, people seem to like it, and then it seems to work out all right. And we can have fun doing it.

So one of the first new things y'all use on this year's badge was the ESP 32. So how was that experience?

It was hit or miss.

Let's list the pros, the pros of the ESP. Okay, so

the pros, though you get a whole lot of specs for nothing. I mean, next and I think $4 For a ESP 32 rover, you have four megabytes of RAM, and for four and a half megabytes of RAM in two. That's why y'all pick picked it right? Yes. And actually, so going back to the screen, right? So if I needed 77 kilobytes of RAM per frame, if I want to fully buffer this, then do animations with the menu. And then do recall at 30 frames per second. I need a lot of RAM. So that's why we

got that was actually on your spec sheet 30 FPS Rick Roll. Yes.

back of the envelope design constraint that we throw up front, so that way we know hey, I need this much memory to do this.

Yeah, we had 1.2 gigabytes of data on the SD cards. And 303 megabytes of that was the Rickroll video. Priorities. Yeah, priorities. And that was also the first thing we tested once, you know once I had the display driver running, you know, how do we how fast can we really push this? The ESP 32 has a really nice spy bus, I think it would go up to 80 megahertz. The year before we did a Nordic NRF 52. That was only eight megahertz. So we were severely limited. Here, we could push video as fast as we could as fast as you wanted. It also had a four bit wide stdio peripheral to the SD card, so we could pull data off the SD card extremely fast. And that's how

it actually was using the four bit SD card interface.

Yeah. So if we want to switch over to the cons of the conversation

I'll be fair, yeah, yeah, go for that. Well, yes,

fourth. So um, so SD cards, if you follow us on Twitter at all, you'll see around maybe mid July when our Kickstarter start getting their badges that we're we got a lot of issues with the SD cards. Once thing, when you buy SD cards, there's a high probability that they're counterfeit, especially if you're buying them in high quantities, and you're looking for the cheapest cost. We had an issue there. So we had a lot that were not meeting their specs, they're just bad from the get go. They worked, we tested them and shipped but then they, you know failed soon after that the four bit mode, there are some issues with some crosstalk or shorts or something that was occurring within the ESP 32 module. Because I think the high ROI HS reflow may have caused caused some issues within the module, which I mean, it is what it is we ended up having replaced some of the modules and fixing that. We were also over specking it were a lot of turns out SD cards can only really go about 26 megahertz, we were pulling it about 40. So we were just pushing things a lot harder than they should ended up. At the end of the day, we ended up dropping to one bit mode, running it 10 megahertz and ignoring the the car detect pin, that was the other one because there are some issues with some with that shorting to ground and not detecting the SD card. In the end it worked. But we had a lot of issues with corruption bet if the badge would make contact with another sometimes it ground out was doing the right which it did every 10 seconds. And that would corrupt the file system. Its overall SD card experience was really, really bad.

You're gonna move over to like spinner disks, right?

Yeah, we're gonna put 120 megabyte hard drives magnetic hard drives on the bed just next year, so they're gonna weigh about 20 pounds. And be great.

Actually, I wonder if you can still buy the hard drives they put in the gen one iPods, little spinners in there.

Oh, those things? Those things are great. I think those are really like two and a half inches or one and a half inches are pretty small. Yeah, one and a half.

Yeah. Tough hard drives. But anyway, I mean, you're bringing up espresso, and you're talking about the SD cards. Just because something allows you to do it, it doesn't mean you should do it. And we implemented it, we are cranking it fast. Because the ESP IDF allows you to do that it tells you here's your limits, only dim when we start getting failures and read into it realize, oh, and you shouldn't do that to an SD card, even though you're allowed to do it. I mean, yeah, and like somewhere like how you brought the STM 32. I mean, sometimes you want to pick something because a lot of people are using it. There's a lot of examples. There's a huge community behind it. And that was nice. So that's a pro in that respect. But we started really realizing that it is a way to it's like a it's like the new kind of Arduino. It's a it's an intermediate level for Arduino, where you can produce some prototype something on ESP 32 board really quickly and really easily. Their SDK has tons of examples. And you can do stuff really fast. But it's like it's made as a plugged in IoT device. Power Management, for example, which I'll kick back over to zap you know, until the drivers or until the low power mode started working, how much power were we drawing just using Bluetooth and the regular, you know, system and package and everything. So I measured

while I was running bling, he was running about 175 milliamps at I think at 2.8 volts in which is phenomenal for two double A's, which was about three to four times what we were pulling the year before we pride ourselves on on battery life that we can provide and that was only maybe about 16 hours of battery life on the two batteries. So yeah,

I mean it can consider you can run a Raspberry Pi W headless at 120 milliamps. So to have that thing cooking at 170 is is nuts.

Yeah. And they saw what they added they added a thing called modem sleep to the the Bluetooth modem they added in late May so after we'd already frozen and so on course that never introduces bugs, because it totally did. That got us down to about 120 milliamps. And then, right before we shipped, I got us down about 96 and was able to fix some bugs. But we'd have issues where we would turn that on. And then suddenly, the console game would crash. We're like, what's going on? Why is this crashing? Well, turns out that the console game was using the UART, peripheral, too long, and it was crashing the Bluetooth modem in their their little binary blob for the modem. And you had no idea. They didn't give you a stack trace because it was just a blob. So no idea where that crash was occurring, or why. And we finally figured out if you just hold down the A key and the console game for like 30 seconds, it would crash the modem, it was just this bizarre interaction. So as hiring said, where they're really good for one thing, like, here's an IoT device that downloads a separate Wi Fi and turns on a light switch or light, that's fine. But when you're doing console, and you're doing Bluetooth, and you're managing this botnet, and you're running LEDs and the display all at the same time, there's all sorts of interesting race conditions interactions that occur there that you didn't expect.

Yeah, it sounds like it's not been vetted really well.

So yeah. So So I mentioned that I the idea of is being updated constantly, they also don't document a lot of the details of the hardware that well, like the way that they allocate memory and DMA is really, really poor. They only you have four megabytes on rover. But really only 500k of that is, is natively accessible, and only about 80k of that 500k is available to DMA, which is what all the peripherals need. Well, if you use a malloc, which almost everything the system does, it allocates the DMA memory first. So you start getting these crazy crashes where you suddenly can't write to the display because you started using an extra 200 Bytes on your stack in your console game, which was another error we had. It just, it'd be nice to get those application notes you see, you know, ti St. Nordic all the all the major vendors that give you these applications, and show you the architectural details behind their their hardware we just didn't get that was expressive, and understood the way that they the way they allocated memory or the way that they prioritize peripherals.

Now one of one of the craziest things that drove me nuts, you were bringing up the UART peripheral. I mean, if we haven't said this already, expressive like the ESP 32. It is a system in package. It's not a system on chip, it's not a single die. So you peel off that metal frame. And you have maybe five or six different manufacturers that supply peripherals to express if they solder it all together and put a cap on it. There you are, and their Wi Fi. Excuse me not there, you're there. Apologize, there are Wi Fi is made by someone else. So when I was doing the Wi Fi, when I was doing the puzzle game, and I had a Wi Fi challenge in one of the puzzles. Normally it would color code log messages, you know, red or green, depending on the level of log. It didn't catch my that they were gray. And so when I finally turned off logging, because what the last thing you want someone to see when they're doing like an embedded challenge at a command line is all that's going on in the background. It was saying oh, the website it was connecting to all the different keys it was doing and I I could not for the life of me figure out what the heck was going on. There. A subcontractor just dumps all their messages to the UART line. They don't use a logging API, they just print F everything. In the SDK was literally running a bash script and this is under cm, you could check it out, it's running a bash script that takes their Wi Fi module and reroutes all the log messages to No, which is probably the dirtiest thing you could do in programming. In a recent fix, they finally updated the firmware for the Wi Fi so it doesn't do that anymore. But you know, they're even though they're open source, the firmware for these peripherals is closed source. You don't get it, you can't do anything. And so I'm sitting there like really you want me to just reroute the print offs to know and waste memory and do all this kind of stuff. Well, that also made me think I want to find off the shelf stuff running expressive and just crack it open and you know, run a serial or USB to UART adapter on other lines because you'll see everything that's going on in there because there was literally no way to turn it off unless you ran a manual bash script and edited all your header files.

Now is that stuff? Is that under the can though or is it Are those your lines come out to the outside of the module? Oh,

they're they're exposed on the pins.

So you can easily hack a Xpress device then. Oh, yeah.

Those lines are just being clogged all the time with your

information. When it when they when they hit a 3.0 there SDK, they finally resolved it with an updated firmware. But for everything for the past two years, it's just been dumping it. Like if you were running the if you had any kind of logging on, it would print it straight to the UART line, which is great. What if you're trying to make a maintenance console on your product? Like you don't want to see that stuff on there?

Well, this is the thing is when well, OEM OEMs won't update their, their files. So, so if you designed it, if you designed it within the last two years, it's going to still have that issue.

Oh, good. They are updating IoT devices. Oh, they're probably already patched.

So it's the one that here's the, here's the craziest thing that they did. And I was so happy at first, when I saw you know, they actually tried to address security. So on most MCU is, you know, usually have a security bit where you can set you know, read back protections 01 Or two. So either prevent it completely, or

you mean, I have no idea where you're sitting Bitsa to man. Damn you.

Crazy Russian Tertiaries call those.

But you know what I mean, we can set readout protection to prevent people from dumping your firmware, looking for strings, things like that. They actually have secure boot and flash encryption built in. So like WhatsApp was talking about, we could put keys on there. So someone can't shim a bootloader in there, or they can't put firmware on there that unless it's trusted. And even the Flash has a aes 256 encryption where unless you have like a chip whisper call and O'Flynn and you want a side channel crack at I mean, that's more security than you see on most IoT devices, except for the fact that they only let you flash four times. So you're given all this consideration to security, you do the initial flash. And if you're are actually security conscious, you're going to provide patch updates and security updates. And that means your product only gets three patches in its lifecycle. And then it's toast. It will not let you flash anymore after that. So that's that's a weird. Yeah, it has thrown it out for fuses. And every time you flash, it burns one of them.

Yeah, I'm not sure what the physical limitation is to that. It's an odd design constraint to me. That's actually why we didn't use it.

So yeah, we were getting ready to do that. So you know, you're doing a puzzle game, you don't want people dumping the flash off your badge and just reading the strings and cheating. And we're like, oh, well encrypted, and you go, Oh, he should I think I copied and pasted in here. It actually says, After the fourth encryption, it is disabled. The fuse has the maximum value and it's permanently disabled. And that's an espresso documentation. Hmm, that's just weird. Yeah. It's weird to see someone put that much effort into security and then say, but you're only limited to four updates on the chip, and then you're done. So, yep, there's IoT security.

Unknown. The thing is, it's what does that preventing someone from doing in the field?

Yeah, it's really only the Rebbeck protection you should care about if somebody overwrites it with their own firmware? Who cares? Right? They broke their device. Yeah, I don't have any of your IP.

I think their consideration was, it's, it's become pretty trivial to trick reback protection. I mean, if you really want to, you can decap a chip and flash the right bit or you can do like a row hammer attack and flip the bit and and still pull something out. They you can do that to STM 32 years pretty easily. But I think instead of using read back protection, they went the route of well, we're just going to encrypt everything on the flash. And who knows why there's a fuse, or four fuses included versus

what I'm wondering is why even put that in there? Someone thought that was a good idea to implement.

Yeah. I think what it comes down to is I am happy I learned about a you know, extends a architecture and ESP 32. There's a lot of like, juicy RAM and storage and all these peripherals on it, but it's because it's not an efficient architecture or SDK, so they got to throw all those RAM and processing at it. So it can handle all that and it's not exactly meant as a wearable. I mean, it cooks a lot of power. So I think we may be going back to arm. Arm is a good architecture for portable devices.

Is it RISC? V or RISC? Five?

I think it's RISC five. Like we keep wanting to say RISC V but yeah, I think it's RISC five.

I would like risk the

Yeah, like risky as well. We're gonna we're gonna start that trend on this podcast. So it's not risky. It's not risky. So So

you guys have already migrated over right? When you say you think you might you already have,

yeah, we're not using espresso again. But who knows if we're gonna go arm or recipe, we're, I think we're down selecting the processor at this point. Yeah, we've

already made one microcontroller change to something a lot more supported. So I mentioned, it's nice having all those application notes. I've got those on next year's microcontroller, so I feel a lot more comfortable. But yeah, definitely off of extends.

And so I guess before we end this podcast, at this Friday, y'all at the Hackaday Supercon, right.

Oh, yeah, that's right. So Friday,

Friday at 330. At headquarters.

Yep. So they're starting Supercon earlier, usually it starts on Saturday. I usually, I mean, this may be the fourth or fifth one, but it's usually Saturday and Sunday. And now they have some talks that start at noon, at their headquarters, and then they have a little party after that. And then they roll into the main conference Saturday and Sunday. So we get to get our talk out of the way. We get to have a little party a little free food, free drinks and be great. And one thing so the talk is on the hardware puzzles that Hirens mentioned. It's also on lolz code. If you haven't heard of Lowe's code, you should google it, L U. L. Xie, CO. D. But that was our little scripting language, we embedded in the badge this year. Basically, it's an extension of LoL code.

You don't need to know C to make your hardware work. You can talk in cat speak, as

I say you can talk to me. Yeah.

So you can write things like is left? Oh, really? Yeah, really. GTFO bed Kitteh OYC. That's got to read out loud. But I spent 10 months writing that writing lols code and extending the language and in writing one of the scripts that actually drove the UI on the badge. So that was a lot of fun. But yeah, the talks gonna be on that. So

that's actually a fun way to troll people too. Because if they wanted to hack the badge and change how it works, you know, sometimes people use micro Python and put it on an SD. All of our source code was read off the SD card in real time from those files. But I wish I could have seen people's faces when they open up the files and go is this

now you got to do it in in either ASCII symbols or in emote emoticons.

Oh, man, we could use the toymakers tis but a scratch but in emoticons. Yes, he passes. De bez.

That's great. So so so we are actually what 1010 months away now from DEF CON. 27. Yeah, I think it usually happens in August, right? August. Oh,

I can feel the crunch already.

Yeah, we're less than 300 days away. But who's counting?

So we would we would love to be the the first with some some juicy secrets on what's happening in DEF CON 27. You want to share anything?

Yeah, we're gonna bring a vegetable form of power to the badge. I don't know if you know about potatoes can provide power. So bring your potatoes your favorites from Idaho. Now we don't we don't have any secrets to share. really weren't playing to share anything, but we're probably going to go stealth mode until the very last minute.

No, no, no Bender themes to give away it.

No. Although things are going to be pretty radically different next year. We've We've done three years, we kind of had a trilogy, I guess you'd say, of Bender themed badges and I think people know were capable of and so you'll see something completely different next year. Although just as much effort just as much polishes we've done in the past.

That sounds pretty rad.

Yeah. So

do you have anything else Steven?

I think I'm good.

I mean, Blitz?

Yeah.

Yeah. That pion would you're like the sinus out?

Yeah, sure thing. Well, that was the macro fab engineering podcast. We're guests. Hi, Ron.

And zap.

And we are your hosts grab foam. And Blitz? You know, this episode? Wasn't that spooky?

No, it wasn't. Well, we talked about ESP 32

Hi, Ron. Yeah, give us give us some more spook.

That's all spooky I can do I need to go carve a pumpkin right now. Gotcha. Later guys.

Take it easy.

See you later

Thank you. Yes, you our listener for downloading our show. If you have a cool idea project topic or bad idea that you want Stephen tonight to discuss, tweet us at macro fab or email us at podcast at macro fab.com. Also, check out our Slack channel. If you're not subscribed to the podcast yet, click that subscribe button. That way you get the latest episode right when it releases in please review us wherever you listen as it helps the show stay visible and helps new listeners find us