Supply Chain Conspiracy Securities

Welcome to the microfiber engineering podcast. We are your guests, Misha Goffin

and Chris church.

And we're your hosts, Steven Craig and Parker Dohmen. This is episode 142.

Our topic for the podcast this week is supply chain hardware and software security. This topic spurs from recent allegations from Bloomberg about the possibility of us hardware supply chain infiltration, and super micro computer Inc, which may have caused security concerns for servers owned by Amazon, Apple and Facebook. This week on the Mac five engineering podcast we have Misha and Chris to talk more about this subject. Chris was on a previous episode of the podcast, which was about recent Chinese tariffs. That was episode 127. So go take a listen to that if you have not listened to that episode yet. And Misha is new to our listeners. So would you like to introduce yourself?

Sure. My name is Misha. I am a new CEO over at microfiber and have been involved with the company for about five years but formally joined only about five months ago. I've mostly stayed in the shadows, but most of my professional career has been spent in cybersecurity. So this super Micro Case suddenly made my long term experience very relevant. So I'm looking forward to the discussion here.

Oh, cool. So I guess could y'all to explain what was the premise of this alleged hardware attack?

Chris, why don't you go ahead?

Yeah. So as the story goes, someone in the Super micro supply chain, put a device on the the motherboards of these servers, after they had been designed and built. At some point, they've altered the design, put this new device in that, in theory allows us to take over the the BMC, or boot management controller of the device.

So how did they find this this attack or alleged attack?

Well, it's unclear how they found it. And it's also unclear whether it's really been found, which is where a lot of the controversy is coming from. I think there there are a couple of things to note about this particular case. One is that we're obviously living if not an age of fake news, then in allegations of fake news. So as soon as this news broke, in look, I follow a lot of cybersecurity giants in various forums, there is a lot of skepticism about whether the story is real. This always happens. But it especially happens with stories that involve national security. There by definition, are usually sourced from anonymous sources. You can't quote where you get the data from, you can't explain where you got it from. The intelligence community is notoriously cagey about what they call sources and methods. So because it's it's not exactly clear how they got this information, although Bloomberg cites something like 17 sources for this right. There is a lot of skepticism about whether the story is real. Yet, at the same time, it's the very first time we've had confirmation of what we've always suspected, which is supply chain security attack, but not just supply chain, but actually compromise point at a point of manufacture, which I'm not aware of ever happening before, there's been a lot of hardware implants. And that's not some of what we should touch on here. What has happened before, what thing what cases have we observed in the past? What's plausible, and how do the intelligence agencies behave? But we've never had a case where we believe the manufacturing supply chain is has been the vector that's been compromised, that's new.

Yeah, I've heard of chip manufacturers being exploited. And when like, let's say you go get made a microcontroller and a sick made, there are cases out there where the company put in their own backdoors into like a basic before like, would go into a modem, which is pre what would be happening on this server, because that's like, you know, at a chip manufacturer, and not at the board level.

That's one of the fundamentally different things about this case, is that the idea that the design of the product itself was modified in some obvious traceable way to inject this new hardware.

I think one thing that everybody is struggling with and I and I think one big reason is because most of the cybersecurity experts are are they're commenting on this are from the Western world and I think they're having trouble. I mean, it's a it's a problem of imagination. I think they're having trouble imagining why somebody go through the third the trouble of compromising the supply chain when there's such easier ways to compromise hardware. And there's been a plenty of one of the reasons why we know that somebody would do this is because it's been done before by us, right? The NSA is one of the most prolific malicious actors out there. And they compromise hardware regularly. The way they do it is by using the access that we have, right? There's, there are cases where Cisco hardware, and I believe Dell hardware has been intercepted, intercepted in transit, right, in implants, or inserted as it was being shipped somewhere else. This is something you do when you have access to a particular particular mode of implantation. So because we're not, we're not a manufacturing center anymore, certainly not for electronics, the place where NSA goes to exploit their targets, is, is the shipping routes, right? They try to, they they intercept them with, with their partners. So a lot of times, it's easy for them to go after somebody, if they're shipping something to another country and drop an implant there. I think one of the things we're failing to imagine is that perhaps for China, that's not an easy thing to accomplish at all, for them, compromising the manufacturing supply chain is actually much easier, right?

But in this situation is actually fairly, a little bit more intense than that correct? Because there's something about what you can intercept something in shipping and do a modification to it. But in this situation, the actual design was modified pre manufacturing was so that makes this a little bit of a unique case in that sense, because first of all, somebody had to know what to do in that situation and how to modify it. And I assume potentially, there's, there was work put into making it less noticeable.

That's right. And I think it's, it's instructive to go back and look at some of the known cases of hardware implementation before, when the NSA dropped implants at the Cisco routers, I think the number of known cases was something like a dozen. And I think the number of beacons that they saw were maybe just over 100. They chose their targets, they had to get physical access to these devices in transit and drop implants on them. I think if somebody was looking closely enough, they could find it. I think the benefit of attacking the design, as Steven just said, is that your footprint is much broader. detection is much harder in you can attack these at your leisure even many years later, when the exploit for your firmware becomes available. Right. So there's been a lot of questions raised, why not just attack the software? The attack for the software may not exist yet. The access is what is a separate problem. And there's usually separate teams that intelligence agencies that are responsible for this, the people that write the software, not the same people that go into field and actually drop the implants physically, right. Are our best software engineers are not the ones that are going in submarines and tapping underwater fiber routes, right?

Does that actually happen with the fiber routes? I actually can imagine that's like really James Bond thing to think about?

No, it looks it absolutely does. And this is what really blew my mind after the Snowden disclosures for me. Security world really splits into before Snowden and after Snowden, I think, you know, if we're looking at the timeline for this, we're very much in a stage where Snowden made a bunch of disclosures, there was a lot of newspaper articles. And then the denial started coming, James Clapper denied strongly, everybody did not as strongly right. What they didn't anticipate is that he had all of the documentation to back it up. And he's already deleted. So before the data dump started happening, for which we learned a staggering number of things, including that, yes, we, in fact, do modify transatlantic routes. And we have both listening stations and ability to inject data on internet links that I thought previously would just mean, to me the thought of getting a nuclear submarine to go top of underwater fiber bundle with just the cost of that was incalculable. And yet, the Black Ops budgets for intelligence agencies, I was just way larger than anybody anticipated. Not only that, but the data that Snowden leaked, didn't just include the description of budgets, but also catalogs that they were used. I mean, at some point, you have to think about the problem of I got these spies that need access to certain technologies. How do I make them aware of what tools are accessible to them, you publish catalog, so the catalogs are public, and they're easily searchable? So if you want to kind of rejigger what you know your your imagination go look up vault seven catalogs look for NSA catalogs the ANT catalog, the implants that are introduced directly into into Ethernet connections, a whole lot of stuff that I just thought physically was impossible because the cost would be prohibitive. They just have access to greater budgets than anybody realized. And everything I thought was off the table wasn't just on the table. It's already been

done. And I think I want to kind of chime in here on one thing real fast. You know, Steve, Steven, you mentioned, you know, the difference of, you know, the impact of it being injected in the manufacturing cycle. And I think, Misha, you talked about, you know, the the targeted activities of the NSA. I think what I'm hearing from a lot of people when I talk about talk to them about this subject, is this sort of fear that this is a broad base attack, right, we go from a highly targeted set of activities that are designed to intercept one particular player. And then we now apply that to a whole segment of a market. And anyone could potentially be a target for that. I think we, if we look at just sort of the the fallout from was it like Stuxnet right? When it when it actually got out of Iran, you know, our own infrastructure was being targeted by that. And the sucks

next was the the attacking Siemens PLCs. And the, the

centrifuges center.

Yeah, that was, that was one time when I was very proud to be an American, because that was a super, I mean, that was a super impressive operation. And the fact that they were able to get specific Siemens PLC is that we used in Iranian nuclear facilities was mind blowing, not a supply chain attack, but definitely required level of access that, again, I would have thought was impossible. So that was a highly sophisticated attack. Now, what Chris is referring to is that it was specifically designed to target a very narrow range of Siemens PLCs. And yet the delivery vector for it was a self propagating malware variant. In at some point, it started ricocheting off the Iranian networks and ended up back in our own networks. And we're still murky about what exactly happened. But either the malware was modified by the Iranians to attack us back using our own mechanisms, or the attack was just broad enough to where it wasn't just this, the Siemens PLCs that were attacking, it just affected more infrastructure than we anticipated.

But you know, now we're talking about this, this vector that may exist everywhere around us, right. And anyone could be a target at any time.

I think we also need to understand the timescale right. When Snowden disclosures came, and that was about five years ago, he leaked the catalog that at that point was, I think, four or five years old. Right.

So so they got a decade of new stuff now. Oh, absolutely.

Right. I mean, the most impressive thing I saw, and that wasn't the Snowden disclosure, that was the Kaspersky Lab. Because, you know, we all pretend like our cybersecurity companies are multinational in nature. But in reality, they have deep connections to the country of origin, right? I mean, Kaspersky Lab works for the Russian government, let's be clear about that. I don't know why we buy it and install it in our own intelligence agencies or even commercial deployments. But that's besides the point, right? They do a lot of research into what NSA does. They found a hardware implant that was done by the NSA, where they broke malware specifically to compromise. Hardware firmware, right, which, again, I thought microcode like that couldn't be compromised, at least not easily, and yet they'd done it. And by the time we knew about it, it was eight years old.

And it's also the A lot people were talking about how the the device that they found on the server boards where it's tiny, it's about the size of an o 603. capacitor. And people are still wondering, like, could you even build something that small that could actually actively do something. And the thing is those you can buy a, you can buy a, a Cortex, our microcontroller that is smaller than that part, that's got like 14 terminals underneath it, you can get Wait, you can get wafer level microcontrollers, that are like that,

which I think is one of the more important questions, I think, you know, when the stories are flying, and the denials start being released. And, you know, look, we're seeing we're seeing Apple, you know, highly respected security people from Apple sending letters to their congressman, there's a lot of Kabuki Theater of denial going on right now. We just don't know if the story is real. But let's look at it this way. Does it really matter if it's real? Because if it hasn't happened yet, how long do we have to wait until it does? Right? So I think technical feasibility is an important question. I'd love to hear your thoughts on on. How would you execute an attack like this one, if it was feasible, and we do have to think about our manufacturing supply chain as a vector that somebody couldn't be going after? We really even prepared for that, right? We've We've started thinking about manufacturing as the, you know, the lowest common denominator and squeezing every little bit of cost out of it. Is that really the smarter Do we know that in the 80s, Russia stole pretty much all of our super computer designs and built their own Intel like, processors, and so on. So, at that time, we did not allow our critical infrastructure to get manufactured somewhere else. And now we assume that all of these countries that were previously enemies are friends, right? Because they trade with us. So if it's feasible, then what do we do about it? Right? Exactly.

Yeah. And I think, you know, one of the things I've been reading a lot when I read about this case, is people are claiming negative on the feasibility, because they're looking for a very complex tool to exist in this piece of hardware by itself. And I think that like, like to say, for example, well, this little tiny piece of hardware is going to run some code that's going to take over the device on its own, or it's going to store a new copy of flash in there or something. And I think that tends to really undermine how these complex attacks really work themselves out, there are a lot of small simple things, right, rather than one big complex piece of code, you take, you take individual little vulnerabilities, and you chain them together. So I'm gonna kind of jump out there and a completely wild thing here. And I have nothing to back this up. This is very, you know, how I would look at that, right? Knowing if they're trying to target the BMC, the boot management controller. This basically lights out management for a data center, right, and we know that they they run unsigned firmware on those, and we know that they update over the internet. So we don't necessarily have to deploy some sort of controlling code on that chip, what we have to do is when that chip gets some sort of external signal, it puts the BMC in a state that it needs to go look for a new firmware. Right? So we start with very simple thing, can we send a simple signal to that chip, can we then use that chip to put the BMC in a state to look for a new firmware, then we can combine that with something like a BGP route pollution, that simply for a very brief period of time for a very small portion of the internet redirects where a particular address is routed to. So if you can combine those things and synchronize them together at once, you can then execute an attack which results in new firmware being deployed to that device that you now control?

So Chris, on the on attack vector side, I think there is a there's definitely precedent for what you're describing. You know, one of the most misunderstood attack methods out there has been spoofing. Everybody talks about it, nobody realizes just, it's not that it's complex. Actually, spoofing is very simple. Spoofing is very easy to detect a throw it, you know, it forces your network cards and your operating systems to throw off just a bewildering number of error codes. So the reason you don't do it is because it's so noisy, it's difficult to avoid detection. Yet, spoofing has been that remained in the middle of attacks much, much more successful. And recently, one of the most, one of the most interesting cases where somebody was scraping user identities was forcing them to go to what they believed was a LinkedIn page, enter their password, let them onto LinkedIn through a proxy, essentially, just so actually authenticating but quietly record their password. This was done without needing to spoof somebody, it was done by impersonating the Billington login page in transit, you need to be able to be in a data path for that. But we've already established that that's something that's possible, there's a number of ways of not forcing your hardware implants to be going all over the place wildly, and look for instructions, right? There are other ways of delivering code to them that I think much more so.

And I was I was thinking about this. And I'm like, well, what could be another reason to do this to to attack the BMC? And it'd be? Well, if you took if you basically stopped the BMC from talking, you could make the piece of hardware not work anymore, a denial of service attack. And so yeah, basically, now you're talking about, well, what if this, this chip is sitting on the BMC bus and it gets one instruction. And so let's say that, because what this device was a, according to the Bloomberg article was a six pin device that's really tiny. It could be just looking for a single instruction, and then it just disconnects the bus. And so then the BMC can't talk anymore. Now you have AWS, you have Apple servers, you have Facebook servers, all these servers just stopped working again, working now. Alright,

so I'm gonna keep throwing out existing cases. Yeah, just to validate that the rationale for that is not as crazy as it sounds. And you guys can tell me to stop because I have a lot more of these stories, right. So we know that that one of the attack methods is not necessarily just to capture somebody's data or inject data somewhere right? In cyber warfare scenarios, right where you're trying to knock out critical infrastructure. This happened most recently in the To the Russian war against Ukraine, which is supposedly not happening, but there it's a well, it's a well documented case where they knocked out power grid infrastructure through a denial of service attack. And these are, you know, these are attacks that you only use a handful of times. Because once you burn your, your implant, it's it's known that it's there, and you know that that device has been compromised. But what would it be worth for somebody to be able to shut down certain aspects of AWS infrastructure for Amazon Web Services, it runs a lot of pretty important stuff at this point. So I think the motivation would be pretty high.

Yeah, especially gov cloud. Well,

the GOV cloud, but also, don't forget, AWS has a $600 million contract with the CIA to run, essentially private infrastructure for CIA, and they're bidding on a much there's a $10 billion contract out there that multiple firms are competing for. So Cloud is no longer exotic. With our intelligence community there, they're

using it. What's funny, as you bring up the Ukraine, Russian, not war in quotes. We actually even I was talking about a security exploit that happened two years ago. And it was with a application that the Ukraine was using on their, I think was the rangefinding for their howitzers. And it was hacked. The light that add the Android libraries that they were using to compile it was had an exploit that the Russians put in to basically it would broadcast where you were using that app, the GPS locations, and then the Russians would come and just, you know, drop a missile on the Howard, sir.

You know, the another point about motivation, I think one thing we're forgetting is we're always looking backwards. And we're looking at at methods of attack based on what of how the stuff used to work in the past, I think you got to think about how the code has been written now, right? If you're looking at what's in data centers, a lot of it is classic, you know, three tier web infrastructure, or even client server applications, right. But if you look at what's being written as modern code, we do this at macro fab, right? We don't have that much physical server infrastructure, we don't even have that much virtual server infrastructure, we are serverless in all the places where we can be. So if your application is serverless, and if it's distributed, if it's a lambda function in AWS, that only runs for five minutes, right? If you need to compromise an application, you can't do it by taking over a server anymore. You know, when when we're talking about NSA, you know, compromising single or double, even triple digits, numbers of servers or routers? That's just not enough, right? Cloud infrastructure is distributed by definition and code can run anywhere. So why would somebody build something at a design level, as Steven said, that maybe the only way to get things done in the future? Right?

You know, actually, so I've got a little bit of a question that's a tad bit of maybe physical and a bit of a rewind, from what we've been talking about here. So if we have a physical attack, you know, if, if something has been designed and implemented on yours, or someone's boards, or let's just take super micro, for an example, they had to go through the process of actually physically getting these units manufactured elsewhere, when they receive these units, I'm curious as to how they were not aware of this, it's, I understand that, you know, server boards, if you ever look at one, it would be a nightmare to check every component. So I'm assuming the the idea was that it was just hidden within there, but it's an actual physical thing that is different from you know, and from the original design and and super micro was, it was it's not like they're building two or three of these, you know, there's there's a significant amount, I would expect that there would be some kind of visual inspection that would catch this but maybe not,

or even functional testing that could have I mean, anytime you anytime you deal with hardware implants, you're you're risk compromising functionality to some degree. So what really surprised me was they were able to get by those functional tests. I mean, I'd love to hear your thoughts on how likely is that and how difficult that is to avoid detection at that level, because at the very least, it should have been misfiring somewhere right? Out of parameters.

So looking at the part, that's the legend part, at least it's a six pin what was it? It's a six pin conditioning signal conditioning, I think they call it a signal coupler. Yes, signal coupler, which is basically a signal coupler is basically a it's a capacitor. Sometimes it has an inductor in it, but it's basically a known it's a very high precision capacitor, inductor network that filters a high speeds signal to basically get rid of a certain frequency. So like, we'll get rid of ringing on a square wave. So what what could happen actually is if this, these motherboards had these parts already on it, and then let's say it was, it was China, China built an ASICs that fit into that package and made it look like that part, they could just drop it into there. And it would work because a signal conditioning device needs power ground for the inside LC network to work. And it needs and basically, two signals go into it, and two signals come out of it. And so if the it was a microcontroller, or an ASIC, that was basically just passing data back and forth. And just looking at the data until it got a trigger signal to do something, you couldn't, you could, you wouldn't see that in testing. I mean, you could probably test like your emissions from the board, the say, oh, for some reason, that frequency is now propagating out the board. But generally, you don't do that after, you know, you already validated your design,

exactly. If this was already in production, that all of that testing would have been done already. If this was if this was full production, the only thing they would have been doing, I shouldn't say only but but what they would have been doing is the actual functional testing of the unit. And if this device is, say, in between the processor and its memory, it can just sniff memory and inject whenever it needs to. Or, you know, we even discussed it earlier, maybe if it just had the ability to just completely short your memory lines, you There you go, you've killed all all thought out of your processor.

And my understanding is that the MBS was not compromised. It had the ability to get compromised through the sampler implant. Right. So theoretically, there's not a lot of change to the system beyond this, this on unexpected part being inserted into the circuit. Right? Correct. And

I think it was that it was originally not on the bill of materials, or something like there was something about how, like, the designers of the boards, didn't think this part was supposed to be there. But it would be very easy to add this part. I mean, all you have to do is draw two traces to get power and ground to it and drop this guy in.

Right. And moreover, you know, it's not uncommon, as the devices going into production for changes to be made to it, that the original designers are completely unaware of, but are normal and part of the process reacting to any changes or any tests that were failing, etc. During the initial production run.

Yeah, knowing how to sub a, you know, how the substitute components are being selected? There's not a lot of vetting that goes into that selection process, right. Somebody has to approve it. But yet at the same time, it's not that unusual that that part, the part numbers change in the manufacturing process, right? Correct.

It all depends on what the application is. And if you are, you know, required to keep specific part numbers, you know, if you work in medical or safety or you know, many other industries, you would be required to have a specific part number, but in a lot of cases, I'm assuming servers would would be one where you could just have an, you know, this component or equivalent on your on your bill of materials. And in that case, they can swap anything out. You know,

there's one other aspect and I've seen this bantered around. And by the way, what one thing that I really learned over the last couple of weeks is just how little certainly cybersecurity experts, but I think you can say that about general it practitioners, how little people know about not just electronics, it people will readily admit, look, I just don't know much about hardware. But I think manufacturing, right is just a black box that people have no idea how it functions. I've learned a ton about it's in being here in five months. But a lot of the comments that I've seen online are largely I wouldn't say ignorant as much as they're just unaware of how this stuff usually works. Right. So there's a there's a lot of supposition that things would that there will be easier, for example, to compromise a component manufacturer as opposed to compromise the design of a motherboard, right. In and I'm not so sure, right. I mean, I've interacted with a couple of component manufacturers. And first of all, their ability to spot counterfeits are is really high actually, they have you know, as I understand it, some of them even have FBI agents on staff because counterfeit parts are such a big problem. So I think counterfeit parts and ability to just go willy nilly reprogram components or compromise them at a component level isn't all that trivial at all actually, even though there's a lot of people disagreeing about it. Now. manufacturing side of it. I don't see nearly as many controls out there there. I doubt there is any FBI agents barked at major factories for example,

I basically how I see how if this was true, how would this hour see this going down is basically China would or one Every agency basically made these fake parts, or these parts to replicate this style part. And one guy went into the factory to the line and said, Oh, don't use those, these are better use these. And where you got the wrong part, right? Yeah, that's all it takes. Because people are like, Oh, this would be such a huge, like, a lot of people would have to know about this attack, right? And so like, the more people know about an exploit or know more about how to game the system, the less likely it is to last longer, right? This takes only a agency, right? To build the part and you don't even know what this part can be used for. Right? And then only one person needs to know what it actually is. And he's the one who takes it into the building and tells the guy to swap the part out.

All it takes is just loading the tape on on one machine, you know, swapping it out for one, one thing. Yeah. Five minute job. Yeah,

I think that that that speaks to our fundamental, when we look at these kinds of issues, we assume well, isn't there one thing you could have just done to prevent this from happening? You know, we asked, we asked earlier, right? Are there tests you can run? Is there a way you can audit the manufacturing process. And the reality is, when you've offloaded or suddenly outsourced, the testing, the automated optical inspection, the assembly, everything to the same place, you give up a lot of that control. And I think if you're to prevent something like this from happening in the future, it's really about taking a more in depth or layered approach. You know, if you were to look at these servers, and realize how unprotected for example, the BMC is on these, right? That's probably the weakest link in them. Right? So it doesn't matter how many checks you put through the hardware, that, you know, one particular service running on there gives you full control over it. And it's not not heavily guarded, right. So no amount of checks, you can put in there really going to give you 100% security. Knowing that there there are so many vectors and so many ways to attack that particular.

Yeah, look, control infrastructure has been a vector for a very, very long time. We know this, because if you look at the NSA and catalog, it's riddled with various controllers that are used to upgrade motherboards and hardware. This is the NSA doing this, right? The target hack, right, that was used, that used an ability to remotely upgrade BMC Software applications in a field, right, so the things that we use to manage the things are usually the very last thing to get upgraded. They probably run ancient Linux kernels that, you know, we pray that somebody that can't access in this implant would give you access to a very old kernel.

Yeah, that that target hack, that was the credit card, point of sales attack, right. That was that was through a funny enough an Internet of Things. Air conditioner, or it was like air control module that was on the same network as the point of sales. And they were able to attack that

that was the initial vector. But ultimately, what they compromised was the point of sale system that ran a BMC different BMC. Right, this BMC, software BMC agent on it, and it was the upgrade infrastructure in that agent that got compromised,

correct. But how it was interesting how they first got into it. And it was like how church was saying earlier, is you start with the smallest thing you can do, and step up into where you need to go.

Yeah, that's, that I think, is the fundamental here, there's no one thing you can do in the hardware to prevent this, what you have to do is be able to think like the attacker, right, and look at all of the different little things that can be combined. And look at each step and say, does that give me one more level of access? Does it give me one more level of access? And, you know, I mean, the reality is, there's, you know, there's probably no chance you could have detected this part, you know, post manufacturing. Right. But the question is, could you have mitigated the effects of any part added to it after it was manufactured?

Yeah, and I think this is where some of the misconceptions about security come into play. I think if you talk to a lot of security people, there's just a lot of, you know, jaded old time guys that are pretty cynical about this stuff. And, you know, the prevailing opinion is that we're, you know, we, we're not winning, we're losing. In fact, I think the opposite is true. The reason these attacks are becoming more sophisticated is because we are getting better at detecting the stuff that happened yesterday, right now, detecting hardware implants, detecting attacks against components on motherboards that operate as their own as their own operating systems and are autonomous essentially, they're not centrally managed, that we're not aware of them. We can't scan for them. Security Technology for that just doesn't exist, right? So why go after it? That's the next frontier, we've gotten pretty good at defending everything else.

Yeah, short of basically randomly sampling your, your, your product and then disassembling everything. And then taking, basically, you'd have take dye samples of everything and compare it to known good die samples, like images. That's the only way I can think of making a your hardware robust in terms of security. And even then you don't know if even your sample images that you're comparing against would be legit scans. Like what if your, your IP the IP you don't know about, so like, let's say your microcontroller already has an exploit put into it,

I think one way to deal with this, and I actually think this, if, if it's not this particular case, it'll be other cases that convince people that it's necessary, but I think one big outcome for this, because I think it's gonna take a very, very long time for us to develop any technology that actually identifies this. So I think the technological solution is not coming for a very long time. But some of the solutions, I think, are just purely structural, right? Nobody forced us to offshore all of our manufacturing, right? Nobody forced us to assume that hardware has no value, and especially the people that manufacture it, bring bring no value to the process, right? It didn't used to be this way. So I think at some point, that simplest change you can make is just, if it's critical, if it's gonna end up in a cloud, if it's going to end up on anything that requires defensive posture, you don't build it in a lowest, you know, in a lowest priced geographical region, you build it somewhere where it's harder for the adversaries to come in and get into your supply chain. I even think that, for example, Mexico would be preferable to China, for example, right? We have a very, very good relationship with Mexico, some of their latest trade challenges, notwithstanding, we're still reliant on them, and they're very reliant on us. I think it just be more difficult to go to a Mexican manufacturer convince him to drop an implant than it would be in China in China. I mean, I don't want to remind you what happens when you say no, right?

What happens when you say no?

You may go away. But look, this is actually something that wasn't a Bloomberg story. They, they didn't exactly go after Super micro itself, super micro offshores their manufacturing to China, and to sub contractors in China as well. So it's those places where the subs are being, you know, being farmed out to other subs, they went after the smaller manufacturers and twisted arms. I think in some cases, they said, Do it for your country. When that didn't work. They said do it for your family, or else. I don't know if anybody turned them down. But I think you use your imagination for what happens in China when you refuse overtures of, of intelligence. Right.

Welcome to the macro fed conspiracy theory, our

Alex Jones will join us next week.

I swear if if that guy says anything about macro injury podcast, we might change.

We might need to get a bigger podcast host.

All right, cool. So we've talked about the underlying premise of the hardware attack for potential hardware attack. general structure of intelligence agencies and like existing public cases. Is there any other cases we show like this?

There's there's a lot of cases that I mean, that just I think people just underestimate the range of hardware implants that have already been developed. And, you know, the ones that are it, look, this is embarrassing. Right. But the ones that have been they've been widely, you know, widely know about, and it's not just Snowden. Right. Snowden disclosures were massive. We're talking about gigabytes of data dumped. But there was also the NSA vault seven, and I forget, I think it was Shadow Brokers that stole just an ungodly amount of data from NSA and CIA. So that wasn't that was supposed Snowden. Right, that, you know, those disclosures just kept coming. So a lot of the, our intelligence agencies lost a lot of their tooling. Now, some of these tools were pretty old, but we also know from eternal blue for example, that that was the the exploit that led to not Petya, and several other really massive attacks later on is that these exploits have been out there unknown for eight years or more. So as long as nobody knows about them, and they're not burned, our our intelligence agencies use them very successfully. So there's been an incredible amount of tooling that they lost and just weren't able to use anymore. So look, it's very instructive to go out there and read about I mean, some of these catalogs are just fun right? I mean, I I feel bad that they're out there because obviously that compromises our ability to to walk the wage that I'm pays when you to wage, but at the same time, it is fascinating reading, right? So one of the most interesting tools that I saw in that catalog is it's a, it compromises the, the ethernet jack, and it broadcasts a 45 megabit stream, what then I think something like a kilometer in distance. So you can, I mean, think about how you would have to use something like this, right, you would have to implant it in a field, but you would also have to collect the data in the field. But get access, you get to it as unconditional, we're talking about all data at very high speed rates. And, you know, this is the this is where you got to really understand how these agencies operate. I think there's 1200 people in, in Tailored Access Operations stationed in San Antonio, and those adjusted guys the right software, right, they're not the guys out there in the field, dropping the implants, that's a totally different unit. And they're operate using a different set of procedures, they have different skills, right. So. So I think, I think, I think one thing, when you just realize that this is done at a really high level, spending a lot of money. It's something that we're gonna have to contend with for a very long time. You know, I think one question that we sort of answered here, but I'd love to restate again. Is there something that electronics designers should do in order to to avoid situations like this? Or is this one of those things there? Were once you, you know, once you produce a design, and once it goes to manufacturing, you kind of lose a lot of control over it, you just have to trust him and the manufacturer is not going to be compromised?

Yeah, I think it's it's a goes into also like counterfeit parts and stuff is you need to vet your supply chain. And you basically have to trust your manufacturer, that your manufacturer has vetted its supply chain. And then the manufacturer has to make sure that its suppliers, its distributors that it's using to buy parts has vetted its supply chain, as well. So there's a there's a long list of trusts that have to, you know, be built up.

Well, so I think you're absolutely right with that, Parker, however, I don't I don't think trust necessarily just comes in a handshake most of the time, correct? Correct. You know, there are, there are things in place that that you can demand from a manufacturer such that they can provide you information on where they purchased things, who they've dealt with. And that that is a little bit more of that that is the trust that you're talking about it? Yeah,

yeah. Like you can go in and make sure that you're getting legit parts. You can make sure you're getting your lock codes to make sure that you know, your your date and your date codes to make sure they're actually real parts. You can vet the parts by sending them back to the manufacturer, to make sure they're they're real. It really comes down to like how far you want to go with that with your design. And, like let's talk about like, it also depends on like, how high level like, if you have a x 86 style device, it's like, well, how do you not know that Intel? Was backdoored, by the NSA

has any of this reflected in the ISO standards in any way? Because I mean, look, here's the way it works with both infrastructure hosting and outsourced data centers, but especially cloud. Maybe it happens with the largest customers, but I don't think people go to Amazon data centers to go audit their security, that's not the way it works. You can't just show up and go look at how their physical security procedures are done. Now, what Amazon Web Services does, though, is publish a really long list of third party audits and regulations they they abide by, there's a pretty strong set of standards published by NIST and the and and I think, actually ISO as well. But all of them are related to data security, none of them are related to supply chain security at all. So is there a regulatory body that governs any of this? Or is this all upcoming in something we need to really think about implementing in the future? If we were to take this seriously, you know, when we

talk about ISO, most commonly we're looking at ISO 9001, which is really about process. I don't think it actually provides any real controls here that could prevent something like this from happening, because it's it's mostly focused on Do you have a process for everything? Yeah.

And it's quality related, right? It's not I mean, process could be very much a, you know, supply chain verification process, right, but either I'm not aware of a whole lot of security being built into the ISO standard, necessarily, right. There's just not a consideration right now.

Well, and the ISO thing actually goes back to what I was talking with Parker, about just a moment ago, say that, if you needed to trace where a component came from, and you were working with an ISO manufacturer, they would be able to provide that information for you. Because that is a requirement throughout ISO to be able to track where things come from where you purchased them, where even in your building, you have them effectively. So the ISO would be more of the quality aspect. Yes, you're right with that media, but also being able to track things that would come through ISO,

but I think I think they still suffer, it's, you know, whether you're talking about ISO or ISO or anything else, they still suffer from the fact that what you're measuring is, is a given process followed, not whether or not a given process is fully capable of catching every issue that could happen. You know, I say, for example, you know, we may have traceability on a part. But when someone's counting that part in, it's not very difficult, when they're sitting there on that that real counter, right, they're loading one reel up, well, they just put a different reel to feed back onto it. Right, they load a different set of parts on there with the same markings. Now your your part is going through the process, it's now been changed out in there. And any any actor in that process could swap that out and leave the labels on there. So

I mean, this is comes down to what happened recently with that the Japanese steel company, Kobe steel. And basically, they were just, they were the origin point for steel. And they were just basically making fake documentation saying their steel was better, or basically taking less quality steel and selling it as higher quality steel, and having the proper documentation. And so Ford was basically like, would go to their distributor and get all their ISO standard stuff. But when the original manufacturer the steel was basically lying about their steel, you know, set of standards could prevent that from happening. And that's,

that's where we get back to, you know, at the end of the day. These, these techniques don't take advantage of just flaws in the process, they take advantage of the flaws in the underlying technology that's being utilized. And given that you can never 100% verify your supply chain. I think it really pushes back to the product designers and the consumers. Those products are designed around technology, which can be safely validated or vetted from end to end.

Yeah, by the way, this is something that I have seen before. I guess I've always thought of counterfeit parts. As you know, components that you know surreptitious manufacturers introduced in the manufacturing process, because they're trying to save a couple of dollars, you actually see this being reflected in some of the stories being written now, what I wasn't aware of was that sometimes it's designer choices that lead them to very similar for example, components, but they're cheaper because they're, you know, they're knockoffs made in China, for example, and that by me, that's where you make any design choice and to trade off cost for a reputable component designer, for example, how prevalent is that? I mean, is that those just one offs that I've seen, or? Or is that not that uncommon, because that's sure is a good way to kind of earn, you know, to design your way into using compromised parts, right. So

I've seen I've never seen a situation where it led to an exploit. But an example would be that I can think of right off the bat would be USB to UART. Like TTL level to communication chips, were the big players back and they were like FTDI, there was another one that sort of the P that was before my time. Man, I can't remember the name of that one. But then there's like Silicon Labs as a microcontroller microchip one. But then there's this company out in or in thing out in China. They make this chip called the ch 302. Very cool. And it's really inexpensive. And it basically does a USB to TX, RX style steel communication. And I know a lot of people choose that I see because it is half the price of that CDI. You know, 20 to 30x, or the silicon lab CS zero, whatever it is. So yeah, they are designed decisions to go with cheaper, but technically functionally equivalent parts in terms of what the bat the black box does, is, but it's like, well, technically, I guess if you could put an exploit in that chip, but I don't I don't I don't see that. There hasn't been one that happened yet.

Yeah, it's kind of like Facebook is a cheap application because we don't pay anything for it. Somebody pays for it. Yeah, right. Somehow Right. So what is cheaper? Is it really because it's cheaper because it's subsidized by other forms of, of financial incentives? Right? Correct. This is my conspiracy side coming out again.

Yeah. And I could see the same thing happen with with microcontrollers. And like, let's say RAM, like going with cheaper RAM, because it's cheaper. I mean, that's it people just don't think of why is it potentially cheaper? Or is it just because it's just, you know, less expensive? Is there another reason in there?

Yeah. So I think this is kind of a layer of, of this layer of controls that we just don't have developed yet. And I think, I think a good parallel to this is the industrial control space, because we've known for a long time that systems that control critical infrastructure, right, first of all, they need to be air gapped. And that's, that's been a requirement for many years, and yet, they're compromised still. Right? So there's a lot of regulations that exist in place in order to govern how those things should work. It doesn't sound like there's much in terms of security and in availability for manufacturing and supply chain security. So I think that's one of the things that we're at a glacial pace, because this always happens at a glacial pace, I have to address at some point, by the way, since we're always up for additional security stories, right? One question you got to ask yourself is, so you have centrifuges in a lab somewhere in Iran, that or trust me, they were not connected to the internet, right? They're not supposed to be I think one of the few places where they actually follow the air gap requirements is probably in a secret nuclear lab somewhere in Iran. So how do you get your malware on there? This one, this one's a personal interest for me, because we're microfibers. here in Houston. Apparently, one of the easiest way to get there is using the scientist because we would trust scientists, right? We don't trust adversaries, we may not trust supply chain even but scientists, that's a different story. They go to conferences, and they meet and they talk about ideas, as I understand that one of the vectors to get and I don't know if this was used specifically in that attack. But I do know that at least one of the attacks on one of those air gapped networks happen here in Houston, because it is the place where nuclear and other energy scientists, a lot of times comes for conferences. The original point of entry was the conference material CD, which tells you when when it happened, this was 2005 or so. But the CD that they brought home with them to read the conference papers was that original was the original point of attack. That's what led through several stepping stones to an air gapped network and eventually delivered the malicious payload to something that should have never been connected to the internet. Right.

So I think there's gonna be a little bit of side question is, because they always talk about, you know, USB devices, like thumb drives or whatever being exploited. Like if you buy a route, like you should never plug in a USB drive you don't know like you found on the ground, or like your, your house reloader like gives you a USB drive that's got like 3d models of the houses that you're going to go look at. It's like, well, or at a conference in Houston, I guess. All right. Yeah. Conference in Houston. So so on, let's say just USB drives, like how can you make sure that you're not going to be compromised? With a USB drive, you

cannot be sure of that whatsoever? You need a dumb USB charger, that is not connected to your to your PC in any way.

I always assume that every USB device is compromised. And, you know, take that approach. It's just not getting plugged into my computer.

I mean, Chris's favorite case is the vaping devices, right? The E cigarettes, right? I would not, I would not plug that into anything that you care about. So, you know, if you don't have a USB hub for this purpose, then you're wide open to some to a supply chain attack.

Yeah, that's that's one thing I want to touch on really fast. You You had mentioned earlier about the concern around components. And you know, the the question here is, you know, what are some of the things we can do? The thing that keeps me up at night around electronics? Are these complex systems on modules, right, the things that they have a microcontroller, they can run a whole operating system, they're running the Wi Fi for your device, or I know you just posted the thing about particle I think we can probably trust particle to some degree here. But there's a lot of cheap Chinese systems on modules.

We're talking about like the ESP Siri Yeah, and stuff like that. Anything where you

you're taking a whole printed circuit board, running its own circuitry with its own control software and putting it directly in your communication chain. I mean, at the end of the day, if I were looking for vectors that I wanted to lock down on my product, I'd start with that. You know, don't put any complex unsecured uncontrollable devices in my product.

Yeah, actually, an interesting point because on on cell phone hardware, your the modem that is in your cell phone is a complete black box, even to the operating system in the firmware on the phone. You can't touch that stuff. Because they understand? Well, it could be that or it is already, you know, it's the exploits are already built in. I'm the kind of person that thinks the NSA already has that should unlock.

We remember remember every, every adversary has adversaries, right? So they may inject their own their own exploits, but at the same time, they wanted to harden that against exploits from others.

Yes, yeah, that's the weird side effect of working in the security industry for a long time. And, you know, Chris, and I have both spent many years in that space. It's not the security doesn't matter. It's just people need to understand that. There's adversaries that are you can deal with, and these are individuals and maybe commercial entities. But nation states are just a different story, right? If we're talking about finding exploits that they've been using for eight, nine years, and having firmware on hard drives, they can implant there, you know, there's look, there's MMS exploits where, you know, somebody's running protests during Arab Spring, and all of their cell phones can be compromised or shut down through an MMS message, right? Nation states just operate not a different plane. So to some degree, security really doesn't matter. So there's a difference between, there's a difference between what you should be doing for yourself personally, right, versus what you should be doing for your company in your business. And I think that brings me to one of the points I wanted to make. I talked a little bit about the mistake we made about thinking about what is possible and what has already happened, which I think in security world is largely irrelevant, what you need to be thinking about is what's going to happen next, we touched on the fact that software is more and more distributed. But what's the next big wave of computing that's coming our way. And that's industrial internet of things, which, you know, I thought it was mostly a BS term for a very long time. But look, there is a lot of industrial computers that are being dropped into places where we don't expect humans operate. And we'd really don't think about them as something that's remotely controllable or accessible. industrial IoT, by definition puts a lot of connected devices into place, the last places where we want them to be right, in water treatment plants, in you know, there's a ton of products being designed now to measure, you know, pressure and pipes and do all sorts of things that could actually shut down physical plants. This is where supply chain security and IoT gonna come crashing into each other at some point, right, the footprint of this stuff is going to be immense. And if we're not taking security of supply chain seriously, there, there'll be a price to pay at some point.

And that it comes back from what church was saying, with the system on module stuff, where you don't really know what's running under the hood to make that Wi Fi tick. That might be something that needs to happen in that space in the IoT space is to start locking down that kind of stuff. One of the interesting movements is with open source microcontrollers like RISC v. And the fact that now okay, you have the Verilog code, or basically the the hardware code that is that physically makes the gate logic, Misha, I don't know if you've ever looked at verilog or VHDL code, but it's it looks like C except it's not executable code. It's basically describing how the hardware is set up in software. And so now you have that now you can say, okay, there's no you can vet that and say, Okay, there's no backdoors in that. That's, that's that's good. The problem is you still got shipped that off and get that chip made.

Yeah, that's the problem is still has this issue where you have to trust the fab in the manufacturing chain, right?

So even if you go that was the biggest thing I saw was like people like, oh, we can fix this by open sourcing everything. I'm like, that gets you as far as to the people who put that code into a crate, what's it called the when they make the masks for the chips? What's that mascot making? A special name for those masks. I just silicone process silicone processing. So you get to that point. And then of course you okay, you can get the masks, but it's like, how do you turn the masks into gates and back into something that's human readable? You can't?

Yeah, and I think I think at the end of the day, what we're gonna see is more and more attention is paid to the hardware. What we're really looking at is a series of auditable sub components that you can use that are auditable, and verifiable, that go through a tightly controlled slip supply chain, to start eliminating those risks in your product, especially with the IoT IoT stuff, right? The idea that we can put a product in a nuclear plant that contains a random sub assembly from a random manufacturer that was bought cheaply off Alibaba, because it was the lowest price component and did the job. I mean, it should be shocking to anyone. Right. So I think

it's I think it's madness. And yet, I think it happens all the time right now without a second thought. I mean, I think the main, the fundamental mistake that I've seen, now that I've joined the manufacturing world is just how everybody's fixated on lowest possible price. And, you know, look, there's a lot of considerations right. In it is sourcing quality. But it's really more about counterfeit parts. I don't think we think too much about whether the components would show chose the right components, whether they are trustworthy, I don't think we're necessarily thinking about whether manufacturers can actually be trusted. I think all of those are secondary, or maybe even tertiary considerations. And all of that is gonna have to change over time. And, you know, that's the reason why I look, I'll be honest, I'm excited. What about Bloomberg story? Because even if they got a lot of technical details wrong in it, there's a distinct possibility that, you know, I think James Comey had a really good quote for why people kind of get details wrong. You know, whoever knows about this attack. And I do believe that that is real, whoever did talk to Bloomberg might have heard about it, you know, not secondhand, but they've been briefed about it the non technical manner, through three different layers of somebody who's actually working on this, technically. So, you know, when Bloomberg publishes a photo of a six pin component, I have no idea if it's a six pin component or not, look, that was an artistic drawing, or something that may or may not actually be in the field, it could very well be that the actual component is radically different. Right? It doesn't mean that that that didn't happen exactly the way to describe. I'd be surprised if Bloomberg went out there and wrote a story with 17 sources and was nothing to it,

right? Or maybe the 17 sources were all planted as part of a disinformation campaign to get us to look away from the

that's probably the most, the clearest case of fake news that I've seen where I think it was a lawyer, I'm not gonna name which lawyer for which party it was, but I think the lawyer leaked the news and then confirmed it to another Newspaper Source. So they became kind of a miracle aberration, which are there. But again, we're talking about 17 sources, that's not easy to do, you know?

Well, I think I think overall, there's there's one thing that we can start with, especially in the IoT, hardware community, let's let's get people to start changing the default passwords on all of these, all of these products, or hey,

that would be a good start, right?

Let's begin there. But it makes managing them so much hard.

It was it about a year and a half ago, was that big botnet of security cameras? Oh, yeah. That's my wife, because it's left the news. And it's the problems been mitigated, basically, by banning giant swaths of IPs, not botnet. So exist, by the way, it's still doing its thing, just,

you know, not even mentioned this, right. So I had to, you know, my mother wanted some security cameras installed in her home this past weekend. And you know, even though our viewers can't see it, I have this wonderful big scratch across my scalp here from the, from the attic doing that, but I thought it was really odd, you know, I got her this well regarded easy to use system, I set everything up. And when it was time to get everything connected together, it's a Wi Fi security cam, all I had to do was press one button on the camera, and it automatically connected up to the network and everything worked fine. And I'm like, that just means that all of the security is pre baked into this, that means it's going to be the same thing from camera to camera. And that's going to make this really easy to exploit.

So if her neighbor were setting up cameras, and click that button, it would connect to their station, they can get on either it's whatever network he decides, but I didn't

think to bring my like my laptop and any, you know, any sort of scanning solution to it to see what was actually going on there. Because I imagine it's just an open network. And that's all it's doing. It's talking to, you know, some sort of, you know, API over an open Wi Fi network that it's got pre configured keys for, or, you know, pre configured to use this SSID and just hits a, you know, unprotected endpoint says, I'm here,

by the way, so I'm not only sitting here spreading conspiracy theories, and I actually contribute something useful to this podcast. Because we're talking about industrial internet of things. It's security. One of the most useful tools out there and this is this kind of falls in the in the domain of open source intelligence. This is a public resource. If you're ever curious about what's out there. What's actually connected and not just connected, but actually indexable and indexed? The Shogun Shogun, that IO, Search Engine is a fascinating resource that basically gives you a rundown of either address space and what's connected to it. Or you can I think you can actually search by device type and actually find what's out there and what the footprint looks like, what versions they're running. There's quite a bit of data in it. It's a fascinating toolkit that, like, Oh, I found a way to look for Siemens s seven devices. So it's, it's a lot of fun to play with,

Didn't somebody use that same thing to set up a thing that randomly accesses unsecured VNC instances, so you can go and use people's cameras and look at them and look at their desktops?

Right? Well, it's also the best way to go out there and look for targets once the exploit code has been published. So when the exploit code for this super micro implant finally gets leaked. And again, in my Alex Jones voice, it's only a matter of time. The very next thing you guys should do is go to the shoden search engine and look for targets before somebody else find them. Because like Parker said, if they're, if they're single use denial of service, implants, then you may have a short window, run the exploit.

And I'll just point out that that Chat Roulette open, unsecured desktop open VNC thing is a great way to spend a bunch of time.

It is it's it's a research tool, but it's also can be used for a lot of

very entertaining at the minimum.

Was there anything else we want to talk about? Or do you want to sign us out?

Now, look, I'd love to hear some of the audience feedback on this, because this is one of those topics that I wouldn't say directly relates to, to engineering, but yet, I think every engineer does care about that, even though it's not entirely clear what engineers can do can do about it. But I suspect supply chain security is going to be a topic for a long time. And you know, this is the first time where we've seen a major story break about about manufacturing as a vector that somebody uses for an attack. I doubt it's the last. I think there is going to be a lot of ideas, there's going to be a lot of interesting viewpoints, it's going to be a ton of skepticism. And ultimately, I think this is going to have to have to be a topic in the future. Because there may not be a whole lot we can do about it now. But I think we're gonna have to contend with this for a long time to come.

How people get a hold of us, Misha is in our Slack channel. So you should definitely you know, go camp out there next couple days.

I plan to especially since HipChat is no longer a viable chat tool. So I think I have to switch to Slack either way.

So I think Misha was the one who signed us in to the podcast so church sign us out.

Yeah, so that was the macro fab engineering podcast. We are your guests Chris church,

and make sure everything and we are your hosts Parker Dolman

and Steven Craig see you let everyone take it easy

Thank you, yes, you our listener for downloading our show. If you have a cool idea, project topic or security question that you want Steven or I or church or Misha to discuss, tweet us at Mac fab or email us at podcast at Mac fab.com Also check out our Slack channel which Misha will actually have to download and make an account. If you're not subscribed to the podcast yet, click that subscribe button. That way you get the latest episode right when it releases and please review us wherever you listen as a helps the show stay visible and helps new listeners find us