That's my guess too. Usually AV software auto-updates to try to keep the latest fingerprints of viruses/malware up to date to try to catch the latest before it hits you (unless you're the lucky one that got hit first). I know some I've used in the past updated multiple times a day, but those weren't DLLs or anything, just fingerprints/definitions. I believe the software updates themselves were prompted at the admin level then you could choose to push them out, but haven't used Crowdstrike so could be different.
If you read through the thread I linked that seems like one of the major issues with IT folks right now. CrowdStrike seems or acts like it has admin privileges, adding the corrupted file in C:\System\Windows32\drivers\CrowdStrke\CS-0000291*.sys but for many users at their workstation in Windows 10/11 setups with Azure logins, they are not the administrator and there were issues with that.
The second and more major confounding issue it seems is with bitlocker keys. Pretty sure by default windows 11 and windows 10 machines if setup correctly (which presumably most IT depts did) have bitlocker encrpyted hard drives. With that you now cannot boot into safe mode without entering that encryption key locally at the work station.
To fix my computer this morning, I had to get a copy of my bitlocker recovery key, enter that just to get into safe mode just so I could delete the file to fix the issue. AFAIK right now at least there is no way around having to do this manually at every affected work station.
Now consider companies who have their own bitlocker key servers, that had crowdstrike on them, that are now inaccessible, and documentation was on another computer that had crowdstrike. Lot of IT nightmare fuel in that reddit thread.
I'm going home and hugging my linux laptop a little bit harder tonight.
7 Likes
Yeah, that's a pretty accurate description of how today went so far.
1 Like
takes notes for when I hear someone suggest replacing our plant PLCs with custom software on Windows servers
8 Likes
We have a couple thousand computers that are affected and the majority of them will not accept the bitlocker keys that IT has provided. Luckily it is possible to boot into safe mode without entering the key.
This is the instructions we are giving
- If your computer is not already at the recovery screen reboot until you see the recovery screen.
- Click See Advanced Repair Options
- Click Troubleshoot
- Click Advanced Options
- Click Command Prompt
- Click Skip This Drive
- Type âbcdedit /set {default} safeboot minimalâ (without the quotes) into the command prompt and press enter
- Close the command prompt
- Click continue, the computer will reboot
- Log into windows with a local admin account
- Navigate to C:\windows\system32\drivers\Crowdstrike
- Search for c-*291 and delete the file that starts with C-00000291 with .sys file ext
- Open the command prompt, type in âbcdedit /deletevalue {default} safebootâ (without the quotes), press enter
- Restart the computer
Hopefully this helps somone!
9 Likes
I wonder how many IT groups at big companies will factor their losses in the total cost of ownership comparisons of Windows vs. other options. And act on those comparisons.
3 Likes
If history is any guide: Disappointingly few, because obviously this was a black swan event, and it'll never happen again. 
4 Likes
Well, for us it happened at 3pm on a Friday, so we got to go to the pub early for a beer 
so I guess I owe CS that much. The day kept giving after that though, as there was also a big storm that took out the power and lights in the gym at night, so that was also fun.
1 Like
I just aborted a return trip home to sit in a hotel to try and rebuild a primary gateway because someone clicked the "recover windows" option nuking the system.
Awesome times.
@pturmel I've been trying to get customers to walk away from Windows... maybe this is enough of a wake-up call that people will do something about it.
2 Likes
Hopefully that someone wasn't from IT...
No it wasn't.
Thankfully I had auto backups going so getting Ignition back up and running was pretty easy. Just need to rebuild everything else...
I've heard IT people say things to the effect of "Windows pays the bills", and I imagine there is a lot of truth to that. How would IT people make bank if the whole world switched to Linux and the servers simply ran? I don't invest in individual stocks, but if I did, I'd say this is probably a good time to invest in CrowdStrike. I hear their stocks are down by like 30% right now, and who knows when their shares will be this cheap again.
I'd be firing Crowdstrike as fast as possible. They've revealed, via this flub, that they are trusted to inject arbitrary code into the kernels of billions of Windows systems. That makes any other persistent hack ever exposed look trivial. (Outside of Microsoft's own automatic update opportunities, of course.) Crowdstrike obviously cannot be trusted with this capability.
I would expect hostile nation states to be targeting Crowdstrike itself, now, as a shortcut to surreptitious access to all those billions of systems.
(Firing Microsoft should be on top of the menu, too, for the same reason.)
6 Likes
This idea that it's acceptable to have forced, unscheduled, updates just baffles me. Let alone updates that reboot your computer with no option to postpone. It shows a total lack of respect for any production environment.
2 Likes
My brother works for Backblaze and their servers I don't believe were hit as they're running Linux, but his work laptop BSOD'd.
I think what Windows/MS has done with updates changed due to all these older Windows systems left vulnerable because no one was doing updates, so they started forcing the updates to try to prevent the spread of viruses from old vulnerabilities. In a way, the old way was probably better because if you decided not to update, that was the risk you were taking rather than MS forcing it on you along with it rebooting while you're working (I've had it do this even though it says it will not do it during active hours)
On a somewhat related yet separate note, I was just thinking, wouldn't it be nice to have a "module manager" similar to a package manager inside the Ignition interface for updating 3rd party modules with the ability to add other repositories. I say this because CirrusLink would probably be more of an official repository that you could enable, but allow repositories from others to either be in an official approved/vetted list, or just the ability to add a URL for let's say @pturmel's modules that when an update is pushed, it periodically checks and lets you know there's an updated module and allow you to just have a couple clicks to update it without much hassle. You could have requirements like min/max version supported, release notes, license agreements, etc that are part of it, but would streamline adding/updating frequently used modules.
1 Like
I image a third party would have to do that. IA's own modules cannot be updated separately from the platform, except for the occasional short-lived beta.
Yeah, I know IA's are tied to gateway version (unless they wanted to build in an updater that would walk you through updating the entire system), but even if they added it just to help out 3rd party modules, it would streamline the update process I think. I use HomeAssistant for my home automation, and a 3rd party "package manager" called HACS that does something similar to the built-in updater, but works really well.
1 Like
anyone have a link to writeup on how the file actually caused BSODs?
It was a division by zero error.