![Bravo Company BCM](/images/2016/banners/sticky/BCM_StickyBarAd_225x40.gif)
![Login](/images/2016/spacer.gif)
Posted: 7/27/2024 1:44:39 AM EDT
Anyone else still dealing with the fallout from ClownStrike’s little misadventure? Learned today that our enterprise GIS systems are corrupted and it’s not going to be an easy fix. Good times.
What fun things are my ARFCOM brethren working through now? |
|
[#2]
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
|
|
[#3]
MS is just Miserable Shit. gates is like a demonic force and always has been. glad I don't have to deal with it much being retired. I laugh at the people that are PC die hards. they love apple butter
|
|
[#4]
|
|
[#5]
|
|
[#6]
Quoted: It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable. View Quote Which is why they'll do it again. They told you it was a bad patch. An error in programming. How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust? To slow productivity. To keep Americans on edge, angry but also fearful? How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm? Is Crowdstrike under serious investigation? Are they being uninstalled by major corporations and governments? Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart? This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again?? |
|
[#8]
Quoted: Anyone else still dealing with the fallout from ClownStrike’s little misadventure? Learned today that our enterprise GIS systems are corrupted and it’s not going to be an easy fix. Good times. What fun things are my ARFCOM brethren working through now? View Quote Yeah, my shop lost a week due to that crap. |
|
[#9]
Hmm, our GIS systems didn't have any issues. What was the corruption?
|
|
[#10]
I got lucky. It only impacted the Azure/.Net side of our business. Everything that I'm responsible for runs on Linux. Well, not entirely true. HL7 document processing stopped for awhile.
|
|
[#11]
I got lucky as well. We had just started a rollout to one of our retail clients.
I launched it on 4 systems at their HQ. One came back on its own. I had to go hands on with 3. Now they are scared (rightfully so) of Crowdstrike. We use Huntress on all monitored machine, beginning to wonder if that along with defender is enough for endpoint protection |
|
[#12]
The shit with the airlines was the worst goat rodeo I've seen in 10 years of pretty consistent flying.
At one point I heard a Delta customer service desk announce "If you all keep coming up here and yelling at us, we will just shut down our computers and walk away." LOL Lines hundreds of people deep. Stacks and stacks of baggage. Crazy. |
|
[#14]
Quoted: Bill Gates stepped down as CEO like 18 years ago. View Quote View All Quotes View All Quotes Quoted: Quoted: MS is just Miserable Shit. gates is like a demonic force and always has been. glad I don't have to deal with it much being retired. I laugh at the people that are PC die hards. they love apple butter Stock price is fairly high also. |
|
[#15]
It's been a week, why have they not recovered from a backup or failed over to a replica?
|
|
[#16]
Apparently you can buy IT insurance so when this shit happens, the insurance covers the costs associated with recovery. It's beyond fucking expensive but companies are now making it a requirement before they will do business with you. Which means unless you are a big company with deep pockets and profits higher than a giraffes asshole, you ain't doing business.
|
|
[#17]
|
|
[#18]
I got lucky and wasn’t impacted one bit, besides the VPs harassing me for status. Some of our customers use it and they were circling the wagons.
I press hard for minimalist approach and want everything we install to have a purpose that has a clear ROI. I’d argue that crowdstrike Falcon is a solution looking for a problem. Makes people feel like they are doing something. You don’t need it. Has it ever actually prevented something? I doubt it. And ironically, most of the impacted systems that made the news (kiosks, checkout machines, gate agent machines) are terminals that should be booting off a remote image. Pushing updates to them is stupid. Even worse, most of these things shouldn’t even be on windows. |
|
[#19]
Quoted: Which is why they'll do it again. They told you it was a bad patch. An error in programming. How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust? To slow productivity. To keep Americans on edge, angry but also fearful? How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm? Is Crowdstrike under serious investigation? Are they being uninstalled by major corporations and governments? Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart? This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again?? View Quote I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually. The error was a null pointer.. basic shit not to fuck up. Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it. So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels |
|
[#20]
Why did everyone patch the EMS client at the same time?
Maybe dont allow auto updating of your systems and do phased upgrades and testing. Reminds me of a DIE hire we had who was in charge of a major front end application update. They did it on Thursday after 3am. Friday morning DIE hire emailed all that the update was a success. Until all the machines that had already rebooted at midnight on thursday rebooted at midnight on friday. Half our frontline stations were bootlooping on Saturday morning. Our busiest time. Same fix as the crowdstrike. But we had to boot into barts PE and remove the file. I was fixing the issue for weeks as many back office machines had the application as well. |
|
[#21]
Quoted: It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable. View Quote If a software vendor writes code that runs in kernel mode, and the code is corrupted, that type of action can crash any system. MS is a blue screen, Linux is a black screen, and Apple has a pink screen. Apple has been compromised recently, and a serious back door exploit has recently been found in new Linux distros. So this begs the question oh thou great IT professional, what O/S do you use? ![]() |
|
[#22]
|
|
[#23]
Quoted: How do you think any of the crowdstrike thing is microsoft's fault? View Quote View All Quotes View All Quotes Quoted: Quoted: It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable. How do you think any of the crowdstrike thing is microsoft's fault? Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen. |
|
[#24]
Quoted: It's been a week, why have they not recovered from a backup or failed over to a replica? View Quote The computers are locked into an endless loop. They don't get to a stable place where a restore can be done. And I'm guessing the issue is with real PCs and not virtual machines. And if any of the databases were corrupted when the computer initially scrammed itself and rebooted, then you need to get the computer clean and then restore the DB. It all takes time when people need to physcially touch each computer in/affected. |
|
[#25]
Quoted: I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually. The error was a null pointer.. basic shit not to fuck up. Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it. So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels View Quote View All Quotes View All Quotes Quoted: Quoted: Which is why they'll do it again. They told you it was a bad patch. An error in programming. How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust? To slow productivity. To keep Americans on edge, angry but also fearful? How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm? Is Crowdstrike under serious investigation? Are they being uninstalled by major corporations and governments? Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart? This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again?? I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually. The error was a null pointer.. basic shit not to fuck up. Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it. So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels The problem is that it was not a code update. It was definitions. So code analysis would not help. It might throw a generic potential issue message, but they would have set that on permanent ignore since it would be expected for cases where it worked. |
|
[#26]
Incidents like this make it hard not to ponder talk of the "competency crisis" and even the "Atlas Shrugged" premise of the lights going out in NYC.
|
|
[#28]
Quoted: Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen. View Quote View All Quotes View All Quotes Quoted: Quoted: Quoted: It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable. How do you think any of the crowdstrike thing is microsoft's fault? Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen. They have had filter drivers for a long time. The problem is forcing them to certify/sign drivers that can "run" unvalidated p-code. Like antimalware programs that push up to the minute definitions. |
|
[#29]
To be quite honest if you haven't recovered from something that, at its worst case, takes 20 minutes to rectify, your IT infrastructure deserves to fail for being poorly constructed.
|
|
[#30]
Quoted: Bill Gates stepped down as CEO like 18 years ago. View Quote View All Quotes View All Quotes Quoted: Quoted: MS is just Miserable Shit. gates is like a demonic force and always has been. glad I don't have to deal with it much being retired. I laugh at the people that are PC die hards. they love apple butter Sexual harassment investigations will do that to a person. Too bad he didn’t stay and just focused on doing evil in the computer world instead of spreading it across everything. |
|
[#31]
if you use someone elses datacenter (cloud) then you need to take some responsibility for your data. using cloud doesnt take away the need for backups, point in time recovery, redundancy etc.. it just makes those processes easier.
|
|
[#32]
Quoted: I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually. The error was a null pointer.. basic shit not to fuck up. Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it. So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels View Quote Null pointers are not fatal in userspace, but yeah - this was apparently in the kernel. I spent a week walking Fortune 100 level companies through rescuing thousands of broke cloud instances. CSB - Microsoft rescue boot media didn't work in the cloud enviro. No time to debug why. A Linux based rescue ISO worked immediately and perfectly. Just had to add NTFS drivers. So it took Linux to unfuck Microsoft. ![]() |
|
[#33]
Thankfully we don’t use Crowdstrike. But some vendors we use for certain services do, so we had issues with those systems. Thankfully our payment processor isn’t one of them. Our disruption was minimal.
|
|
[#34]
Quoted: Hmm, our GIS systems didn't have any issues. What was the corruption? View Quote From ESRI: However, after recovery, ArcGIS Enterprise users reported issues with components not starting properly. These issues were traced back to the corruption of ArcGIS Enterprise configuration files, which were impacted by the operating system crash. This issue is tracked by Esri as BUG-000169285: "Update to the CrowdStrike software results in failure of ArcGIS Enterprise components installed on affected machines." |
|
[#35]
|
|
[#37]
Quoted: It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable. View Quote ![]() Click To View Spoiler DEI is a smokescreen for deliberate sabotage. Even in the Secret Service. ![]() |
|
[#38]
Quoted: Or just remove the offending files. It’s really easy to recover from this. View Quote View All Quotes View All Quotes Quoted: Quoted: It's been a week, why have they not recovered from a backup or failed over to a replica? Or just remove the offending files. It’s really easy to recover from this. I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping. |
|
[#40]
Quoted: I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping. View Quote View All Quotes View All Quotes Quoted: Quoted: Quoted: It's been a week, why have they not recovered from a backup or failed over to a replica? Or just remove the offending files. It’s really easy to recover from this. I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping. because of this, our keys are stored in local AD, on a file server, and in our RMM. I need to confirm that we are getting a copy properly stored in EntraID (Azure AD) |
|
[#42]
I'm still waiting for the hacker/ransomware fallout from everyone disabling it, to hit.
|
|
[#43]
|
|
[#44]
Quoted: I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping. View Quote View All Quotes View All Quotes Quoted: Quoted: Quoted: It's been a week, why have they not recovered from a backup or failed over to a replica? Or just remove the offending files. It’s really easy to recover from this. I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping. Yep, that might happen. Best keep backups somewhere. |
|
[#45]
|
|
Sign up for the ARFCOM weekly newsletter and be entered to win a free ARFCOM membership. One new winner* is announced every week!
You will receive an email every Friday morning featuring the latest chatter from the hottest topics, breaking news surrounding legislation, as well as exclusive deals only available to ARFCOM email subscribers.
AR15.COM is the world's largest firearm community and is a gathering place for firearm enthusiasts of all types.
From hunters and military members, to competition shooters and general firearm enthusiasts, we welcome anyone who values and respects the way of the firearm.
Subscribe to our monthly Newsletter to receive firearm news, product discounts from your favorite Industry Partners, and more.
Copyright © 1996-2024 AR15.COM LLC. All Rights Reserved.
Any use of this content without express written consent is prohibited.
AR15.Com reserves the right to overwrite or replace any affiliate, commercial, or monetizable links, posted by users, with our own.