Warning

 

Close

Confirm Action

Are you sure you wish to do this?

Confirm Cancel
BCM
User Panel

Posted: 7/27/2024 1:44:39 AM EDT
Anyone else still dealing with the fallout from ClownStrike’s little misadventure? Learned today that our enterprise GIS systems are corrupted and it’s not going to be an easy fix. Good times.

What fun things are my ARFCOM brethren working through now?
Link Posted: 7/27/2024 2:02:39 AM EDT
[#1]
Fuck me

Sorry dude.
Link Posted: 7/27/2024 2:10:28 AM EDT
[#2]
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
Link Posted: 7/27/2024 8:53:27 AM EDT
[#3]
MS is just Miserable Shit.  gates is like a demonic force and always has been.  glad I don't have to deal with it much being retired.  I laugh at the people that are PC die hards. they love apple butter
Link Posted: 7/27/2024 8:58:59 AM EDT
[#4]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
MS is just Miserable Shit.  gates is like a demonic force and always has been.  glad I don't have to deal with it much being retired.  I laugh at the people that are PC die hards. they love apple butter
View Quote

I love ARFCOM.  I come across something new every day.
Link Posted: 7/27/2024 9:00:59 AM EDT
[#5]
Link Posted: 7/27/2024 9:02:47 AM EDT
[#6]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
View Quote


Which is why they'll do it again.

They told you it was a bad patch. An error in programming.

How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust?

To slow productivity. To keep Americans on edge, angry but also fearful?

How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm?

Is Crowdstrike under serious investigation?

Are they being uninstalled by major corporations and governments?

Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart?

This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again??
Link Posted: 7/27/2024 9:28:35 AM EDT
[#7]
My company doesn't use Crowdstrike.
Link Posted: 7/27/2024 10:00:02 AM EDT
[#8]
Quoted:
Anyone else still dealing with the fallout from ClownStrike’s little misadventure? Learned today that our enterprise GIS systems are corrupted and it’s not going to be an easy fix. Good times.

What fun things are my ARFCOM brethren working through now?
View Quote

Yeah, my shop lost a week due to that crap.
Link Posted: 7/27/2024 10:06:28 AM EDT
[#9]
Hmm, our GIS systems didn't have any issues.  What was the corruption?
Link Posted: 7/27/2024 10:10:31 AM EDT
[#10]
I got lucky.  It only impacted the Azure/.Net side of our business.  Everything that I'm responsible for runs on Linux.  Well, not entirely true.  HL7 document processing stopped for awhile.
Link Posted: 7/27/2024 10:15:25 AM EDT
[#11]
I got lucky as well. We had just started a rollout to one of our retail clients.
I launched it on 4 systems at their HQ. One came back on its own. I had to go hands on with 3.

Now they are scared (rightfully so) of Crowdstrike.

We use Huntress on all monitored machine, beginning to wonder if that along with defender is enough for endpoint protection
Link Posted: 7/27/2024 10:25:40 AM EDT
[#12]
The shit with the airlines was the worst goat rodeo I've seen in 10 years of pretty consistent flying.

At one point I heard a Delta customer service desk announce "If you all keep coming up here and yelling at us, we will just shut down our computers and walk away." LOL

Lines hundreds of people deep. Stacks and stacks of baggage. Crazy.
Link Posted: 7/27/2024 10:28:42 AM EDT
[#13]
Link Posted: 7/27/2024 10:31:17 AM EDT
[#14]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Bill Gates stepped down as CEO like 18 years ago.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
MS is just Miserable Shit.  gates is like a demonic force and always has been.  glad I don't have to deal with it much being retired.  I laugh at the people that are PC die hards. they love apple butter
Bill Gates stepped down as CEO like 18 years ago.


Stock price is fairly high also.
Link Posted: 7/27/2024 10:32:05 AM EDT
[#15]
It's been a week, why have they not recovered from a backup or failed over to a replica?
Link Posted: 7/27/2024 10:35:38 AM EDT
[#16]
Apparently you can buy IT insurance so when this shit happens, the insurance covers the costs associated with recovery. It's beyond fucking expensive but companies are now making it a requirement before they will do business with you. Which means unless you are a big company with deep pockets and profits higher than a giraffes asshole, you ain't doing business.
Link Posted: 7/27/2024 10:36:33 AM EDT
[#17]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
View Quote

How do you think any of the crowdstrike thing is microsoft's fault?
Link Posted: 7/27/2024 10:39:42 AM EDT
[#18]
I got lucky and wasn’t impacted one bit, besides the VPs harassing me for status. Some of our customers use it and they were circling the wagons.


I press hard for minimalist approach and want everything we install to have a purpose that has a clear ROI.  I’d argue that crowdstrike Falcon is a solution looking for a problem.  Makes people feel like they are doing something.


You don’t need it.  Has it ever actually prevented something?  I doubt it.

And ironically, most of the impacted systems that made the news (kiosks, checkout machines, gate agent machines) are terminals that should be booting off a remote image.  Pushing updates to them is stupid.   Even worse, most of these things shouldn’t even be on windows.



Link Posted: 7/27/2024 10:40:22 AM EDT
[#19]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


Which is why they'll do it again.

They told you it was a bad patch. An error in programming.

How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust?

To slow productivity. To keep Americans on edge, angry but also fearful?

How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm?

Is Crowdstrike under serious investigation?

Are they being uninstalled by major corporations and governments?

Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart?

This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again??
View Quote


I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually.

The error was a null pointer.. basic shit not to fuck up.

Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it.

So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels
Link Posted: 7/27/2024 10:43:18 AM EDT
[#20]
Why did everyone patch the EMS client at the same time?
Maybe dont allow auto updating of your systems and do phased upgrades and testing.

Reminds me of a DIE hire we had who was in charge of a major front end application update.
They did it on Thursday after 3am.
Friday morning DIE hire emailed all that the update was a success.
Until all the machines that had already rebooted at midnight on thursday rebooted at midnight on friday.
Half our frontline stations were bootlooping on Saturday morning.
Our busiest time.
Same fix as the crowdstrike.
But we had to boot into barts PE and remove the file.
I was fixing the issue for weeks as many back office machines had the application as well.
Link Posted: 7/27/2024 10:44:31 AM EDT
[#21]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
View Quote


If a software vendor writes code that runs in kernel mode, and the code is corrupted, that type of action can crash any system.  MS is a blue screen, Linux is a black screen, and Apple has a pink screen.  
Apple has been compromised recently, and a serious back door exploit has recently been found in new Linux distros.  So this begs the question oh thou great IT professional, what O/S do you use?

Link Posted: 7/27/2024 11:14:49 AM EDT
[#22]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?
View Quote

this
Link Posted: 7/27/2024 11:16:00 AM EDT
[#23]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

How do you think any of the crowdstrike thing is microsoft's fault?
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.

How do you think any of the crowdstrike thing is microsoft's fault?

Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen.
Link Posted: 7/27/2024 11:37:54 AM EDT
[#24]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?
View Quote


The computers are locked into an endless loop.  They don't get to a stable place where a restore can be done.  And I'm guessing the issue is with real PCs and not virtual machines.

And if any of the databases were corrupted when the computer initially scrammed itself and rebooted, then you need to get the computer clean and then restore the DB.

It all takes time when people need to physcially touch each computer in/affected.
Link Posted: 7/27/2024 11:48:02 AM EDT
[#25]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually.

The error was a null pointer.. basic shit not to fuck up.

Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it.

So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:


Which is why they'll do it again.

They told you it was a bad patch. An error in programming.

How easy would it be to intentionally write a bad patch to weaken the American economy. To cause havoc and mistrust?

To slow productivity. To keep Americans on edge, angry but also fearful?

How easy is it to manipulate people who believe every story fed to them by the people actually doing the harm?

Is Crowdstrike under serious investigation?

Are they being uninstalled by major corporations and governments?

Is there an alternative to Crowdstrike? Would people be motivated to accept a new company and operating system, or would it just be easier to just let Crowdstrike continue o and just believe they have our best interest at heart?

This outage affected less than 10% of all applied Microsoft operating systems and look at the chaos it caused? Now imagine 50% of the operating systems affected to include power, water and delivery systems. Do YOU trust the masters who are whipping you to not whip you again??


I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually.

The error was a null pointer.. basic shit not to fuck up.

Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it.

So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels

The problem is that it was not a code update. It was definitions. So code analysis would not help. It might throw a generic potential issue message, but they would have set that on permanent ignore since it would be expected for cases where it worked.
Link Posted: 7/27/2024 11:48:17 AM EDT
[#26]
Incidents like this make it hard not to ponder talk of the "competency crisis" and even the "Atlas Shrugged" premise of the lights going out in NYC.
Link Posted: 7/27/2024 11:49:38 AM EDT
[#27]
I want the DNC server!  

Fuckers buried it.
Link Posted: 7/27/2024 11:50:52 AM EDT
[#28]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.

How do you think any of the crowdstrike thing is microsoft's fault?

Microsoft is blaming the EU by forcing MS to permit AV companies from running in kernal mode that allows for this problem to happen.

They have had filter drivers for a long time. The problem is forcing them to certify/sign drivers that can "run" unvalidated p-code. Like antimalware programs that push up to the minute definitions.
Link Posted: 7/27/2024 11:52:05 AM EDT
[#29]
To be quite honest if you haven't recovered from something that, at its worst case, takes 20 minutes to rectify, your IT infrastructure deserves to fail for being poorly constructed.
Link Posted: 7/27/2024 11:54:01 AM EDT
[#30]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Bill Gates stepped down as CEO like 18 years ago.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
MS is just Miserable Shit.  gates is like a demonic force and always has been.  glad I don't have to deal with it much being retired.  I laugh at the people that are PC die hards. they love apple butter
Bill Gates stepped down as CEO like 18 years ago.


Sexual harassment investigations will do that to a person.

Too bad he didn’t stay and just focused on doing evil in the computer world instead of spreading it across everything.

Link Posted: 7/27/2024 11:59:09 AM EDT
[#31]
if you use someone elses datacenter (cloud) then you need to take some responsibility for your data.  using cloud doesnt take away the need for backups, point in time recovery, redundancy etc.. it just makes those processes easier.  

Link Posted: 7/27/2024 12:01:15 PM EDT
[#32]
Discussion ForumsJump to Quoted PostQuote History
Quoted:


I'm not a developer but I've fucked with enough C/C++ over the years just being an open source software user back in the days where you had to compile everything yourself or actually apply source patches manually.

The error was a null pointer.. basic shit not to fuck up.

Pretty much every compiler should warn or abort on a null pointer reference. Or if they are running some code analysis tool like Coverity against their code it would flag it.

So it sounds to me like their build process intentionally sets compiler flags to ignore certain warning levels
View Quote


Null pointers are not fatal in userspace, but yeah - this was apparently in the kernel.

I spent a week walking Fortune 100 level companies through rescuing thousands of broke cloud instances.

CSB - Microsoft rescue boot media didn't work in the cloud enviro. No time to debug why. A Linux based rescue ISO worked immediately and perfectly. Just had to add NTFS drivers.

So it took Linux to unfuck Microsoft.
Link Posted: 7/27/2024 12:02:50 PM EDT
[#33]
Thankfully we don’t use Crowdstrike.  But some vendors we use for certain services do, so we had issues with those systems.  Thankfully our payment processor isn’t one of them.  Our disruption was minimal.
Link Posted: 7/27/2024 2:09:39 PM EDT
[#34]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Hmm, our GIS systems didn't have any issues.  What was the corruption?
View Quote



From ESRI:

However, after recovery, ArcGIS Enterprise users reported issues with components not starting properly. These issues were traced back to the corruption of ArcGIS Enterprise configuration files, which were impacted by the operating system crash. This issue is tracked by Esri as BUG-000169285: "Update to the CrowdStrike software results in failure of ArcGIS Enterprise components installed on affected machines."
Link Posted: 7/27/2024 2:13:42 PM EDT
[#35]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?
View Quote

Or just remove the offending files. It’s really easy to recover from this.
Link Posted: 7/27/2024 4:36:53 PM EDT
[#36]
Do we have a choice in regards to Crowdstrike?
Link Posted: 7/27/2024 4:40:37 PM EDT
[#37]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
View Quote
Microsoft rejected DEI, 5 days later Crowdstrike who prides itself on DEI(and is owned 15% by Blackrock/Vanguard) releases an update that takes Microsoft down.

Click To View Spoiler
Link Posted: 7/27/2024 4:51:20 PM EDT
[#38]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

Or just remove the offending files. It’s really easy to recover from this.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?

Or just remove the offending files. It’s really easy to recover from this.

I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping.
Link Posted: 7/27/2024 4:54:38 PM EDT
[#39]
CrowdStrike isn't going to miss you.
Link Posted: 7/28/2024 4:21:41 PM EDT
[#40]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?

Or just remove the offending files. It’s really easy to recover from this.

I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping.

because of this, our keys are stored in local AD, on a file server, and in our RMM.  I need to confirm that we are getting a copy properly stored in EntraID (Azure AD)
Link Posted: 7/28/2024 4:24:19 PM EDT
[#41]
Link Posted: 7/28/2024 4:27:37 PM EDT
[#42]
I'm still waiting for the hacker/ransomware fallout from everyone disabling it, to hit.
Link Posted: 7/28/2024 4:32:06 PM EDT
[#43]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
I want the DNC server!  

Fuckers buried it.
View Quote


This.

How a company that tied themselves to the hips of the DNC and Hillary so completely is trusted by ANYONE is beyond me.  
Link Posted: 7/28/2024 4:32:21 PM EDT
[#44]
Discussion ForumsJump to Quoted PostQuote History
Quoted:

I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping.
View Quote View All Quotes
View All Quotes
Discussion ForumsJump to Quoted PostQuote History
Quoted:
Quoted:
Quoted:
It's been a week, why have they not recovered from a backup or failed over to a replica?

Or just remove the offending files. It’s really easy to recover from this.

I've seen a company where the bitlocker keys were stored in AD and all of the DCs took this update and started bootlooping.

Yep, that might happen. Best keep backups somewhere.
Link Posted: 7/28/2024 4:43:57 PM EDT
[#45]
Discussion ForumsJump to Quoted PostQuote History
Quoted:
It wasnt just Crowdstrike, Microsoft needs to wake the fuck up as well. The whole thing was pants on head retarded and easily preventable.
View Quote


This one wasn't MS's fault. If you want to blame anyone but Cloudstrike, blame the EU.
Close Join Our Mail List to Stay Up To Date! Win a FREE Membership!

Sign up for the ARFCOM weekly newsletter and be entered to win a free ARFCOM membership. One new winner* is announced every week!

You will receive an email every Friday morning featuring the latest chatter from the hottest topics, breaking news surrounding legislation, as well as exclusive deals only available to ARFCOM email subscribers.


By signing up you agree to our User Agreement. *Must have a registered ARFCOM account to win.
Top Top