Raising hell is the American way...maybe the CEO should give back some of his pay recent raise to cover decent IT security.

I cannot argue any of that, because I suffer from the same lack of information that you do.

Maybe the speculation is true. People can spin up all sorts of ideas about what it could be. I tend to fall back on Occam's Razor and think it more likely that someone has the common cold than Dengue Fever. So, while it could be a cyber attack or ransomware, etc., it strikes me as more likely some kind of hardware failure or unhandled software or db exception that they are ill-prepared to address.

Over the weekend I was in the local grocery store looking at the expiring meat (what I am cheap and not working) and the butcher does a double take and said he thought I was his dad.

I guess I could have a 20 year old... but no one told me! I did end up shaving most of the beard off then

The reality is servers failing or data corruption is easily and quickly recovered by a professional organization  with off site backups so I doubt that is what happened including the doubt of it being a professional organization that prioritizes data and server redundancy

first this :

"we are expeditiously working with outside industry experts to address the issue"

now this:

"IT is not located at HQ. It is all remote to them."

given in the context from Emoto that HQ doesnt know what is going on.

so apparently not easily contacted which I take to mean "outsourced"  and ARRL is not actually working on it and actually dont know what happened or what is being done about it.

My guess is that they are not spending a bunch of money to have data and server duplicates and backups or security.

and they probably dont have the resources to recover from whatever happened. They certainly dont have the resources to recover quickly because quickly has already passed.

I would question if they even have a functional backup of the data at this point, because if they did they probably would have been back up and running with at least minimal services many days ago

Well going after the backups has been a ransomware thing for a while now, if they didn't have some offline backup somewhere...

The admin contact for arrl.org is

DomainsByProxy.com

which is also not responding



ETA. that is just the godaddy admin privacy reference

Yep!

Before I retired one of my last projects was to honcho the company's migration from an in-house IT plant that had become old, brittle and unsupportable, to a Microsoft 365 cloud solution. You would think that a cloud service automatically includes backup services, but that is not the case. That is an extra-cost service. And their were tiers of that as well.

how long has it been down?

7 days ?!?!


I wonder what the “ principal sponsor “
thinks about this ???

Attached File

I have to agree with what Mach and the rest of you are saying.

They have posted an update, reproduced in its entirety below. The 2nd paragraph, second sentence is new information.

Update on the ARRL site:

Updated 5/22/2024

We are continuing to address a serious incident involving access to our network and systems. Several services, such as Logbook of The World® and the ARRL Learning Center, are affected.

We have heard from many LoTW® users, asking about the status of the service and its data. This is not an LoTW server issue, and LoTW data is secure.

Our editorial and production team is preparing the July issue of QST magazine, which is still going to press. It may be delivered a few days late to members who receive print subscriptions. The digitial edition should be published on time.

We appreciate your continued patience as our staff and others work tirelessly to restore affected systems.

This story will be updated with new developments.
---

ETA:

1. Let this 'incident' prompt you to back up your own stuff.
2. Why do they capitalize the T in LotW? Conventional English language title rules would have the t as a lower case letter, as I have written it.

So it isn’t a server issue and the LOTW data is secure, but they still don’t say what the issue is.


Good chance IMO it is ransom ware. Data is secure, they just can’t access it anymore and they don’t have any backup servers, but maybe the data is backed up or maybe it isn’t.

They sound like they are fucked.It has been 8 days and they still will not articulate what the issue is, except that it is serious.

Nobody give a hoot about the July QST...just fix the damn LoTW website.

Nothing but corporate double speak and additional deflection by the ARRL on LoTW.

Could be. I mean, who the heck knows.



LOL. Yup.

I hope most people upload their logs to QRZ as well.  But I know many don't.

It should be trivial to export an .adif file from QRZ and upload to any other logging service.

I agree with the earlier comments about the LOTW security measures being overkill for a hobby.  

I don't keep full backups of my logs because they just aren't that important to me.  I qualify for mixed WAS but didn't bother to join ARRL just to purchase the paper award certificate.  My QRZ page reflects the same achievement and that's enough for me.  So does my POTA info as well.  Even if all of the data disappeared overnight, I would still feel comfortable telling people I'd worked all states, and if they demanded proof, I would feel like they are taking the game way too seriously.

I think it is pretty clear by now, they CAN'T fix LOTW.

But they can publish QST so that is what they are going to do.

They can probably also collect dues.

The fact that 9 days later they still will not state the issue is very telling. This isn't a hardware or software or DNS issue.

They state it is an access issue. They don't have access. The rest is left up to our imagination.

Personally,  I image a platoon of blondes in short sundresses and heels with guns have secured the server room and are demanding sex and the management of ARRL have said no way and are standing their ground. This standoff might go on for ever.

But I could be wrong.

Sir, does this mean Anne Margret is not coming ???

Attached File

An Admin for the "LoTW (Unofficial)" FB group is telling me that she has inside sources in ARRL who tell her that they were hacked. Obviously, I cannot know if that is true or not.

So when it the mobility scooter mafia marching on the ARRL HQ?  

Sucks if they were hacked. Hope everyone has their stuff backed up. I was going to pay for my membership this year and seen the price went up quite a bit. Needless to say I do not think I will be doing that anytime soon.

after 9 days, I think it is the most probable situation.



And now LOTW’s just a word that you read in history

meanwhile at clublog

Attached File

A guest W1AW operator in the QRZ forums said he went to W1AW and found all the computers were off as directed by the ARRL staff. The computers are configured with the N1MM logging software in a way where he was unable to use the amplifier or cw. That may explain why the routine, computerized W1AW messages haven't been transmitted since May 14.

The most apparent reason to me to turn off the computers for an entire campus would be a virus or attack. In the ARRL's 5/22 update, they assured us 'This is not an LoTW server issue, and LoTW data is secure'. A competing theory I have is the board decided to shut down LOTW until users begged and cried, then turned it back on and introduce a new fee structure with a monthly subscription plus fee per upload, and then pat themselves on the back for being the hero.

It would be trivial to export your log from QRZ/Clublog, but it'd be far from trivial for DXPeditions and silent keys to QSL.

Now why in the bloody hell would you want to start a damned rumor like that?

Estimated probability of full recovery based on days down:   1/days

2 days down = 50%
4 days down = 25%
8 days down = 13%
16 days down = 6%

It's most likely gone.

About the only hope of fundraising after this is for ARRL to sell "I survived the great LOTW disaster of 2024" T-shirts.  Ain't nobody gonna pay dues just for QST and substandard, ineffective lobbying.

I was at a club meeting Tuesday eve and a section manager said they they were "hacked".   He used the term hacked.

It's quite obvious no due to the time it's been down regardless if the ARRL admits it or not. They are most likely in full panic mode.

If the data is lost, then LOTW is done for. I guarantee you that there are QSOs from years gone by that could never be re-entered by the DX. Lots of DXpeditions from decades ago and silent keys.

Attached File


This seems to be the ARRL's current "serious incident" response plan.

My favorite is the counter accusations, the wilder the better.

maybe they are all fighter pilots.

Standard Single Seat defensive tactics when accused of something you actually did on purpose:

1. Act surprised.
2. Deny Deny Deny
3. DEMAND AN APOLOGY!!

See post #9.

Of course they will say that. I really hope no one believes that!

They are going to tell everyone right now the database is “Safe”. I am sure it is, probably has been encrypted and being held for ransom.

This is the ARRL right now.
Attached File


I need someone with photoshop skills to make a hat with the ARRL callsign and add a bunch of patches like ARES and IOTA POTA on his uniform!

LoTW is absolutely secure ... it's jut that the ARRL doesn't have the keys to decrypt it.

It would seem more likely to me that they would avoid putting a total falsehood into writing on their web site, but would instead use weaselly corporate-speak to avoid directly addressing it. So, I do not know what is true or false, but I think bald faced lies are unlikely. I could be wrong, of course.

All kidding aside, I can imagine

1. The ARRL will fix it soon, with full recovery.

2. ARRL never fixes it.

3. ARRL fixes it, but we have to upload all our logs again, and dead guy QSO's are gone forever.

4. ARRL fixes it, but starts charging money to use it.

Indeed they are. I decided long ago to not go after awards. But its nice for the guys that want them. That is the great thing with radio hobby. There is endless facets if things you can do and what you like. I use to use EQSL and club log as well. Now days i only upload to QRZ and LOTW. I do it for the guys that want to do it.  Especially a lot of guys working EME, VHF and Microwave. That is a huge chunk of my contacts. They chase grid squares, FF 6m, DXCC on VHF and above.  Start charging and guys not chasing those awards stop uploading, making it more difficult.

As convient as it is for people to use LOTW, there is something to be said about the old days of ham radio. Its was really designed so we could communicate without being dependent on  a 3rd party for infrastructure to make it happen. Hard to hack good ole fashion QSL cards. The longer this goes on the more people realize how irrelevant the ARRL is becoming. Hell QST has be become nothing but an ad rag. Most of the projects are either so simple that there is no point to making it, or so complex with nonexistent surplus parts that maybe 3 people in the country own, that no one is going to attempt it.

This is what happens when organizations like this get people into power that are all about money and power. Same thing happened with the NRA. From what I am hearing the new CEO is Wayne La Pierre V2.0. Mistakes are mad so they do not have to spend money. Where most of the money coming in go to the people in the organization and not for lobbying or expanding the hobby. Lets face it, if the ARRL closed tomorrow, how much would it affect everyone? There are more VEC,s out there other than the ARRL. Do not get me wrong, I would hate to see it, but some stuff needs to change there if they want to continue as an organization. I am willing to bet that even if they recover and lets say they get LOTW back up and recover the data, they are still going to hemorrhage membership. Now if they do not recover it and all that data is lost, it may be a death knell. I hope it works out in the long run but I am not going to hold my breath.

I just sent off a couple of weeks ago to renew my VE status, going to have to make sure I get on with some other VECs soon.

DXCC has always been my gold standard and I don't think I would have ever been able to get DXCC without using LoTW.  It would have been cost prohibitive for me to confirming around 20,000 contacts.

I've never sent off for any of the awards. Might do at some point, out of boredom. Or not.

I've worked 290 DX entities in the last three years. I have cards from almost every one of them, so while it would suck if LOTW went belly up, I'd still be in pretty good shape. Laugh if you will, but I've worked Argentina more times than I can count, yet not one of them ever had the decency to send a card in return.

double-tap.

Still down. No update on the ARRL site for four days.

I am reminded of this:

Could the ARRL be playing the Y2K gambit, using it as an excuse to drop something they really didn't want to be doing anyway?

Yep, the longer a problem goes on, and you don't "fess up" to what happened, the worse it gets. Someone (or many people) screwed up in some way. But the longer you hide the truth, then people's confidence wanes. Not a good thing. now everyone thinks the data is "secure", yeah secure and password protected by a ransomware criminal that you will never get decrypted unless you pay the ransom.  Whether it is true or not, that is what people think is happening.  If the servers are free and clear, why cant you get them up on the net? A network issue? Yeah what is that? What does that even mean?

Always a good idea to make and keep backups.

Your comment about developers prompts me to point out why that is the case.

Often, software development is incompetently managed by people who have zero technical expertise. The smart ones realize this and hire a "software development project manager". I was one for a number of years, and ran projects for a couple of divisions of a large company, whose CTO would call me in when there was a problem. In other words, I got shit done with good results and was trusted at high levels. Not meant as a brag, just how it was.

Anyway, the first thing often ignored is setting detailed written "requirements" that must be met. Leaving them vague (or verbal only) is a recipe for disaster. Part of this is setting and defining the scope of what is covered. The project manager finds out from the non-technical business managers what it is that they want to end up with, and translates that into tight requirements.

Resources, i.e., developers, quality testers, tech writers, etc., are assigned.

Next is a detailed project plan that has each step necessary for the software to be designed, design reviewed, written, built, tested, accepted, and deployed, all laid out on a timeline with all dependencies linked, and names and dates next to each task. This is like an outline, and detail added as sub-tasks as needed, so everyone can see what has to happen.

Then you execute and keep close tabs on progress, adjusting the plan based upon what happens.

Obviously, this is a gross simplification, but if you want good software, you can't just turn a developer loose and say "do what is needed" and leave it at that. Even the best developers like to have a map for what is needed.

I have no idea how LotW was developed or whether the ARRL has anyone with any technical management skills. The long outage would suggest that they don't have anyone with skills.