# Site Outage Log



## Gandalfe

*Weekend Outage*

Down for a number of hours Sunday night. Harri figured out what it was and fixed it post haste.


----------



## Harri Rautiainen

*Server reboot*

Down maybe for 15 minutes Sunday 12:35 - 12:50 EST.


----------



## Gandalfe

*Closed Sunday for an hour*

Down due to SQL database problems this morning.


----------



## Harri Rautiainen

*April 29-30 site outage*

The site has been out from 04-29-2006 midnight (EST) almost 45 hours.

This was due to fact that (first time) the SOTW site exceeded the 60 Gigabytes monthly limit set forth in our service package.

It was unfortunate that the ISP cut off the service without a warning, and that I was out of town during this week-end.

Anyway, now we are back, and did not have to wait for beginning of the month of May (Dallas time).

My sincere apologies for the inconvenience,
-Harri


----------



## Saxhound

Sounds like we need to do more "Donate the proceeds from a sax mouthpiece sale" stuff!

60 Mb is ugly. Is that average or peak?


----------



## Harri Rautiainen

Saxhound said:


> Sounds like we need to do more "Donate the proceeds from a sax mouthpiece sale" stuff!
> 
> 60 Mb is ugly. Is that average or peak?


60 gigabytes per month is quite reasonable. Yet, because we do not down-load big files, etc., it is quite a lot of ASCII-character information.

It is a monthly not-to-exceed limit. First time in the SOTW history we now did exceed it, and that lead into trouble.
The ISP should not have turned off the switch without e-mailing first, considering that the SOTW is an old reliable customer.


----------



## Saxhound

I was thinking that 60 Mb was the circuit bandwidth - i.e. > T3. That would be a pricey connection. Glad to hear it is a cumulative thing.


----------



## BayviewSax

Before you go paying that extra $10/month, might I suggest checking out www.netfirms.com. I get quite a bit of space and their MAX fee is $14.95/month for a boatload of transfer.


----------



## JfW

> The ISP should not have turned off the switch without e-mailing first, considering that the SOTW is an old reliable customer.


Hosts generally have this type of thing automated.


----------



## kcp

how rude... they coudn't even send an automated message


----------



## jazzbluescat

If it'll help anything, I'll send you a message.


----------



## Pete

<opinion> While it's annoying that they disabled the account and may not have sent an e-mail saying that you were reaching quota -- Harri, check your JUNK MAIL folder -- it's also something that's not exactly difficult for the owner of the website to check: it's on the frontpage of the control panel (Kim, etc. can't check, unless Harri has given them passwordage). </opinion>

That being said, I'm pretty darn happy with the lack of downtime with ASO, even though the other provider mentioned is pretty darn cheap.

But that's just me: I didn't even know the Forum was down until someone e-mailed me .


----------



## Harri Rautiainen

*Outage on Wed May 10*

Recovery Alert

Your Web Site is no longer on error

URL: http://www.saxontheweb.net/ 
Time: Wed May 10 15:49:24 EDT 2006 
Total Time on Error: 12 minutes, 43 seconds


----------



## Pete

Am I insane or has the website been up and down like blinky Chrismas tree lights for the past couple weeks? I only go here once or twice a day and I've seen lots of "Contact Billing Department" and/or "There is a Problem with the Database" errors.


----------



## Hurling Frootmig

We're checking into why this has happened occassionally.


----------



## fballatore

saxpics said:


> Am I insane or has the website been up and down like blinky Chrismas tree lights for the past couple weeks? I only go here once or twice a day and I've seen lots of "Contact Billing Department" and/or "There is a Problem with the Database" errors.


Yep, I knew I wasn't the only one. I've been waiting for someone to say something about it. Don't want to be a pain in the you know what. There was a period of time last week that it was down for a 2-3 hour stretch.

:?


----------



## Harri Rautiainen

*Outage Jun 13, 2006*

There was an outage some 90 minutes ago. I cannot tell the length yet, but my guess is 30-40 minutes based on time stamps on new messages. A database repair cured it.

"Contact Billing Department" was strictly administrative in nature. ASO did not know how to invoice the extra bandwidth I purchased because of the previous cut in service.


----------



## Hurling Frootmig

We just had a problem with the hosted server. It looks like it was rebooted.


----------



## Gandalfe

There sure have been a lot of problems accessing the SOTW site in the last couple of months. So far this week, every day I have been unable to access the site for some time. Have we gotten too big for our britches?


----------



## Harri Rautiainen

*reply from ASO*



Hurling Frootmig said:


> We just had a problem with the hosted server. It looks like it was rebooted.


reply from ASO (the service provider):


> Harri,
> This is often the problem with being on a shared server. The server
> contains many domains and around that time of day there are a lot of
> http, email, mysql, etc requests that cause the server to have some
> issues. The servers load is low now and your domain is resolving without
> issues.
> 
> Regards,
> Jayson


If this is becoming a daily problem what shall we do; go to a dedicated server perhaps?


----------



## Hurling Frootmig

Dedicated would be great but I'm sure that would cost a lot more than what we pay now.


----------



## Gandalfe

Harri Rautiainen said:


> If this is becoming a daily problem what shall we do; go to a dedicated server perhaps?


Depends on whether we can afford the cost. It was my recommendation last year (and others too? I can't remember) to consider a dedicated server solution. Wasn't the cost less than ten bucks a month more?


----------



## jazzbluescat

Time to *cough* *cough* it up?


----------



## Hurling Frootmig

It appears that our server had an issue with mysql a little bit ago and the service had to be restarted. My guess is that we were down for about 30 minutes due to this issue. I was posting something on the forum when it occured so I can tell you that the issue was caught very quickly


----------



## fballatore

Hurling Frootmig said:


> It appears that our server had an issue with mysql a little bit ago and the service had to be restarted. My guess is that we were down for about 30 minutes due to this issue. I was posting something on the forum when it occured so I can tell you that the issue was caught very quickly


I don't mean to be a pain, but it was more like an hour and a half. I was trying to get on to respond to a post around 10AM EST and couldn't. About 1/2 hour later, I directly emailed the other SOTW member I was trying to reach. According to the post times, it didn't come back until after 11:30. This has been a fairly consistent issue over the past few weeks. I'd highly recommend a dedicated server if that will end these issues.

Frank


----------



## Hurling Frootmig

You're probably right it was more like an hour and a half. I just checked my e-mail logs from dealing with the event. As a computer guy what seems like 30 minutes to me is normally about 3 times longer in real time. My wife has figured this out but I still haven't. I like living in my false sense of time. Something Brubeck about it.


----------



## Pete

Hurling: leave the jokes to the professionals .


----------



## jazzbluescat

Don't listen to'im, Hurling. It was a very good joke. You got potential.


----------



## Hurling Frootmig

Well we just had another outage. This one lasted somewhere under 30 minutes but more than 10 minutes. I had to optimize one large table which made the outage last a few minutes longer.

We're working on a couple of ways to fix this.


----------



## saxymanzach

Another outage at 4:10? Lasting possibly 15 mins.


----------



## Harri Rautiainen

During past couple of hours there were two short service breaks due to installing system software on the new SOTW server.


----------



## Harri Rautiainen

*Site outage*

Yesterday at 11PM Californian time there has been a major burst in ingoing and outgoing traffic. After that it looks like the system has been down for 11 hours. I am trying to have some vacation, and checking the systen only once per day. I apologize for the long delay in rebooting the server.

I have asked our new techie guy to study the log files in order to get more illumination to the problem.

-Harri


----------



## Harri Rautiainen

Harri Rautiainen said:


> Yesterday at 11PM Californian time there has been a major burst in ingoing and outgoing traffic. After that it looks like the system has been down for 11 hours. I am trying to have some vacation, and checking the systen only once per day. I apologize for the long delay in rebooting the server.
> 
> I have asked our new techie guy to study the log files in order to get more illumination to the problem.
> 
> -Harri


Tuesday night server outage is still a mystery. Nobody logged on right before it. However during past 5 days there has been :
4 dictionary break attempts and 1 tarpit attack.

As a cautionary measure I am going to make the interval between postings to the forum longer.


----------



## Harri Rautiainen

From our ISP on November 16, 2006:


> Dear Harri Rautiainen,
> 
> At 11:15am, the UPS system protecting our C2 datacenter experienced a catastrophic system failure. During this failure, the system control board were damaged, which prevented the cutover to generator power. Power was down to the facility for less than 5 minutes as the UPS was bypassed and generator power was implemented.
> 
> Cari.net network service personnel began a server by server inspection to ensure that all customer servers were back in operation. Hardware replacements and reboots were handled as top priorities, and customer phone calls and cases were answered in a first come, first serve basis.
> 
> All UPS systems which support Cari.net datacenters are constantly maintained and serviced by industry experts. A replacement UPS is on order as a precaution, and a certified repair team will be onsite first thing tomorrow (11/16/06) to assist in bringing the UPS back online without further incident. Further information will be provided as it becomes available.
> 
> We apologize for any issues that this outage may have caused. We are working hard to ensure that issues of this nature do not resurface.
> 
> Thank you.
> 
> Chris Orlando - Vice President
> Cari.net


----------



## Harri Rautiainen

*Site outage*

There was a site outage that started yesterday (Dec. 14) around 5 PM Californian time and lasted for seven hours.
Cari.net NOC Technicians did not react to my first request to re-boot the system. No explanation offered.

On the positive note: 5 months of uninterrupted running is better than what we had before.


----------



## Harri Rautiainen

*Interrupt of Service*

I have got reports from the forum users recently of some strange and unexpected sw responses on the forum. I thought that they were caused by some database problems. On Jan. 15 around 11:30 californian time I saw some problems on the SOTW main site, too, plus a sudden increase of the system load.

So, what do the computer types do, if their car will suddenly stop on the high way. They all leave the car, close the doors, and then enter the car again trying to start it. So I decided it is time for the LINUX reboot to solve the problems. However, the server did not reboot under command line control. The cari.net technicians and my tech person (my son) had to work one hour to get it back to the net. It was before 1 PM.

The forum came back again, and the problems it exhibited are gone. There were soon a record number of simultaneous on line users, almost 300. Now the number of users is dropping fast, but the server load is still unusually high. Also there are some hard disk errors verified. I am afraid that we are for another scheduled system stoppage some 12 hours from now.


----------



## Harri Rautiainen

*Another short one*

Another short stopping of the service trying to get rid of the extra system load.

Looks better now.


----------



## Razzy

Last night me and at least two other forum members experienced a 4 hour plus downtime. About 11pm to 3am EST.


----------



## Harri Rautiainen

Razzy said:


> Last night me and at least two other forum members experienced a 4 hour plus downtime. About 11pm to 3am EST.


There was a shot interrupt (the server was down 15-20 min) around 4:30 AM PST. So it was 1:30 AM your time.

However, because the system was not very stable before and after that it may very well be and you were not able to receive the pages.

More about the latest long ones in the next post. I still have some loose ends to tie up.


----------



## Harri Rautiainen

*SOTW Forum will go down*

Although cari.net got the system back it is unusually slow.

I am afraid that we have to replace a hard disk on the server.
We are forced to have another interrupt in service.


----------



## Harri Rautiainen

*Back in Business*

18 hours later, here we are.......

It has been quite exciting 30 hours altogether. (I got 7 hours sleep in between, though.) :banghead:

The latest service break involved replacing server's hard disk and installing the operating system from scratch. My son is still doing some tuning of the software, but the forum and the resource site seem to be operational.

Please let me know or post here, if you experience any problems or anomalies.

Some of you may hae felt some withdrawal feelings.  
Thank you all for your patience.


----------



## Eric Bessette

Harri Rautiainen said:


> 18 hours later, here we are.......
> Some of you may have felt some withdrawal feelings.
> Thank you all for your patience.


Withdrawal indeed. I wish I could at least say I got more work done.

Thanks for all your hard work to make this all happen Harry.

eab


----------



## Harri Rautiainen

*Admin e-mail problems*

Just realized the the mail server on saxontheweb.net did not start properly until now. So I am missing hundreds of junk mail from past 36 hours.

The bad news that your e-mail sent to admin_at_saxontheweb.net did not come thru, sent from the forum or otherwise.

Please do not panic. Most e-mail servers try several times over several days before giving up. However, if you see that the issue you reported is not getting taken care over next few days, please resend.


----------



## Harri Rautiainen

*Server outage Jan. 31, 2007*

The SOTW server was out of operation between 2:30 and 5:30 AM PST today.

Sorry for the inconvenience. Everything seems to be smooth right now.


----------



## martysax

Harri Rautiainen said:


> The SOTW server was out of operation between 2:30 and 5:30 AM PST today.
> 
> Sorry for the inconvenience. Everything seems to be smooth right now.


Thanks for clearing that up. I thought my wife sabotaged my browser!

She lives.


----------



## tjontheroad

"The server is too busy at the moment. Please try again later."

Sadly, this message is getting all too common.


----------



## Harri Rautiainen

*server load?*



tjontheroad said:


> "The server is too busy at the moment. Please try again later."
> 
> Sadly, this message is getting all too common.


Are you saying that you have seen it more than once?

Anyway, I turned off the searching for guests so that should help a little.

Looks like time to look for a server upgrade :?


----------



## tjontheroad

Harri Rautiainen said:


> Are you saying that you have seen it more than once?
> 
> Anyway, I turned off the searching for guests so that should help a little.
> 
> Looks like time to look for a server upgrade :?


Yes, quite often. I even had trouble posting this. I had to refresh the thread 3x to get it to come up.


----------



## Carl H.

I'm on my 2nd try in posting to this thread now too. 

Maybe a virus gumming up the works?


----------



## Harri Rautiainen

I do not see anything extraordinay right now.

The server load is reasonable 2 and the CPU is running 60%....80% idle. :?:


----------



## Carl H.

WOW, what was that, almost 6 hours !


----------



## G-dawg

tjontheroad said:


> "The server is too busy at the moment. Please try again later."
> 
> Sadly, this message is getting all too common.


This was the only message I saw last night trying to logon. From about 7:30 pm until 10 pm local time....

Didn't sleep well without my SOTW fix.


----------



## Harri Rautiainen

Carl H. said:


> WOW, what was that, almost 6 hours !


Carl H., G-dawg and many others:
I feel like owing you an explanation. After the recent system failure I wanted to identify forum's performance bottle-necks. I changed some search parameters, and was instructed to rebuild the entire search index after that. I knew that it was a big task, but did not realize that it will consume all CPU resources for several hours. Almost 100 registered members and more than 100 guests who were in, stayed in forum and got some level of service response. However, the new ones were not permitted to log-in and received the "server busy" message.

I certainly picked the wrong time to do a major maintenance task (peak time for the Americas members.) As a consolation, I am going to study the search situation and performance generally speaking further, but do not plan for any drastic maintenance in near future.


----------



## Carl H.

> Your search is in progress and you will be taken to the results in a moment. Thank you for
> your patience.
> 
> Click here if your browser does not automatically redirect you.​





> Thank you for posting! You will now be taken to your post. If the administrator has selected to moderate all posts in this forum, you will be taken back to the forum and your post will be displayed presently. If you opted to post a poll, you will now be allowed to do so.
> 
> Click here if your browser does not automatically redirect you.​


New additions?

BTW I have gotten a couple "busy" messages this morning too.

Not complaining, mind you, just letting you know.


----------



## kavala

There still seems to be a few problems Harry.

You cant use the 'Back' button sometimes. Extra dialogs popping up.

I liked the old SOTW better. Progress - sheeesss.


----------



## Harri Rautiainen

Carl H. said:


> New additions?
> 
> BTW I have gotten a couple "busy" messages this morning too.
> 
> Not complaining, mind you, just letting you know.


Carl and kavala,

The "user assurance" message removed.
The server load limit increased.

Appreciate your messages, keep us posted if problems occur.


----------



## jazzbluescat

Harri Rautiainen said:


> ......Almost 100 registered members and more than 100 guests who were in, stayed in forum and got some level of service response. However, the new ones were not permitted to log-in and received the "server busy" message.
> 
> .....


I must be an exception. I was on the forum for about 1/2 hour, until I tried to post, then I got the "server busy" message. I got the same message upon trying to access any/all the forums, effectively shut out, not allowed to return to SOTW.

Glad you got the ship sailing again.


----------



## Harri Rautiainen

*Server overload*

There was a server overload situation today strating around 7:45 AM EST.
This caused the server to "freeze". The situation was corrected at 9AM, and apache parameters tweaked to avoid this in the future.

Sorry for the inconvenience,


----------



## Harri Rautiainen

*Database error*

A database error occurred on Saturday, March 17th 2007 around 5 PM Pacific time.
It was caused by a third-party plug-in which is now temporarily disabled.

It took me 50 minutes to find a fix and this is cutting already to my sleepy time (3 AM here right now.) I had a 12 hour old back-up of the entire forum data, but did not want to resort to it in first place.

Sorry of the inconvenience,


----------



## jazzbluescat

Harri Rautiainen said:


> .....It took me 50 minutes to find a fix and this is cutting already to my sleepy time (3 AM here right now.) I had a 12 hour old back-up of the entire forum data, but did not want to resort to it in first place.
> 
> Sorry of the inconvenience,


Thanks Harri; get your beauty sleep.


----------



## bari_sax_diva

Hey, Harri... 16:00 California time, and I've been getting "Forum too busy" errors all day. Seems odd for a Monday.

BTW, hope you got some sleep last night...


----------



## Gandalfe

Hmm... I've been on and off the forum from 3:30 Pacific time till almost 9 PM now. I am unable to repro your experience.

I've a couple of questions:

1. What ISP are you using? 
2. Is your speed DSL or Cable?
3. Do you have more than one computer on the feed line from your ISP?
4. What Internet Browser are you using and version?

Also, is anyone else having these problems? If you are could you let us know the answers to these questions?

Thanks everyone.


----------



## saxmanglen

I've seen that a few times today but a refresh usually has has corrected it.


----------



## retread

I had the problem at, more or less, 1700 CDT yesterday. On Roadrunner Cable, only computer in the home, Mac with the latest Safari.


----------



## bari_sax_diva

I've got cable through Charter Communications. I'm using Firefox 1.5 on Mac. There's another computer on my network, but we're rarely surfing at the same time.


----------



## Mark5047

I am using IE 6 as well as IE 7 on different computers - different ISP's and have had an occasional "Forum too busy" error messages. Nothing so far today, but it happened about 3 times in a row last night. It didn't clear up until I hit refresh a couple of times. 
I was clicking on the 'New Posts' link each time, if it helps.


----------



## Mark5047

1:44 Eastern Time (GMT - 5) I received the following:

vBulletin Message 
The server is too busy at the moment. Please try again later.

Not a complaint, just passing along info.

Mark


----------



## Harri Rautiainen

*Service interrupt*

The saxontheweb.net site was down approx. 30 minutes.

This had nothing to do with the recent server change (I hope), because the ISP's site was down, too.

No word from cari.net, yet.
===================================
Add 3,5 hours later:

There were tho relatively short interruptions today. The First was over about 4 AM PST and the latter 7:10 AM PST.

No explanations, yet.


----------



## Harri Rautiainen

*From ISP*

From ISP (cari.net):


> Dear Customer,
> 
> We experienced a momentarily power failure, we have our technicians working on it and at the moment we are working on bringing all the servers online, we will take care of updating this case as soon as we have more news.
> 
> Thank you very much for your patience
> Cari Net Support


----------



## jazzbluescat

Wonder why they don't have backup power?


----------



## Harri Rautiainen

*Site outage*

June 10, 10:30PM U.S. Eastern time, and came back up June 11, around 3:00AM.


----------



## Harri Rautiainen

*Service outage*

July 15 starting 10AM PST gradual degradation of service.
Totally down appr. 10:30 and service restored 10:55 AP PST.


cari.net said:


> I'm terribly sorry but our connection outward has been shaky during this time and sending out a massive notice would currently be unreliable. We have been working with our upstream providers to isolate each offending IP address and null route them. Currently Two out of Three of our providers is back on line and synchronized. We are expecting the last one to be up shortly.
> 
> Please stand by until this is done, we are terribly sorry for the inconvenience.
> 
> Thank you for your patience and understanding,
> Technical Support


----------



## Harri Rautiainen

*Service outage today*



> Recovery Alert
> 
> Your Web Site is no longer on error
> 
> URL: http://www.saxontheweb.net/
> Time: Wed Dec 05 17:32:32 EST 2007
> Total Time on Error: 1 hour, 24 minutes, 38 seconds


No reason, yet, but I can assure that this was not done by the Turkish hackers group who caught the saxgoumet site. We do have safeguards against such attacks.


----------



## Harri Rautiainen

*Jan 27, 2008 - DDoS attack*

The recent service break lasted less than two hours.


> *Recovery Alert*
> 
> Your Web Site is no longer on error
> 
> URL: http://www.saxontheweb.net/
> Time: Sun Jan 27 15:27:35 EST 2008
> Total Time on Error: 1 hour, 27 minutes, 52 seconds


It was not caused by server problem but a massive distributed denial-of-service attack (DDoS attack) toward our service providers network.
_
added later:_
It looks like the access to SOTW was not totally blocked. Anyway, I was monitoring it in the net, and most of the time all continents were blocked:


> Hi Harri,
> 
> I read your outage report about a Denial of Service attack today. I was on line nearly all day, and didn't experience this. What time did it happen?
> 
> The only odd thing that I did see was at one point this afternoon (central US time zone), there were only 3 or 4 members on line, which is very strange indeed.


----------



## Harri Rautiainen

*Re: Jan. 27, 2008*

More illumination:


> Dear Harri Rautiainen,
> 
> Date & Time (PDT): Sunday, Jan 27 2008, 11:13 - 13:00 (This issue has been resolved)
> 
> A network related issue affected a number of Cari.net customers this afternoon. We have provided a brief update below. Further information will be provided as soon as it is available.
> 
> Details:
> At approximately 11:30am today, one of Cari.net core routers began operating in a degraded mode, resulting in HSRP issues within specific corporate VLANS. Some customer experienced caching DNS issues, intermittent control panel access, and other minor issues with internal resources such as bandwidth graphing servers were reported. After the router was reset, the Cari.net customer network was restored to full functionality.
> 
> The routing advertisements caused by the router reset uncovered a routing issue in the Cox subscriber network. This was immediately escalated to the Cox engineering team and when they could not resolve the issue, the Complex Drive network operations team made the decision to close down the Cox transit connection. This action resolved all Cox subscriber access issues.
> 
> At this time the network is fully operational, with the Cox circuit disconnected until the connectivity issue is resolved. A window will be scheduled with notice to customers in order to bring the Cox connection back online.
> 
> We will provide further information regarding this incident as soon as it is available. Any customers who have questions or concerns should contact our customer service department to open a work order. You may also reply to this message with non-emergency information.
> 
> We apologize for any inconvenience.
> 
> Mark Ortenzi
> President
> Cari.net


----------



## Harri Rautiainen

*90 mins on March 13,2008*

Approx. between 8:30 and 10 AM Pacific time.

No reason given by our service provider.


----------



## SuperAction80

*Re: 90 mins on March 13,2008*



Harri Rautiainen said:


> Approx. between 8:30 and 10 AM Pacific time.
> 
> No reason given by our service provider.


I thought that I was having some browser trouble on my end of the line, but the other vbulletin sites were up and running. Ain't technology grand?


----------



## Harri Rautiainen

*March 13, 2008*

The bulletin came out:


> Dear Harri Rautiainen,
> 
> At 8:51am Cari.Net's network began experiencing significant routing and BGP issues. The problems continued until 10:10am, during which time customers experienced a variety of connectivity problems ranging from DNS to direct Internet connectivity issues. At this time all services have been restored.
> 
> We apologize for the outage and any inconvenience it may have caused. A full scale inquiry with external resources is being conducted and details will be provided within the next 2 hours.
> 
> If you are still experiencing connectivity issues, please call our 24/7/365 Support Center or open a trouble ticket.
> 
> Thank you.
> 
> Cari.Net Network Staff


----------



## Harri Rautiainen

*March 18, Outage*

From Cari.net:


> Dear Harri Rautiainen,
> 
> This afternoon Cari.net experienced a network outage that appears to have similar root cause as the previous incident that occurred on Thursday of last week. During the 48 hours following Thursday's incident, Cari.net along with our partners came up with a number of network changes that were necessary to resolve recent network reliability issues.
> 
> Cari.net held a Network Maintenance Window on Friday March 15th to implement the first and most critical set of changes which were deemed necessary. While these initial preventative measures were rolled out as planned, it would appear that the overall root cause of the issue was further reaching than previously understood.
> 
> Today the issue was able to re-manifest itself and again it was propagated to both primary and redundant routing legs within our network.
> 
> Within minutes of the incident beginning, Cisco's Tier 1 Support along with a 3rd party consulting organization worked in conjunction with Cari.net's Networking Team to combat the issue. At the moment, the Network is stable, and senior members of each team will remain on site until permanent resolution can be proven.
> 
> Further updates will be made as soon as they are available. Details regarding root cause and preventative measures taken will be made available as soon as possible.
> 
> We apologize for the problems and issues this incident has caused.
> 
> Best Regards,
> 
> Mark Ortenzi
> President - Cari.net


----------



## Harri Rautiainen

*Tue Aug 19, 2008*

Time: Tue Aug 19 16:36:42 EDT 2008 
Error Type: Time Out

Back Time: Tue Aug 19 17:12:47 EDT 2008 
Total Time on Error: 40 minutes, 58 seconds

The SOTW dedicated server is running also email services for saxontheweb.net.
Apparently there was a situation where a spammer has been faking _admin (at) saxontheweb.net_ as a sender address and spamming in large scale. The SOTW server was bombarded with error mail messages cause by a huge number of spam sent to non-existing addresses. That took so much of the memory that all other tasks ground to halt.

The situation cleared now.


----------

