Friday, May 17, 2019

Grid outage autopsy



April Linden has blogged about the outage this week that saw all users unable to connect to Second Life for four hours.

You can see the full blog here: https://community.secondlife.com/blogs/entry/2550-the-road-to-downtime-was-paved-with-good-intentions/

It reads..


Hi Residents!

We had one of the longest periods of downtime in recent memory this week (roughly four hours!), and I want to explain what happened.

This week we were doing much needed maintenance on the network that powers Second Life. The core routers that connect our data center to the Internet were nearing their end-of-life, and needed to be upgraded to make our cloud migration more robust.

Replacing the core routers on a production system that’s in very active use is really tricky to get right. We were determined to do it correctly, so we spent over a month planning all of the things we were going to do, and in what order, including full rollback plans at each step. We even hired a very experienced network consultant to work with us to make sure we had a really good plan in place, all with the goal of interrupting Second Life as little as we could while improving it.

This past Monday was the big day. A few of our engineers (including our network consultant) and myself (the team manager) arrived in the data center, ready to go.  We were going to be the eyes, ears, and hands on the ground for a different group of engineers that worked remotely to carefully follow the plan we’d laid out. It was my job to communicate what was happening at every step along the way to my fellow Lindens back at the Lab, and also to Residents via the status blog. I did this to allow the engineering team to focus on the task at hand.

Everything started out great. We got the first new core router in place and taking traffic without any impact at all to the grid. When we started working on the second core router, however, it all went wrong.

As part of the process of shifting traffic over to the second router, one of our engineers moved a cable to its new home. We knew that there’d be a few seconds of impact, and we were expecting that, but it was quickly clear that something somewhere didn’t work right. There was a moment of sheer horror in the data center when we realized that all traffic out of Second Life had stopped flowing, and we didn’t know why.

After the shock had worn off we quickly decided to roll back the step that failed, but it was too late. Everyone that was logged into Second Life at the time had been logged out all at once. Concurrency across the grid fell almost instantly to zero. We decided to disable logins grid-wide and restore network connectivity to Second Life as quickly as we could.

At this point we had a quick meeting with the various stakeholders, and agreed that since we were down already, the right thing to do was to press on and figure out what happened so that we could avoid it happening again. We got a hold of a few other folks to communicate with Residents via the status blog, social media, and forums, and I kept up with the internal communication within the Lab while the engineers debugged the issue.

This is why logins were disabled for several hours. We were determined to figure out what had happened and fix the issue, because we very much did not want it to happen again. We’ve engineered our network in a way that any piece can fail without any loss of connectivity, so we needed to dig into this failure to understand exactly what happened.

After almost four very intense hours of debugging, the team figured out what went wrong, worked around it, and finished up the migration to the new network gear. We reopened logins, monitored the grid as Residents returned, and went home in the middle of the night completely wiped out.

We’ve spent the rest of this week working with the manufacturer of our network gear to correct the problem, and doing lots of testing. We’ve been able to replicate the conditions that led to the network outage, and tested our equipment to make sure it won’t happen again. (Even they were perplexed at first! It was a very tricky issue.) As of the middle of the week we’ve been able to do a full set of tests including deliberately disconnecting and shutting down a router without impact to the grid at all.

Second Life is a really complex distributed system, and it never fails to surprise me. This week was certainly no exception.

I also want to answer a question that’s been asked several times on the forums and other places this week. That question is “why didn’t LL tell us exactly when this maintenance was going to happen?”

As I’ve had to blog about several other times in the past, the sad reality is that there are people out there who would use that information with ill intent. For example, we’re usually really good at handling DDoSes, but it requires our full capacity being online to do it. A DDoS hitting at the same time our network maintenance was in progress would have made the downtime much longer than it already was.

We always want what’s best for Second Life. We love SL, too. We have to make careful decisions, even if it comes at the expense of being vague at times. I wish this wasn’t the case, but sadly, it very much is.

We’re really sorry about this week’s downtime. We did everything we possibly could have to try to avoid it, and yet it still happened. I feel terrible about that.

The week was pretty awful, but does have a great silver lining. Second Life is now up and running with new core routers that are much more powerful than anything we’ve had before, and we’ve had a chance to do a lot of failure testing. It’s been a rough week, but the grid is in better shape as a result.

Thanks for your patience as we recovered from this unexpected event. It’s been really encouraging to see the support some folks have been giving us since the outage. Thank you, you’ve really helped cheer a lot of us up. ❤️

Thursday, May 16, 2019

Saturday: Eurovision Song Contest - Live in London City






On Saturday,  it is time for the Eurovision Song Contest again!  The show that is so bad, it is great!

This is the third Eurovision since Britain voted to leave the European Union (without leaving).  Last year they gave us a few points, to show there is no hard feelings.  This year, we're hanging around in Europe like a bad smell, so how will voters react?

You can enjoy the grizzly spectacle of the campest show on earth, The Eurovision Song Contest, in London City on Saturday.

From  12pm SL (8pm UK) get your space on the dance floor as we celebrate by-gone British entries, historic winners and then listen live to this year's hopefuls as they each croon their way into the inevitable oblivion that awaits them.

The half time show features previous Euro winners Brotherhood of Man and previous disaster, Bonnie Tyler!

Come and experience Eurovision, here in  London City, with all the entrants singing their fabulous songs and the good ole UK getting its usual 1 or 2 points.

The campest show on earth is here  http://slurl.com/secondlife/London%20City3/136/123/23 starting at 12pm SL (8pm UK) with a prize of 300L for Best in Your Countries Colours.

Don't miss it!

Tuesday, May 14, 2019

Eurovision Replay of 2018 - We Was Robbed!




It's the Eurovision song contest on Saturday!

Here is a Eurovision replay.  This what happened last year..

[originally blogged 13 May 2018]

British entry SuRie, sang like a German band in a performance that was exciting for all the wrong reasons.  In the last minute of her song, what police describe as "a European citizen but not British", ran on to the stage snatching the singers microphone and appeared to shout, "Nazis of the UK media, we demand freedom."

Apparently, this as yet unnamed scumbag, also invaded the stage in three similar incidents during the National Television awards, The Stephen Noland show and the final of last year's, The Voice.

Someone really should get his name and stop inviting him to these functions.

Although our entry was probably not the winning number of the night, it deserved to do a little better than it did, ending in 24th place, beating just Finland and Portugal.  To put this into perspective when  Bonnie Tyler represented us we came 19th, and with Engelbert we came 25th of 26.

Eurovision ceased to be anything about music a very long time ago. In fact a dreadful entry from Israel featuring an overweight Chinese lady at a Takeaway, scooped top prize by winning  the so-called popular vote, having been in an obscure place on the leader board only moments before.

We are used to losing this contest, and in recent years the BBC has twice played canned cheering when our contestants take to the stage to drown out the booing in the auditorium.  So although we never expect to win due to political reasons (ie the rest of EU not liking us for Brexit), we always have fun at this event and the indignation we suffer each year, is actually just part of the cabaret that is Eurovision.

We had a terrific evening in London City, enjoying old Eurovision acts Brotherhood of Man and Bonnie Tyler during the intermission as we collectively watched the Live show, whilst sucking our teeth.  Hope you enjoyed it too!









Monday, May 13, 2019

Second Life Maintenance reminder..



Starting today and running for 4 days, the grid is undergoing essential maintenance.

Linden Lab blogged:

Our engineers will be performing maintenance on the Second Life network May 13 -to- 16. 

We hope to perform most of the maintenance early in this window, but it may extend several days if needed.  Residents may experience problems connecting to, being disconnected from, or an inability to log-in during this time, as well as possible issues rezzing objects, teleporting, or crossing regions. 

We hope to keep these disruptions to a minimum and greatly appreciate your patience during this time as we work to make Second Life more robust. We will resolve this status once the maintenance has been fully completed.



Friday, May 10, 2019

Jessie J Tribute Band on Saturday



The British singing sensation tribute to Jessie J is on Saturday.

Cornish singer/songwriter, Jessie J, born Jessie E in 1988, grew up and studied muzak at the Brit School where she penned the dead classy tune Do You Like it Dude? which made its way into the charts at Number 2.

After Jessie's number two, she followed up with a number one which she called Price Tag, then a debut album Who Are You in 2011.
The Party starts at 12pm SL (8pm UK) and the show is an hour later at 1pm SL.

In the words of Jessie J, "Be wiv us!"

Location http://maps.secondlife.com/secondlife/London%20City/114/111/23


Thursday, May 09, 2019

Biden or Bernie



We asked who you would like to see take on the great Orange leader, out of the two most likely candidates..

And your sense of humour has most certainly prevailed as 32 (76%) of 42 total votes, want to see Bernie Sanders in the ring with Mr Trump as America goes to Election next year.

Sleepy Joe did not do anywhere near as well as Larry King lookalike Bernie.

Another fun poll soon.  Thanks for playing!

Monday, May 06, 2019

It's a Baby!




Congratulations to the Duke and Duchess of Sussex on the news of the arrival of their first child, a baby boy.

Her Majesty the Queen, is said to be 'jubilant' at the news of the arrival of her eighth Great Grandchild.

The Royal Instagram account reads, "We are pleased to announce that Their Royal Highnesses The Duke and Duchess of Sussex welcomed their firstborn child in the early morning on May 6th, 2019. Their Royal Highnesses’ son weighs 7lbs. 3oz. The Duchess and baby are both healthy and well, and the couple thank members of the public for their shared excitement and support during this very special time in their lives."



Friday, May 03, 2019

Frankie Valli & The Four Seasons - Saturday



This Saturday Frankie Valli & The Four Seasons will be on stage in London City.

Known for smash hit songs including, Sherry, Big Girls Don't Cry, Beggin', Working My Way Back To You, Grease and many many more chart toppers, Frankie Valli is an enduring phenomenon and has been performing for over fifty years.

The Party starts at 12pm SL (8pm UK), with the show at 1pm SL time.

Visit London City here: http://slurl.com/secondlife/London%20City/115/110/24




Thursday, May 02, 2019

Another Bank Holiday, Another Party



London City West - The Beach


This public holiday is well overdue.  It has been almost two weeks since the last one.

All this partying makes for thirsty work.  We can expect lashings of cheap grog and last minute sales from all the major supermarkets, with cheap sausages and burgers thrown in for the inevitable Bank Holiday Barbie.

As usual, as part of our public service, we've looked around for the best star deals on booze for this weekend as the boozy Brits sup their pints like there is no tomorrow.

 And they are.. Peroni, 12 bottles for £14 and Stella, 18 cans for £14, both from Sainsbury's.

Carling, 18 cans for a tenner, from Asda!

London City will have a whole host of drunken happenings for everyone to enjoy throughout this holiday!

That concludes this public service announcement!