Server Software Upgrade
Many of you who have been members of Ricochet long enough have experienced the dreaded "login problem." Sometimes (about once a month?) the site will simply log you out and keep you out for a day or two at a time, no matter how many times you log in again. We have not been able to reproduce it in-house long enough to debug it, and it comes and goes; it's been a bit of a ghost problem. We love our content management software platform overall, but we were not sure if it was to blame, or if it was the interaction between other layers in the server software stack.
A month ago, a new release of the software came out. We decided to upgrade, as we were now two versions behind. We were hoping this would resolve this problem and others. Thanksgiving week seemed to be a good time to do it, as traffic is a bit lighter.
After testing the upgrade on a development server all weekend, this morning I rolled out the upgrade on our production servers. No matter how you test, however, you can never reproduce the exact conditions of a running site with a significant amount of live traffic. We immediately found that the login problem actually became universal: no one could stay logged in, or it seemed sporadically to log you in and out.
So, first of all, apologies to all of you who had to suffer through that. I rolled the servers back to our original version, but to really test and debug the problem, I had to switch over to the upgraded version several times throughout the day. I finally realized I needed to upgrade another component of the server software stack, after which the problem seems to have gone away.
Our server is now running on the latest version of the content management software, and the login problem seems to be resolved. We hope it is resolved for good, but do please let us know if you run across it again.
There are many things for which each of us can be thankful if we care to think about it. My job as architect, sysadmin and lead developer of this site is hard at times, but I mostly love my work and am very thankful for this opportunity. I hope each of you is somehow enriched by Ricochet.
Thanks for being readers and members! You are the reason we are here, and the reason I get to work on such a great site every day.
Happy Thanksgiving!
- Comment (22)
- · Quote
- · UnfollowFollow (0)
- Pages:
- 1
- 2
- Pages:
- 1
- 2
Comments :
Sep '10
Re: Server Software Upgrade
As long as there's no full body scan on login, I really can't complain.
Nov '10
Re: Server Software Upgrade
You IT people are all the same... ;-)
Sep '10
Re: Server Software Upgrade
Again with the emoticons.
May '10
Re: Server Software Upgrade
I wonder if this will help the "denied" problem on Member Feed uploads.
Feb '10
Re: Server Software Upgrade
Let me know either way.
Aug '10
Re: Server Software Upgrade
Everything's good this morning, and the system is running faster. Comments are processed quicker.
Is there any chance I can load a graphic in the Member Feed sometime soon?
Edited on Nov 23, 2010 at 4:26amMay '10
Re: Server Software Upgrade
Happy Thanksgiving, BSA, and thanks for keeping us all together. Logged in (again) this morning. Fingers crossed that it's the last time. Now to see if, like River, my comment gets processed more quickly....
May '10
Re: Server Software Upgrade
Yup, much more quickly!
I'll try it on the mighty iPad later.
Edited on Nov 23, 2010 at 4:46amJun '10
Re: Server Software Upgrade
Still having the logon problem this morning. I am not logged on automatically when I open the site, despite have checked the "keep me logged on" feature. When I logon the main feed, I am kicked back to unlogged when I click the member feed tab. After a couple of trys it works (I'm here aren't I?) Could be my imagination but it seems to work better if I don't click "keep me logged on" on reattempts to logon. Also, I can post a comment but get a "error creating preview" Signed, Patience
May '10
Re: Server Software Upgrade
Thank you, BSA! I hope you and your family have a very happy, meaningful, and restful Thanksgiving.
Aug '10
Re: Server Software Upgrade
So - you're saying my magic screen that talks with people from around the world by sniffing their words from thin air will now be even more magical?
Awesome.
May '10
Re: Server Software Upgrade
As Pilgrim reported, I too have seen the "login problem" again today, but not as frequently as yesterday. Some sort of issue maintaining session state. Are y'all running on a server farm now? Keeping sessions synced across a farm can be a real challenge, and when it fails it looks just like this...
Nov '10
Re: Server Software Upgrade
I think that you do a fine job with a difficult medium and am grateful for your taking it on. Keeping so many top flight minds comfortable with this means of communication has to be a rum go. Thanks again and a fine Thanksgiving to you and yours.
Feb '10
Re: Server Software Upgrade
That's exactly what is going on as far as I can tell, but again, it's really hard to reproduce reliably enough to test and get to the root. The servers in the cluster are supposed to work with each other, keeping sessions in a central database; but it doesn't always seem to work. I upgraded our load balancer to pin each individual client to one particular web server in the cluster, and that seems to help.
Edited on Nov 23, 2010 at 11:53amFeb '10
Re: Server Software Upgrade
I should have pointed out more explicitly that the login problems yesterday were largely due to my testing and trying to figure out a solution. I'm interested to hear about any that have happened since sometime in the evening on November 22. The data point about "Keep me logged in" not working is useful. I'll have to check up on that.
Sep '10
Re: Server Software Upgrade
Busy System Admin
That's exactly what is going on as far as I can tell, but again, it's really hard to reproduce reliably enough to test and get to the root. The servers in the cluster are supposed to work with each other, keeping sessions in a central database; but it doesn't always seem to work. I upgraded our load balancer to pin each individual client to one particular web server in the cluster, and that seems to help. · Nov 23 at 11:51am
Edited on Nov 23 at 11:53 am
I have some thoughts to share, but will wait until private messaging is enabled, so as not to bore the audience or annoy you unnecessarily. Nothing earth shattering that you likely already haven't thought of, but some workarounds you may find interesting.
Feb '10
Re: Server Software Upgrade
I'm interested. You can reach me at support@ricochet.com as always.
May '10
Re: Server Software Upgrade
Well it still logs me out all the time. I can get it, but I don't seem to stay in.
Jun '10
Re: Server Software Upgrade
Busy System Admin
I'm interested. You can reach me at support@ricochet.com as always. · Nov 23 at 12:17pm
I don't think the folks on this thread are bored with any thoughts or ideas. It doesn't sound to me like anybody is less than appreciative of BSA's efforts and challenges -- in fact it kind of sounds like we would all like to kick in and work on the site with him -- what a cool job! Not much sleep but still cool.
May '10
Re: Server Software Upgrade
My iPad got logged off, and it (the iPad) still processes relatively slowly. I'll check in from the computer after the cannoli.