Server move and downtime, take 2 April 19, 2020 8:38 AM   Subscribe

In a second attempt at moving MetaFilter to a new AWS account, the site will be offline between 22:00 and 01:00 PDT Sunday evening.

The moderator e-mail (via admin@mefi.us) will still be working during this time.

In other time zones:
GMT:   05:00-08:00
EDT:   01:00-04:00
BST:   06:00-09:00
CEDT:  07:00-10:00
IST:   10:30-13:30
CST:   13:00-16:00
AET:   15:00-18:00
(hopefully these are all correct this time)
posted by frimble (staff) to Uptime at 8:38 AM (104 comments total) 8 users marked this as a favorite

2 Down 2 Time
posted by cortex (staff) at 8:57 AM on April 19, 2020 [18 favorites]


CANNOT CONNECT TO JRUN SERVER
posted by thatwhichfalls at 9:00 AM on April 19, 2020 [4 favorites]


Ganbattene!
posted by Going To Maine at 9:41 AM on April 19, 2020 [1 favorite]


In other time zones:
Zulu dammit.
posted by adamvasco at 10:25 AM on April 19, 2020 [2 favorites]


Plugging that into the Event Time Announcer widget: Mefi outage 2.
posted by zamboni at 11:13 AM on April 19, 2020 [2 favorites]


At least it’s not MeFi Outrage 2 200,000....
posted by GenjiandProust at 1:13 PM on April 19, 2020 [2 favorites]


Good luck. Don’t forget for the upgrade this time that the Moon today is in a Waning Crescent phase. The naysayers will say it *literally* doesn’t matter..........but I ask you - if the naysayers were true to their belief, wouldn’t they just say “nay”?

Also you probably shouldn’t be a few cans in when you start the upgrade like the rest of us are.
posted by inflatablekiwi at 1:51 PM on April 19, 2020 [9 favorites]


take 2

Good luck, we're all counting on you.
posted by zamboni at 2:45 PM on April 19, 2020 [8 favorites]


It's a deja vu feeling all over again.
posted by Splunge at 3:03 PM on April 19, 2020 [2 favorites]


take 2

Electric boogaloo!
posted by grumpybear69 at 3:36 PM on April 19, 2020 [3 favorites]


Ziipity bop
posted by cortex (staff) at 4:03 PM on April 19, 2020 [5 favorites]


I just wanted to tell you both, good luck. We’re all counting on you.
posted by mwhybark at 6:07 PM on April 19, 2020 [8 favorites]


This time, it's personal.
posted by justsomebodythatyouusedtoknow at 6:52 PM on April 19, 2020 [9 favorites]


Sunday SUNDAY Sunday
posted by Night_owl at 9:28 PM on April 19, 2020 [5 favorites]


So, we're up and running on the new servers this time. Most stuff should be up and running. Posting, commenting and the contact form definitely are, chat is not for the moment.

Throughout the day, I'll be poking the further things that need to be manually tweaked, but we're hopefully done with any more outages for the next while. Sorry it took two goes and thank you all for your patience with the move.
posted by frimble (staff) at 11:41 PM on April 19, 2020 [25 favorites]


We're there! It's the bright shiny future!

Thanks for sticking the landing, frimble.
posted by mightshould at 2:04 AM on April 20, 2020 [4 favorites]


I just want to tell you both, good luck. We're all counting on you.
posted by applesurf at 2:27 AM on April 20, 2020 [7 favorites]


Thanks, frimble.
posted by nangar at 2:31 AM on April 20, 2020 [3 favorites]


Never leave us again! It’s cold, and there are wolves after us!
posted by Ghidorah at 3:33 AM on April 20, 2020 [6 favorites]


Thank you for sorting this! Is there a timeframe for chat?

I'm trying to get it up and running today.
posted by frimble (staff) at 5:16 AM on April 20, 2020


And chat should be up again. Sorry about that: The new chat server was still trying to connect to the old database server and I spent an hour or so on the question, "why can't it connect to the new database server?" before realizing that that was the wrong question entirely.
posted by frimble (staff) at 5:33 AM on April 20, 2020 [8 favorites]


So anyone else seeing a "site is down" screen when going to metafilter.com? I'm able to reload pages I had up, and get here, obviously, but, not the main page...
posted by Windopaene at 8:37 AM on April 20, 2020


Yup, same here
posted by trig at 8:54 AM on April 20, 2020


You may need to do a force-refresh to get that to empty out the cached page.
posted by cortex (staff) at 8:57 AM on April 20, 2020


I'm getting a "scheduled maintenance" page if I load metafilter.com from my home wifi, but it's up when I use cellular data.
posted by ectabo at 9:19 AM on April 20, 2020 [2 favorites]


I just want to tell you, good luck. One more strike and it's a perfect game. Don't be nervous.
posted by Splunge at 9:33 AM on April 20, 2020


I tried to visit during the downtime and was utterly charmed by the "here are some assorted sites to check out instead!" page that popped up. It felt very old-school webring-y.
posted by showbiz_liz at 9:57 AM on April 20, 2020 [5 favorites]


Just to confirm / reiterate: I needed to do a force refresh (Ctrl-F5) on any browser that saw the maintenance page.
posted by mark k at 10:06 AM on April 20, 2020 [1 favorite]


I have the same problem as ectabo. On wifi, whether laptop or phone, front page or individual post, Ctrl-F5 or using a different browser, no dice; using cellular data on my phone, everything's fine.
posted by ManyLeggedCreature at 10:18 AM on April 20, 2020


I don’t think I’m having cache issues. I get a scheduled maintenance page on the front page and 404 errors on every other page over wifi, and it works normally on my phone over cellular or if I tether my phone to my laptop.
posted by ectabo at 10:21 AM on April 20, 2020


Yep, exactly the same here. Frustrating, as I have a princely one bar of 3G to play with.
posted by ManyLeggedCreature at 10:23 AM on April 20, 2020


Sometimes the DNS will cache the old page. You can try flushing its cache by trying the steps on this page.
posted by jessamyn (retired) at 10:49 AM on April 20, 2020


Thanks, Jessamyn, but no joy, sadly. The problem affects any device connected to my wifi (laptop, phone, iPad), so presumably the router's cached something really unhelpful.
posted by ManyLeggedCreature at 11:10 AM on April 20, 2020


This post brought to you by the number 2. 2! 2!
posted by Melismata at 11:40 AM on April 20, 2020


I had to flush my MetaFilter cookies, but then that holding page went away.
posted by chavenet at 11:54 AM on April 20, 2020


Thanks, Jessamyn, but no joy, sadly. The problem affects any device connected to my wifi (laptop, phone, iPad), so presumably the router's cached something really unhelpful.

If your ISP is caching an old DNS entry, you could set your device(s) to use the DNS servers of 1.1.1.1 (scroll down to Setup) or Google instead. If that doesn't work, most likely your devices are still caching old entries in their OS or browser.
posted by matthewr at 12:07 PM on April 20, 2020


For me, the website is intermittently available via home wifi. I have had no trouble with it on via 5G. Sometimes if the wifi won’t load, a switch to 5G and then back works.

I especially don’t get the last behavior, and the intermittency is weird too, but I mostly expect this will lessen as various DNS lookups get replaced.
posted by nat at 1:01 PM on April 20, 2020


Yeah, it looks like my ISP has cached an old DNS entry. I ended up enabling DNS over HTTPS in Firefox, which fixed the problem there, and I'll wait for the right entry to propagate to my ISP
posted by ectabo at 1:14 PM on April 20, 2020


Sweet! Sorry if this has been asked/answered, but what was the benefit to moving to a new account? I'm interested in this from a learning tech/programming sort of angle.
posted by SpacemanStix at 1:16 PM on April 20, 2020 [1 favorite]


In a very old version of Firefox (53) I see the down page (love the color). Same machine, Firefox 73, I am typing this. Computers are weird.
posted by JawnBigboote at 1:59 PM on April 20, 2020 [1 favorite]


Sorry if this has been asked/answered, but what was the benefit to moving to a new account? I'm interested in this from a learning tech/programming sort of angle.

The dull answer is there was very little benefit, just tying up one of the more stubborn loose ends of the change of ownership a few years ago. A lot of the way MetaFilter stuff was set up while mathowie was building the site over the years was tied into his personal accounts with various services. Most of those we were able to cleanly transfer or sever custody of when the site changed hands, but the AWS stuff was a big exception. Matt and I both put some effort into finding a solution where Amazon could shuffle things around or prise the AWS bit away from Matt's account, but no dice unfortunately.

So after failing to find a less hardcore approach to disentangling MeFi from Matt's Amazon stuff, we worked out a plan with AWS and one of their contracting partners to just clone/migrate the whole schmear to a fresh AWS account. Which is sort of a lot of work for the goal of nothing particularly changing, but Amazon had a program for reimbursing essentially all of the contracting costs in future credits to the new account, so getting it finally done made sense and is one more thing off that short list of dangling threads, and means Matt doesn't have to worry about a MeFi thing coming back around somehow and we don't have to worry about something somehow happening with Matt's Amazon stuff.

We did (and "we" here basically means frimble) manage to clean up a few bits of cruft in the process, though. Nothing we couldn't have done just to do anyway, but it's a bit like getting ready to change apartment units in the same building; you're gonna find a few things to throw away while you're cleaning.
posted by cortex (staff) at 2:42 PM on April 20, 2020 [21 favorites]


Did you run across the Corningware Dutch oven I left there in 2011?
posted by Johnny Wallflower at 2:55 PM on April 20, 2020 [8 favorites]


*shifting to hide huge lump under shirt* no
posted by cortex (staff) at 2:55 PM on April 20, 2020 [23 favorites]


Just to confirm / reiterate: I needed to do a force refresh (Ctrl-F5) on any browser that saw the maintenance page.

For the record this also happened to me just now, which is kind of a surprisingly long time after the switchover was accomplished. Obviously the mods / admins can't do anything about this, but probably they should be prepared for more concerned emails about the topic.

Could the "site is down" page perhaps be modified to reflect that it sometimes takes hours, possibly a day or more, for DNS entries to update? If nobody's bugging you about it then I suppose it doesn't matter much - just an idea.
posted by Joey Buttafoucault at 4:08 PM on April 20, 2020 [1 favorite]


I've had no problem accessing the site on 5G but after I did I started getting a really harsh cough and my temperature spiked.
posted by 99_ at 4:15 PM on April 20, 2020 [4 favorites]


cortext writes interesting tech things

That was interesting and informative. Thanks for sharing!

I think it would be cool to hear more about stuff like this every once in awhile, and it wold be very interesting perhaps to others on the site, too. "A day in the life" that shares a bit about the tech questions you guys deal with to keep the site running. And before you say, "boring!", no way. It'd be great.
posted by SpacemanStix at 4:25 PM on April 20, 2020 [3 favorites]


I had the persistent maintenance page problem last time too. I tracked it up from my clients and through various devices and as far as I can see my ISP is doing the cacheing. Getting them to flush it may be difficult.

I can rearrange DNS servers but

a) I want to make sure I understand your process. You guys are changing the metafilter.com CNAME to point to a different server(-array) that just shows the maintenance page, and then changing it back when you're done?

b) I seem to recall that DNS is really unhelpful this way, but is there a way you can "encourage" cache invalidation when you change the entry?
posted by Tell Me No Lies at 4:26 PM on April 20, 2020 [1 favorite]


I cleared the browser history on my phone (Safari is my Metafilter browser) and am still getting the maintenance page. Not only that, but I lost all my tabs, my precious tabs...

Anybody, Bueller?
posted by computech_apolloniajames at 5:50 PM on April 20, 2020 [1 favorite]


I am still having the persistent maintenance page issue as well. It is definitely a DNS issue. Using the dig tool I determined that my ISP's DNS server (my ISP is Cox) is returning a different result than Google's. My guess either a TTL issue (some DNS servers reject SOA setttings with very low expire/refresh values and substitute a default value to prevent DNS thrash) or the fact that the serial number was not changed as part of the move.

Cox: dig @68.105.28.13 SOA metafilter.com
;; ANSWER SECTION:
metafilter.com.		299	IN	SOA	ns-488.awsdns-61.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
Google: dig @8.8.8.8 SOA metafilter.com
;; ANSWER SECTION:
metafilter.com.		899	IN	SOA	ns-1113.awsdns-11.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
posted by RichardP at 6:32 PM on April 20, 2020 [1 favorite]


Another issue I noticed is that the TLS certificate for mefi.us (which hosts the static content) apparently expired on Friday, April 17, 2020. For me this means that when logged with the "use secure browsing" option turned on I see errors. For example, the URL https://mefi.us/images/mefi/theme-modern.png reports an expired certificate.
posted by RichardP at 6:36 PM on April 20, 2020 [2 favorites]


Metafilter only works on my phone network now, not on my wifi. Will just waiting a few days make this go away? Or is there an obvious thing I need to do to be able to access the site again on my home internet? It's happening across all my devices and I don't want to delete all my history etc for naught.
posted by k8lin at 8:52 PM on April 20, 2020


I just wanted to tell both of you, good luck. We’re all counting on you.
posted by mwhybark at 8:53 PM on April 20, 2020 [4 favorites]


Just wanted to say, I hear you that people are still running into trouble, and I'm sorry about that. Once frimble's online again for the day, they may have more ideas.
posted by LobsterMitten (staff) at 9:31 PM on April 20, 2020 [1 favorite]


After all day of status page it finally works for me. WHEW
posted by aubilenon at 10:10 PM on April 20, 2020 [1 favorite]


For what it worth, as of not long ago everything is now working for me as well. Google and my ISP's DNS servers are now returning the same IP address for metafilter and the TLS certificate for mefi.us is valid.
posted by RichardP at 10:26 PM on April 20, 2020 [1 favorite]


On my iPhone, everything in the land of Mefi is fine and dandy but on my iPad, I get the red screen of Mefi doom. It’s down and has been now for what feels like 24 hours. I’m in Sydney. Any idea how to fix my borked iPad?
posted by Jubey at 11:54 PM on April 20, 2020


Wow, magically sorted and all I had to do was enquire here!
posted by Jubey at 1:22 AM on April 21, 2020 [1 favorite]


Well, the DNS problem laughed at my attempts to resolve it by rebooting or resetting the router, and my phone signal at home is so ropy that I gave up trying to update the thread to say so... but it's gone away by itself overnight. All's well that ends well.

I envy those of you with 5G signals. When my broadband goes out I'm down to 1 bar of 3G, if I stand in exactly the right place and don't breathe.
posted by ManyLeggedCreature at 2:20 AM on April 21, 2020


Whew, you’re back! I too had zero access yesterday, or even through early this morning when I couldn’t sleep. Multiple devices on wifi, but I never tried cellular.
posted by jon1270 at 3:18 AM on April 21, 2020 [2 favorites]


OK, now my android phone is looking weird on metafilter. https://imgur.com/a/eZwk3K6 - all the subsites have the large icons.

Forcing a refresh doesn't fix it.

Fine on laptop.
posted by freethefeet at 3:37 AM on April 21, 2020 [1 favorite]


Yeah, I'm still getting the maintenance page when I try to get to the Blue.
posted by bardophile at 3:37 AM on April 21, 2020


I'm having the same issue as freethefeet.
posted by poxandplague at 4:01 AM on April 21, 2020


I can't Telnet in and the Gopher server is glitchy.
posted by Wordshore at 6:13 AM on April 21, 2020 [8 favorites]


I'm having the same issue as freethefeet.
posted by lollusc at 6:37 AM on April 21, 2020


Was it just me or was the site down for most of Monday, too? *clings* don't ever do that to me again mefi
posted by MiraK at 6:40 AM on April 21, 2020 [5 favorites]


Still down for me too, but from my phone it’s good
posted by lazaruslong at 7:01 AM on April 21, 2020


I cleared my browser cache and everything is good
posted by photoslob at 7:37 AM on April 21, 2020


Freethefeet, poxandplague, lollusc, I've just poked the CSS so that it should load for you. Are you able to load the site normally now?
posted by frimble (staff) at 7:39 AM on April 21, 2020


That fixed it for me.
posted by blurker at 7:40 AM on April 21, 2020


Still red for me on Chrome on my laptop. I can get the blue on my cell and on Opera on the laptop.

I've cleared the DNS from the the command prompt. I've restarted the laptop. Are there more things I could do that might help, or should I just wait?
posted by Nancy Lebovitz at 10:05 AM on April 21, 2020


Still red for me on Chrome on my laptop. I can get the blue on my cell and on Opera on the laptop.

Sounds like a Chrome problem — flush the Chrome DNS cache by going to chrome://net-internals/#dns and clicking Clear Host Cache.
posted by matthewr at 10:16 AM on April 21, 2020


matthewr, thank you, but I'm still getting the red screen.

I'm assuming that hitting the Clear Host Cache button is enough-- it doesn't give feedback.
posted by Nancy Lebovitz at 10:20 AM on April 21, 2020


It's working! It's blue!
posted by Nancy Lebovitz at 10:24 AM on April 21, 2020


😊👍
posted by matthewr at 10:28 AM on April 21, 2020


For those curious folks, like me, can we see an archived view of the "offline" red page?
posted by filthy light thief at 10:52 AM on April 21, 2020 [1 favorite]


cleared cache and still get the left hand (!) in URL bar.
posted by clavdivs at 10:55 AM on April 21, 2020


I'm still getting a red page on my laptop. Firefox, latest version, latest windows OS, all standard. Works on android phone's chrome browser
posted by a power-tie-wearing she-capitalist at 11:08 AM on April 21, 2020


For those curious folks, like me, can we see an archived view of the "offline" red page?

Here!
posted by matthewr at 11:10 AM on April 21, 2020 [3 favorites]


wait nm it just came back for me. yesssss
posted by a power-tie-wearing she-capitalist at 11:20 AM on April 21, 2020 [1 favorite]


It's back! Hooray, thank you!
posted by k8lin at 11:46 AM on April 21, 2020 [1 favorite]


Having conquered Cat Island the last time around, yesterday I acquired a new vacation destination in the Bludenz-Innsbruck train video. I imagine it's still a pretty good ride even if you only ever get to look out the side windows.
posted by rhizome at 1:52 PM on April 21, 2020 [2 favorites]


So, for me MeFi appeared to be down for several days, and I didn't realize I was having a cache issue until I randomly loaded MeTa earlier today from my phone. In fact, the browser on my computer was still showing the System Is Down page until I did a hard refresh just now. Why did this happen with MeFi? And is there a way to fix it? Is this also happening with other sites and I just don't know about it?

Mind, blown...
posted by panama joe at 2:58 PM on April 21, 2020 [2 favorites]


Metafilter: Flush Your Cookies
posted by Chairboy at 3:00 PM on April 21, 2020 [3 favorites]


Yep CSS issue fixed, thanks frimble!
posted by freethefeet at 3:32 PM on April 21, 2020


That fixed it for me frimble. Thanks :)
posted by poxandplague at 3:37 PM on April 21, 2020


(!) why is this still in the address bar when I access metafilter.
posted by clavdivs at 5:00 PM on April 21, 2020


Thanks Frimble! It was all good when I woke up this morning
posted by lollusc at 6:12 PM on April 21, 2020


(!) why is this still in the address bar when I access metafilter.

If this is Firefox, are you sure it isn't (i)? There should also be a green lock. Chrome should just have a black lock to the left of the URL.
posted by rhizome at 6:43 PM on April 21, 2020


chrome, it is an 'l' and no lock image.
posted by clavdivs at 6:53 PM on April 21, 2020


Thanks for all your hard work gang.
posted by turbid dahlia at 7:49 PM on April 21, 2020


clavdivs, if you click on the "Preferences" tab is the "Use secure browsing?" checkbox on or off?
posted by RichardP at 7:57 PM on April 21, 2020


can't find preference tab, not on a computer, nothing about secure browsing.

wtf
posted by clavdivs at 8:51 PM on April 21, 2020


19 years here never had a problem like this. care to sort this out, frimble.

android, chrome.
posted by clavdivs at 9:27 PM on April 21, 2020


I’m no longer able to charge my Amazon Prime orders to Matt’s account. Does this have something to do with the server migration? If so, can we move it back?
posted by Slarty Bartfast at 9:28 PM on April 21, 2020 [6 favorites]


clav i love you but i feel like you may have blanked on a lot of years of just constant random JRun errors if you feel like being able to actually comment in MetaTalk about a browser tic is the worst state you've found the site in during our long shared tenure
posted by cortex (staff) at 9:35 PM on April 21, 2020 [4 favorites]


ok, I get that. but not saying the site is in worstest state. I have the tech problem, mostly likely JRUN fever clouding my ability to think it through but clearing the cache friggled with the fries but I solved that.

I see your take and should have been clearer.
so, help a filterine out.

love ya too boss.
posted by clavdivs at 9:43 PM on April 21, 2020


There was that time Matt put a Basilisk on the front page for the hell of it. I was offline that day but we lost two thirds of the membership in one morning.
Three of them were Nobel prize winners.
posted by thatwhichfalls at 9:45 PM on April 21, 2020 [3 favorites]


I just wanted to wish you both good luck. We’re all counting on you.
posted by mwhybark at 9:49 PM on April 21, 2020 [3 favorites]


clavdivs, if you can't find the MetaFilter "Preferences" link (should be somewhere at the top of the page) to check the state of the "Use secure browsing?" checkbox, try the following direct link: Preferences.
posted by RichardP at 9:52 PM on April 21, 2020


too many pop flys in left field on sunnydays.

I'm hunting for chrome preference.
ok, box is checked in profile edit, reset it, log out/in and nope. gonna clear cache again see if that....
posted by clavdivs at 10:34 PM on April 21, 2020 [1 favorite]


Not sure if this is related, or some trouble on my end, but I just started getting certificate invalid errors and being unable to make an https:// connection
posted by thelonius at 7:08 PM on April 22, 2020


and still the (I) in the address bar.
posted by clavdivs at 9:51 AM on April 23, 2020


congrats and kudos for grinding thru what must've been just the <sarcasm>funnest</sarcasm> slog into aws console land.
posted by sammyo at 10:50 AM on April 23, 2020


The highest compliment I can pay to Frimble et al. is that I visit Metafilter just about every day, on various devices, and had no idea this was happening until I read about on MeTa just now.
posted by thenormshow at 1:48 PM on April 23, 2020 [1 favorite]


« Older Metatalktail Hour: what's growing?   |   XXV: Fucketh Newer »

You are not logged in, either login or create an account to post comments