Home Forums LOBBY Soft Opening

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #515
    Mulch
    Keymaster

    Hello There,
    Yes, we quietly went live a few weeks ago. It has been a stress test of sorts, to deal with bots and randoms who found their way here, under consistent load. Cache response seems over 90% stable and implemented based on things learned during this initial stress testing (bots are RELENTLESS).

    We have 100% kept spam posts away from registering and deflected a ridiculous and persistent series of attacks. I am not promising security; not even our governments can do that!

    This is about speed and stability, considering the “homebrew” limitations of this setup, and WITHOUT a CDN to help defray bandwidth. I am quite pleased by the many, MANY behind the scenes developers who have helped create the skeleton of this amazing content & community management software.

    We are still not in “Grand Opening” mode quite yet (The Dark Souls Giveaway Happening will be the “Real” launch of this site, follow us on Twitter and on Steam to stay informed).

    There are a few changes I would like to make, to improve speed and responsiveness, that will mostly use tracking and redirects to handle the persistent theme of “all kinds of bots” hammering the site “at all times.”

    Right now, I would say caching and software has us at about 90% efficiency, but I think we can do better. You will notice google analytics being deployed shortly for that purpose. The first step will be seamlessly integrating it with both the dynamic and cached static content, as well as the template. This will allow us to begin to manage what resources unidentified bots discover when they get here, thereby lowering their overall impact.

    We are now entering a phase of diminishing returns of responsiveness and stability via caching, without significantly throttling probe bots. So filtering traffic may become the final strategy to combating this pest. I think we are on the right track in the long term thinking, in that regards, and can migrate to, say, AWS, without much problem.

    Again, I am quite pleased with the overall gains made in responsiveness and stability because of this stress testing, but there are still tweaks and I guess we are still in beta as there are these new approaches I am going to examine that may delay the actual “launch.” Periods of instability may persist during these last days.

    I will say, we are open to the public, and any humans who find us should have no problem whatsoever joining, making friends, sending messages, forming groups, or joining servers. There may still be a few more bumps to go, but overall, we are near the infrastructure similar to the one that I was initially trying to create. Now to see if we can build an ecosystem. Welcome to Mulchworlds!

    #523
    Mulch
    Keymaster

    one step forward, 2 steps back

    no sooner had I posted the optimistic “almost ready to launch” post, I discovered that my new solution for object caching was failing similar to my first; specifically for logged in users which were the primary beneficiaries of this form of caching. I had previously tried redis, and while it was blistering fast when used under the right circumstances, it had completely random, unexpected side effects mostly around permissions and being logged in.

    I *thought* that I had a new object cache solution with an APC plugin, but after some testing, the same wildly random, permission, and authenticating errors popped up, so we might be a little further away from the very critical “object caching” solution that NEEDS to be implemented for acceptable speed and responsiveness and stability, even while scaled.

    In other words, we might still be firmly in beta until and a bit further away from the “grand opening” then my optimistic last post alluded to

    I apologize. no misdirection was intended. The problems surfaced soon after my post, and quick and dirty solutions proved useless, so it might be back to the drawing board.

    If you have experience with object caching for dynamic content, with either redis or apc, please contact me if you have the time. This is a major issue in performance, particularly to those logged in participants who are so valued. I would like to get this settled so we can move on to the sequestration of the bots.

    And on that front, expect google analytics to show up soon as planned, even if they are out of sequence to what I mentioned in the last post behind caching. I still see tremendous value in optimizing resources by more carefully controlling these random probe bots access and limiting their impact on overall resources.

    The object caching, analyzing traffic, and directing the traffic to more resourcely benign places are the next 2 critical focuses before “grand opening.”

    then we can worry about customizing game, voice, and media servers, recruiting more of our friends, and fleshing out what mulchworlds will ultimately become. I so very much look forward to each and every one of your individual contributions and ideas about what we actually do become. Thanks for your support.

    #525
    Mulch
    Keymaster

    yeah, the bots are absolutely HAMMERING this site (my sincere apologies for lag and unresponsiveness since the 1st post in this thread as I have since had to remove object caching, which is crucial to a responsive social, discussion, multimedia, game serving CMS system that we have been putting together and has drastically slowed the site since then). i was so thrilled at the ridiculous improvement to responsiveness before i was let down and reality set in and showed me that we needed a different approach to object caching

    yes we are indexed in google (probably bing and a few others i strategically submit to for a different style of results with some similar, but with key differences for tracking purposes).

    This was strategic in a sense, as there was no SEO done, no advertising or marketing in any way beyond the limited alpha/beta closed opening about a year ago; but the bots would find us and put us through our paces… and they did

    no, there has been no real human marketing of our existence. i truly want the tech, in terms of speed, robustness, and responsiveness, to be the top notch priority and maximized by the time I invite and encourage random, ordinary real people to take part in this experiment.

    it was unexpected and sad that object caching has been such a failure so far for us, considering its outstanding reputation (it is actually prolly operator error, soz), to me personally.

    I do have a homebrew server strung together with spare parts and hacks, that i use nonstop, connected via a fiber optic residential ISP (50/50), that you are connected to right now if you use the mulchworlds.com url on any of our many servers.

    without getting into too many specifics (there isnt really a reason to do that), I do host multiple servers of multiple varieties, and in some cases, there are some shared libraries and resources that have unintentionally been put into a shared cache, resulting in wildly random outcomes, particularly with logged in users.

    the combination of gaming, voice, web, and other servers potentially prevent effective object caching… and i dont know what to do away with/modify, if that is indeed the cause; i would prefer salting the cache as i have when sharing infrastructure, but i have yet to test this

    i fear that all of my previous attempts at optimizations featuring caching goes haywire. My html based logged out cache has been a God send, especially for bots. But object caching to serve dynamic content to logged in users… the real challenge that I am working on, has been consistently elusive

    I haven’t tried salting the cache in apc (which may very well be the solution once configured, or not), but i did attempt that procedure with redis with somewhat limited results; they mitigated a large amount of the instability and authentication issues with impressive speed, but not all of them, and those that remained were pretty stubborn and important to boot, so that had to vacate production at the current time

    we arent there yet, as optimistic as my 1st post in this thread was. object caching is a much more important aspect than dealing with the bots (which is an important aspect, dont get me wrong, they are on us at all times, night and day. i could post notifications of 22 hours of failed login attempts using the most ridiculous usernames you ever saw even when they were IP blocked after 5 incorrect login attempts)

    successful object caching is the PRIMARY goal!

    in other words, i aint trying to make this big until after we can completely deal with both the bots and object caching.

    i do have a dummy site, but right now, i would rather use the main site to “sandbox test” attempts to deal with both the bots and object caching in a real time, in a live, “production” environment that is relentlessly taking hits without me even trying to get them there to help me stress test (simulated, for the most part, via bots that i did not invite), all the while gaining the benefits of learning how to identify and mitigate this threat

    hang in there, it might get a bit worse before it gets better, but it WILL get better because, as has been stated, things like bots and spam dont even exist here. the current focus is on site response, speed, and human experience.

    Bots are a big problem but I think they can be dealt with. The more pressing problem is object caching, which PRIMARILY effects members and logged in participants. That is EXACTLY the people that i want to take this from a “blog” to a community, so my 1st goal is to get object caching working to ensure they are the main beneficiaries of the work and enjoy the improvements that we are working towards

    the bots hurt everyone, so they are on my list to. i have prevented them from using known exploits via a variety of proven best practices. I have employed the current best crop of anti spam registrations and commenting.

    What I cant do is to stop them from just using the sites resources by visiting, requesting files, and attempting to register and post through my stacked layers of honey traps only bots will follow.

    Even though these measures are keeping random bots from registering and posting, they are still taking resources just by visiting the site.

    SO again, I can say this out in the open because the bot operators wont be reading this (as there is nothing to be gained on this beta test site to gain them anything whatsoever, at any time), my goal with these bots is to either confuse them, get them off this site (and by consequence, get them to stop using our resources), trap them in cacheland where at most there will be slight, very controlled bandwidth usage while they are sequestered from the hard drive and cpu and wander aimlessly forever through ram based proceduraly generated labyrinths that lead no where, pose no security risk, use extremely minimal resources (mostly ram based, which is rarely in short supply these day)

    #535
    Mulch
    Keymaster

    redis has been deployed and configured and appears to be functioning correctly

    YAY!

    please let me know if you experience any weirdness, particularly issues regarding remaining logged in, stale pages, or impact on notifications.

    Thanks!

    now to battle the bots!

    #537
    Mulch
    Keymaster

    redis has become the process using the most resources on the machine, HOWEVER, the entire load on the machine has gone down overall, and the site seems snappier when logged in, with no weirdness.

    it looks like progress!

    #653
    Mulch
    Keymaster

    In the further hardening of security, before official “launch” (hahaha), several additional security precautions were put in place with the specific goal of weeding out bots.

    Now, my friends, the most vulnerable aspect of this space is you.

    What I am saying is very soon, just before “launch,” I will force a password reset with more stringent password requirements on all members. This is legit. This is not a hack or a phishing expedition.

    Thanks again for every ones support during closed and now open beta. I hope that you enjoy the results of these efforts.

Viewing 6 posts - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.
MulchWorlds