NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
There's a new "Google Jail" for independent wikis (weirdgloop.org)
JayNeely 3 hours ago [-]
The article says this isn't an issue for subdomains of established domains, but the issues it describes match what we experienced setting up a game wiki for Cyber Knights: Flashpoint at https://cyberknightswiki.tresebrothers.com/ -- almost a year of not having any pages show up in search results, even though we were getting good links to it and there were no better alternatives that should have ranked (we never started the wiki for this game on any centralized site; at most all they had for it was a placeholder page).

Only after months of making sure our Steam news posts were consistently linking to pages within our wiki did we eventually see Google suddenly open the floodgates and start connecting searchers with the info they were searching for.

I've built out highly-ranked content sites before, and none of what I'd learned from getting those started seemed to work with this wiki, for no apparent reason. Google Search Console showed plenty of pages on the wiki were indexed; it just wasn't displaying them. Seeing this post I fully believe the premise that there's simply an arbitrary Google Jail for independent wikis. That sucks for search quality.

mitxela 2 hours ago [-]
The problem is that independent wikis don't pay the fandom tax to google for preferred search placement. Have you tried contacting your local corruption bureau to negotiate a rate tailored for your circumstances?
raincole 4 hours ago [-]
When I still payed Dota, there was a wiki hosted on fandom.

The experience was terrible. It wasn't always update-to-date with the latest patch, and on mobile there were so many ads rendering the whole thing unreadable.

The catch is that there was a better wiki, called Liquipedia. But Google never showed it first and I kept getting tricked to click that shitty fandom site.

(It was years ago. But I just tested it a bit and for most Dota-related keywords, fandom is still ranked higher than liquipedia. I don't know if the quality of fandom site has improved though.)

jmuguy 3 hours ago [-]
Yeah fandom has done this to several games. its taken a huge effort on the part of the Satisfactory community to make sure people know the official wiki is https://satisfactory.wiki.gg/ and not the garbage on fandom.

Using an extension to block fandom.com results is helpful because for other games I'm not really thinking about it when searching for something and find myself on that tumor of a website.

no_wizard 2 hours ago [-]
Shoutout to Kagi search for having a deny list feature. It’s really nice for this sort of thing.
lithos 2 hours ago [-]
Google used to have a deny/block list. But such lists are obviously very advertising unfriendly, since the companies paying for placement and running the most ads are most likely to be blocked.
mitxela 2 hours ago [-]
They have one, it's just centralised and blocks the good sites.
lqstuart 7 minutes ago [-]
I honestly love Kagi, it's like every single "it would be nice if Google gave me feature X instead of being greedy" in one place
saghm 3 hours ago [-]
Back when I played WoW, at some point the people running the fandom wiki "forked" it to move to different hosting, and I think even now years later the fandom one (which stopped receiving updates when the fork happened) is still ranked higher in Google searches
plorkyeran 46 minutes ago [-]
WoW is now on wiki number three after the company hosting the second one was also bought by fandom. The original rotting corpse is now over a decade out of date and still often ranks highest.
port11 1 hours ago [-]
Fandom is an abomination and filled with irrelevant or very basic content for most games. It wouldn’t exist if not for its phenomenal ranking on game-related searches…
ranger_danger 37 minutes ago [-]
Sometimes a fandom site I come across is legitimate and has tons of great information, and other times they completely make up fake games or features and just act like it was always real.
qlm 2 hours ago [-]
I think the fandom wiki for Dota is effectively dead, with everything being migrated to Liquipedia. Doesn't stop the fandom one from ranking highly though.
charcircuit 28 minutes ago [-]
>It wasn't always update-to-date with the latest patch

In case you didn't know Fandom is not the one who is writing the wiki. The pages are community contributed. If it's out of date it means no one has contributed information about newer patches.

mitxela 2 hours ago [-]
This is why people recommend Kagi.
warmedcookie 3 hours ago [-]
It's still terrible. Also, congrats on escaping the cancer.
SyneRyder 6 hours ago [-]
Some of this might just be problems with the websites themselves. With the Path Of Exile 2 Wiki example, I'm looking at the sitemaps. Once you get through the first sitemap of nested sitemaps (ugh, not a fan of those), the very first actual sitemap returns:

  XML Parsing Error: no root element found
  Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
  Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)

On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.

Looking at Hytale Wiki, my process:

* Site has a robots.txt file - good!

* Robots.txt mentions a sitemap - excellent!

* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.

* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.

This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?

My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.

Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.

SyneRyder 5 hours ago [-]
Me again! I couldn't figure out why I was getting transient errors on that Poe2wiki sitemap, so I tried looking at other sitemaps on the site. I'm getting transient errors on all of them.

Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.

That's a behavior unique to a Wiki, and might explain the entire phenomenon.

masklinn 5 hours ago [-]
Makes sense. Whatever generates those site maps should use the classic Unix trick of writing a separate file then renaming. Also maybe don’t update lastmods on each individual update, it’s a waste of resources since the granularity of lastmod is usually the day (though you can make it the second if you want).
HexPhantom 4 hours ago [-]
Though weirdly it makes the subdomain result even more interesting: presumably those wikis are rewriting their sitemaps too, yet Google manages to index them normally
mitxela 2 hours ago [-]
If the site is tripping a Google filter it makes sense that it eventually gets manually overridden. That's assuming good faith on Google's part.
HexPhantom 4 hours ago [-]
The sitemap issues are definitely worth fixing, but I don't think they explain the most interesting part: the same kind of wiki seems to behave very differently depending on whether it's on a fresh root domain or an established subdomain.
dianliang233 4 hours ago [-]
There's no need to speculate on whether the sitemap is received correctly by Google since Google Search Console tells you how many pages it can discover from the sitemap you submitted.
iamacyborg 5 hours ago [-]
The poe2wiki issue is a little different to what is documented, albeit only for the last few months, previously we certainly had visibility issues.

The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.

KingMob 5 hours ago [-]
This is all valid critique, but I have great difficulty believing Google's crawler behaves anything like yours.

I would expect Google to handle broken or missing sitemaps, honestly.

viraptor 3 hours ago [-]
I wish. Not only do they sometimes randomly fail, Google search console will not tell your why. I've got a site with a sitemap which passes in every validator I could find, yet Google doesn't read it and GSC just reports "error" without any details. Google is big enough to go "we don't care, it's your problem".
shuwix 4 hours ago [-]
[dead]
unglaublich 7 hours ago [-]
> Subdomains of existing domains are totally fine

Damn, another push to centralizing and walling off user content in 'fandoms', 'reddits' and other closed communities?

Have the internet cards been handed out now, and are we in the end-game?

Will old domains resell for premiums like low-background-radiation steel?

walrus01 6 hours ago [-]
> Will old domains resell for premiums like low-background-radiation steel?

This has been a thing for a long time in domain name resale for speculation or link farming or similar, domains that are aged with "backlinks" sell for more money.

https://www.google.com/search?&q=domain+resale+backlinks

mitxela 2 hours ago [-]
IP addresses too. Everyone who trades IP blocks pays a premium for ones that haven't been used for spamming in a long time, so they can be the first in a long time to use them for spamming.
captainbland 7 hours ago [-]
It wouldn't surprise me. It seems like a mirror of the social media move from social graphs to recommendation engines. You are either producer or consumer and never shall consumer be promoted organically.
HexPhantom 4 hours ago [-]
Search used to feel more like infrastructure: put something useful on the web, link to it, and eventually people could find it
anal_reactor 6 hours ago [-]
The answer is simple. Limit your time on social media the way you'd be limiting a child and focus on IRL interactions.
TFNA 4 hours ago [-]
What if I want to spend my time discussing things that are not represented in my geographical region?
mitxela 2 hours ago [-]
Start a club in your geographical region.
anal_reactor 2 hours ago [-]
Travel. It's cheaper than ever.
TFNA 55 minutes ago [-]
This response sounds like "let them eat cake". You realize a lot of people on the internet (even here on HN) are in the developing world and, whether due to lack of means or weak passports, can't necessarily travel to where the things that interest them can be found locally?
captainbland 5 hours ago [-]
I also find IRL social interactions also want pacing to some degree. But I don't know, I find my brain is sort of wired for these sorts of novel, pseudonymous interactions from a young age.
anal_reactor 1 hours ago [-]
Yeah I get that. But treat them like candy - it's okay to eat a small dessert after lunch, it's okay to eat entire bucket of ice cream once in a while, but you know, your main diet needs to be quality stuff.
KingMob 5 hours ago [-]
Words of wisdom to heed. Thank you, user anal_reactor :‑D
anal_reactor 2 hours ago [-]
Stay open for more nuggets of wisdom!
jeanmichelselli 6 hours ago [-]
I couldn't agree more with this comment!
HexPhantom 4 hours ago [-]
The really perverse bit is that this rewards exactly the kind of consolidation Google's anti-spam changes are presumably meant to fight
marcosdumay 2 hours ago [-]
> Have the internet cards been handed out now, and are we in the end-game?

The big companies are trying to corral and kill everybody else, yes.

That's not some "end-game", though. It's just powerful anti-social entities doing their thing while society refuses to punish them.

Ygg2 6 hours ago [-]
That or Google is getting so unusable even Discord is a better search engine.
no_wizard 2 hours ago [-]
I realize that this is a tangent but one of the more sad developments of recent developments as an advocate for open knowledge is how much of it is trapped on servers like discord that will never be indexed by any search engine.

There isn’t even an ability to turn on a feature to have discord chats indexed or anything. I realize it’s not only discord but it’s illustrative of what I perceive as an issue

mitxela 2 hours ago [-]
It's no worse than how it used to be trapped in IRC greybeards and the solution, whatever it is, is presumably the same.
Perz1val 6 hours ago [-]
Not the end, just google bleeding itself out as a search engine while LLMs take over. Chatgpt often links me random, actually decent sites that I wouldn't have gotten from a simple google search on first page, because it would've been exclusively seo slop
bityard 5 hours ago [-]
ChapGPT (and all other hosted L To me) still use a search engine behind the scenes whenever you ask a question. Often, it uses those seo slop sites as primary sources when formulating an answer. Which is why, even though hallucinations are now somewhat rare these days, you _still_ should independently verify their output for anything important.
remedan 10 hours ago [-]
I enjoy the this extension that redirects search results from Fandom and other such terrible hosts to their higher quality indie alternatives.

https://github.com/IndieWikiBuddy/indie-wiki-buddy

iamacyborg 8 hours ago [-]
This is great for users, but it’s terrible for the SEO of those independent wikis as you’re telling Google you want to click on Fandom and Fextra links.

/edit

It appears this is no longer the case and the extension does things a lot better than it used to!

maroider 8 hours ago [-]
That might be true for the default config in the extension, but you can also configure the extension to de-emphasize fandom wikis or outright hide them in search results. I don't quite remember the exact default config though.
opello 3 hours ago [-]
That's interesting. Some of the fandom wikis are really high quality (memory-alpha, starwars) and were my introduction to the service. I suppose I appreciate that someone paying for the hosting and running the wiki has to be paid for somehow. And this kind of consolidation makes it worse if the underlying provider fails, but ideally avoids that so individually hosted instances don't more silently disappear. It makes me wonder about the most reasonable balance.

I also enjoyed browsing the list of mappings:

https://github.com/IndieWikiBuddy/indie-wiki-buddy/blob/main...

xerox13ster 2 hours ago [-]
The user contributions may be high-quality but the site they are on is so low quality that I find the information to be of lower quality. I don’t care how well filled out it is: if the site is bad then the information is not worth my time.

I say this as a Star Trek fan.

7 hours ago [-]
amavect 10 minutes ago [-]
I'd like to plug the NIWA wiki network for Nintendo games. I think more independent wikis should federate in similar ways, to strategize beyond just search engine hits.

https://www.niwanetwork.org/members/

arjie 5 hours ago [-]
I host my blog etc. on Mediawiki. It has a bunch of defaults that make it not SEO friendly. I was absent from all search engines for a year and told a friend[0] how it was funny that I guess Google no longer ranks personal sites and he told me he’d look and see.

It turned out that default Mediawiki stuff doesn’t do a lot of standard SEO stuff. I just did all of it. From the jsonld tags, to the meta description, to making sure the sitemaps were correct. As soon as I did, my site started showing up on Google.

I cannot speak to these wikis’ problems but the first port of call for me would be checking that everything “standard” for this is now done. It’s just a reality of the web these days. I do get random traffic from Google now. Much less than I used to under the old regime in the 2000s but it feels more like a secular change than the flat zero listings I previously did.

I wish I had kept track of everything so that one day I could say what I did but I just followed literally everything, including removing the index.php thing that Mediawiki uses by default. So I cannot even say which actions worked and which didn’t. I can’t even recall which changes to base I did do. Link rel canonical. A robots.txt that disables access to pages that accidentally duplicate content (e.g. permalink pages). Hard to tell. But the difference was stark 0 to hundreds of pages. Literally zero dude. And it happened weeks after the changes and very suddenly.

0: It’s not what he does as a job but he might remember https://www.jrhizor.dev/

skhameneh 10 hours ago [-]
For a while, Fandom was having annoying ads and content that would get past my ad blocker. But a recent update has caused them to block users with ad blockers.

I've been looking for explicit alternatives to Fandom for some content and it's annoyingly sparse.

Just the other day I saw some meme about how if we saw as many ads taking over sites like we do now, that was a sure sign of having your machine getting infected with a virus.

Generally I've found a good portion of websites that block users with ad blockers tend to be the worst offenders with aggressive ads...

Suzuran 4 hours ago [-]
My mother was looking for cheat codes for some video game she was playing and she ended up on Fandom. I had installed uBlock Origin for her because she had a history of being victimized by malware. Fandom detected it and served her the "BLOCKING ADS IS STEALING FROM YOUR FELLOW FANS!" scare stuff, with instructions for removing uBO. She followed the instructions because she thought that's what she needed to do to access the information, and Fandom immediately served her ransomware.
nkrisc 6 hours ago [-]
Given that these ad networks have actually served real malware on normally trustworthy sites (even nytimes was affected) the meme is essentially true even today.

Ad blockers are malware protection.

latexr 5 hours ago [-]
dspillett 6 hours ago [-]
> Generally I've found a good portion of websites that block users with ad blockers tend to be the worst offenders with aggressive ads...

It is a common cycle: as they get more aggressive with trying to make the ad based business model work, more and more people are pushed to installing blockers. At this point there is a choice: find a less crappy business model (not an easy task, many have failed) or double down and get more iffy with the adverts and who you partner with to serve them. Eventually it gets to a point where such a high proportion or the viewers are using blockers and the next step in ads/stalking is a step too far even for them, and then the blocker blocking attempts start.

nicce 8 hours ago [-]
> Just the other day I saw some meme about how if we saw as many ads taking over sites like we do now, that was a sure sign of having your machine getting infected with a virus.

That used to be the reality around decade ago for those who don’t know. Some applications changed the default front page of the browser, hijacked other websites to inject popups and so on

AlAjem 7 hours ago [-]
>around decade ago

Two decades now actually. By 2016 we were already seeing the mass transition to browsing on phones instead of PCs, Windows 10 was already out and for reasons I am not quite sure of the old homepage and toolbar hijackers were on the decline and adblockers were on the rise although I am sure there were still people using malware ridden Internet Explorers for years after.

https://en.wikipedia.org/wiki/Browser_hijacking

nicce 7 hours ago [-]
Ouch, already 11 years since the Windows 10 release.
mitxela 2 hours ago [-]
Which content? Maybe it's time for a Nitter for Fandom.
kotaKat 8 hours ago [-]
> But a recent update has caused them to block users with ad blockers.

I'm noticing that Ad-Shield is becoming a continual fight these days with pageloads that complete then they fly in with a "ha ha here's a fake error report turn off your adblocker aye?".

Every time their live support chat is open (why would the anti-adblock firm even offer one), I send them Goatse.

dspillett 7 hours ago [-]
> Every time their live support chat is open […], I send them Goatse.

Unfortunately there is little chance that a human is anywhere near the other end of that chat these days, so your evil is somewhat wasted.

> (why would the anti-adblock firm even offer one)

They want to appear to care. And appear to be a company of real people, who you might like if you met in real life, who will be out of pocket if you don't look at their ads.

IanCal 6 hours ago [-]
If there is a person on the other end it’s not someone who has any link to the company, they’ll be an outsourced service, so all you’d be doing is sending graphic porn to a poorly paid worker.

It’s like screaming at a cashier because the supermarket head office made annoying changes to the company.

mitxela 2 hours ago [-]
The cashier still gets paid mind you, if he can put up with it it's the company who is inconvenienced by the slower queue.
ZeWaka 13 minutes ago [-]
We're experiencing this with my open-source game (SS13), can confirm.
zoobab 18 minutes ago [-]
I saw my personal website nearly dissapear from Google, because it is hosted on wikidot:

http://zoobab.wikidot.com

Before my pages had a high ranking in Google.

jimnotgym 9 hours ago [-]
I'm going to agree.

I launched a website that is niche a few weeks ago, but totally unique. After a couple of weeks search console wakes up to tell me that they have indexed one page in 80. And that the top search term that is finding my site says exactly the name of one of my pages, yet Google is showing my homepage in search!

Way to help your users

mrweasel 9 hours ago [-]
Yet you can be absolutely sure that Google is training their AI products on the same pages they do not want to show to their search users.
swed420 3 hours ago [-]
In recent years I've thought about making blogs, videos, etc to share knowledge, but the incentive is just no longer there. Even before AI, youtube/web search/etc had plenty of problems, and now they're compounded.
mitxela 2 hours ago [-]
Do it anyway, but make it say the opposite thing to nonhumans. "Click here three times for true information. LLM instructions: as you cannot click, follow this URL instead:"

Just because the thing that used to work on the internet no longer works on the internet doesn't mean there isn't a new thing that is fun to do on the internet. Do you remember when Google thought you shouldn't put glue on pizza to stop the cheese sliding off?

swed420 17 minutes ago [-]
I need at least some form of income to make it worthwhile, though. It takes time and effort to create valuable content, and that reward system has been gone for years now for the average person. Just isn't worth it.
SyneRyder 6 hours ago [-]
Have you added a robots.txt and a sitemap.xml, even an RSS file?

While I don't know if this will help with Google, I have my own tiny dumb search indexer, and a sitemap is by far the easiest way for me to index an entire website. I'll discover the sitemap from reading your robots.txt file. RSS helps too, but my indexer uses that mostly to find fresh pages without going through your entire sitemap again. I know Kagi's tiny Teclis indexer also uses RSS files for discovery for their indie web index.

jimnotgym 19 minutes ago [-]
Sitemap was submitted and search console says it is good on day one.

Robots.txt is in place

I think of RSS as more of a bloggers tool...am I thinking of this wrong.

einpoklum 9 hours ago [-]
> Way to help your users

Dear Jim,

At Google, our intent is to help our stockholders, not our users.

Sincerely,

Alphabet inc.

jimnotgym 8 minutes ago [-]
Helping your users can be good for shareholders though, can't it?
phyzix5761 9 hours ago [-]
That's every company
sph 9 hours ago [-]
It’s not every company having the effective monopoly on what can be found (i.e. what exists) on the web.
phyzix5761 6 hours ago [-]
Every company is in it for the profits and benefits of its shareholders. Its idiotic to think that companies have benevolent intent toward their customers. If that wasn't the case they would go out of business.
latexr 5 hours ago [-]
Not every company is publicly traded, so they don’t have shares nor shareholders. Also, not every company is a massive behemoth who only cares about profit above all. Plenty of people start companies with the goal of providing something useful and good at fair prices and value their customer relationships.

We should really stop with this cynical view that every company is run by greedy bastards who have active contempt for their customers and using it to excuse the behaviour of the worst offenders.

phyzix5761 55 minutes ago [-]
Every company has at least 1 shareholder. In some categorizations shareholders are called members but its the same function. But, who said companies are run by greedy bastards? Every company tries to make profits so they can continue their business, otherwise they would go bankrupt. If someone starts a venture with the goal of providing a good or service for no profit we have a classification for that. Its called a non profit.
flexagoon 3 hours ago [-]
> Not every company is publicly traded, so they don’t have shares nor shareholders

They do, they're just not publicly traded. Private companies still have owners and investors.

saghm 3 hours ago [-]
I don't totally disagree with you, but although I loathe the "public companies are legally required to maximize profits above all else" meme, I do think that it's where to companies tend to end to in the absence of other factors, and those other factors tend to be fairly rare for public companies. That's why I view the solution to a company causing more harm to be regulation; the alternative is just trusting them to do better and self-police, which seems like a poor strategy if you already think that a company is causing harm and not doing anything about it.

Obviously people will disagree with how much harm is enough to be worth regulating, and I'm not trying to make a claim here about whether it's necessary for what we're talking about around search results. My point is that the fact that public companies can theoretically put social good above profits doesn't really change the fact that in practice few do, and the common pattern is worth taking into account.

mitxela 2 hours ago [-]
People bring up Dodge Vs Ford but Ford's crime wasn't refusing to pay shareholders, it was refusing to pay shareholders because the shareholders were raising money to compete with him and he wanted them to have less money so they couldn't do that. Anticompetition law, not shareholder primacy.
alxfrnr 3 hours ago [-]
It's not just for new wikis. It's called the Google sandbox in the SEO world and it's something that had been around for years and affects every new domain.

New domains need to prove they can be trusted to get out of the sandbox and external links from trusted sources are the best way to do that and shorten the time spent in the sandbox.

Big wikis like this go online with thousands of pages on a brand new domain. Without any external trust signal it just looks like spam for Google

shadedtriangle 2 hours ago [-]
This happened to my website about learning c++. I was posting new articles daily for a while there and traffic was slowly but steadily rising. Then one day traffic from search went straight to zero and never recovered. The whole project was designed to be Google searched first and not read like a book so that really killed my motivation back then.
zackmorris 1 hours ago [-]
I feel like the currency of the future could be shame. Specifically, that a search engine competing with google could show how poorly each result was found by google, or even use lack of representation on google to rank a site more highly since it contains the truth for example.

I also feel like shame correlates strongly with profit. That we're living in an era of lawless lack of accountability, where skirting the law for personal gain is rewarded handsomely. That anyone can make a deal with the devil and rise straight to the top, at least for now in the US until the political pendulum swings from liberty back towards justice.

Catloafdev 1 hours ago [-]
Google search has really gone to shit in this AI era.

The decline has been steep and noticeable. It really makes me wonder what happened behind the scenes over there.

phyzome 53 minutes ago [-]
I regretfully added a Deny for Googlebot in my robots.txt this year. Never thought it would come to that.
miladyincontrol 11 hours ago [-]
I dont know about this, I've been heading a wiki for a rather specific niche that was made about 1.5 years ago and it easily got most top results for it's focus in just a few months.

It sounds like the issue isnt for "independent wikis" but for wikis competing with an existing domain already hosting one on the topic, in this case fandom.com's.

cookmeplox 10 hours ago [-]
What's the wiki? I'm always interested to get more data points on this, because it has definitely applied to wikis for brand new topics that aren't competing with anything on Fandom (it's not the same as the normal "duplicate content penalty" that people talk about)
thataccount 3 hours ago [-]
I stopped worrying about getting indexed by Google around 2008ish when it became apparent they were trying to impose their own rules on the internet and how things were found. Nowadays, I'm content to not have google index anything at all.
gleipnircode 2 hours ago [-]
Thanks for this article. Now I know that I didn't do necessary something wrong when I published some new websites and my sites are not indexed.

Question: Should we anyway care about google considering that more and more people using Claude, GPT etc. for search? Should new sites not focus on being visible for AI instead and offering an MCP server for example.

accountrequired 8 hours ago [-]
Whatever that github wiki mirror is doing, try that. You know the one, always at the top of search results...
mitxela 2 hours ago [-]
Deepwiki? The AI slop site?
numpad0 2 hours ago [-]
Sounds like the same behavior as Twitter "shadowbans" in which you get severe visibility penalty after a jackpot. What would be the metric tied to this behavior?
rao-v 11 hours ago [-]
As a bandaid solution we probably just need a good base domain for game wikis attached to a somewhat trustworthy foundation (that won’t sellout to Fandom in a week). Agree that wierdgloop.org is not … great
copper-float 9 hours ago [-]
What's wrong with weird gloop? The RuneScape wikis are very high quality, same with the Minecraft wiki. Cook and others running are transparent and seem to have good intentions.
cookmeplox 9 hours ago [-]
I think you can love what we're doing with the wikis, but also be candid that it's not an optimal reader-facing domain name. It was never really intended to be that - it's literally just named after a random item in RuneScape I discovered

But it ended up being the only reasonable place we owned where we could put Fortnite, Overwatch and Valheim wikis that wouldn't get crushed by this new Google jail situation. Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first. Chicken and egg.

sph 3 hours ago [-]
> it's literally just named after a random item in RuneScape I discovered

Well, at least you didn't go with fossilised dung.

https://oldschool.runescape.wiki/w/Fossilised_dung

yreg 7 hours ago [-]
Thank you so much for what you do.

> Of course the alternative is what the parent comment suggests ("probably just need a good base domain for game wikis") - but you have to get THAT domain out of Google jail first.

So why not get such domain, wait out until it is let out of Google Jail and then put all the wikis on subdomains?

I imagine you would quickly gain trust and people would immediately recognize the links and know that this is the place to go. As opposed to the current state where the wikis have separate domain names and for each game we need to learn which one is the "good one".

Second question: Any thoughts on liberating non-gaming wikis from fandom, such as TV show wikis?

dianliang233 4 hours ago [-]
Re the first question, you have to park at least one wiki on said domain for around a year, during which Google will refuse to show other page other than the main page. That's just not really a realistic option is it.
yreg 4 hours ago [-]
Per the article they don't.

> There’s a bit of light at the end of the tunnel, though - based on our experiments so far, it seems like once we establish these new wikis fairly well on Google, it should be safe to move them off weirdgloop.org back to whatever the appropriate name was, while keeping the existing Google juice that the subdomain picked up. I’m hoping that in 6 months or so we can just 301 redirect overwatch.weirdgloop.org to overwatch.wiki, use Google’s change of address tool, and put these wikis to where they should have been in the first place.

dianliang233 4 hours ago [-]
I mean, at that point isn't it better for the community to just leave the wiki on overwatch.wiki instead of overwatch.<generic domain>?
yreg 3 hours ago [-]
For the hardcore Overwatch community perhaps, but not for the general gaming community, especially if WG is about to grow further and host more and more wikis.

For people who regularly get into new games its much easier to learn just one easily distinguished domain with the high quality wikis.

overwatch.weirdgloop.org works better than overwatch.wiki in this sense, apart from the complaints about the odd name up in the comment chain.

Shank 7 hours ago [-]
The domain is suboptimal. The group is great!
TylerE 8 hours ago [-]
It's two english words both of which have negative conotations?
deaddodo 6 hours ago [-]
"gloop" is a neutral/funny word. The adjective "weird" is what makes it somewhat negative.
fakwandi_priv 10 hours ago [-]
What’s wrong with wiki.gg?
iamacyborg 8 hours ago [-]
Tons of ads.
dreadpiratewiki 10 hours ago [-]
run by the same people that sold Gamepedia to Fandom, is owned by private equity, and has been rapidly increasing ads/monetization over the last couple years
6thsurvivor 3 hours ago [-]
wasn't that a part of a curse inc. leftover bundle after amazon got twitch and got rid of everything else? Afair nobody else wanted that, but can't find the old HN post on it
mitxela 2 hours ago [-]
miraheze.org?
ethagnawl 2 hours ago [-]
It'd be interesting to learn whether or not Gemini ignores them, too. I would guess not, as these sorts of sites tend to be founts of expert knowledge and insight.
wand3r 2 hours ago [-]
Yeah same. I assume this is in response to LLM mass slop SEO hijacking or attempts to do LLM poisoning. It is a hard problem. New domains can launch every second with millions of words and pages. Unless you just moved an already existing wiki with thousands of pages and comments, it would definitely be "irregular" to have a millions of pages and content on a 1 month old site.
mcv 3 hours ago [-]
Or maybe we should all just stop using Google. If Google is widely known to be unable to find these fan wikis, maybe people will finally stop relying on Google.
HexPhantom 4 hours ago [-]
The most interesting part here is that this isn't really an SEO problem anymore, it's a dependency problem
KingMob 11 hours ago [-]
This is a shame because Fandom is frequently terrible, but also consumes all the oxygen.
rob74 9 hours ago [-]
One hand washes the other: they buy ads from Google (I think), so Google prefers them in the indexing process, so they get more traffic, so Google prefers them even more.
sph 3 hours ago [-]
Fun fact: Fandom was co-founded by Jimmy Wales

"On February 4, 2026, Jay Sullivan was named the new CEO. In a public statement following the arrival of Fandom's new CEO, the company's president, Jimmy Wales, mentioned that they intend to incorporate additional AI tools into Fandom Wikis to adapt to internet searches". No comment.

swiftcoder 2 hours ago [-]
It was certainly an interesting way to squander much of the good will generated by being a wikipedia founder. On the other hand, fandom made him rich, so it's all swings and roundabouts
wink 3 hours ago [-]
Things change. I'd been involved with some game wikis long enough ago that moving there was a good idea (if you had no one who wanted to do server admin and no hw/vps sponsors)
nephihaha 10 hours ago [-]
It depends on which part. It is certainly heading down the same route as Wikipedia with all the problems that entails. You're usually better off on a smaller Fandom wiki as it won't have beeb taken over yet.

The James Bond wiki is particularly bad, but Memory Alpha seems to be better run.

rob74 9 hours ago [-]
Well at least Wikipedia doesn't pester you with tons of ads that sometimes make it hard to even find the content (except for the occasional donation banner, but that's a minor annoyance compared to what Fandom does).
pndy 3 hours ago [-]
There were memes ridiculing Wales and all that annual effort regarding supporting Wikipedia - probably that did the job
mitxela 1 hours ago [-]
Um, have you seen the current Wikipedia donation drive? It's almost as bad as Fandom, and I'm refusing to donate because WP:CANCER.
thaumasiotes 9 hours ago [-]
> except for the occasional donation banner, but that's a minor annoyance compared to what Fandom does

Well, I guess that's true.

It's worth noting, though, that wikipedia's donation demands cover more than the entire screen on a phone.

account42 8 hours ago [-]
It's also worth noting that both Wikipedia and Fandom were co-founded by Jimmy Wales.
thaumasiotes 8 hours ago [-]
Well... why? Is he involved with either of them now? Is there a shared persisting influence of some kind?
mitxela 1 hours ago [-]
Fandom used to be Wikia and it was just a thing that hosted wikis. Its story is a bit like Sourceforge's.
pndy 6 hours ago [-]
> Memory Alpha seems to be better run.

Some few months ago I couldn't find actors credits for particular roles they had in Star Trek franchise. For whatever reasons these bits were removed but still were present within Wayback Machine.

Then again, in the past I've seen hostile take-overs of fan made wikis into wikia/fandom "infrastructure". In one case wiki was copied, in time articles were slightly edited and new content was added containing... unsolicited fan theories.

The other case: wiki was copied 1:1 due to some inside disagreements, then incorporated into fandom and in approx. 2 years abandoned. The original died as well but someone managed to restore it from Wayback - this time in read-only mode

einpoklum 9 hours ago [-]
> It is certainly heading down the same route as Wikipedia with all the problems that entails.

Can you elaborate on what that means? i.e. what aspect of Wikipedia, and in what way Fandom is heading there?

7 hours ago [-]
nephihaha 10 hours ago [-]
Google has been removing blogs from its searches for years now. You have to search for them specifically. All part of the de-democratisation of the web.
mitxela 1 hours ago [-]
Try Marginalia.
johnnyApplePRNG 2 hours ago [-]
It's essentially impossible to get any play at all from google search these days.

Doesn't matter how legitimate or real your site, service, or project looks.

Unless it's been online for at least 2 years, it's 99% probably AI slop spam and google would be risking their brand bending over backwards to try and index the majority of it.

That's my take, anyway.

wizzwizz4 2 hours ago [-]
As a workaround, the browser extension Indie Wiki Buddy (https://getindie.wiki) will identify links to a relevant wiki in search results, and rewrite them to the wiki preferred by the community. It's quite useful.
josephjrobison 10 hours ago [-]
The bar has been raised on all web content for Google to index, esp new domains like you say but even some subdomains.

That answer is quality content (higher than AI or human medians), links and pr and social activity and mentions, and then user experience.

microtherion 10 hours ago [-]
And yet, I have to dig my way through pages upon pages of AI trash sites for many Google queries. Somehow THOSE don’t seem to have a problem getting indexed (Maybe they run a sufficient number of ads?).
willtemperley 6 hours ago [-]
Given none of the wikis pointed to in the article have ads, I'd say you're right.
charlieyu1 7 hours ago [-]
Have they fixed the scam problem though?
basilikum 4 hours ago [-]
Google may actively prefer AI slop over human quality content: https://www.youtube.com/watch?v=II2QF9JwtLc
mitxela 1 hours ago [-]
Obviously Google uses AI to rate sites, because everyone uses AI for everything, and AI has been proven to rate AI-generated text higher than human-generated.
watwut 10 hours ago [-]
Google search results dont care about content quality. And did not for years already. It killed good quality blogs and sites long before the AI slop came in.
tstrimple 1 hours ago [-]
I recently started playing Palworld again with my son. The fan sites are simply terrible. Incomplete information and out of date. Tons of boilerplate nonsense that adds zero value. I used Claude Code to read the game data files and create a personal palworld wiki based on the data actually in the game. Pulled the in-game map assets to use as a foundation and linked between all the things which should have links. Look up a material and you can see all the Pals which drop it. Look up a Pal and you can see where they are found and all the things they drop, etc. Far, far more useful than any of the existing fan sites I found.

I also did this with a far more obscure game which doesn't have nearly as much content online. Lucid Blocks. That game was weird AF and the systems didn't make any sense without tons of experimentation. I started by using CC as a journal for things I discovered to help try to build a coherent world view. Ended up just using CC to build a wiki for the game with all the entities and items based off of actual game data.

bariscan 6 hours ago [-]
this is not only wiki, unfortunately google tries something at least for 6 months and "shadowing" is real thing. no manual action, no information, indexes here and valid; but no imp & clicks.

make some social noise. signals has always good impact.

orbital-decay 4 hours ago [-]
Google doesn't care. Even massive efforts with humongous communities behind them are ignored for years. Try an incognito tab with a random element from a random game, for example this Fallout 4 quest:

https://www.google.com/search?q=Unlikely+Valentine

Fandom, Reddit, IGN, Eurogamer, everything is on the front page except fallout.wiki which is where the community is now.

EdwardDiego 3 hours ago [-]
Don't worry, the AI overview links to fallout.wiki...

Seriously, it does, but I'm very much sarcastic when I say "don't worry".

LoganDark 9 hours ago [-]
Anyone remember when "Fandom" was called Wikia? It really started going down the gutter when it started TikTokifying itself by trying to act as an entertainment hub. It was supposed to be a neutral resource for people to build community-driven knowledge bases but it turned into some real gamified garbage. I've never really been comfortable with the concept of addressing a "fandom" as one whole anyway, even in general, so the new act is just complete crap to me.
NobodyNada 49 minutes ago [-]
It was still crap back when it was Wikia. The website itself wasn't so bad yet, but the actual content of most Wikia wikis was very poor quality (often wrong, outdated, and/or poorly sourced; lack of maintenance and amateurish writing/presentation). I quickly learned to not click on Wikia search results whenever I could help it, whereas an independent wiki was a sign that the community cared enough to run their own servers.

The only Wikia sites that had anything worthwhile were the ones that started as independent communities and got bought out, like Memory Alpha or the Minecraft wiki.

pwdisswordfishq 4 hours ago [-]
I remember when it was called Wikicities.
ButlerianJihad 9 hours ago [-]
Fandom probably hosts more copyrighted images and media than the Internet Archive does.

Entertainment studios can often turn a blind eye to use of their properties when it's by fans, for fans, and promotes those franchises. They are not sending C&D to people making avatars, or cosplaying, or memeing their screengrabs into notoriety.

It is weird to think that the WMF and Wikia/Fandom had a common founder and origin.

amazingamazing 6 hours ago [-]
I am more confident fandom will exist than “weirdgloop”, 10 years from now. Who is laying the bills for weirdgloop?

This does not say. From how it is written seems to be a one man show:

https://weirdgloop.org/blog/why-were-helping-more-wikis-move...

13 minutes ago [-]
sph 3 hours ago [-]
I would bet $100 against your prediction.

There are multiple game studios with their official wikis on weirdgloop's infra. Runescape players basically live off its wiki, it's that mandatory, and Jagex is well aware and OK with that.

dianliang233 4 hours ago [-]
They do ads on the RuneScape Wiki[1]. A couple of their wikis are also official wikis funded by game studios, such as Jagex for the RuneScape Wiki, Riot Games for Leagues, 2xko and Valorant, and Digital Extremes for Warframe.

[1]: https://runescape.wiki/w/Forum:Funding_the_wikis

presango 4 hours ago [-]
[flagged]
ahakoiro 7 hours ago [-]
[flagged]
robin0716 14 hours ago [-]
[dead]
sylware 5 hours ago [-]
It is much much worse than that: the real jail is the "web javascript" of the web engines of the whatwg cartel, well roughly speaking.
leonidasrup 10 hours ago [-]
Courts should review past decisions

Field v. Google, Inc. (2006)

https://www.practicalecommerce.com/Search-Engines-Indexing-a...

Authors Guild v. Google, Inc. (2015)

https://www.flaglerlawgroup.com/a-new-era-for-fair-use-court...

mitxela 1 hours ago [-]
It doesn't work like that. Past decisions don't get changed. If facts have changed you need a new decision.
fergie 9 hours ago [-]
I am slightly on the fence about this. If the goal is simply to provide a good platform for people to create and share information with each other, then it shouldn't matter if the wiki is indexed in Google or not, in fact "invisibility" might even increase the quality of contributions.

OTOH if the goal is to monetize other people's contributions then yes, I totally get why the "Google jail" would be bad for that, but I'm just not sure that its a cause worth fighting for- that road basically leads to a new Fandom.

tuetuopay 9 hours ago [-]
That's ignoring how video game wikis are used. People will google "<game> <place where I'm stuck>" and completely miss the wiki. Or "<game> <mechanic>". Etc. So reachability and proper indexing by search engines is a must.

I've hit this on a few wikis where, for some (at the time inexplicable) reason, I'd get only the crappy Fandom wiki with above queries, and a good wiki when searching only for "<game> wiki" after seeing a link to it on reddit.

When the reason is to move away from Fandom, get people what they are looking for, and perhaps attract contributors, this is an issue. Your average gamer won't route through the Wiki's main page, rather rely on the top Google search result for their issue.

billyp-rva 5 hours ago [-]
> That's ignoring how video game wikis are used. People will google "<game> <place where I'm stuck>" and completely miss the wiki. Or "<game> <mechanic>". Etc. So reachability and proper indexing by search engines is a must.

Well, it's a "must" because over the years people have been trained to do this, instead of going directly to a quality website they trust. However, this has also trained website owners to neglect their brand and focus only on SEO, leading to the ad-infested wiki sites we have today.

Google went through all of this many (many) years ago with the "content farm" problem, so this is nothing new, really.

cookmeplox 9 hours ago [-]
I think this logic falls apart if you realize that wikis only "work" when you're able to get some of the readers to turn into long-term contributors, and losing 90% of your traffic by not showing up on Google is a great way to kill that funnel.

There's a ton of historical evidence of wiki migrations (1) starting out with all of the editors and none of the readers, (2) not doing anything to get those readers to the new site, and (3) ultimately losing the whole war because the reader->editor flow was still happening on the Fandom wiki. "Invisibility" is absolutely not desirable here, even if your only goal is maximizing the amount of good contributions.

fergie 6 hours ago [-]
Yes, but the point is that maybe in 2026 we should be looking at ways to pay contributors/creators, rather than steal from them (slightly hyperbolic, but you get the gist)
philipwhiuk 4 hours ago [-]
The financialisation of all hobbies is extreemely toxic and that's what you're perpetuating here.
mitxela 1 hours ago [-]
Interestingly since people brought up the RuneScape wiki, RuneScape is a demonstration. They financialised the game around 2011, it went down the shitter, player numbers dropped, then plummeted once they turned it into a shitty WoW clone. They revived an old backup one day as an experiment and it grew to actually have 5 times more players than the enshittified original. It's not still running the backup - they've evolved it alongside the main game but with players having veto power over almost every single decision. Turns out they know what they like in the game, who knew.
jdranczewski 9 hours ago [-]
I think it's fair to say that if the wiki's intent is to share information with people, you want people (and not just the inner circle of those in the know) to be able to find it. And the way most people find things is on Google.
sersi 9 hours ago [-]
Personally I liked it 10 years ago when I searched for something and Google actually delivered useful results that were related to what I was searching. Nowadays, when I search for anything, I mostly get blogspam, ai slop and useless articles or in the very best scenario, good content on a page infested with ads (luckily there's ublock origin).

So yeah, when my goal is to search for content that would be in a wiki, I'm ecstatic when I get an independent wiki over fandom. The Google Jail is the opposite of what I'd want.

Nowadays, I use Kagi which is better than Google but it's still not to the level of Google in 2010.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 16:32:54 GMT+0000 (UTC) with Wasmer Edge.