Open-source community maintained bang repository

Browsing6853

DuckDuckGo is not updating them in a good frequency (there are innumerous reports about that on Reddit) and there is a huge set of applications depending exclusively on the hidden JavaScript in their website with the list of bangs and URLs to work. In the long term, this will help decrease the number of issues regarding on bangs and every other application because of open-source nature. The main problem is moderating the requests on the beginning, but considering the work done by Kagi to fix bangs and the possible visibility of this project, I think it will be worth it.

Few examples of topics related to bangs:
https://kagifeedback.org/d/208-bangs-wayback-waybackmachine-not-parsing-url-correctly
https://kagifeedback.org/d/479-add-kagi-bang
https://kagifeedback.org/d/465-make-you-bang-using-youcom-search-engine-instead-of-youtube
https://kagifeedback.org/d/410-fix-oed-ox-bang
https://kagifeedback.org/d/375-gm-bang-not-redirecting-to-google-maps-search
https://kagifeedback.org/d/151-bbcgf-bang-does-not-use-proper-search-query-for-bbc-good-food

Vlad

@z64 what do you think?

z64

Vlad I think there are a few facets here.

Issues with existing bangs (bugs)
Additions of new bangs or altering existing ones
Unlike DDG we would like to add user-defined bangs

Because of (3), users will have the ability to do (1) & (2) without any intervention or waiting on our end in most cases, they just override the stock bang. Very few bang issues have required a change that could not be issued from user-level config - 5/6 from the OP.

This leaves the questions of:

Flowing down these template fixes from users to our default set
Admitting new bangs or changes into the default set. For example some sites have moved domains, but support redirecting to the new one as to not break any backlinks.

(1) could be resolved with a "report bang" option. This could provide a form to submit some additional details - like how to fix the bang if they know - effectively optimizing submitting a bug report either here or on a Git repo. This would hook into our existing feedback mechanisms (IA 👍️ / 👎️)

(2) is the only thing here that may warrant creating a repo (or anything similar) as a vehicle for moderating and accepting these changes. It should probably be done long-term, but the urgency is greatly reduced in our case.

A further mitigation is allowing users to share bangs (like sharing lenses), then users can maintain their own repos.

z64

Arguably 6/6 of the posts OP linked could be fixed by user-level config if we also exposed URL encoding options, as some services have different expectations around this.

ploum

Just to add my own experience here: DDG supports "!ploum" to search on my own blog : https://ploum.net.

Unfortunately, this was done at a time when I was using Wordpress. It’s currently a static website, without search engine. I’ve asked DDG multiple times to either remove the bang or to make it a site:ploum.net search (which would be my prefered outcome) and never had any answer.

Which tells me that it might be a sensible option to "fork" DDG bang list (considered that it doesn’t really evolve anymore) and allows kagi to modify it.

As pointed out, this is nothing that could not be fixed at the individual level. But I believe that this about more tangent usecases. Nobody but myself will configure a bang to search on my own blog. But many of my readers, if aware, may use a preconfigured blang to find one of my article they remember and now want to share (an usecase which exists according to mails I receive).

Making an open and dynamic bang list that user can contribute to might be a really good way to foster competition. (For example, I’ve just learned about a French healt-based search engine and created a custom bang about it. This could definitely benefit other French speaking users)

Of course, it is always a matter of priorities.

Vlad

This is something we explored doing before and it does make sense (especially as many of DDG bangs we inherited are broken and are subject to constant fixing).

I think the repo would be organised with basically two files. One is external bangs and one is for itnernal (kagi specfic bangs). it would accept PRs to modify both (obviosuly itnernal bangs would have more scrutiny).

is this what everyone would like?

KagiForMe

Vlad This would be awesome! I would love it! Any plans to implement this soonish? 🙈

ploum

Looks nice to me (but I’m aware that I’m doing bikeshedding here, I’m not sure this is a killer feature)

mlissner

Hi all, I'm the director of Free Law Project, a non-profit that works to make legal data more broadly accepted. We host CourtListener.com, which has several free legal search engines (case law, filings, judges, oral arguments, etc.).

I came here because I'd love to get bangs set up for our website in Kagi so that Kagi users can easily query it. I can imagine queries like:

!cases, !legal, !law — for searching case law
!recap, !pacer — for searching our collection of filings (called the "RECAP Archive")
!judges — for searching judges
!oa — for searching oral arguments

But as I write that, I recognize the need for some sort of guidelines. I know a lot of people would use the heck out of !law, and !recap, but I question how many people would use a bang for our oral argument search engine (it's not that useful!). I also imagine that a system this would soon get a request for the !porn bang, or who knows what else.

So I think I have two questions:

How do we decide if a site merits a built-in bang? Traffic? Notability? Purpose? Maybe this doesn't have to really be defined, because in practice a benevolent dictator will be fine, but I thought I'd raise this issue.

As a site owner, I'd like to have clear guidelines though. DDG does not provide a guideline, so the decision (or lack thereof) is opaque — a bummer.
Is there a way the community can help make this feature a reality? I guess half of this project is setting up the repo with the two files described above, but the harder "half" is using those files inside Kagi itself. Is there a way to get this started?

Thanks for any thoughts people have!

Vlad

mlissner We could do this relatively quickly, the main thing missing are sensible guidelines. We do not want to be in the business of moderating bangs.

I would be careful about allowing english word bangs like !law as that immediately begs the questions who gets to 'own' this and why.

Most bangs are shortcuts of their website name. For example !yt is shortcut for youtube.

!cl could be therefore appropriate default bang for courtlistener. The user is free to rename it via custom bang to !law if they so wish.

Is there a way to get this started?

Coming up with a set of guidelines would be a great starting point. After that it would take us less than a week to have the full system in operation as suggested above.

leftium

mlissner I came here because I'd love to get bangs set up for our website in Kagi so that Kagi users can easily query it. I can imagine queries like:

!cases, !legal, !law — for searching case law
!recap, !pacer — for searching our collection of filings (called the "RECAP Archive")
!judges — for searching judges
!oa — for searching oral arguments

Can you give me the search URL's that would be associated with each bang trigger above?

I will create a custom branch of bangtastic! that supports these triggers and can be hosted publicly. (As a fully functional POC.)

The service can be hosted on and accessed from your own domain: CourtListener.com
Even non-Kagi users can use it (although that is already true for all non-Kagi bangs)

I think open-source bangs means bangs can be executed on any site, not just a few limited sites like Kagi, DDG, Brave, etc

Each site can determine themselves which bangs they support, and how they are executed.
Each site can also choose to support the bangs supported by other sites via scoped bangs:
- @kagi.com!images summons the Kagi bang for images; @duck.com!images would summon DuckDuckGo's version.
- Likewise, @courtlistener.com!cases would summon the bang you suggested.
- On a domain, the default scope is the domain itself.
- To support custom user bangs, their could also be user-level scopes like @mlissner:kagi.com!law, which could be shortened to @mlissner!law on Kagi.com, since that is the default domain. (And logged in with your own account, you can also omit your own username: just !law)

After proposing scoped bangs, I realized the competition is just moved to desirable scope names (albeit at a smaller scale). Domain Names as Handles in Bluesky is a nice solution to this:

Kagi.com already controls/manages both the domain name and bangs on their own domain. (Ditto for duck.com, brave.com)
The bang data can be stored in a standard location on each domain, like favicons, robots.txt, etc. (Perhaps and/or an API since there can be lots of bangs?) The scope tells you exactly where to retrieve the scope's data from.

The exact bang scope syntax isn't final (like lots of sites use email addresses as user id's, which include @ and . characters), but I think the idea of scoped bangs is pretty solid.

Bangtastic! is a personal project I'm developing for myself. I don't need the scoped bang functionality so I will probably not develop it myself. But I'm throwing the idea out there!

KagiForMe

@Vlad
Guideline Proposal:

Everyone can propose a bang for a website. Not just the website owner.
The website has to have some kind of search feature; else people will just use it as a bookmark, which is not the use case.
No (proper, i.e. !in is fine) English words are allowed, no country code bang should be overwritten i.e. !fr.
First come first served, except where (Kagi?) traffic is substantially higher. Users can still change the bang manually.
Kagi will either (a) not accept NSFW bangs i.e. !porn or (b) will only make them work with Safe Search disabled. This would be cool, but a bit more effort from Kagi's side.
Kagi is not expected to fix broken bangs. Broken bangs get removed, users can add a PR that fixes any bugs.
There is a GitHub template that makes people list the link, the expected users, the name of the bang, NSFW status etc in orderly fashion.
...

Do the bangs need to have an expected amount of users?

leftium

I'm more interested in using this bang repository in my own project more than in governing how bangs are added/updated (on Kagi), but I'll share my thoughts:

Right now, I am using bang data from DuckDuckGo for my own project (https://webapps.stackexchange.com/a/162253/1530). I understand it's a little old and things may be broken, but it's better than nothing. I have already made great progress in analyzing all the bangs, merging duplicates, etc.

I heard Kagi had fixed a lot of bangs and modified them to work with Kagi (vs DuckDuckgo). If I had access to these bangs, it would save me a lot of effort since I also want my project to work with Kagi. (My project provides a bang search interface like https://duckduckgo.com/bangs or https://search.brave.com/bangs. But better ^__^. It actually fulfills a lot of open suggestions in this feedback forum.)

In turn, Kagi might find the work I've done massaging DuckDuckGo's bang.js data file useful for their own purposes. And they might even be able to use my project for their own kagi.com/bangs page.

So my opinion is a little biased, but I say just release the data. What's the worst that can happen?

Maybe a flood of github issues requesting new bangs and fixes to existing ones
But Kagi is not obligated to apply changes to the open-source repository to their own search engine
Kagi is not even obligated to respond to any of the issues
Maybe the worst that can happen is Kagi kills any momentum in growing a community to support this bang repository.
But I think waiting for guidelines kills just as much momentum.

On the other hand:

I think the guidelines are overengineering for problems that likely won't happen. (There won't be a flood of github issues asking for new bangs.)
If people do submit issues/pull requests on github, it will give a better idea of what type of guidelines are actually needed. Solve problems that actually exist.
Community members other than Kagi might step up and start maintaining the bang repository. No requirement for Kagi to do it themselves.

It seems one of the main issues the guidelines are supposed to address is akin to problems with domain names like squatting and determining who gets the "valuable" ones. I don't see this as a real problem (until the non-techy majority starts using bangs to navigate the web like domain names.) In addition, Kagi (and my project) allow over-riding bang triggers if you don't like Kagi's default bangs.

Here is a suggestion Kagi can use for managing bangs on their site:

Allow anyone to register a bang that doesn't exist yet. (Like NPM modules)
The bang can be used right away, but it is "hidden" until it passes moderation.
Moderation can be done manually by an admin, or a voting system like this feedback forum.
If the bang is found not useful/against the rules/etc the bang is immediately retired.
After passing moderation, the bang can be found via autocomplete/search/etc.
If bang usage dips below a certain threshold, it is automatically retired so someone else may register a new URL.
The bang creator can retire their own bangs. Updates are more questionable...
Perhaps there is some moderation/voting mechanism to update/delete existing bangs.

leftium

Oh, another issue I thought of: aren't bangs technically DuckDuckGo's intellectual property?

So are there any copyright/patent issues?

I don't think DuckDuckGo plans on litigating other people/companies' use of bangs, but it's something to consider.

Perhaps involving DuckDuckGo would mitigate any risk~

KagiForMe

leftium involving DuckDuckGo

🍿

leftium

Here are some example guidelines from open source projects that take and manage similar contributions from users:

Since Kagi has a feature to author/manage custom bangs, that would be a good place for users to suggest new bangs:

User creates custom bang (https://help.kagi.com/kagi/features/bangs.html#custom-bangs)
User toggles new "public" bang setting to publish to a Kagi bang directory
Other users can search public bangs and add them to their own custom bangs. Kagi users adding the custom bang (and regularly using it) becomes the voting mechanism.
If a significant number of users add a certain public bang, Kagi may consider adding it to their default set.

leftium

Bang triggers are similar to npm package names, so their guidelines might help: https://docs.npmjs.com/package-name-guidelines

Interestingly, npm has the concept of scoped packages: https://docs.npmjs.com/package-name-guidelines

Scopes help fix the problem of "all the good, short names (bang triggers) are already taken."
Unscoped bangs could be determined by usage rate of scoped bangs. If a scoped bang @leftium!gg is very popular and used often while the unscoped bang !gg is rarely used, the scoped bang could be ((semi?) automatically) promoted to an unscoped bang.

Scoped bang triggers might be a useful feature:

Scoped bangs are prefixed like @kagi!images @ddg!images @leftium!images
- Postfixed scopes might be useful, too: !images@kagi
- Or standalone to say all following bangs have this scope: @kagi !images (if/when multiple bangs are supported)
If you overrode kagi's default !images bang with your own custom bang, @kagi!images would allow you to occasionally use the default bang without touching your settings
@username like @leftium would allow for autocomplete of your custom Kagi bangs.
- Perhaps @!images without a scope name defaults to your own user scope.
- It could also allow simple sharing of public custom bangs with others
@category!bang like @entertainment!... could limit autocomplete to bangs from certain categories (as categorized by duckduckgo)
Now you can choose which service to use (I found over 169 bangs that could be serviced by multiple providers:
- mostly the site: search operator...

leftium

I was interested in an open-source version of Kagi's bang data so I could reference any fixes/enhancements Kagi made.

However, my request was blocked by this ticket... Since Kagi won't open up their data (any time soon), I decided to open up my data so maybe Kagi (and others) can use my project as a starting point for discussion.

I went ahead with my project, referencing the only data I have: DuckDuckGo's bang.js data.

I have done some light analysis and merged/cleaned up entries.
I am also designing an extension to the format
Hopefully it won't be too difficult to retrofit Kagi's data/extensions into my project's data if/when they finally release anything.

Eventually, Bangtastic will be an alternate search box for initiating bang searches. Both Kagi and DuckDuckGo will be supported. It should even work from your browser's address bar! Very early prototype here: https://bangtastic.vercel.app/

Easier to enter ! on a mobile device without easy access to the ! key
Try (double tapping) the space key (when the search box is empty).

Bangtastic currently supports:

Searching/browsing bangtastic data: https://bangtastic.vercel.app/bangs
Browsing bang.js source data: https://bangtastic.vercel.app/bangs/sources

Vlad

leftium

Since Kagi won't open up their data

This is not what is going on here. This has not even reached a decision making point (becuse we still lack clear guidelines). WE are more leaning towards doing this then not, but it will not get done on its own. Reality is that this is just one of hundreds open issues on this forum that we have to consider at any given point in time.