Open-source community maintained bang repository

KagiForMe

Vlad This would be awesome! I would love it! Any plans to implement this soonish? 🙈

mlissner

Hi all, I'm the director of Free Law Project, a non-profit that works to make legal data more broadly accepted. We host CourtListener.com, which has several free legal search engines (case law, filings, judges, oral arguments, etc.).

I came here because I'd love to get bangs set up for our website in Kagi so that Kagi users can easily query it. I can imagine queries like:

!cases, !legal, !law — for searching case law
!recap, !pacer — for searching our collection of filings (called the "RECAP Archive")
!judges — for searching judges
!oa — for searching oral arguments

But as I write that, I recognize the need for some sort of guidelines. I know a lot of people would use the heck out of !law, and !recap, but I question how many people would use a bang for our oral argument search engine (it's not that useful!). I also imagine that a system this would soon get a request for the !porn bang, or who knows what else.

So I think I have two questions:

How do we decide if a site merits a built-in bang? Traffic? Notability? Purpose? Maybe this doesn't have to really be defined, because in practice a benevolent dictator will be fine, but I thought I'd raise this issue.

As a site owner, I'd like to have clear guidelines though. DDG does not provide a guideline, so the decision (or lack thereof) is opaque — a bummer.
Is there a way the community can help make this feature a reality? I guess half of this project is setting up the repo with the two files described above, but the harder "half" is using those files inside Kagi itself. Is there a way to get this started?

Thanks for any thoughts people have!

Vlad

mlissner We could do this relatively quickly, the main thing missing are sensible guidelines. We do not want to be in the business of moderating bangs.

I would be careful about allowing english word bangs like !law as that immediately begs the questions who gets to 'own' this and why.

Most bangs are shortcuts of their website name. For example !yt is shortcut for youtube.

!cl could be therefore appropriate default bang for courtlistener. The user is free to rename it via custom bang to !law if they so wish.

Is there a way to get this started?

Coming up with a set of guidelines would be a great starting point. After that it would take us less than a week to have the full system in operation as suggested above.

leftium

mlissner I came here because I'd love to get bangs set up for our website in Kagi so that Kagi users can easily query it. I can imagine queries like:

!cases, !legal, !law — for searching case law
!recap, !pacer — for searching our collection of filings (called the "RECAP Archive")
!judges — for searching judges
!oa — for searching oral arguments

Can you give me the search URL's that would be associated with each bang trigger above?

I will create a custom branch of bangtastic! that supports these triggers and can be hosted publicly. (As a fully functional POC.)

The service can be hosted on and accessed from your own domain: CourtListener.com
Even non-Kagi users can use it (although that is already true for all non-Kagi bangs)

I think open-source bangs means bangs can be executed on any site, not just a few limited sites like Kagi, DDG, Brave, etc

Each site can determine themselves which bangs they support, and how they are executed.
Each site can also choose to support the bangs supported by other sites via scoped bangs:
- @kagi.com!images summons the Kagi bang for images; @duck.com!images would summon DuckDuckGo's version.
- Likewise, @courtlistener.com!cases would summon the bang you suggested.
- On a domain, the default scope is the domain itself.
- To support custom user bangs, their could also be user-level scopes like @mlissner:kagi.com!law, which could be shortened to @mlissner!law on Kagi.com, since that is the default domain. (And logged in with your own account, you can also omit your own username: just !law)

After proposing scoped bangs, I realized the competition is just moved to desirable scope names (albeit at a smaller scale). Domain Names as Handles in Bluesky is a nice solution to this:

Kagi.com already controls/manages both the domain name and bangs on their own domain. (Ditto for duck.com, brave.com)
The bang data can be stored in a standard location on each domain, like favicons, robots.txt, etc. (Perhaps and/or an API since there can be lots of bangs?) The scope tells you exactly where to retrieve the scope's data from.

The exact bang scope syntax isn't final (like lots of sites use email addresses as user id's, which include @ and . characters), but I think the idea of scoped bangs is pretty solid.

Bangtastic! is a personal project I'm developing for myself. I don't need the scoped bang functionality so I will probably not develop it myself. But I'm throwing the idea out there!

KagiForMe

@Vlad
Guideline Proposal:

Everyone can propose a bang for a website. Not just the website owner.
The website has to have some kind of search feature; else people will just use it as a bookmark, which is not the use case.
No (proper, i.e. !in is fine) English words are allowed, no country code bang should be overwritten i.e. !fr.
First come first served, except where (Kagi?) traffic is substantially higher. Users can still change the bang manually.
Kagi will either (a) not accept NSFW bangs i.e. !porn or (b) will only make them work with Safe Search disabled. This would be cool, but a bit more effort from Kagi's side.
Kagi is not expected to fix broken bangs. Broken bangs get removed, users can add a PR that fixes any bugs.
There is a GitHub template that makes people list the link, the expected users, the name of the bang, NSFW status etc in orderly fashion.
...

Do the bangs need to have an expected amount of users?

leftium

I'm more interested in using this bang repository in my own project more than in governing how bangs are added/updated (on Kagi), but I'll share my thoughts:

Right now, I am using bang data from DuckDuckGo for my own project (https://webapps.stackexchange.com/a/162253/1530). I understand it's a little old and things may be broken, but it's better than nothing. I have already made great progress in analyzing all the bangs, merging duplicates, etc.

I heard Kagi had fixed a lot of bangs and modified them to work with Kagi (vs DuckDuckgo). If I had access to these bangs, it would save me a lot of effort since I also want my project to work with Kagi. (My project provides a bang search interface like https://duckduckgo.com/bangs or https://search.brave.com/bangs. But better ^__^. It actually fulfills a lot of open suggestions in this feedback forum.)

In turn, Kagi might find the work I've done massaging DuckDuckGo's bang.js data file useful for their own purposes. And they might even be able to use my project for their own kagi.com/bangs page.

So my opinion is a little biased, but I say just release the data. What's the worst that can happen?

Maybe a flood of github issues requesting new bangs and fixes to existing ones
But Kagi is not obligated to apply changes to the open-source repository to their own search engine
Kagi is not even obligated to respond to any of the issues
Maybe the worst that can happen is Kagi kills any momentum in growing a community to support this bang repository.
But I think waiting for guidelines kills just as much momentum.

On the other hand:

I think the guidelines are overengineering for problems that likely won't happen. (There won't be a flood of github issues asking for new bangs.)
If people do submit issues/pull requests on github, it will give a better idea of what type of guidelines are actually needed. Solve problems that actually exist.
Community members other than Kagi might step up and start maintaining the bang repository. No requirement for Kagi to do it themselves.

It seems one of the main issues the guidelines are supposed to address is akin to problems with domain names like squatting and determining who gets the "valuable" ones. I don't see this as a real problem (until the non-techy majority starts using bangs to navigate the web like domain names.) In addition, Kagi (and my project) allow over-riding bang triggers if you don't like Kagi's default bangs.

Here is a suggestion Kagi can use for managing bangs on their site:

Allow anyone to register a bang that doesn't exist yet. (Like NPM modules)
The bang can be used right away, but it is "hidden" until it passes moderation.
Moderation can be done manually by an admin, or a voting system like this feedback forum.
If the bang is found not useful/against the rules/etc the bang is immediately retired.
After passing moderation, the bang can be found via autocomplete/search/etc.
If bang usage dips below a certain threshold, it is automatically retired so someone else may register a new URL.
The bang creator can retire their own bangs. Updates are more questionable...
Perhaps there is some moderation/voting mechanism to update/delete existing bangs.

leftium

Oh, another issue I thought of: aren't bangs technically DuckDuckGo's intellectual property?

So are there any copyright/patent issues?

I don't think DuckDuckGo plans on litigating other people/companies' use of bangs, but it's something to consider.

Perhaps involving DuckDuckGo would mitigate any risk~

KagiForMe

leftium involving DuckDuckGo

🍿

leftium

Here are some example guidelines from open source projects that take and manage similar contributions from users:

Since Kagi has a feature to author/manage custom bangs, that would be a good place for users to suggest new bangs:

User creates custom bang (https://help.kagi.com/kagi/features/bangs.html#custom-bangs)
User toggles new "public" bang setting to publish to a Kagi bang directory
Other users can search public bangs and add them to their own custom bangs. Kagi users adding the custom bang (and regularly using it) becomes the voting mechanism.
If a significant number of users add a certain public bang, Kagi may consider adding it to their default set.

leftium

Bang triggers are similar to npm package names, so their guidelines might help: https://docs.npmjs.com/package-name-guidelines

Interestingly, npm has the concept of scoped packages: https://docs.npmjs.com/package-name-guidelines

Scopes help fix the problem of "all the good, short names (bang triggers) are already taken."
Unscoped bangs could be determined by usage rate of scoped bangs. If a scoped bang @leftium!gg is very popular and used often while the unscoped bang !gg is rarely used, the scoped bang could be ((semi?) automatically) promoted to an unscoped bang.

Scoped bang triggers might be a useful feature:

Scoped bangs are prefixed like @kagi!images @ddg!images @leftium!images
- Postfixed scopes might be useful, too: !images@kagi
- Or standalone to say all following bangs have this scope: @kagi !images (if/when multiple bangs are supported)
If you overrode kagi's default !images bang with your own custom bang, @kagi!images would allow you to occasionally use the default bang without touching your settings
@username like @leftium would allow for autocomplete of your custom Kagi bangs.
- Perhaps @!images without a scope name defaults to your own user scope.
- It could also allow simple sharing of public custom bangs with others
@category!bang like @entertainment!... could limit autocomplete to bangs from certain categories (as categorized by duckduckgo)
Now you can choose which service to use (I found over 169 bangs that could be serviced by multiple providers:
- mostly the site: search operator...

leftium

I was interested in an open-source version of Kagi's bang data so I could reference any fixes/enhancements Kagi made.

However, my request was blocked by this ticket... Since Kagi won't open up their data (any time soon), I decided to open up my data so maybe Kagi (and others) can use my project as a starting point for discussion.

I went ahead with my project, referencing the only data I have: DuckDuckGo's bang.js data.

I have done some light analysis and merged/cleaned up entries.
I am also designing an extension to the format
Hopefully it won't be too difficult to retrofit Kagi's data/extensions into my project's data if/when they finally release anything.

Eventually, Bangtastic will be an alternate search box for initiating bang searches. Both Kagi and DuckDuckGo will be supported. It should even work from your browser's address bar! Very early prototype here: https://bangtastic.vercel.app/

Easier to enter ! on a mobile device without easy access to the ! key
Try (double tapping) the space key (when the search box is empty).

Bangtastic currently supports:

Searching/browsing bangtastic data: https://bangtastic.vercel.app/bangs
Browsing bang.js source data: https://bangtastic.vercel.app/bangs/sources

Vlad

leftium

Since Kagi won't open up their data

This is not what is going on here. This has not even reached a decision making point (becuse we still lack clear guidelines). WE are more leaning towards doing this then not, but it will not get done on its own. Reality is that this is just one of hundreds open issues on this forum that we have to consider at any given point in time.

leftium

Vlad

I've had the feeling this ticket is and will forever be stuck in unnecessary "analysis paralysis." (I predict this ticket will remain unimplemented one year; five years from now). I think I finally figured out a good explanation:

The main problem that started this thread is:

Many bangs are broken and should be updated on Kagi.com (partially solved by custom bangs)
People want to add new bangs on Kagi.com
- For personal use (solved by custom bangs)
- For "exposure" to others like a domain name

The current way to request updates/additions to Kagi's global bangs is this feedback forum, but that has been deemed inadequate. An open-source repository of bangs has been suggested as the better method.

Why is this feedback forum inadequate?
How will an open source repository of bangs fix these inadequacies?

I argue an open source repository of bangs has no advantage over this feedback forum. The repository would probably be like a github repo. So there are two ways to contribute:

Clone the repo and submit a pull request.
- This will exceed the technical skill level of most users
- These steps consume far more time than a simple forum post
- How can can the pull request be confirmed to work properly?
Submit an issue. This is about the same as posting to this forum.
- However, now you have two places where bang requests can be sent: this forum or the github repo.
- Kagi will end up doing most, if not all of the fixing due to the challenges of cloning and submitting a PR.

I think Kagi's custom bang interface is a much better way to propose new bangs and updates to existing bangs.

Requires relatively little technical skill
Can be tested by the user themselves
All that is required is adding a method to share custom bangs.
Bangs may be proposed either via this forum or in the custom bang UI.

Guidelines (for public bangs available on Kagi.com) is a separate issue.

The lack of guidelines is (already) an issue whether there is an open source bang repository or not.
Users are free to suggest new bangs on this feedback forum, right?
Kagi has already decided which bangs will be available on their site, and will presumably always have final say on future bangs. So Kagi should just document their implicit guidelines.
The guidelines can be "we will not accept new bangs, only updates to existing bangs" if that is the current policy. This guideline can be changed in the future, if/when Kagi is ready.

These thoughts have helped me realize Bangtastic will become the truly open-source bangs. So please help the development of Bangtastic!

https://bangtastic.vercel.app/
https://github.com/Leftium/bangtastic
Currently simply forwards requests to Kagi.com
In the (near) future, will execute bangs without Kagi.com (so custom bangs can be updated/added)
Open-source here means anyone can fork the project and publish their own site that executes custom bang searches.

Vlad

This is live now, thanks everyone for feedback!

https://github.com/kagisearch/bangs

leftium

Vlad

Wow, I didn't think this would happen any time soon, but I'm glad to be proven wrong! Thank you~

I will consider including the fmt field as I design the next generation of bangs ^^

Thibaultmol

Vlad consider adding the link to the repo in the 'answer' of this thread as well

Vlad

leftium Underpromise/overdeliver.

Please use repo to post issues related to it.

« Previous Page