AI answers may improve traditional search

Isaac Asimov, the writer of science and science fiction, described his experience with publishing houses as a writer. People had warned him to stay away from the publishing world, telling him that it was full of unscrupulous opportunists who would take advantage of him. Yet his experience was a good one; the publishers, editors, and others he worked with were (for the most part) honest, hard-working, and ethical.

Asimov had a conjecture about this. He surmised that for some time prior to his arrival as a writer, the publishing industry did have a large number of unscrupulous opportunists, and they gave the industry a bad reputation. He further theorized that when he started as an author, those individuals had moved on to a different industry. Not because of his arrival, but because there was a newer, larger, and more lucrative industry to take advantage of individuals. It was the movie industry that provided a better "home" for those individuals. Once they saw that movies were the richer target, they abandoned the publishing industry, and left the ethical people (who really wanted to work in publishing) behind.

I don't recall that Asimov proved his conjecture, but it has a good feel to it.

What does this have to do with software? Well, not much for the programming world, but maybe a lot for the online search world.

Search engines (Google, Bing, Duck-duck-go, and others) make a valiant attempt to provide good results, but web sites use tricks to raise a web site's ranking in the search engines. The result is that today, in 2023, many searches work poorly. Searches to purchase something work fairly well, and some searches for answers (when does the Superbowl start) tend to be relevant, but many queries return results that are not helpful.

As I see it, web site operators, in their efforts to increase sales, have hired specialists to optimize their ranking in search engines, leading to an endless race of constantly outdoing their competition. The result is that search engines provide little in the way of "organic" lists and too many "sponsored" or "optimized" responses.

The situation with search engines is, perhaps, similar to the pre-Asimov era of publishing: full of bad operators that distort the product.

So what happens with the new AI-driven answer engines?

If people switch from the old search engines to the new answer engines, we can assume that the money will follow. That is, the answer engines will be popular, and lead to lots of ad revenue. When the revenue shifts from search engines to answer engines, the optimizations will also shift to answer engines. Which means that the efforts to game search engines will stop, and search engines can drift back to organic results.

This change occurs only if the majority of users switch to the answer engines. If a sizable number of people stay on the older search engines, then the gains from optimizing results will remain, and the optimization games will continue.

I'm hoping that most people do switch to the new answer engines, and a small number of people -- just enough for search engines to remain in business -- keep using the older engines.

A new old web

One idea I have been pondering is a retro version of the world wide web. This "new old web" would be a world-wide web as it originally existed: a collection of static web pages with links to other web pages, either on the same site or on other sites.

The defining aspects of this new old web are what it doesn't have: HTTPS, certificates, cookies, and JavaScript. It would be a simpler version, and an unsecure version, of today's web.

Why do this? Why re-create the old web, one that does not have HTTPS and therefore security?

In a word, permanence.

The current web all but requires HTTPS, which in turn requires security certificates, which in turn expire and therefore require replacements. All of that means that a web site needs maintenance, every 12 months or whenever the certificates expire.

What I am considering is a web that lets one set up a web server and leave it running with no maintenance. Perhaps one could apply updates to the operating system and occasionally blow dust out of the box, but that's it. No annual dance for certificates. Maybe one does not even update the operating system.

Why do this? Mostly as a thought experiment, to see where it leads us. So let's start with these conditions and see where we go. 

This new old web could have web sites that exist for years, or even decades.

Of course, without certificates, one cannot support HTTPS.

Without HTTPS, one cannot transact business. No banking, no credit card statements, and no purchases.

Without HTTPS, one cannot securely log in to a web site, so no personalized web sites. No Facebook, no Twitter, no e-mail.

Such a web would need a new web browser. Current web browsers dislike HTTP connections, and warn that the page is insecure. (We may be a few years away from requiring HTTPS for all links and URLs.) So with current web browsers deprecating HTTP, perhaps we need a new HTTP-only browser.

A new HTTP-only browser would request and load pages over HTTP connections. It would not request an HTTPS connection. A link to HTTPS would be considered an ill-formed link and not valid.

If I'm building a new browser, I can make other changes.

I banish cookies. This prevents third-party cookies and tracking. Overall, this is an enhancement to privacy.

Scripts are also forbidden. No JavaScript or any scripts of any type. The HTML <script> tag must render as text. This eliminates the threat of cross-site scripting.

Web pages may contain text, HTML, and CSS.

One could use PHP, JSP, ASP or ASPX on the server side to render web pages, although the possible uses may be limited. (Remember, no logins and no user IDs.)

It seems that such a web would be mostly static web pages, serving documents and images. I suppose one could serve videos. One could, of course, link from one page to the next.

My idea is not to replace the existing web. The existing web, while it started as this earlier, static web, has evolved into a different thing, one that is quite useful.

My idea is to create a second web, one that exists in parallel to the current web. I would like to try it, just to see what people would do with it. Instead of a web with private areas for private data (e-mail, Facebook, online banking, music that has been purchased, etc.) we would have a web where everything is available to everyone.

How would we act in such a space? What would we create?

That is what I have been pondering.