"Of course you realize, this means war!" -- Bugs Bunny -- "Bully for Bugs" (1953)

Greetings. Believe it or not, when I discussed the ongoing problems of the newspaper industry yesterday in From Hell It Came: Will "Free" Destroy the Internet As We Know It?, I didn't realize that the Newspaper Association of America (NAA) meeting was being held in San Diego this week, or that the Associated Press board would choose their meeting at that venue to declare war against Internet news aggregators.

Some of the language being bandied about is quite remarkable. AP's chairman, apparently channeling character Howard Beale's famous monologue from the film Network (1976), was quoted as exclaiming to applause that "We are mad as hell and we are not going to take it any more!" Notably, AP also took this occasion to announce some price breaks for their media clients, who have reportedly sometimes been using similar language to explain their recent cancellations of AP contracts.

Robert Thomson of the Wall Street Journal was quoted in an Australian newspaper as saying that, "There is no doubt that certain Web sites are best described as parasites or tech tapeworms in the intestines of the Internet."

"Madness!" "Parasites!" See, I wasn't kidding around with my horror movie analogy yesterday.

The battle preparations apparently go beyond the usual threats of lawsuits, to include talk of diverting users of news aggregation sites away from their selected stories to special "landing pages" instead (no word on how this might be accomplished, though "referrer"-based diversions seem like one likely technique).

While Google was not specifically named in most of the pronouncements as The Enemy per se, it's clear to all observers that Google -- and their "Google News" service -- are at the top of the pack when it comes to the designated villain list.

Coincidentally (?), Google CEO Eric Schmidt is scheduled to deliver the keynote tomorrow at the NAA meeting. (Free advice -- check for improperly grounded microphones at the podium before handling them!)

Seriously, these controversies are indeed Big Issues, but to my mind remain essentially orthogonal to the situations that I was describing yesterday regarding the "free" vs. "paid" Internet content dilemma. Search indexing is not the same as being a content provider, given any normal usage of these terms -- and treating Search like conventional content could be devastating to people's legitimate access to information.

That Google and others derive economic benefit from Search is a given, obviously. But the logical result of a major move toward a "pay to index" methodology on the Internet would likely not only be a massive distortion of how Web pages are indexed, but also the triggering of a major and harmful skew to the basic nature of search engines themselves.

It's easy to see why. As things stand right now, any Web site, including at any newspaper, can use the "robots.txt" exclusion method to control how any well-behaved indexing robot will crawl (or not crawl) their site. Any site that wants to keep Google or other search engines out completely, or even limit them individually or collectively to indexing specific pages, can do so.

But if the Internet paradigm shifts to a "pay me or you can't index my site" structure, we've taken an already complicated situation and stirred it up
into a maelstrom.

Search engines would be faced with making economically-based value judgments regarding which sites could affordably be included in their index for users to query. "Those sites are garbage, but they're free -- so they get in." "These sites have the quality info, but they're too expensive this year -- they get left out."

Meanwhile, many Web sites themselves might be tempted to try game the search system in various new ways -- perhaps by making "teaser" materials free to index and then charging the big bucks for anyone wanting to index the real meat --
Web site indexing brought down to the level of late night infomercials.

Immediately lost in a regime where many search engine index crawling decisions move to a "pay indexing" basis would be the trust of search engine users. They could never really be sure if search results actually represented an honest reality, or if those results had been hopelessly limited, distorted, and skewed by the complexities and charge-based bottom lines of the associated indexing permission fee structures imposed by Web sites onto search engines.

I have a great deal of admiration for Associated Press. Their reporters and other personnel are top notch in the truest sense, and in my opinion represent the quintessential essence of quality journalism.

I also fully support the right of any site to control if and where it is indexed, and by whom. That's what robots.txt is for.

But if we open up the Pandora's box of what would effectively be "indexing charges" and associated fee-based indexing limitations, I believe that we'd be going down a path that would be extremely damaging to Internet users, and we risk ultimately decimating the quantity and quality of important information available to us all.

--Lauren--