On the legality of pretending to be "googlebot"?

[George Swan]

A friend recently sent me a link to a site described how to make your browser pretend it is googlebot.

Apparently sites that require registration, and possibly a subscription, will allow google's web-crawler robots through, so their pages can be indexed by google.

While this might seem tempting at first, doesn't US law, and possibly the law in other advanced nations, have such draconian anti-hacking laws that even this relatively trivial setting could be considered a crime?

If this could be considered a crime, what about actively managing the cookies sites leave on your computer?  I presume deleting them is OK.  What about editing the cookies, to make them report something other than what the site that left them wanted to find on subsequent visits? 

[Howard C. Berkowitz]

A friend recently sent me a link to a site described how to make your browser pretend it is googlebot.

Apparently sites that require registration, and possibly a subscription, will allow google's web-crawler robots through, so their pages can be indexed by google.

While this might seem tempting at first, doesn't US law, and possibly the law in other advanced nations, have such draconian anti-hacking laws that even this relatively trivial setting could be considered a crime?

If this could be considered a crime, what about actively managing the cookies sites leave on your computer?  I presume deleting them is OK.  What about editing the cookies, to make them report something other than what the site that left them wanted to find on subsequent visits? 

To the best of my knowledge, and I do follow the fields, none of the items you raise is illegal under U.S. law.

[Tom Morris]

There's no law against me sending an HTTP request to a server with a different user agent than it expects.

[George Swan]

To the best of my knowledge, and I do follow the fields, none of the items you raise is illegal under U.S. law.

But then there are guys like Senator Stevens, of Alaska, who sits on the committee that regulates telecommunication, who called the internet, "a collection of tubes".  I presume he meant vacuum tubes.

It might seem obvious to us that spoofing a site that requires a subscription, by pretending to be a googlebot, is in a whole different class than hacking into Wall Street and stealing a billion dollars.  But, I suggest, this is not obvious to those less familiar with technology, like Senator Stevens.

During the Reagan administration some bright spark came up with the notion that work on encryption technologies should be "born classified".  The same bright sparks decided that work on encryption should be shoehorned into the same laws that prohibited the export of high tech weapons.

There was a lot of discussion over the truly bizarre implicaitons of this law in Peter Neumann's RISKS digest.  Let me repeat my recollection of a story, as it was first published in RISKS.

One correspondent on the digest had been involved in the development of some new cell phones.  Remember this was about twenty years ago.  The cell phones he worked on had a facility to encrypt conversations.  He was planning a trip to Europe, and he wanted to bring a pair of these phones with him, to show some colleagues.

He realized that he could probably just have packed them in his luggage, with no one the wiser.  But he decided, as an exercise, to try to fully respect the existing law, and go through all the right channels, and do all the right paperwork, to bring the phones with him legitimately.

He described doing a lot of work, many phone calls, many letters.  Finally, he thought he had taken all the required steps, but one.  One of his contacts said something like: "There is one final form you have to fill out -- form mumble-mumble-mumble.  But you can ask for that form, and fill it out, at the airport."

So, when he gets to the airport, he tells his story, and explains to a customs official that he needs a "form mumble-mumble-mumble". 

"Form mumble-mumble-mumble?  Could you please come with me sir?"  Well, he follows the official down a couple of deserted hallways, the official opens a door for him, he enters -- and the official locks the door behind him! 

He is locked in this room -- and left to stew for a long time.  I can't remember whether he missed his flight.  As he waits he can't really hear anything that would give him a clue as to how long he would have to wait.  Finally, he can hear, for a minute or so, the footfalls of a guy walking down those deserted hallways.  A tough-looking official enters.  Our hero writes that the guy reminded him of the Sergeant Joe Friday, from the old TV show Dragnet.  The official looks at him, and says, "So -- you are the guy trying to carry a bomb?"

I think he was eventually able to carry the phones with him to Europe.  I can't remember what steps he took to prepare for his return.

[Tom Morris]

Senator Ted Stevens ignorance is his problem, not mine. I'll send any old HTTP headers I want. If you don't like it, reconfigure your server.

[Howard C. Berkowitz]

To the best of my knowledge, and I do follow the fields, none of the items you raise is illegal under U.S. law.

But then there are guys like Senator Stevens, of Alaska, who sits on the committee that regulates telecommunication, who called the internet, "a collection of tubes".  I presume he meant vacuum tubes.

It might seem obvious to us that spoofing a site that requires a subscription, by pretending to be a googlebot, is in a whole different class than hacking into Wall Street and stealing a billion dollars.  But, I suggest, this is not obvious to those less familiar with technology, like Senator Stevens.

During the Reagan administration some bright spark came up with the notion that work on encryption technologies should be "born classified".  The same bright sparks decided that work on encryption should be shoehorned into the same laws that prohibited the export of high tech weapons.

No, it wasn't under Reagan, although the original law there is more ambiguous than the Atomic Energy Act, which specifically assumed nuclear weapons information was "born classified".  While I can't immediately find the citation, there is something earlier that deals with communications intelligence and cryptography; what I can find is the Export Control Act of 1949, which gives the incumbent Administration broad authority to define what may, or may not, be a "munition" and barred from export. Later legislation, the Export Administration Act of 1969, made the authority a bit more clear and required more justification.

Separate from export control is patent law. The U.S. Patent and Trademark Office has the authority, after consultation with agencies with the relevant technical knowledge, to classify a patent application for an indefinite time, and only issue the patent after there is security approval. In many but not all cryptographic cases, the material was created by a government employee (e.g., William Friedman). There is a classified patent application(s) by Teller and Ulam for thermonuclear weapons.

I'd have to check it, but, from memory, there may be some relevant material in (again IIRC) Section 206 of the PATRIOT Act.

Probably outside your query, but the Communications Act of 1934 established the principle of privacy of communications, although in Smith vs. Maryland (1979), the Supreme Court of the U.S. ruled that there is no expectations of privacy for one's calling records.

[George Swan]

Senator Ted Stevens ignorance is his problem, not mine. I'll send any old HTTP headers I want. If you don't like it, reconfigure your server.

Feel free to make yourself a test case.

Would I ever consider making myself a test case?  I wouldn't rule it out.  But I feel a lot less induced to do so than when I was younger

In general, I'd like to make an informed choice, and know when I was at risk of being a test case.

Cheers!