Comments on: Shim Crawler bot

By: Oldes

Oldes — Fri, 27 Oct 2006 09:27:52 +0000

This bot visited me from:
inetnum: 157.82.0.0 – 157.82.255.255
netname: UTNET2
country: JP
descr: University of Tokyo

identified as:
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp)

By: Forchheim

Forchheim — Sat, 21 Oct 2006 19:31:03 +0000

I found another IP of the Shim-Crawler bot: 157.82.254.46.

By: Herschel

Herschel — Tue, 25 Oct 2005 00:31:01 +0000

The IP was 65.57.245.11. Googling it turns up that a ton of hits for that IP, but it’s hard to tell what it is. It doesn’t belong to Google’s address space, it’s assigned to Level 3, but I saw a few pages that, if you add them all up, seem to indicate that it most likely belongs to Google’s corporate proxy that their users go through to get to the net. At least, that seems like the most likely explanation right now.

I fired off an e-mail and we’ll see what happens. I didn’t get an immediate bounce, so it’s probably a legitimate address.

By: Olliver

Olliver — Mon, 24 Oct 2005 18:44:15 +0000

Seems like a strange bot indeed, I’ve never stumbled upon it so far in the logs I have access to. Do you still have its ip address as well? I think if it actually was a genuine Google crawler it should have started its journey from the Google address space, shouldn’t it? In case it’s a fake then at least it’s not that silly as the GoogleBot fake from 209.67.212.18 (h17.plesklogin.net), which totally ignores robots.txt, but that’s a different story