[{"data":1,"prerenderedAt":377},["ShallowReactive",2],{"blog-gptbot-log-analysis":3,"blog-prev-gptbot-log-analysis":208,"blog-next-gptbot-log-analysis":304},{"id":4,"title":5,"author":6,"body":7,"canonical":193,"category":194,"cover":193,"date":195,"description":196,"draft":197,"extension":198,"keywords":193,"meta":199,"navigation":200,"noindex":197,"ogImage":193,"path":201,"seo":202,"seoDescription":193,"seoTitle":193,"stem":203,"tags":204,"updated":193,"__hash__":207},"blog\u002Fblog\u002Fgptbot-log-analysis.md","服务器日志里的真 GPTBot：四个识别特征和一次全量统计","元析智能",{"type":8,"value":9,"toc":183},"minimark",[10,14,18,30,36,42,48,52,55,58,61,150,153,156,159,162,165,168,171,174,177,180],[11,12,13],"p",{},"光看 user-agent 判断 AI 爬虫，会有大概两成是假的。2025 年 11 月整月，我们统计了四个客户站的日志，AI 爬虫请求合计 4.2 万次，其中自称 GPTBot 的 1.9 万次，核对 IP 之后只剩 1.1 万次是真的。",[15,16,17],"h2",{"id":17},"四个识别特征",[11,19,20,24,25,29],{},[21,22,23],"strong",{},"一、user-agent 字符串。"," GPTBot 的完整串是 ",[26,27,28],"code",{},"Mozilla\u002F5.0 AppleWebKit\u002F537.36 (KHTML, like Gecko); compatible; GPTBot\u002F1.2; +https:\u002F\u002Fopenai.com\u002Fgptbot","。注意后面那个说明 URL，正规爬虫都会带。",[11,31,32,35],{},[21,33,34],{},"二、来源 IP。"," OpenAI 公开了爬虫的 IP 段，维护在 openai.com\u002Fgptbot.json 这类文件里。这一段是关键，UA 能伪造，IP 段伪造不了。",[11,37,38,41],{},[21,39,40],{},"三、请求路径分布。"," 真爬虫的路径分布很散，会顺着内链爬。假爬虫常常只打首页、sitemap 和几个热门 URL，因为它在扫。",[11,43,44,47],{},[21,45,46],{},"四、时间分布。"," 真爬虫按队列来，节奏均匀，凌晨也有。假爬虫扎堆在某个小时。",[15,49,51],{"id":50},"怎么核对-ip","怎么核对 IP",[11,53,54],{},"两步。先把日志里的 IP 提出来做反向解析，看 PTR 记录是否落在官方域名下；再拿 IP 去官方公布的段里做匹配。两步都过才算真。",[11,56,57],{},"我们一开始偷懒只做了 UA 过滤，得出\"GPTBot 抓了 1.9 万次\"的结论，还挺高兴。后来做 IP 核对，发现 8000 次的来源是几个云主机 IP，UA 一模一样，路径全在首页。那批是有人在批量采集。",[15,59,60],{"id":60},"一个月的数据",[62,63,64,80],"table",{},[65,66,67],"thead",{},[68,69,70,74,77],"tr",{},[71,72,73],"th",{},"爬虫",[71,75,76],{},"声称次数",[71,78,79],{},"核实后",[81,82,83,95,106,117,128,139],"tbody",{},[68,84,85,89,92],{},[86,87,88],"td",{},"GPTBot",[86,90,91],{},"19000",[86,93,94],{},"11000",[68,96,97,100,103],{},[86,98,99],{},"OAI-SearchBot",[86,101,102],{},"4100",[86,104,105],{},"3400",[68,107,108,111,114],{},[86,109,110],{},"ChatGPT-User",[86,112,113],{},"1200",[86,115,116],{},"900",[68,118,119,122,125],{},[86,120,121],{},"PerplexityBot",[86,123,124],{},"2300",[86,126,127],{},"1900",[68,129,130,133,136],{},[86,131,132],{},"ClaudeBot",[86,134,135],{},"3200",[86,137,138],{},"2800",[68,140,141,144,147],{},[86,142,143],{},"Bytespider",[86,145,146],{},"17000",[86,148,149],{},"16000",[11,151,152],{},"Bytespider 是字节的爬虫，量大是正常的。OAI-SearchBot 值得单独说：它负责给 ChatGPT 搜索抓内容，抓得比 GPTBot 频繁，路径也更集中在对比页和文档页。",[15,154,155],{"id":155},"这个统计能干什么",[11,157,158],{},"第一，判断你的文档有没有人在读。如果一个站 GPTBot 三个月没来，不是它屏蔽你，是你没内链、没 sitemap、或者全站都在 robots 里挡着。",[11,160,161],{},"第二，看它抓了什么。OAI-SearchBot 如果反复抓你的定价页和对比页，说明你的对比类内容进了它的候选池，这时候把这两页做得更可引用，收益最直接。",[11,163,164],{},"第三，发现抓取异常。有个客户站 11 月中旬 GPTBot 访问量突然掉 90%，查出来是他们上了 CDN 的防爬规则，把 AI 爬虫一起挡了。这种事故不看日志永远发现不了。",[15,166,167],{"id":167},"怎么把这件事做成常态",[11,169,170],{},"我们现在给客户的建议是每月跑一次，写成一个固定脚本，输出三行字：本月 AI 爬虫总请求、核实后占比、抓取最多的五个路径。三行字放在月报第一页，比任何流量曲线都直观。",[11,172,173],{},"有个客户靠这三行字发现了另一件事：他们的文档站被 OAI-SearchBot 抓得很勤，但博客基本没人碰。原因是博客全是\"行业趋势\"这类内容，没有能被直接搬走的答案。他们后来把博客砍掉一半，人挪去做文档，两个月后文档的引用进了 AI 回答。",[11,175,176],{},"日志不会告诉你该写什么，但它会告诉你什么没被读。后者更有用。",[15,178,179],{"id":179},"一句提醒",[11,181,182],{},"别为了好看去改 UA 统计口径。我们见过代理商把 Bytespider 全算成\"AI 搜索流量\"，数字翻了三倍。跟老板汇报前，先问一句这批 IP 核实过没有。",{"title":184,"searchDepth":185,"depth":185,"links":186},"",2,[187,188,189,190,191,192],{"id":17,"depth":185,"text":17},{"id":50,"depth":185,"text":51},{"id":60,"depth":185,"text":60},{"id":155,"depth":185,"text":155},{"id":167,"depth":185,"text":167},{"id":179,"depth":185,"text":179},null,"技术SEO","2025-11-20","2025 年 11 月一个月，四个站共有 4.2 万次 AI 爬虫请求，冒充 GPTBot 的占了 9000 次。四个特征能分辨真假，光看 user-agent 会被骗。",false,"md",{},true,"\u002Fblog\u002Fgptbot-log-analysis",{"title":5,"description":196},"blog\u002Fgptbot-log-analysis",[88,205,206,194],"AI爬虫","日志分析","Gdg0-E4uIHGlku3NNATeUfLNo0XBKOhBEq2iwm1_Uck",{"id":209,"title":210,"author":6,"body":211,"canonical":193,"category":292,"cover":193,"date":293,"description":294,"draft":197,"extension":198,"keywords":193,"meta":295,"navigation":200,"noindex":197,"ogImage":193,"path":296,"seo":297,"seoDescription":193,"seoTitle":193,"stem":298,"tags":299,"updated":193,"__hash__":303},"blog\u002Fblog\u002Fkeyword-library-four-tables.md","把两万个关键词压成四张表，我们的词库分层法",{"type":8,"value":212,"toc":283},[213,216,219,222,225,228,231,234,238,241,244,247,250,253,256,259,262,265,268,271,274,277,280],[11,214,215],{},"词库建得越大，越没人用。我们的做法是反过来的：两万一千个词进来，最后只留四张表，加起来不到 1200 个词。四张表各有用途，每周五更新一次，谁都能打开看懂。",[15,217,218],{"id":218},"四张表分别放什么",[11,220,221],{},"第一张是核心词，10 到 20 个，决定首页、导航和产品页的定位。第二张是页面词，每个词绑定一个确定的 URL。第三张是长尾池，不绑页面，按主题挂着。第四张是排除词，写明为什么不写。",[11,223,224],{},"四张表之外，别的词都进回收站。需要时再翻，不占位置。",[15,226,227],{"id":227},"核心词这张表最重要",[11,229,230],{},"核心词是客户会用来描述你这个品类的词，通常不超过 20 个。它决定首页标题、导航结构，也决定你未来两年跟谁竞争。",[11,232,233],{},"这张表我们半年才动一次。动一次要重新看竞品和客户说法，不是加一个词那么简单。",[15,235,237],{"id":236},"页面词要写到-url-一级","页面词要写到 URL 一级",[11,239,240],{},"第二张表每行必须有一个 URL，没有 URL 的词打回去。这张表是给排期用的，写稿的人拿到行就能开工。",[11,242,243],{},"我们 2025 年 10 月统计过，这张表里 68% 的词已经有对应页面，剩下 32% 就是下个季度的选题来源。",[15,245,246],{"id":246},"长尾池只记三个字段",[11,248,249],{},"词、意图、归属主题。不记搜索量，不记难度分。长尾池会持续变大，字段一多就没人维护。",[11,251,252],{},"每周五我们从搜索后台导一次新查询，人工过一遍，能用的扔进池子，不能用的进排除词表。",[15,254,255],{"id":255},"排除词表是被低估的一张",[11,257,258],{},"排除词表写清楚“免费、破解、下载、招聘、外包”这一类，也写下“这个词跟我们的产品无关”“这个词搜的人不会买”。",[11,260,261],{},"新同事接手时不看这张表，会把去年删掉的词重新挖回来。我们遇到过两回。",[15,263,264],{"id":264},"我们走过的弯路",[11,266,267],{},"2025 年上半年，我们把词库按 20 个行业分成 20 张表，做得很规整。结果三个月后没人打开，因为找不到该看哪张。",[11,269,270],{},"下半年改成四张表，第一周就有人主动用了。分类不是越细越好，是让人一眼知道去哪张表里找东西。",[15,272,273],{"id":273},"四张表之间怎么流动",[11,275,276],{},"新词默认进长尾池，被证明有承接价值才升进页面词表，被证明没人搜才降进排除词表。每个月做一次升降，只动 20 到 30 个。",[11,278,279],{},"这个流动过程本身就是选题会。词往上走，说明有活儿要干；词往下走，说明以前判断错了。",[11,281,282],{},"词库不是资料，是排期表的上游。上游乱，下游一定乱。",{"title":184,"searchDepth":185,"depth":185,"links":284},[285,286,287,288,289,290,291],{"id":218,"depth":185,"text":218},{"id":227,"depth":185,"text":227},{"id":236,"depth":185,"text":237},{"id":246,"depth":185,"text":246},{"id":255,"depth":185,"text":255},{"id":264,"depth":185,"text":264},{"id":273,"depth":185,"text":273},"SEO","2025-11-19","词库的价值在删不在存。两万一千个词经过三层过滤，最后只留四张表：核心词、页面词、长尾池、排除词。每张表有明确用途和负责人，每周五更新一次。",{},"\u002Fblog\u002Fkeyword-library-four-tables",{"title":210,"description":294},"blog\u002Fkeyword-library-four-tables",[300,301,302],"词库搭建","关键词研究","内容排期","0X3kYHpm_KvMf_jzr50-tPVqtF9vZm2Trw36Yh0kMNk",{"id":305,"title":306,"author":6,"body":307,"canonical":193,"category":365,"cover":193,"date":366,"description":367,"draft":197,"extension":198,"keywords":193,"meta":368,"navigation":200,"noindex":197,"ogImage":193,"path":369,"seo":370,"seoDescription":193,"seoTitle":193,"stem":371,"tags":372,"updated":193,"__hash__":376},"blog\u002Fblog\u002Fpaid-industry-award-worth-it.md","花三万块买行业榜单排名，我们为什么劝客户放弃",{"type":8,"value":308,"toc":360},[309,312,315,321,327,333,336,339,342,345,348,351,354,357],[11,310,311],{},"大部分\"行业榜单\"卖的是 logo 位，不是背书。去年 11 月，一个做客服系统的客户收到报价，三万块保进某榜单前五，还送一个奖杯和一篇通稿。我们看了榜单页面，劝他别买。四个月后，同预算花在案例内容和作者页上，品牌词月搜索从 130 涨到 250。",[15,313,314],{"id":314},"榜单的三种形态",[11,316,317,320],{},[21,318,319],{},"媒体评选类。"," 由编辑评审，免费或只收报名费，标准公开，去年的获奖页面还在。这种有真实价值，因为读者知道入选有门槛。",[11,322,323,326],{},[21,324,325],{},"商业榜单类。"," 付费保排名，报名费和名次挂钩。这类榜单的问题不在贵，在于读者也知道它是买的。你挂在官网上，懂行的人一眼看穿，反而扣分。",[11,328,329,332],{},[21,330,331],{},"平台方发布的方案榜。"," 云厂商、应用市场、渠道商发布的入选名单。这类榜单的价值不是外链，是渠道和信任，往往还带实际的客户来源。我们看这类榜单时，会顺便查一下榜上有没有我们的直接竞争对手，如果有，说明这个平台确实在认真选厂商，值得争取。",[11,334,335],{},"判断的时候别只看报价单上的头衔，去看发榜方的平时在做什么。一个每年只发一次榜、其余时间没有任何内容的机构，和一个月更十篇行业分析的媒体，可信度完全不同。",[15,337,338],{"id":338},"我们为什么不买",[11,340,341],{},"第一，页面留不住。榜单每年重做，旧页面 404 是常态。2023 年这个客户买过一次，第二年榜单改版，页面直接没了，当初说的\"长期外链\"一条不剩。",[11,343,344],{},"第二，页面本身没有搜索流量。我们查了那个榜单页的关键词和预估流量，月访问不到 400，还集中在榜单品牌词上，不是你的产品词。",[11,346,347],{},"第三，三万块有更好的去处。同一个客户，我们把钱拆成三份：一篇带数据的客户案例、两个作者页的履历补充、一次行业媒体的深度投稿。四个月后，品牌词搜索翻倍，案例页带来 7 个询盘。",[15,349,350],{"id":350},"怎么判断一个榜单值不值",[11,352,353],{},"问三个问题就够了。榜单页自己有没有自然搜索流量？评选标准是否公开、评委是谁？去年的获奖页面现在还能打开吗？",[11,355,356],{},"三个都过，报名费在 2000 以内，可以试。只要有一项不过，尤其\"保排名\"这三个字出现，就别买了。",[11,358,359],{},"奖杯摆在前台没人看，客户看的是你的案例和你的搜索结果。这笔钱花在哪，反映的是你信不信搜索。",{"title":184,"searchDepth":185,"depth":185,"links":361},[362,363,364],{"id":314,"depth":185,"text":314},{"id":338,"depth":185,"text":338},{"id":350,"depth":185,"text":350},"行业观察","2025-11-21","去年 11 月一个客户收到某行业榜单的报价，三万块保前五。我们拦下来了，把这三万花在别处，四个月后品牌词月搜索从 130 涨到 250。这篇讲榜单的三种收费形态和判断方法。",{},"\u002Fblog\u002Fpaid-industry-award-worth-it",{"title":306,"description":367},"blog\u002Fpaid-industry-award-worth-it",[373,374,375],"行业榜单","品牌背书","预算决策","p21-4QNColia5TaT5fvHxws3qrUoMVnPBGI_otvyifk",1794088110696]