昨天发布了 【社煤挖掘：大数据告诉我们，希拉里选情告急】 ，鉴于大选的临近和选情的瞬息万变，我们决定用我们的社煤挖掘的核武器，每日跟踪大数据选情。 美国大选大数据一日一更新，11/1/2016 前24小时，看FBI事件发酵后的走势最新动态： 嗨 过去 24 小时，克林顿赶上来了也：两人打平，都是 -12%。热议度克林顿更甚，这也难怪，FBI 重启以后，议论焦点从老川转移到老喜身上。看看BPI这图，这一对真是冤家啊，纠缠在一起： 川大叔整个被喜大妈包住了，严严实实，比孙悟空的紧箍圈还厉害。Note：里面的圈是川普，外面的圈是希拉里，貌似希拉里气场如今大过老川了。照这个趋势，克林顿希望蛮好。 昨天晚上看新闻，说虽然 FBI 重启对克林顿选情影响很大，传统的新闻民调 CNN poll 还是希拉里领先五个百分点，其他的民调有曾一度只领先一个百分点的记录。虽然都比以前的领先幅度缩小，但仍然领先。川普阵营批判说这些个民调都是被操纵的，他们那边的民调是川普领先。这些个极小数据的民调极易偏差，公婆各有理，还是 put aside，咱们看真正的大数据：这是川普与希拉里最近24小时的 big data summary 对比 回顾重温一下一周来（10/25-11/1）的走向，作为希拉里选情起伏的背景： 到现在为止的一周平均 net sentiment，Trump 是 2%，Clinton 是 -12%，可见希拉里的反击，主要不是把自己的 social rating 提升了（过去一天还是 -12），而是把对手拉下水了，让川普从周平均的 +2 拉到现在的冰点以下 -12。克林顿用的是什么伎俩赶上来的呢? 朋友说，大招来了：原来 拉川普下水是找到了川普与普京勾搭的新证据啊： A Veteran Spy Has Given the FBI Information Alleging a Russian Operation to Cultivate Donald Trump Donald Trump Used Legally Dubious Method to Avoid Paying Taxes 约: 有点标题党，内容还算靠谱: 希拉里这次要坐牢？ 施: 这次选举是测试大数据有效性的一个试金石，我感觉可能无效.... 另：美帝国主义的人民群众也太不成熟了，一点自己的信念都没有？都受舆情影响，吃瓜群众表示不懂 南: 关键是很多选民都没有被社交媒体覆盖到吧 施: 情绪和投票时间的关系是什么样的？ Nick: 没错。伟哥说这么多没用，就一句话：谁能上。 张: 看样子是川普了，我很好奇这个家伙上来会是什么结果 我: 我这才是实事求是，动态跟踪，全方位大数据信息。“谁能上”那算个啥啊？ 在胶着的选情下，那就是赌命，有没有大数据，都可以一赌，也都有不小的概率猜中，或猜不中，没有半点营养。如果是非胶着状态，大数据预测比其他预测更准。我坚信。 要学那个AI大嘴巴，谁不会？他们根本连技术细节都没有，不过是制造了一个话题，顶了一个AI的帽子，利用普罗和媒体对AI的敬畏 。我的选情追踪和分析，比那个高出不知几个数量级，这还真不是吹的。今天的选情趋势如果能够持续，大选日前没有新的定时炸弹被引爆，我预测克林顿当选的可能性可达80% Nick: @wei 是骡子是马，拉出来溜溜。就一句话：谁赢。 我: 这样吧，大选日前一天，我做个预测，根据一直到那一刻的综合大数据 analytics，现在不行，选情还在变化，并且显然有胶着的迹象。 Xi: @wei ， 别那么保守！ 得老莫者， 得天下！ 肯定是Hillary赢了。。。 Nick: @wei 这算什么本事？ 我: 尼克是星座骗女青年骗惯了，只知道短平快 如何得手，顾不了失手的后果了。 反正我有大数据 有平台 有深度parisng 我就这么每日追踪 不打无准备之仗。 以唐老师的说法，得老墨者得天下，那是克林顿无疑了， 西班牙语舆情那是一面倒，克林顿高高在上，从来没有下来过 。 白: 伟哥这是要把谁能上做成红学的节奏。 最后，谁能上不重要了，为了谁能上而秀肌肉的人互撕。 我: 重在过程 不在结果。 这次大选好 富有戏剧性和悬念, 具有观赏性和互撕性， 跌宕起伏 精彩纷呈 阿: 我开了个盘口 目前二人押川普 四人押希太 欢迎加入 重在结果 不在过程 我: 问一句 为什么希拉里推特说的三点facts 第一条说 fbi 并未重启电邮门调查，只是提议重启。 Nick: @wei 加入盘口，eat your own dog food 我: 第二个 fact 是 fbi director 自己并不清楚新发现的邮件有多少相关 据信很可能是已经审查过的邮件的另一个拷贝。 这个 director 涉嫌扰乱大选，对一个不知结果的新线索 可以按程序重启调查 但在大选前造成舆论 难逃干扰大选的怀疑，他可能也有违法乱纪的麻烦。 【相关】 【社煤挖掘：大数据告诉我们，希拉里选情告急】 CNBC: AI system finds Trump will win the White House and is more popular than Obama in 2008 Trump sucks in social media big data in Spanish Did Trump’s Gettysburg speech enable the support rate to soar as claimed? 【社煤挖掘：川普的葛底斯堡演讲使支持率飙升了吗？】 【社煤挖掘：为什么要选ta而不是ta做总统？】 Big data mining shows clear social rating decline of Trump last month Clinton, 5 years ago. How time flies … 【社媒挖掘：川大叔喜大妈谁长出了总统样？】 【川普和希拉里的幽默竞赛】 【大数据舆情挖掘：希拉里川普最近一个月的形象消长】 欧阳峰： 论保守派该投票克林顿 【立委科普：自动民调】 【关于舆情挖掘】 《朝华午拾》总目录 【关于立委NLP的《关于系列》】 【置顶：立委NLP博文一览】 【 立委NLP频道 】
大数据舆情挖掘，看图说话。 先看近一个月来在社会媒体上的希拉里和川普的品牌形象对比图： 看点三： 1 川普的 buzz 大过 希拉里一倍多，川普是话题中心（圈的大小表明热议度） 2. 普罗对川普比对希拉里，情绪更趋激烈：表现在 Y 轴的 passion intensity 上 3. 两人总体都不讨人喜欢，川普更加让人厌恶，表现在 x 轴上的 Net Sentiment（也就是褒贬对比的度量）。两人都在冰点之下，社会媒体的形象不佳。 如果我们要自动调查过去一个月时间的趋向和形象消长，可以考虑把数据分割为两段或三段来看此消彼长，先一分为二来看图： 看到了吧，过去一个月，随着总统大选辩论和丑闻的揭示和宣传，川普的媒体形象显著恶化，表现在舆情圈圈从右（x轴上的右是评价度高 love like，左边是评价度低 hate dislike）向左的位移。本来评价度clearly比希拉里要好，终于比希拉里差了。同时，希拉里的社会媒体形象有所改善，圈圈在从左向右位移。两个人始终都是冰点以下，吐槽多于赞美，但是就在一个月前，还是喜妈更不受待见： 不是民众更喜欢老川，而是普罗更厌恶喜妈 。 这个品牌对比图示表达了四维信息： 1. net sentiment 评价度 x 轴 2. passion intensity 舆情烈度 y 轴 3. buzz 圈圈的大小，是热议度 4. 一分为二的两个圈是时间的粗线条切割的维度 在二维的图纸上，要表达四维的信息，的确不是很容易。 要是嫌第四维时间太粗线条，咱们一分为三看看： 三个圈，浓度的深浅表达的是时间的远近。当短短的一个月的时间，被一分为三的时候，我们看到了什么趋向呢？请注意颜色的深浅，对应的是时间的远近。我们看到，喜妈的三个圈圈是左下角到右上（还是visualization设计不到家，不同品牌应该用不同的颜色区分才好）。原来喜妈的评价是先好，后坏，最后回到中间。而老川在同一个时间点，是先中，后略好，最后跌入深渊。 以上是利用我们自创的品牌对比图（有美国专利的）来看候选人的形象消长。 社会媒体数据的来源呢？Twitter 为主： 这是一个月来的舆情总结： 的确是大数据了，一个月的随机的社会媒体数据样本里面，两人的 mentions 就有近两亿，眼球数共计高达3万6千亿。川普占7成，喜妈才三成。川普跟冰冰类似，都是话题之王。 总体社会评价，川普零下20%，喜妈零下18%。 下面是有关川普的社煤数据选摘： Bill Clinton disgraced the office with the very behavior you find appalling in Trump. In closing, yes, maybe Trump does suffer from a severe case of CWS. Instead, in this alternate NY Times universe, Trump’s campaign was falling apart. Russian media often praise Trump for his business acumen. This letter is the reason why Trump is so popular Trump won I'm proud of Trump for taking a stand for what's right. Kudos to Trump for speaking THE TRUTH! Trump won I’m glad I’m too tired to write Trump/Putin fuckfic. #trump won Trump is the reason Trump will lose this election. Trump is blamed for inciting violence. Breaking that system was the reason people wanted Trump. I hate Donald Trump for ruining my party. 32201754 Trump is literally blamed by Clinton supporters for being too friendly with Russia. Another heated moment came when Trump delivered an aside in reponse to a Clinton one-liner. @dka_gannongal I think Donald Trump is a hoax created by the Chinese.... Skeptical_Inquirer The drawing makes Trump look too normal. I'm proud of Donald Trump for answering that honestly! Donald grossing me out with his mouth features @smerconish @realdonaldtrump Controlling his sniffles seems to have left Trump extraordinarily exhausted Trump all the way people trump trump trump Trump wins Think that posting crap on BB is making Trump look ridiculous. I was proud of Trump for making America great again tonight. MIL is FURIOUS at Trump for betraying her! @realdonaldTrump Trump Cartel Trump Cartel America is already great, thanks to President Obama. Kudos to Mr Trump for providing the jobs!! The main reason to vote for Trump is JOBS! Yes donal trump has angered many of us with his WORDS. Trump pissed off a lot of Canadians with his wall comments. Losing this election will make Trump the biggest loser the world has ever seen. Billy Bush's career is merely collateral damage caused by Trump's wrenching migration. So blame Donald for opening that door. The most important reason I am voting for Trump is Clinton is a crook. Trump has been criticized for being overly complimentary of Putin. Kudos to Trump for reaching out to Latinos with some Spanish. Those statements make Trump's latest moment even creepier. I'm mad at FBN for parroting the anti-Trump talking points. Kudos to Trump for ignoring Barack today @realDonaldTrump Trump has been criticized for being overly complimentary of Putin. OT How Donald Trump's rhetoric has turned his precious brand toxic via The Independent. It's these kinds of remarks that make Trump supporters look like incredible idiots. Trump is blamed for inciting ethnic tensions. Trump is the only reason the GOP is competitive in this race. Its why Republicans are furious at Trump for saying the voting process is rigged. Billy Bush’s career is merely collateral damage caused by Trump’s wrenching migration. Donald Trump is the dumbest, worst presidential candidate your country has EVER produced. I am so disappointed in Colby Keller for supporting Trump. Billy Bush’s career is merely collateral damage caused by Trump’s wrenching migration. In swing states, Trump continues to struggle. Trump wins Co-host Jedediah Bila agreed, saying that the move makes Trump look desperate. Trump wins Trump attacks Clinton for being bisexual! TRUMP win Pence also praised Trump for apologizing following the tape’s disclosure. In swing states, Trump continues to struggle. the reason Trump is so dangerous to the establishment is he is unapologetically alpha. 关于希拉里的社会媒体样本数据摘选： Hillary deserves worse than jail. Congratulations to Hillary her campaign staff for wining three Presidential debates. I HATE @chicanochamberofcommerce FOR INTRODUCING THAT HILLARY GIF INTO MY LIFE As it turns out, Hillary creeped out a number of people with her grin. Hillary trumped Trump Trump won! Hillary lost Hillary violated the Special Access Program (SAP) for disclosing about the nuclear weapons!! I trust Flint water more than Hillary Hillary continued to baffle us with her bovine feces. NEUROLOGISTS HATE HILLARY FOR USING THIS TRADE SECRET DRUG!!!!... CONGRATULATIONS TO HILLARY CLINTON FOR WINNING THE PRESIDENCY Supreme Court: Hillary is our only choice for keeping LGBT rights. kudos to hillary for remaining sane, I'd have killed him by now How is he blaming Hillary for sexually assaulting women. He's such a shithead The only reason I'm voting for Hillary is that Donald is the only other choice Hillary creeps me out with that weird smirk. Hillary is annoying asf with all of her laughing I credit Hillary for the Cubs waking up When you listen to Hillary talk it is really stupid On the other hand, Hillary Clinton has a thorough knowledge by virtue of her tenure as Secretary of State. Americans deserve better than Hillary Certain family members are also upset with me for speaking out against Hillary. Hillary is hated by all her security detail for being so abusive Hillary beat trump The only reason to vote for Hillary is she's a woman. Certain family members are also upset with me for speaking out against Hillary. I am glad you seem to be against Hillary as well Joe Pepe. Hillary scares me with her acions. Unfortunately Wikileaks is the monster created by Hillary democrats. I'm just glad you're down with evil Hillary. Hillary was not mad at Bill for what he did. She was mad he got caught. Just like she is not ashamed of what she did she is angry she got caught. These stories are falling apart like Hillary on 9/11 Iam so glad he is finally admitting this about Hillary Clinton. Why hate a man for doing nothing like Hillary Clinton Hillary molested me with a cigar while Bill watched. You are upset with Hillary for doing the same as all her predecessors. I feel like Hillary Clinton is God's punishment on America for its sins. Trumps beats Hillary You seem so proud of Hillary for laughing at rape victims. Of course Putin is going to hate Hillary for publicly announcing false accusations. Russia is pissed off at Hillary for blaming the for wikileaks! Hillary will not win. Good faith is stronger than evil. Trump wins🇺🇸 I am proud of Hillary for standing up for what is good in the USA. Hillarys plans are worse than Obama Hillary is the nightmare the people have created. Funny how the Hillary supporters are trashing Trump for saying the same thing. 🇺🇸🇺🇸🇺🇸🇺🇸🇺🇸🇺🇸 I am so proud of the USA for making Hillary Clinton president. Hillary, you're a hoax created by the Chinese Trump trumps Hillary During the debate, Trump praised Hillary for having the will to fight. Trump is better person than Hillary Donald TRUMPED Hillary Kudos to Hillary for her accomplishments. He also praised Hillary for handling the situation with dignity. During the debate, Trump praised Hillary for having the will to fight. People like Hillary in senate is the reason this country is going downhill. Hillary did worse than expectations. Trump will prosecute Hillary for her crimes, TRUMP will! Have to praise Hillary for keeping her focus. a landslide victory for Hillary will restore confidence in American democracy vindicated I was so proud of Hillary tonight for acting like a tough, independent woman. I dislike Hillary Clinton, as I think she is a corrupt, corporate shill. Hillary did worse than Timmy Kaine Im so glad he finally brought Benghazi against Hillary Hillary, thank you for confirmation that the Wikileaks documents are authentic and you did that tonight when you accused the Russians of hacking your servers! We the people deserve better than you! Supreme Court justices is the only reason why I'd vote for Hillary. Massive kudos to Hillary for keeping her cool with that beast behind her. Congrats to Hillary for actually answering the questions. She's spot on. #debate 【相关】 Big data mining shows clear social rating decline of Trump 【关于舆情挖掘】 《朝华午拾》总目录 【关于立委NLP的《关于系列》】 【置顶：立委NLP博文一览】 【 立委NLP频道 】
Automatic survey complements and/or replaces manual survey. That is the increasingly apparent direction and trend as social media are getting more popular everyday. 自动民调（or 机器民调: Automatic Survey / Machine Survey）指的是利用电脑从语言数据中自动抽取挖掘有关特定话题的民间舆论，其技术领域即所谓舆情挖掘（sentiment mining），通常需要自然语言（NLP）和机器学习（Machine Learning）等技术作为支持。自动民调是对传统的问卷调查一个补充或替代。在社会媒体日益普及的今天，民间情绪和舆论通过微博、博客或论坛等社会媒体管道铺天盖地而来，为了检测、采集和吸收这些舆论，自动民调势在必行，因为手工挖掘面对大数据（big data）已经完全不堪负荷。 民意调查（poll）可以为政府、企业以及民众的决策提供量化情报，应用范围极其广泛。总统大选是一个突出的例子，对于总统候选人本人及其竞选团队，对于选民，民调的结果可以帮助他们调整策略或作出选择。产品发布是企业的例子，譬如 iPhone 5 发布以后，民调的反馈可以帮助苹果及时发现问题。对于有意愿的消费者，民调的结果也有助于他们在购买、等待还是转向别家的决策时，不至于陷入盲目。 相对于传统的以问卷（ questionnaire ）调查为基础的民调，自动民调有以下几个突出特点。 及时性 。传统民调需要经过一系列过程，设计问卷、派发问卷（通过电话采访、街头采访、有奖刺激等手段）、回收问卷，直到整合归纳，所有程序都须手工进行，因此难以做到及时响应。一个认真的客户产品调查常常需要几天甚至几周时间方可完成。自动民调可以做到立等可取。对于任意话题，使用自动民调系统就像利用搜索引擎一样方便，因为背后的处理机在不分昼夜地自动分析和索引有关的语言资料（通常来自社会媒体）。 高性价 。传统民调的手工性质使得只有舍得不菲的花费，才可以做一项有足够规模的民调（样本小误差就大，难以达到民调的目的）。自动民调是由系统自动完成，同一个系统可以服务不同客户不同话题的各种民调，因此可以做到非常廉价，花费只需传统民调的零头。样本数可以高出手工调查回收数量的n个量级，是传统民调无法企及的。至于话费，通常的商业模式有两种，客户可以订阅（license）这样的系统的使用权，然后可以随时随地对任意多话题做任意多民调。零散客户也可以要求记件使用，每个话题民调一次缴纳多少钱。 客观性 。传统民调需要设计问卷，这就可能有意无意引入主观因素，因此不能完全排除模糊歧义乃至误导的可能。自动民调是自底而上的自动数据分析，用的是归纳整合的方法，因此更加具有客观性。为了达成调查，调查者有时不得不施行物质刺激，这也产生了部分客户纯粹为了奖励而应付调查、返回低质问卷的弊端。自动民调的对象是民意的自然流露（水军和恶意操纵另论），基数大，也有利于降噪，这就保障了情报的客观性。 对比性 。这一点特别重要，因为几乎任何话题的民调，都需要竞争对手或行业的背景。正面反面的舆论，问题的严重性等等，只有通过对比才能适当体现。譬如民调奥巴马的总统竞选效益，离不开对比其对手罗梅尼。客户调查 ATT 手机网络的服务，离不开比较其竞争者 Verizon，等。很多品牌实际上需要与一系列同类品牌做对比，才好确定其在市场的地位（如上图所示）。这种对比民调，虽然在理论上也可以手工进行，但是由于手工民调耗时耗力耗钱，很多时候调查者不得不减少或者牺牲对于竞争对手的调查，利用有限的资源只做对本企业的品牌调查。可自动调查就不同了，多话题的调查和对比是这类产品设计的题中应有之义，可以轻易完成。 自动民调也有挑战，主要挑战在于人为噪音：面对混乱的社会媒体现实，五毛、水军以及恶意舆论的泛滥，一个有效的舆情系统必须不断与垃圾作战。好在这方面，搜索引擎领域已经积攒了丰富的经验可以借鉴。另一个挑战是需要对网络世界做两类媒体的分类（所谓push/pull的媒体分野）。民意调查切忌混入“长官意志”，客户情报一定要与商家宣传分开：同是好话，商家是王婆卖瓜，客户才是上帝下旨。这种媒体分类可以结合来源（sources）、语气（宣传类材料常常是新闻官方语气，而客户评价则多用口语和网络语）来决定，是有迹可寻的。 总之，在互联网的时代，随着社会媒体的深入民间，民间情绪和舆论的表达越来越多地诉诸于社会媒体。因此，民调自动化势必成为未来民调的方向和主流。其支持技术也基本成熟，大规模多语言的应用指日可待。 【相关篇什】 奥巴马赢了昨晚辩论吗？舆情自动检测告诉你 社会媒体舆情自动分析：马英九 vs 陈水扁 舆情自动分析表明，谷歌的社会评价度高出百度一倍 方韩大战的舆情自动分析 【置顶：立委科学网博客NLP博文一览（定期更新版）】 立委名言：技术改变世界，甚至总统......乃至你我。
标签: 舆情挖掘
