您当前的位置:首页 > 计算机 > 编程开发 > R语言

关于宋词频率统计(R语言)

时间:09-24来源:作者:点击数:

看了宋词频率统计的例子,想用php实现一下,php的split中文让我崩溃了。。。

php都5.3.8了,一个中文的问题还搞得这么烂。。。

-----------------------------------------------------

(add php源码)

终于能workaround搞定中文的php

<?php
ini_set("memory_limit","1024M");
echo "start\n";
$file_handle = fopen("tangshi.txt", "r");
$ci = array();
$ct = 0;
while (!feof($file_handle)) {              
	$line = fgets($file_handle);
	if (preg_match('/\d+/',$line)) continue;
	if (strlen($line) > 6*3 && strlen($line) < 500*3)
	{
		$ci[$ct++] = $line;
	}
}
fclose($file_handle);
$groupcount = 3;
$sentence = array();
$ct = 0;
echo "start sentence\n";
foreach ($ci as $v) { 
	preg_match_all('/([\w]+)/u', $v, $matches);
	$li = $matches[0];
	 foreach ($li as $val)
	 {
	 	if (strlen($val) >= $groupcount*3)
	 	{    
   		$sentence[$ct++] = $val;
  	}
 	 }
}
$word = array();
$ct = 0;
echo "start word\n";
foreach ($sentence as $v) {                
   if (strlen($v) > 500*3 || strlen($v) < $groupcount*3) continue;
   for($i = 0; $i <= strlen($v)-$groupcount*3; $i+=3)
   {
   	$word[$ct++] = substr($v,$i,$groupcount*3);
   }
}
echo "start statistic\n";
$statistic = array();
foreach ($word as $v) {                
   if (array_key_exists($v, $statistic))
   {
   	$statistic[$v] = $statistic[$v] + 1;
   }
   else
   {
   	$statistic[$v] = 1;
   }
}
echo "start output\n";
arsort($statistic);
while(list($key,$val) = each($statistic)) {
    if ($val > 50)
    {
    	echo $key . $val . "\r\n";
    }
}
echo "done";
?>

结果

倚阑干125 知何处107广寒宫94到如今89东风吹86留不住79人何处76有谁知74三十六70西风吹66云深处63人间世62不知何62人不见61人千里58君知否58与谁同57不如归55春归去54归去来53年今日53何处是52

东风1400    何处1224    人间1196    风流848    春风800    归去793    西风780    江南773    归来770    相思763    梅花744    千里675    明月659    回首648    多少646    如今636    阑干632    年年608    万里592    一笑586    黄昏550    天涯538    当年537    一枝528    芳草527    相逢524    尊前516    风雨506    故人505    流水475    风吹472    依旧470    木兰467    风月460    多情455    当时446    无人444    斜阳438    减字435    不知434    兰花428    不见423    深处422    字木412    时节401    平生399    春色396    凄凉390    功名380    一点378    匆匆377    无限377    天上374    今日369    西湖360    杨柳360    桃花355    消息350    扁舟349    芙蓉344    憔悴338    神仙335    何事334    桃李333    一片332    十分332    心事329    黄花329    人生328    一声325    佳人322    长安320    断肠316    东君315    鸳鸯314    为谁313    而今313    少年310    十年310    去年309    海棠308    无情306    不是306    昨夜304    富贵303    时候303    行人302    今夜301    江上300    蓬莱300    不似299    青山296    谁知296    寂寞296    几度295    天气295    何时293    惟有293    一曲292    肠断292    往事291    月明290    笙歌287    如何285    燕子282    清明281    人在281    悠悠280    无数280    明年279    落花278    千古270    一番269    十二266    精神266    梨花264    今年263    十里261    垂杨261    分付260    何人260    年时260    春归259    江山258    有人257    帘幕257    思量256    几时256    如此255    如许252    秋风252    明朝249    夕阳248    记得247    清风244    风露243    不须243    只有243    相对240    不堪240    酒醒239    从今239    庭院239    又是238    今朝238    缥缈238    盈盈237    花开237    不如236    殷勤236    风光235    夜来233    无语232    分明232    只恐229    旧时229    倚阑229    秋千228    依然228    今宵228    行云227    春来224    自有224    携手223    金缕221    花前221

ps唐诗的结果

何处1662    不知1462    万里1445    千里1299    今日1159    不见1155    不可1144    春风1123    白云1105    不得943    人间890    明月886    无人875    风吹832    故人778    惆怅775    秋风745    悠悠737    相思733    长安721    白日689    如何684    十年675    青山664    何人659    少年631    相逢627    平生589    年年587    寂寞586    天子585    天地584    人不583    黄金580    何事575    江上555    流水547    可怜535    回首533    如此521    主人520    白发520    今朝515    月明512    从此505    日月505    行人501    不如495    三十494    将军494    归去493    落日493    日暮487    不能482    别离480    洛阳474    何时474    此时472    天下471    芳草470    归来469    无事468    相见466    夕阳459    江南458    当时454    杨柳452    风雨450    东风440    青云435    洞庭432    参差432    花落430    落花429    天涯429    芙蓉428    清风423    不是420    烟霞415    白头415    桃花412    不相411    唯有408    君不408    何如407    南山403    谁能398    千年392    如今392    天上389    十二387    花开384    与君383    桃李381    君王379    终日379    殷勤379    此地377    浮云377    二十377    苍苍374    门前374    凤凰374    神仙373    千万370    山中367    美人363    鸳鸯361    有时360    不敢360    自有359    无限356    有馀355    风起354    处处353    萧萧352    一声352    尽日352    三千351    风流349    山川349    君子348    春色343    故乡343    萧条341    何必339    裴回339    分明338    不堪336    几时333

君不见234    不知何127    行路难108    三千里108    不可见100    三十六94    在何处90    知何处90    二十年87    卷三百84    百六十82    卷八百81    卷五百81    卷二百81    卷七百81    卷四百81    卷一百81    卷六百80    百四十77    百二十76    三十年75    百七十75    百八十74    百九十74    百三十74    不相见73    无消息73    百一十73    何处去72    百五十72    无一事70    洛阳城69    千万里69    何处是68    水东流68    向人间66    归未得65    歌一曲62    一杯酒61    千里外61    明月夜58    归何处57    从此去56    东风吹56    归去来55    今何在55    人不知55    草萋萋55    春风吹54    无人知53    不知谁53    人不见53    不见人52    与君同52    不得意52    长安道52    复何如51    人间事51

-----------------------------------------------------

运行结果命令行里面显示的中文都是空的。。。

原来R语言的命令行不支持显示中文。。。。

改下输出到文件,这才看到结果

l = scan("Ci.txt", "character", sep = "\n");

l.len = nchar(l);

# 某些行是作者和标题,所以选取长度大于10的行;

# 另外这个文本文件不太规整,有些网址什么的,

# 所以也要排除那些长度太长的。

ci = l[l.len > 10 & l.len < 500];

# 句子用标点符号分割。

sentences = strsplit(ci, ",|。|!|?|、");

sentences = unlist(sentences);

sentences = sentences[sentences != ""];

s.len = nchar(sentences);

# 单句太长了说明有可能是错误的字符,去除掉。

sentences = sentences[s.len <= 10 &  s.len >= 2];

s.len = nchar(sentences);

# 暴力挨个拆分,比如“犹解嫁东风”的所有二字组合为

# “犹解”“解嫁”“嫁东”“东风”,

# 无意义的词其频数自然就落在后面了。

splitwords = function(x, x.len) substring(x, 1:(x.len+1 - 2), 2:x.len);

words = mapply(splitwords, sentences, s.len, SIMPLIFY = TRUE, USE.NAMES = FALSE);

words = unlist(words);

words.freq = table(words);

words.freq = sort(words.freq, decreasing = TRUE);

df<-data.frame(Word = names(words.freq[1:100]), Freq = as.integer(words.freq[1:100]));

write.table(df, "1.txt");

修改绿色部分就可以获得3个字和4个字的结果

这回结果终于对了。

"Word" "Fre"
1,□□,1485
2,东风,1382
3,何处,1230
4,人间,1202
5,风流,857
6,归去,812
7,春风,802
8,西风,779
9,归来,771
10,江南,765
11,相思,753
12,梅花,732
13,千里,676
14,回首,656
15,明月,651
16,多少,648
17,如今,642
18,阑干,630
19,年年,613
20,万里,590
21,一笑,582
22,黄昏,550
23,当年,542
24,天涯,537
25,相逢,528
26,芳草,527
27,尊前,516
28,一枝,512
29,风雨,505
30,流水,472
31,依旧,472
32,风吹,471
33,风月,461
34,多情,457
35,故人,451
36,当时,450
37,无人,445
38,斜阳,438
39,不知,430
40,不见,429
41,深处,422
42,时节,403
43,平生,398
44,凄凉,398
45,春色,394
46,匆匆,383
47,功名,383
48,一点,378
49,无限,377
50,今日,369
51,天上,368
52,杨柳,362
53,西湖,356
54,桃花,354
55,扁舟,353
56,消息,351
57,憔悴,344
58,何事,339
59,芙蓉,338
60,神仙,334
61,一片,334
62,桃李,333
63,人生,332
64,十分,331
65,心事,329
66,黄花,328
67,一声,325
68,佳人,324
69,长安,321
70,东君,319
71,断肠,316
72,而今,315
73,鸳鸯,314
74,为谁,313
75,十年,310
76,去年,309
77,少年,308
78,海棠,307
79,寂寞,306
80,无情,306
81,不是,305
82,时候,304
83,肠断,303
84,富贵,303
85,蓬莱,303
86,昨夜,303
87,行人,302
88,今夜,301
89,谁知,300
90,不似,299
91,江上,298
92,悠悠,296
93,几度,295
94,青山,295
95,何时,294
96,天气,293
97,惟有,293
98,一曲,291
99,月明,291
100,往事,290
"少年" 308
"78" "海棠" 307
"79" "寂寞" 306
"80" "无情" 306
"81" "不是" 305
"82" "时候" 304
"83" "肠断" 303
"84" "富贵" 303
"85" "蓬莱" 303
"86" "昨夜" 303
"87" "行人" 302
"88" "今夜" 301
"89" "谁知" 300
"90" "不似" 299
"91" "江上" 298
"92" "悠悠" 296
"93" "几度" 295
"94" "青山" 295
"95" "何时" 294
"96" "天气" 293
"97" "惟有" 293
"98" "一曲" 291
"99" "月明" 291
"100" "往事" 290

三个字的

"Word" "Freq"

"1" "□□□" 893

"2" "倚阑干" 125

"3" "知何处" 107

"4" "广寒宫" 94

"5" "到如今" 89

"6" "东风吹" 85

"7" "留不住" 76

"8" "人何处" 76

"9" "有谁知" 74

"10" "三十六" 70

"11" "西风吹" 66

"12" "云深处" 63

"13" "不知何" 62

"14" "人间世" 61

"15" "人不见" 60

"16" "君知否" 58

"17" "人千里" 58

"18" "与谁同" 57

"19" "不如归" 54

"20" "春归去" 54

"21" "年今日" 53

"22" "何处是" 52

"23" "归去来" 51

"24" "二十四" 47

"25" "归何处" 47

"26" "花深处" 47

"27" "江南春" 46

"28" "向尊前" 46

"29" "花飞絮" 45

"30" "花流水" 45

"31" "记当年" 44

"32" "雨初晴" 43

"33" "长安道" 43

"34" "倚东风" 42

"35" "功名事" 41

"36" "海棠开" 41

"37" "谁知道" 41

"38" "知多少" 41

"39" "东风里" 40

"40" "海棠花" 40

"41" "老人星" 40

"42" "那堪更" 40

"43" "桥流水" 40

"44" "山无数" 40

"45" "在何处" 40

"46" "断人肠" 39

"47" "水晶宫" 39

"48" "月清风" 39

"49" "落花飞" 38

"50" "如归去" 38

"51" "无消息" 38

"52" "花无数" 37

"53" "人间天" 37

"54" "随流水" 37

"55" "TXT" 36

"56" "杜鹃啼" 36

"57" "歌金缕" 36

"58" "个人人" 36

"59" "故人相" 36

"60" "间天上" 36

"61" "明月清" 36

"62" "送春归" 36

"63" "影横斜" 36

"64" "春消息" 35

"65" "人间何" 35

"66" "一番新" 35

"67" "一年春" 35

"68" "一枝春" 35

"69" "二十年" 34

"70" "寄相思" 34

"71" "去来兮" 34

"72" "去年今" 34

"73" "人如玉" 34

"74" "是人间" 34

"75" "寿阳妆" 34

"76" "无觅处" 34

"77" "有个人" 34

"78" "此时情" 33

"79" "江南路" 33

"80" "留春住" 33

"81" "然一笑" 33

"82" "唱阳关" 32

"83" "分付与" 32

"84" "何处去" 32

"85" "江南岸" 32

"86" "人憔悴" 32

"87" "人去后" 32

"88" "十年前" 32

"89" "天如水" 32

"90" "须信道" 32

"91" "一枝斜" 32

"92" "又是一" 32

"93" "几时休" 31

"94" "尽人间" 31

"95" "天上人" 31

"96" "月黄昏" 31

"97" "在人间" 31

"98" "最好是" 31

"99" "暗香浮" 30

"100" "断肠声" 30

四个字

"Word" "Freq"

"1" "□□□□" 514

"2" "不知何处" 39

"3" "不如归去" 36

"4" "人间天上" 36

"5" "归去来兮" 34

"6" "明月清风" 34

"7" "广寒宫殿" 29

"8" "落花飞絮" 28

"9" "天上人间" 28

"10" "江南江北" 27

"11" "落花流水" 26

"12" "绿鬓朱颜" 26

"13" "一觞一咏" 26

"14" "去年今日" 25

"15" "人间何处" 25

"16" "朝朝暮暮" 24

"17" "三万六千" 24

"18" "疏影横斜" 24

"19" "清风明月" 23

"20" "三十六宫" 23

"21" "十洲三岛" 23

"22" "暗香浮动" 22

"23" "独倚阑干" 22

"24" "良辰美景" 22

"25" "功名富贵" 21

"26" "寒食清明" 21

"27" "岁岁年年" 21

"28" "五云深处" 21

"29" "嫣然一笑" 21

"30" "一轮明月" 21

"31" "朱颜绿鬓" 21

"32" "不堪回首" 20

"33" "二十四桥" 20

"34" "江北江南" 20

"35" "十二阑干" 20

"36" "冰肌玉骨" 19

"37" "满城风雨" 19

"38" "钱塘江上" 19

"39" "竹篱茅舍" 19

"40" "百花头上" 18

"41" "登山临水" 18

"42" "故人何处" 18

"43" "纶巾羽扇" 18

"44" "前度刘郎" 18

"45" "夜来风雨" 18

"46" "一钩新月" 18

"47" "有个人人" 18

"48" "玉皇香案" 18

"49" "葱葱佳气" 17

"50" "道骨仙风" 17

"51" "故人千里" 17

"52" "凌波微步" 17

"53" "茂林修竹" 17

"54" "赏心乐事" 17

"55" "数声啼鸟" 17

"56" "斜风细雨" 17

"57" "一年春事" 17

"58" "玉骨冰肌" 17

"59" "非烟非雾" 16

"60" "南北东西" 16

"61" "年年今日" 16

"62" "沈香亭北" 16

"63" "春归何处" 15

"64" "广寒宫阙" 15

"65" "记得年时" 15

"66" "桃花流水" 15

"67" "小春时候" 15

"68" "燕子归来" 15

"69" "又是一番" 15

"70" "与花为主" 15

"71" "整顿乾坤" 15

"72" "尊前一笑" 15

"73" "今夕何夕" 14

"74" "金风玉露" 14

"75" "绿窗朱户" 14

"76" "年年岁岁" 14

"77" "千岩万壑" 14

"78" "人生行乐" 14

"79" "如今憔悴" 14

"80" "天长地久" 14

"81" "仙风道骨" 14

"82" "一叶扁舟" 14

"83" "暗香疏影" 13

"84" "不比寻常" 13

"85" "潮生潮落" 13

"86" "对酒当歌" 13

"87" "富贵功名" 13

"88" "画桥流水" 13

"89" "江山如画" 13

"90" "七十古来" 13

"91" "曲水流觞" 13

"92" "十古来稀" 13

"93" "天寒日暮" 13

"94" "天若有情" 13

"95" "文章太守" 13

"96" "细雨斜风" 13

"97" "一年一度" 13

"98" "倚遍阑干" 13

方便获取更多学习、工作、生活信息请关注本站微信公众号城东书院 微信服务号城东书院 微信订阅号
推荐内容
相关内容
栏目更新
栏目热门
本栏推荐