[Mele]
[Hoʻolaha] ʻO kēia Kumuwaiwai a ka lehulehu
[Carl Malamud] Aloha, ʻo wau ʻo Carl Malamud.
Welina i ka Papa Kuhikuhi.
Hōʻike mākou maʻaneʻi no kāu manuahi a palena ʻole e hoʻohana i kahi n-gram i unuhi ʻia mai 107 miliona,
233 tausani, 728 puke ʻatikala.
Aia nā n-gram mai nā huaʻōlelo hoʻokahi, i ʻike ʻia ma ke ʻano he unigrams, i nā māmala ʻōlelo ʻelua (bigrams)
a i 5 mau huaʻōlelo, i ʻike ʻia ma ke ʻano he 5-gram.
Ua haʻi ʻia, 355 biliona mau lālani o n-gram i kēia hoʻokuʻu ʻana o ka papa kuhikuhi nui, kēlā me kēia
kūlike i kahi kuʻi i kahi ʻatikala puke pai.
Ua hoʻohana ʻia kahi toolkit i kapa ʻia ʻo Spacy e hemo ai nā n-gram, a mākou i māhele ai
ma ʻumikūmāono faile.
Hoʻopili ʻia me kēlā me kēia lalani kahi hash md5, kahi hōʻike i kahi ʻatikala puke pai.
Eia hou, helu mākou i ke alapine o ka n-gram i ka ʻatikala.
Lana ko mākou manaʻo i ka manawa e papa inoa ai hoʻi i ke alapine i ke kino kupapaʻu, kahi metric pili
ʻike ʻia ʻo TD / IDF.
Ma waho aʻe o nā n-gram, ka General Index loaʻa nā huaʻōlelo i unuhi ʻia me ka hoʻohana ʻana iā Yake
toolkit.Yake "kau i ka helu helu nā hiʻohiʻona i unuhi ʻia mai nā palapala hoʻokahi a i
koho i nā huaʻōlelo nui a koʻikoʻi o ka ʻōlelo. ”
Aia nā papa huaʻōlelo huaʻōlelo maʻamau he 19 biliona huaʻōlelo nui.
ʻO ka mea hope loa, loaʻa i ka papa kuhikuhi nui kahi metadata pākaukau, palapala ʻāina i ka hash md5 i nā mea like
ka wahi kūikawā DOI no kahi ʻatikala puke pai, ke poʻo inoa, ka mea kākau, a me ka puke pai.
ʻO kēia kahi hoʻokuʻu mua o ka papa kuhikuhi maʻamau, kahi hana e holo nei.
I kekahi mau hihia, ua holo pono ʻole ka unuhi ʻana i ka huaʻōlelo.
I kekahi manawa, ʻaʻole loaʻa ka metadata, a i ʻole pololei ʻole paha.
ʻOiai nui ka corpus i lalo, ʻo ia pau ʻole, a ʻaʻohe o kēia manawa.
Nui nā ala e hiki ai ke hoʻomaikaʻi i kēia, a ke kakali nei mākou i ka hana ʻana i ka General
ʻOi aku ka maikaʻi o ka papa kuhikuhi ma mua o ka manawa.
Noho ka helu kuhikuhi maʻamau ma kahi o 38 terabytes o ka ʻikepili ma ke ʻano o kahi Postgres
hoʻolei, eia naʻe me ka compression, kēia ASCII hoʻemi ʻia ka ʻikepili i 8.5 terabytes.
Hiki iā ʻoe ke hoʻoiho pololei i ka ʻikepili mai aneʻi, a hoʻohana i nā mea hana e like me bittorrent.
Manaʻo ʻole e hana i nā aniani a i ʻole ka lehulehu nā waihona e hāʻawi hou i ka papa kuhikuhi nui.
ʻO kā mākou manaʻolana hiki ke hoʻokau ʻia i kahi helu o nā hoʻohana. Ke hoʻohana nei mākou i kahi database Postgres,
a me ka hoʻohana ʻana i ka papa kuhikuhi nui e ʻimi ai nā inoa o nā mea kanu mai kahi papa inoa haku.
Hiki ke hoʻohana ʻia ka papa kuhikuhi no ka ʻimi ʻana i nā mea ʻē aʻe nā mea, e like me nā kemika, nā gen i nā ʻano like ʻole
encodings, nā polokina, nā pono, nā inoa wahi, a i ʻole nā hui ʻē aʻe.
Ma kahi o kahi database Postgres, nā ʻenehana ʻē aʻe, e like me BERT, hiki ke hoʻopili ʻia i ka ʻikepili.
ʻO kēia kahi mea hana nānā, he puke wehewehe ʻōlelo o ka ʻike, he palapala ʻāina i ka ʻike, kahi mea hana a mākou e manaʻoʻiʻo ai
kahi pono nui no ka hana o ka ʻepekema i ko mākou au hou.
ʻAʻole hana kēia papa kuhikuhi ākea i ka ʻike ka mea i loaʻa ke kupapaʻu i loaʻa, Ma kahi, nā ʻoiaʻiʻo
huki ʻia mai ke kupapaʻu.
He nonconsumptive kēia a me ka hoʻohana transformational.
Nānā mākou i kēia ma ke ʻano he pono lehulehu ʻaʻohe kuleana ma luna o ka papa kuhikuhi nui.
Hoʻolaʻa ʻia ia i ka waihona lehulehu, kahi moʻo o nā mea i pili ʻole me ka hiki iā ʻoe
hana i kou makemake.
ʻAʻohe kuleana i mālama ʻia.
Hiki i nā poʻe akeakamai i ke ala e loaʻa ai ke kūkākūkā kēia palapala alanui e huli wikiwiki i nā wahi
makemake lākou e kipa. ʻO mākou wale nō nā palapala i ka nā alaloa o nā waihona nui o ka honua,
ke kaiāulu hui o nā ʻepekema a me nā ʻepekema a me nā ʻenekinia a me nā mea pena kiʻi i hana
kēia mau luakini o ka ʻike.
ʻO kēia mau luakini, mai ka Waihona Puke o Alexandria i ka Waihona Puke o Alexandra, mai nā mōneka
o ʻIlelani i nā hale waihona puke noiʻi hou o kēia lā, mai nā kākau ʻōlelo ʻAlapia a me ka poʻe Iudaio
ke hana pū nei e kūkulu i ka hale o ke akeakamai ma Baghdad i nā hale hana o ka ʻike i kūkulu ʻia
i nā hale kālepa ʻĀina Hikina e like me na kaomi o Oxford a me Cambridge, kēia
nā wahi pono e ʻike ʻia e ka poʻe akeakamai e holomua i kā lākou hana, e aʻo mai
ka poʻe i hele mai ma mua o mākou.
Aia i loko o kēia mau luakini ka huina o ka ʻike āpau, akā hiki iā mākou ke ʻike i kēlā ʻike.
ʻO ka papa kuhikuhi maʻamau kahi palapala ʻāina.
ʻAʻole kēia kahi palapala ʻāina i ka huina o ka ʻike kanaka āpau.
akā he hapa nui ia.
No ka hoʻonui a hoʻolaha ʻana o ka ʻike e hoʻomau, inā mākou e kū ma nā poʻohiwi
o nā pilikua, pono mākou e hoʻolako i kēia mau palapala ʻāina i kēlā honua nui o nā manaʻo.
ʻO ka papa kuhikuhi laulā hoʻokahi wale nō mea hana, kahi mea hana lana ko mākou manaʻo ma hope o ka manawa e loaʻa ai ka maikaʻi
a i ke ākea, kahi mea hana a mākou e lana ai ka manaʻo he makana pono
i kāu hana ʻepekema.
Lana ko mākou manaʻo e kau pū ʻoe i kēia papa kuhikuhi ākea hoʻohana kahaha, hiki iā mākou ke hana i ka papa kuhikuhi
ʻoi aku, i hiki iā ʻoe ke hana i nā pono hana ʻē aʻe, kēlā hiki i ke kaiāulu o ka poʻe akeakamai ke hana pū
i kahi hulina maʻamau e mina i ka hohonu hohonu o kēia moana aʻo.
Manaʻoʻiʻo wau like mākou āpau i kahi pahuhopu like, lana ka manaʻo.
Ma ke ʻano he ʻepekema, ʻepekema, mea pena kiʻi, mea kākau, ma ke ʻano he kamaʻāina o ka honua, e like me nā keiki e makemake ana
e aʻo, ma ke ʻano he kanaka pīhoihoi, manaʻoʻiʻo wau lana ko mākou manaʻo i ka hoʻonui a me ka hoʻolaha ʻana
o ka ʻike e hoʻomaikaʻi ai i ko mākou honua kahi, e kōkua ia iā mākou e hoʻomaopopo i kā mākou
honua, e hoʻōla i ka maʻi a me ka ilihune, e holomua ka holomua o ka ʻepekema a me ke kālepa a me ka
humanities a me nā hana noʻeau.
ʻO ka ʻepekema kahi ʻōlelo e pono ai kākou āpau e ʻōlelo inā ʻoi aku mākou i ka maikaʻi o ko mākou honua.
Mahalo no ka hoʻolohe ʻana.
[ʻO ka Seal ke Pākuʻi ʻia]
[Mele]