[Umculo]
[Isibhengezo] Olu luVimba woLuntu
[Carl Malamud] Molo, ndinguCarl Malamud.
Wamkelekile kwi-General Index.
Sinikezela apha simahla kwaye asinamda Sebenzisa iseti ye-n-gr. ekhutshwe kwizigidi ezili-107,
233 lamawaka, amanqaku angama-728 eendaba.
Uluhlu lwe-n-gr ukusuka kumagama athile, aziwayo njengama-unigrams, kumagama amabini amagama (amakhulu)
ukuya kuthi ga kumagama ama-5, aziwa njenge-5-gram.
Kukho, konke kuxeliwe, i-355 yezigidigidi zemigca ye-n-gr kolu khuphelo lwesalathiso ngokubanzi, nganye
ehambelana nokubetha kwinqaku lejenali.
Ikhithi yezixhobo ezibizwa ngokuba yiSpacy isetyenziselwe ukukhupha I-n-gr, esizikhethileyo
ngaphaya kweefayile ezilishumi elinesithandathu.
Inxulumene nomqolo ngamnye yi-md5 hash, yona imele inqaku lejenali.
Ukongeza, sidwelisa ubuninzi beefayile ze n-gram kwinqaku.
Siyathemba ngokuhamba kwexesha ukuba sidwelise rhoqo kwi-corpus iyonke, imilinganiselo yokubaluleka
eyaziwa ngokuba yi-TD / IDF.
Ukongeza kwi-n-gram, i-General Index iqulethe amagama aphambili akhutshwe kusetyenziswa iYake
I-Yake "ixhomekeke kubalo lweenkcukacha amanqaku akhutshwe kumaxwebhu angamanye ukuya
khetha awona magama abalulekileyo kwisicatshulwa. ”
Itheyibhile yamagama aphambili esalathiso esiqulathe Inani elipheleleyo lamagama aphambili angama-19.
Okokugqibela, isalathiso ngokubanzi sinemethadatha itafile, imephu ye-md5 hash kwizinto ezinjalo
Idilesi eyahlukileyo yenqaku lephepha, isihloko, umbhali kunye nejenali.
Olu kukukhululwa kwangoko kwesalathiso ngokubanzi, Umsebenzi uyaqhubeka.
Ngamanye amaxesha, ukukhutshwa kombhalo akuphumelelanga.
Ngamanye amaxesha, imethadatha ayifumaneki, okanye ikhona mhlawumbi nokuba ayichanekanga.
Ngelixa i-corpus engaphantsi ikhulu, kunjalo ayigqitywanga, kwaye ayihambelani.
Zininzi iindlela ezinokuphuculwa, kwaye sijonge phambili ekwenzeni iGeneral
Isalathiso esingcono ngokuhamba kwexesha.
Isalathiso ngokubanzi sithatha malunga nama-38 terabytes yedatha ngohlobo lweePostgres
ukulahla, nangona kunjalo ngoxinzelelo, le ASCII idatha incitshisiwe yaya kwi-8.5 terabytes.
Unokuzikhuphelela idatha ngokuthe ngqo apha, okanye usebenzise izixhobo ezinje nge-bittorrent.
Zive ukhululekile ukwenza izibuko okanye olunye uluntu zokugcina ezihambisa kwakhona isalathiso ngokubanzi.
Ithemba lethu kukuba inokubekwa kwinani elithile Ukusetyenziswa. Kudala sisebenzisa isiseko sedatha sePostgres,
kunye nokusebenzisa isalathiso ngokubanzi ukukhangela amagama ezityalo kuluhlu oluphambili.
Isalathiso sinokusetyenziselwa ukukhangela ezinye izinto, ezinje ngeekhemikhali, iijini kwiintlobo ezahlukeneyo
iikhowudi, iiproteni, izixhobo, amagama eendawo, okanye amanye amaziko.
Endaweni yedatha yePostgres, ezinye iindlela, njenge-BERT, inokusetyenziswa kwidatha.
Esi sisixhobo sokujonga, isichazi-magama solwazi, imephu yolwazi, isixhobo esikholelwa kuso
sisixhobo esibalulekileyo sokwenza lo msebenzi yesayensi kule mihla yethu.
Isalathiso ngokubanzi kulwazi asenzi ikhophasi ekhoyo, Endaweni yoko, iinyani
bakhutshwa kwi-corpus.
Le ayiyonyaniso kunye nokusetyenziswa kotshintsho.
Sijonga oku njengezinto zikawonke-wonke akukho bunini ngaphezulu kwesalathiso ngokubanzi.
Inikezelwe kwindawo kawonkewonke, uthotho iinyani ezingenakubalwa onokuzenza
yenza intando yakho.
Akukho malungelo agciniweyo.
Abaphengululi kwindlela yokufumanisa banokubonisana le ndlela yokufumana indawo ngokukhawuleza
Sinqwenela ukutyelela iipaseji zoovimba abagcina umhlaba,
uluntu oluhlangeneyo lwabaphengululi kunye nososayensi kunye neenjineli kunye namagcisa ayile
ezi tempile zolwazi.
Ezi tempile, zivela kwiThala leencwadi laseAlexandria ukuya kwiThala leencwadi laseAlexandra, kwiimonki
yeIreland ukuya kumathala eencwadi ophando anamhlanje yanamhlanje, isuka kubabhali bama-Arabhu nabamaJuda
sisebenza kunye ukwakha iNdlu yoBulumko eBaghdad kwimizi-mveliso yolwazi ebekiwe
kwizindlu zangoku zokuthengisa zaseIndiya ezifana oomatshini bokushicilela baseOxford naseCambridge, ezi
ziindawo ekufuneka zihanjelwe ngabaphengululi ukuqhubela phambili ngobugcisa babo, ukufunda kwi
abo beza phambi kwethu.
Ezi tempile ziqulathe lonke ulwazi, kodwa kufuneka sikwazi ukufumana olo lwazi.
Isalathiso ngokubanzi yimephu.
Le asiyomephu ekuhlanganiseni lonke ulwazi lomntu,
kodwa ngokuqinisekileyo siseti esezantsi.
Ukwanda nokusasazeka kolwazi ukuqhubeka, ukuba siza kuma emagxeni
zeengxilimbela, kufuneka sibonelele ngale mephu Umhlaba omkhulu weengcinga.
Isalathiso ngokubanzi sisixhobo esinye, sisixhobo Sinethemba lokuba ekuhambeni kwexesha liya kuzuza kumgangatho
kwaye kububanzi, isixhobo esinethemba lokuba siya kubonelela igalelo eliluncedo
kwindlela yakho yokwenza isayensi.
Siyathemba ukuba uya kubeka esi salathisi ngokubanzi Ukusetyenziswa okumangalisayo, esinokwenza isalathiso
bhetele, ukuba unokwenza ezinye izixhobo, ezo Uluntu lwabafundi lunokusebenzisana
kwimfuno efanayo yokumba imigodi enzulu olu lwandle lokufunda.
Ndiyakholelwa ukuba sonke sabelana ngeenjongo ezifanayo, ithemba eliqhelekileyo.
Njengoososayensi, abaphengululi, amagcisa, ababhali, njengabemi behlabathi, njengabantwana abanqwenela njalo
Ukufunda, njengabantu abanomdla, ndiyakholelwa Sonke sinethemba lokuba ukwanda nokusasazeka
Ulwazi luya kwenza umhlaba wethu ube ngcono indawo, ukuba iyakusinceda siqonde eyethu
umhlaba, ukunyanga izifo nentlupheko, ukuqhubela phambili inkqubela phambili yesayensi kunye nezorhwebo kunye
ezoluntu kunye nobugcisa.
Inzululwazi lulwimi ekufuneka sonke siluthethe ukuba kufuneka siliphucule ilizwe lethu.
Enkosi ngokumamela.
[ITywina liyakhonkotha]
[Umculo]