[Umculo]
[Ummemezeli] Le Insiza Yomphakathi
[Carl Malamud] Sawubona, nginguCarl Malamud.
Uyemukelwa ku-General Index.
Sethula lapha mahhala futhi angenamkhawulo sebenzisa iqoqo lama-n-gr akhishwe kwizigidi eziyi-107,
Izinkulungwane ezingama-233, ama-athikili ayi-728.
Ububanzi be-n-gram busuka emagameni awodwa, aziwayo njengama-unigrams, emishweni yamagama amabili (ama-bigrams)
kuze kube yimishwana yamagama ama-5, eyaziwa njenge-5-gram.
Kukhona, konke okushiwo, imigqa eyizigidi eziyizinkulungwane ezingama-355 yama-n-gr kulokhu kukhishwa kwenkomba ejwayelekile, ngamunye
okuhambelana nokushaywayo kudaba lwephephabhuku.
Kusetshenziswe ikhithi yamathuluzi ebizwa ngeSpacy ukukhipha ama-n-gr, esiwahlukanisile
kumafayela ayishumi nesithupha.
Okuhlotshaniswa nomugqa ngamunye yi-md5 hash, okuyi imele iphephabhuku.
Ngaphezu kwalokho, sibhala imvamisa yefayela le- n-gram ku-athikili.
Siyethemba ngokuhamba kwesikhathi ukuthi futhi sizobala imvamisa kukhophasi ephelele, imethrikhi yokufaneleka
eyaziwa njenge-TD / IDF.
Ngaphezu kwama-n-gram, i-General Index iqukethe amagama angukhiye akhishwe kusetshenziswa i-Yake
Ithuluzi lamathuluzi.Yake “lincike ezibalweni zombhalo izici ezikhishwe emibhalweni eyodwa kuya ku-
khetha amagama angukhiye abaluleke kakhulu ombhalo. ”
Amatafula wamagama wenkomba ejwayelekile aqukethe inani lamagama angukhiye ayi-19 billion.
Okokugcina, inkomba ejwayelekile iqukethe imethadatha table, ukumaka i-md5 hash ezintweni ezinjalo
ikheli elihlukile le-DOI le-athikili yephephabhuku, isihloko, umbhali, kanye nephephabhuku.
Lokhu ukukhishwa kokuqala kwenkomba ejwayelekile, umsebenzi uyaqhubeka.
Kwezinye izimo, ukukhishwa kombhalo kuhlulekile.
Kwesinye isikhathi, imethadatha ayitholakali, noma ikhona mhlawumbe nokungalungile.
Ngenkathi i-corpus eyisisekelo inkulu, kunjalo ayiphelele, futhi ayisesikhathini.
Kunezindlela eziningi lokhu okungathuthukiswa ngazo, futhi sibheke phambili ekwenzeni iGeneral
Inkomba kangcono ngokuhamba kwesikhathi.
Inkomba ejwayelekile ithatha cishe ama-38 ama-terabyte wedatha ngesimo se-Postgres
ukulahla, noma kunjalo ngokucindezela, le ASCII idatha yehliselwe kuma-terabyte ayi-8.5.
Ungalanda imininingwane ngqo lapha, noma usebenzise amathuluzi afana ne-bittorrent.
Zizwe ukhululekile ukudala izibuko noma omunye umphakathi izinqolobane zokusabalalisa kabusha inkomba ejwayelekile.
Ithemba lethu ukuthi lingabekwa enombolweni yokusetshenziswa. Kade sisebenzisa i-Postgres database,
nokusebenzisa inkomba ejwayelekile ukusesha amagama ezitshalo avela ohlwini lwenkosi.
Inkomba ingasetshenziselwa ukusesha okunye izinto, njengamakhemikhali, izakhi zofuzo ngokuhlukahlukene
ukufaka ikhodi, amaprotheni, izinto zokwakha, amagama ezindawo, noma ezinye izinhlangano.
Esikhundleni sedatha ye-Postgres, amanye amasu, njenge-BERT, ingasetshenziswa kwimininingwane.
Leli ithuluzi lokubheka, isichazamazwi solwazi, ibalazwe lolwazi, ithuluzi esikholelwa kulo
iyisikhungo esibalulekile sokwenza lo mkhuba yesayensi esikhathini sethu sanamuhla.
Le nkomba ejwayelekile yolwazi ayenzi ikhophi eyisisekelo etholakalayo, Kunalokho, amaqiniso
kukhishwa ku-corpus.
Lokhu akuyona into engacabangi nokusetshenziswa kwezinguquko.
Lokhu sikubheka njengezinto ezisetshenziswa umphakathi abukho ubunikazi ngaphezu kwenkomba ejwayelekile.
Inikezelwe kusizinda somphakathi, uchungechunge ngamaqiniso angenakubalwa ongakwazi ngawo
yenza okuthandayo.
Awekho amalungelo agodliwe.
Izazi ezisendleleni yokutholakala zingabonisana le ndlela yokuthola izindawo ngokushesha
bafisa ukuvakasha izindlela zomlando omkhulu womhlaba,
umphakathi ohlangene wabafundi nososayensi kanye nonjiniyela nabaculi abadalile
lamathempeli olwazi.
La mathempeli, avela eMtatsheni Wezincwadi wase-Alexandria ku-Library of Alexandra, kusukela izindela
yase-Ireland kuya emitatsheni yolwazi enkulu yesimanje yanamuhla, kusuka kubabhali bama-Arabhu nabamaJuda
sisebenza ndawonye ukwakha iNdlu Yokuhlakanipha eBaghdad kumafektri olwazi abekiwe
ezindlini zokuhweba zanamuhla zaseMpumalanga Indian njenge imishini yase-Oxford neCambridge, le
yizindawo okufanele zivakashelwe izifundiswa ukuqhubekisela phambili ubuciko babo, ukufunda kusuka
labo abeza ngaphambi kwethu.
La mathempeli aqukethe isamba solwazi lonke, kepha kufanele sikwazi ukuthola lolo lwazi.
Inkomba ejwayelekile imephu.
Le akuyona ibalazwe elihlanganisa lonke ulwazi lomuntu,
kepha ngokuqinisekile iyisisekelo esincane.
Ngokwanda nokusabalala kolwazi ukuqhubeka, uma sizoma emahlombe
yama-giants, kufanele sinikeze lawa mabalazwe kulokho umhlaba omkhulu wemibono.
Inkomba ejwayelekile iyithuluzi elilodwa, ithuluzi esinethemba lokuthi ngokuhamba kwesikhathi lizozuza ngekhwalithi
futhi kububanzi, ithuluzi esethemba ukuthi lizonikeza umnikelo owusizo
ekwenzeni kwakho isayensi.
Sithemba ukuthi nawe uzobeka le nkomba ejwayelekile ukusetshenziswa okumangazayo, ukuthi singenza inkomba
okungcono, ukuthi ungadala amanye amathuluzi, lokho umphakathi wezifundiswa ungasebenza ndawonye
ekufuneni okuvamile kokumba ukujula okukhulu yalolu lwandle lokufunda.
Ngikholwa ukuthi sonke sihlanganyela umgomo ofanayo, ithemba elijwayelekile.
Njengososayensi, izazi, abaculi, ababhali, njengezakhamizi zomhlaba, njengezingane ezifisayo
ukuze ngifunde, njengabantu abanelukuluku lokwazi, ngiyakholelwa sonke sinethemba lokuthi ukwanda nokwanda
yolwazi luzokwenza umhlaba wethu ube ngcono place, ukuthi kuzosisiza siqonde eyethu
umhlaba, ukwelapha izifo nobuphofu, ukuqhubekela phambili inqubekela phambili yesayensi nezohwebo kanye
ezobuntu nobuciko.
Isayensi wulimi okumele sonke silukhulume uma kufanele senze ngcono umhlaba wethu.
Siyabonga ngokulalela.
[Uphawu lukhonkotha]
[Umculo]