[Nyimbo]
[Wolengeza] Izi ndi Zothandiza pagulu
[Carl Malamud] Moni, ndine Carl Malamud.
Takulandirani ku General Index.
Tikukuwonetsani pano zaulere komanso zopanda malire gwiritsani ntchito magalamu a n-magalamu ochokera ku 107 miliyoni,
Zolemba 233 zikwi 233, 728.
N-magalamu amachokera m'mawu amodzi, odziwika monga ma unigrams, kumatanthauzira mawu awiri (akulu)
mpaka 5-mawu, omwe amadziwika kuti 5-magalamu.
Pali, zonse zafotokozedwa, mizere 355 biliyoni ya n-magalamu pakutulutsidwa kwa index kwakukulu, aliyense
yofanana ndi kugunda m'nkhani yankhani.
Chida chotchedwa Spacy chidagwiritsidwa ntchito kutulutsa n-magalamu, omwe tagawa
kudutsa mafayilo sikisitini.
Yogwirizana ndi mzere uliwonse ndi md5 hash, yomwe imayimira nkhani yolemba.
Kuphatikiza apo, tikulemba pafupipafupi mafayilo a n-gram m'nkhaniyi.
Tikukhulupirira pakapita nthawi kuti tilembenso pafupipafupi mu Corpus yonse, mtengo woyenera
wotchedwa TD / IDF.
Kuphatikiza pa n-magalamu, General Index muli mawu osakira omwe achotsedwa pogwiritsa ntchito Yake
Yake "amakhala pamanambala owerengera mawonekedwe omwe adatengedwa kuchokera pazolemba limodzi kupita ku
sankhani mawu ofunikira kwambiri. ”
Magulu amawu achinsinsi ofunikira ali ndi okwana 19 biliyoni mawu.
Pomaliza, index yonse imakhala ndi metadata tebulo, ndikujambula md5 hash kuzinthu zotere
adilesi yapadera ya DOI yolemba nkhani, mutu, wolemba, ndi magazini.
Uku ndikutulutsa koyambirira kwa index yonse, ntchito ili mkati.
Nthawi zina, kutulutsa mawu kwalephera.
Nthawi zina, metadata sikupezeka, kapena imapezeka mwina ngakhale zolakwika.
Ngakhale kuti corpus yayikulu ndi yayikulu, ndi osakwanira, komanso sankhani zatsopano.
Pali njira zambiri zomwe zingathandizire, ndipo tikuyembekezera kupanga General
Lembani bwino pakapita nthawi.
Mndandanda wonsewo umakhala pafupifupi 38 ma terabyte amtundu wa Postgres
dambo, komabe mokakamiza, ASCII iyi deta yafupika mpaka 8.5 terabytes.
Mutha kutsitsa deta kuchokera apa, kapena gwiritsani ntchito zida monga bittorrent.
Khalani omasuka kupanga magalasi kapena anthu ena onse zosungira kuti mugawenso index yonse.
Chiyembekezo chathu ndikuti chitha kuyikidwapo za ntchito. Takhala tikugwiritsa ntchito nkhokwe ya Postgres,
ndikugwiritsa ntchito index yonse kusaka Mayina azomera pamndandanda wazambiri.
Mndandanda ungagwiritsidwe ntchito kusaka zina zinthu, monga mankhwala, majini osiyanasiyana
encodings, mapuloteni, zida, mayina amalo, kapena mabungwe ena.
M'malo mosunga database ya Postgres, njira zina, monga BERT, itha kugwiritsidwa ntchito pazambiri.
Ichi ndi chida chofufuzira, dikishonale ya chidziwitso, mapu odziwa, chida chomwe timakhulupirira
ndi malo ofunikira pochita izi za sayansi m'badwo wathu wamakono.
Chidziwitso chonse cha chidziwitso sichipanga mtembo wopezeka, M'malo mwake, zowona
amachokera pamtembo.
Izi sizopanda pake ndi ntchito yosintha.
Tikuwona izi ngati zothandiza pagulu alibe umwini pamndandanda wonsewo.
Ikuperekedwa kwa pagulu, mndandanda zazinthu zopanda umboni zomwe mungathe
chitani zomwe mukufuna.
Palibe ufulu wotetezedwa.
Akatswiri panjira yodziwitsa amatha kufunsa mapu awa kuti mupeze komwe akupita mwachangu
Tili chabe maofesi mu mayendedwe azakale zazikulu kwambiri padziko lapansi,
gulu lonse la akatswiri ndi asayansi ndi mainjiniya ndi ojambula omwe apanga
akachisi awa azidziwitso.
Akachisi awa, ochokera ku Library ya Alexandria kupita ku Laibulale ya ku Alexandra, kuchokera kwa amonke
of Ireland kupita kumalaibulale ambiri amakono ofufuza lero, kuchokera kwa alembi achiarabu ndi achiyuda
akugwirira ntchito limodzi kuti amange Nyumba ya Nzeru ku Baghdad kupita kumafakitole azidziwitso omwe amakhala
m'nyumba zamakono zamakono za East Indian monga makina osindikizira a Oxford ndi Cambridge, awa
ndi komwe akatswiri amayenera kukachezera kupititsa patsogolo luso lawo, kuphunzira kuchokera
iwo omwe adatsogola.
Kachisi uyu ali ndi chidziwitso chonse, koma tiyenera kudziwa izi.
Mndandanda wonse ndi mapu.
Awa si mapu owerengera chidziwitso chonse chaumunthu,
koma ndichinthu chofunikira kwambiri.
Kukula ndi kufalikira kwa chidziwitso kupitilirabe, ngati tikuyenera kuyimirira pamapewa
Ziphona, tiyenera kupereka mapu awa kwa izo dziko lalikulu la malingaliro.
Chizindikiro chonse ndi chida chimodzi, chida zomwe tikuyembekeza kuti popita nthawi zidzapindulanso
ndikukula, chida chomwe tikuyembekeza chomwe chidzakupatseni chopereka chothandiza
ku machitidwe anu asayansi.
Tikukhulupirira kuti mudzaikanso mndandandandawu ntchito zodabwitsa, kuti titha kupanga index
bwino, kuti mutha kupanga zida zina, kuti gulu la akatswiri limatha kugwira ntchito limodzi
mufuna komwe kumafikira kuzama kwakukulu za nyanja iyi yophunzirira.
Ndikukhulupirira tonse tili ndi cholinga chimodzi, chiyembekezo chofala.
Monga asayansi, akatswiri, ojambula, olemba, monga nzika za dziko lapansi, monga ana ofunira
kuti ndiphunzire, monga anthu achidwi, ndikukhulupirira Tonsefe tikuyembekeza kuti chiwonjezeko ndikufalikira
ya chidziwitso idzapangitsa dziko lathu kukhala labwino malo, kuti zitithandize kumvetsetsa zathu
world, kuchiza matenda ndi umphawi, kupita patsogolo kupita patsogolo kwa sayansi ndi malonda ndi
zaumunthu ndi zaluso.
Sayansi ndi chilankhulo chomwe tonsefe tiyenera kuyankhula ngati tifunika kukonza dziko lathu.
Zikomo pomvera.
[Chisindikizo Kukuwa]
[Nyimbo]