[Umuziki]
[Uwatangaje] Ubu ni umutungo rusange
[Carl Malamud] Mwaramutse, Ndi Carl Malamud.
Murakaza neza kurutonde rusange.
Turerekana hano kubuntu kandi butagira imipaka koresha urutonde rwa n-garama yakuwe muri miliyoni 107,
Ibihumbi 233, 728 ingingo zamakuru.
N-garama itandukanijwe namagambo amwe, azwi nka unigrams, kumagambo abiri yamagambo (bigrams)
kugeza kumagambo 5 yamagambo, azwi nka garama 5.
Hariho, byose byavuzwe, miliyari 355 umurongo wa n-garama muri uku kurekura urutonde rusange, buri
bihuye na hit mu kiganiro cyikinyamakuru.
Igitabo cyitwa Spacy cyakoreshejwe mugukuramo n-garama, twatandukanije
hejuru ya dosiye cumi na gatandatu.
Yifatanije na buri murongo ni md5 hash, iyo ahagarariye ikinyamakuru.
Mubyongeyeho, turondora inshuro za n-garama mu ngingo.
Turizera ko igihe kizagera no gutondekanya inshuro muri rusange corpus, ibipimo bifatika
uzwi nka TD / IDF.
Usibye n-garama, Indangagaciro rusange ikubiyemo ijambo ryibanze ryakuweho ukoresheje Yake
toolkit.Yake “ishingiye ku mibare y'inyandiko ibiranga byakuwe mu nyandiko imwe kugeza
hitamo ijambo ry'ingenzi ry'ingenzi ry'inyandiko. ”
Imbonerahamwe rusange yijambo ryibanze imbonerahamwe ikubiyemo yose hamwe miliyari 19 zijambo ryibanze.
Ubwanyuma, urutonde rusange rurimo metadata imbonerahamwe, gushushanya md5 hash kubintu nkibyo
aderesi idasanzwe ya DOI kubiganiro byikinyamakuru, umutwe, umwanditsi, n'ikinyamakuru.
Nibisohoka hakiri kare urutonde rusange, umurimo urimo gukorwa.
Rimwe na rimwe, gukuramo inyandiko byarananiranye.
Rimwe na rimwe, metadata ntabwo iboneka, cyangwa ni birashoboka ko atari byo.
Mugihe corpus iri munsi nini, ni ntabwo byuzuye, kandi ntabwo bigezweho.
Hariho inzira nyinshi ibi bishobora kunozwa, kandi dutegereje gukora Jenerali
Ironderero neza mugihe runaka.
Ibipimo rusange bifata hafi 38 terabaýt yamakuru muburyo bwa Postgres
guta, icyakora hamwe no kwikuramo, iyi ASCII amakuru yagabanutse kugera kuri 8.5 terabaýt.
Urashobora gukuramo amakuru kuva hano, cyangwa ukoreshe ibikoresho nka bittorrent.
Wumve neza ko ukora indorerwamo cyangwa izindi rusange ububiko bwo kugabura indangagaciro rusange.
Ibyiringiro byacu nuko bishobora gushyirwa kumubare y'ikoreshwa. Twagiye dukoresha ububiko bwa Postgres,
no gukoresha urutonde rusange kugirango ushakishe amazina y'ibimera kuva kurutonde rwibanze.
Ironderero rishobora gukoreshwa mugushakisha izindi ibintu, nk'imiti, genes muburyo butandukanye
kodegisi, poroteyine, ibikoresho, amazina y’ahantu, cyangwa izindi nzego.
Aho kuba ububiko bwa Postgres, ubundi buhanga, nka BERT, irashobora gukoreshwa kumakuru.
Iki nigikoresho cyo gushakisha, inkoranyamagambo yubumenyi, ikarita y'ubumenyi, igikoresho twemera
ni ibikoresho by'ingenzi mu myitozo ya siyansi muri iki gihe cyacu.
Iyi ngingo rusange yubumenyi ntabwo ikora umurambo wibanze urahari, Ahubwo, ukuri
bakuwe muri corpus.
Ibi ntibisanzwe no gukoresha impinduka.
Turabibona nkibikorwa rusange.Turemeza nta nyirubwite hejuru yibipimo rusange.
Yeguriwe urwego rusange, urukurikirane by'ibintu bitabarika ushobora kubikora
kora icyo ushaka.
Nta burenganzira bwihariwe.
Intiti kumuhanda wo kuvumbura zirashobora kugisha inama iyi igishushanyo mbonera kugirango byihuse ubone aho ujya
bifuza gusura.Turi ariko docents muri koridoro yububiko bunini bwisi,
umuryango rusange w'intiti n'abahanga n'abashakashatsi n'abahanzi baremye
izi nsengero zubumenyi.
Izi nsengero, ziva mu isomero rya Alegizandiriya ku isomero rya Alexandra, uhereye ku bamonaki
ya Irilande kugeza ku masomero manini yubushakashatsi y'uyu munsi, uhereye ku banditsi b'Abarabu n'Abayahudi
gukorera hamwe kubaka Inzu y'Ubwenge i Bagidadi ku nganda zubumenyi zubatswe
mumazu yubucuruzi yubuhinde bugezweho nka imashini za Oxford na Cambridge, izi
niho intiti zigomba gusura guteza imbere ibihangano byabo, kugirango bige
abaje imbere yacu.
Izi nsengero zirimo igiteranyo cyubumenyi bwose, ariko tugomba gushobora kubona ubwo bumenyi.
Igipimo rusange ni ikarita.
Iyi ntabwo ari ikarita yumubare wubumenyi bwabantu bose,
ariko rwose ni igice kinini.
Kwiyongera no gukwirakwiza ubumenyi gukomeza, niba dushaka guhagarara ku bitugu
y'ibihangange, tugomba gutanga amakarita kuriyo isi nini y'ibitekerezo.
Igipimo rusange ni igikoresho kimwe, igikoresho ko twizera ko igihe kizagenda cyunguka ubuziranenge
no mubipimo, igikoresho twizera ko kizatanga umusanzu w'ingirakamaro
kubikorwa byawe bya siyansi.
Turizera ko uzashyira iyi ngingo rusange gukoresha bitangaje, ko dushobora gukora indangagaciro
byiza, ko ushobora gukora ibindi bikoresho, ibyo umuryango wintiti urashobora gukorera hamwe
mubushakashatsi rusange bwo gucukura ubujyakuzimu y'iyi nyanja yo kwiga.
Nizera ko twese dusangiye intego, ibyiringiro rusange.
Nkabahanga, intiti, abahanzi, abanditsi, nk'abenegihugu b'isi, nk'abana babishaka
kwiga, nkabantu bafite amatsiko, ndizera twese twizera ko kwiyongera no gukwirakwizwa
y'ubumenyi bizatuma isi yacu iba nziza ikibanza, ko bizadufasha gusobanukirwa ibyacu
isi, gukiza indwara nubukene, gutera imbere iterambere rya siyanse nubucuruzi nu
ubumuntu n'ubuhanzi.
Siyanse ni ururimi tugomba kuvuga niba tugomba kuzamura isi yacu.
Urakoze kumva.
[Ikidodo kirasimba]
[Umuziki]