[Mumhanzi]
[Mushamarari] Ichi Chinyorwa cheRuzhinji
[Carl Malamud] Mhoroi, ndiri Carl Malamud.
Kugamuchirwa kuGeneral Index.
Isu tinopa pano kune ako emahara uye asingagumi shandisa seti yen-gramu yakatorwa kubva pamamirioni 107,
233 zviuru, 728 zvinyorwa zvinyorwa.
Iyo n-magiramu anotangira kubva kune rimwe chete mazwi, anozivikanwa seunigrams, kune maviri-mazwi mitsara (makuru)
anosvika mashanu-mazwi mitsara, inozivikanwa se5-gramu.
Iko kune, zvese zvataurwa, 355 bhiriyoni mitsara ye n-magiramu mune uku kuburitswa kweyakajairika indekisi, imwe neimwe
inoenderana nekurohwa mune chinyorwa chinyorwa.
Toolkit inonzi Spacy yakashandiswa kuburitsa iyo n-gramu, iyo yatakatsemura
pamafaera gumi nematanhatu.
Yakabatana nemutsara wega wega ndeye md5 hash, iyo inomiririra chinyorwa chemagazini.
Uye zvakare, isu tinonyora kuwanda kweiyo n-gram muchinyorwa.
Isu tinovimba nekufamba kwenguva kuti tinyorewo iyo frequency mune iyo yakazara corpus, kukosha kwemetriki
inozivikanwa seTD / IDF.
Pamusoro peiyo n-magiramu, iyo General Index ine mazwi akakosha akatorwa uchishandisa Yake
Yake "inotsamira pane zvinyorwa zvinyorwa maficha akaburitswa kubva kune rimwe gwaro kuenda ku
sarudza mazwi akakosha ezvinyorwa. ”
Iyo yakajairika index kiyiwordword matafura ane anokwana mabhiriyoni gumi nematanhatu mazwi akakosha.
Chekupedzisira, iyo general index ine metadata tafura, mepu iyo md5 hash kuzvinhu zvakadaro
iyo DOI yakasarudzika kero yemagazini chinyorwa, musoro wacho, munyori, uye magazini.
Uku ndiko kuburitswa kwekutanga kweiyo general index, basa riri kuenderera mberi.
Mune zvimwe zviitiko, kuburitsa zvinyorwa kwakundikana.
Dzimwe nguva, metadata haiwanikwe, kana iri pamwe zvisiri izvo.
Ipo iyo yepasi iyo kosi iri hombe, iri isina kukwana, uye haina kufambirana.
Kune nzira dzakawanda izvi zvinogona kuvandudzwa, uye isu tinotarisira kuita General
Index zvirinani nekufamba kwenguva.
Iyo general index inobata ingangoita makumi matatu neshanu terabytes yedhata muchimiro chePostgres
dump, zvakadaro nekumanikidza, iyi ASCII data rakaderedzwa kusvika 8.5 terabytes.
Unogona kurodha pasi data kubva pano, kana kushandisa maturusi akadai sebittorrent.
Inzwa wakasununguka kugadzira magirazi kana imwe yeruzhinji repositories kugovera patsva index.
Tariro yedu ndeyekuti inogona kuiswa kunhamba zvekushandisa. Tanga tichishandisa dhatabhesi yePostgres,
uye kushandisa iyo general index kutsvaga mazita ezvirimwa kubva kune tenzi runyorwa.
Iyo indekisi inogona kushandiswa kutsvaga imwe zvinhu, zvakaita semakemikari, majini mune akasiyana
encodings, mapuroteni, zvishandiso, mazita enzvimbo, kana zvimwe zvinhu.
Panzvimbo pePostgres dhatabhesi, mamwe maitiro, senge BERT, inogona kuiswa kune iyo data.
Ichi chishandiso chekutarisa, duramazwi reruzivo, mepu yekuziva, chishandiso chatinotenda
inzvimbo yakakosha yekudzidzira yesainzi munguva yedu yemazuva ano.
Iyi general index kune zivo haigadzi iyo yepasi kosi inowanikwa, Asi, chokwadi
inotorwa kubva ku corpus.
Uku hakusi kwekufungidzira uye shanduko yekushandisa.
Isu tinoona izvi sechishandiswa paruzhinji hapana muridzi pamusoro peiyo general index.
Iyo yakatsaurirwa kune yeruzhinji domain, akateedzana yezvinhu zvisina kuverengwa zvaunogona nazvo
ita zvaunoda.
Iko hakuna kodzero dzakachengetedzwa.
Nyanzvi dziri munzira yekutsvaga dzinogona kubvunza iyi mepu yemugwagwa kukurumidza kuwana kwaunosvika
isu tinongoda kushanya.Tiri kungoita maitiro mu nzira dzekuchengetedza kukuru kwepasirese,
nharaunda yakabatana yevadzidzi uye vesaenzi uye mainjiniya uye maartist akasika
aya matembere eruzivo.
Aya matembere, kubva kuLaibhurari yeAlexandria kune Raibhurari yeAlexandra, kubva kumamongi
yeIreland kumaraibhurari akakurisa ekutsvagisa yanhasi, kubva kune vanyori vechiArabhu nevechiJuda
kushanda pamwe chete kuvaka Imba yeUchenjeri muBaghdad kumafekitori eruzivo akavakirwa
mune dzazvino East Indian dzimba dzekutengesa senge michina yekuOxford neCambridge, aya
ndidzo nzvimbo dzinofanirwa kushanyirwa nevadzidzi kufambisa hunyanzvi hwavo, kudzidza kubva
avo vakauya pamberi pedu.
Tembere idzi dzine huwandu hweruzivo rwese, asi isu tinofanirwa kukwanisa kuwana iyo zivo.
Iyo general index imepu.
Iyi haisi mepu kune huwandu hweruzivo rwevanhu,
asi zvirokwazvo chakadzama subset.
Nekuwedzera uye kupararira kwezivo kuenderera mberi, kana tichifanira kumira pamafudzi
yemahofori, isu tinofanirwa kupa aya mepu kune izvo nyika yakakura yemazano.
Iyo indekisi yakajairika ingori chishandiso chimwe, chishandiso iyo yatinotarisira nekufamba kwenguva ichawana muhunhu
uye muchiyero, chishandiso chatinotarisira kuti chinopa mupiro unobatsira
kune yako tsika yesainzi.
Tinovimba iwe uchaisa iyi general index zvakare mashandisiro anoshamisa, kuti isu tinogona kugadzira index
zvirinani, kuti iwe ugone kugadzira zvimwe zvishandiso, izvo nharaunda yevadzidzi inogona kushanda pamwechete
mukutsvaga kwakafanana kwekuchera kwakadzika kwakadzika yegungwa iri rekudzidza.
Ndinotenda isu tese tinogovana chinangwa chimwe, tariro yakajairika.
Semasayendisiti, vadzidzi, maartist, vanyori, sevagari venyika, sevana vanoda
kudzidza, sevanhu vanoda kuziva, ndinotenda isu tese tinotarisira kuti kuwedzera nekupararira
yeruzivo ichaita kuti nyika yedu ive nani nzvimbo, kuti ichatibatsira kuti tinzwisise yedu
nyika, kurapa chirwere nehurombo, kufambira mberi kufambira mberi kwesainzi nekutengeserana uye iyo
vanhu uye nehunyanzvi.
Sayenzi mutauro watinofanira kutaura tese kana isu tiri nani nyika yedu.
Kutenda nekuteerera.
[Chisimbiso chiri Kuhukura]
[Mumhanzi]