[Music]
[Announcer] This is Public Resource
[Carl Malamud] Hello, I’m Carl Malamud.
Welcome to the General Index.
We present here for your free and unlimited use a set of n-grams extracted from 107 million,
233 thousand, 728 journal articles.
The n-grams range from single words, known as unigrams, to two-word phrases (bigrams)
up to 5-word phrases, known as 5-grams.
There are, all told, 355 billion rows of n-grams in this release of the general index, each
corresponding to a hit in a journal article.
A toolkit called Spacy was used to extract the n-grams, which we have split
across sixteen files.
Associated with each row is an md5 hash, which represents a journal article.
In addition, we list the frequency of the n-gram in the article.
We hope over time to also list the frequency in the overall corpus, a relevance metric
known as TD/IDF.
In addition to the n-grams, the General Index contains keywords extracted using the Yake
toolkit.Yake “rests on text statistical features extracted from single documents to
select the most important keywords of a text.”
The general index keyword tables contains a total of 19 billion keywords.
Lastly, the general index contains a metadata table, mapping the md5 hash to items such
the DOI unique address for a journal article, the title, author, and journal.
This is an early release of the general index, a work in progress.
In some cases, text extraction failed.
Sometimes, metadata is not available, or is perhaps even incorrect.
While the underlying corpus is large, it is not complete, and is not up-to-date.
There are many ways this can be improved, and we look forward to making the General
Index better over time.
The general index occupies approximately 38 terabytes of data in the form of a Postgres
dump, however with compression, this ASCII data is reduced to 8.5 terabytes.
You can download the data directly from here, or use tools such as bittorrent.
Feel free to create mirrors or other public repositories to redistribute the general index.
Our hope is that it can be put to a number of uses. We have been using a Postgres database,
and using the general index to search for names of plants from a master list.
The index could be used to search for other objects, such as chemicals, genes in various
encodings, proteins, materials, place names, or other entities.
Instead of a Postgres database, other techniques, such as BERT, can be applied to the data.
This is a lookup tool, a dictionary of knowledge, a map to knowledge, a tool that we believe
is an essential facility for the practice of science in our modern age.
This general index to knowledge does not make the underlying corpus available, Rather, facts
are extracted from the corpus.
This is a nonconsumptive and transformational use.
We view this as a public utility.We assert no ownership over the general index.
It is dedicated to the public domain, a series of unencumbered facts with which you can
do what you will.
There are no rights reserved.
Scholars on the road to discovery can consult this roadmap to more quickly find destinations
they wish to visit.We are but docents in the hallways of the vast archives of the world,
the collective community of scholars and scientists and engineers and artists that have created
these temples of knowledge.
These temples, from the Library of Alexandria to the Library of Alexandra, from the monks
of Ireland to the vast modern research libraries of today, from the Arab and Jewish scribes
working together to build the House of Wisdom in Baghdad to the factories of knowledge housed
in modern East Indian trading houses like the presses of Oxford and Cambridge, these
are the destinations that scholars must visit to advance their craft, to learn from
those that came before us.
These temples contain the sum of all knowledge, but we must be able to find that knowledge.
The general index is a map.
This is not a map to the sum of all human knowledge,
but it is certainly a substantial subset.
For the increase and diffusion of knowledge to continue, if we are to stand on the shoulders
of giants, we must provide these maps to that vast world of ideas.
The general index is but one tool, a tool that we hope over time will gain in quality
and in scope, a tool we hope that will provide a useful contribution
to your practice of science.
We hope you will put this general index too surprising uses, that we can make the index
better, that you can create other tools, that the community of scholars can work together
in a common quest to mine the vast depths of this ocean of learning.
I believe we all share a common goal, a common hope.
As scientists, scholars, artists, writers, as citizens of the world, as children wishing
to learn, as curious human beings, I believe we all hope that the increase and diffusion
of knowledge will make our world a better place, that it will help us understand our
world, to cure disease and poverty, to advance the progress of science and commerce and the
humanities and the arts.
Science is a language we must all speak if we are to better our world.
Thanks for listening.
[The Seal is Barking]
[Music]