Libre AI Datasets

rs-parrot commented

2023-11-15 14:11:01 -07:00

(Migrated from spacecruft.org)

See:

https://spacecruft.org/deepcrayon/parrot-datasets

See: https://spacecruft.org/deepcrayon/parrot-datasets

rs-parrot commented

2023-11-15 14:11:09 -07:00

(Migrated from spacecruft.org)

Like the models, many of the training data sets that claim to be "open source" aren't actually using open source licenses.

Find useful truly "open source" data sets.

Note which notable large data sets claim to be "open source", but don't have open source license.

Like the models, many of the training data sets that claim to be "open source" aren't actually using open source licenses. Find useful truly "open source" data sets. Note which notable large data sets claim to be "open source", but don't have open source license.

rs-parrot commented

2023-11-15 14:11:18 -07:00

(Migrated from spacecruft.org)

Might not suck:

"RedPajama is a clean-room, fully open-source implementation of the LLaMa dataset."

https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

Lists these licenses for what they scraped:

License

Please refer to the licenses of the data subsets you use.

Common Crawl Foundation Terms of Use
C4 license
GitHub was limited to MIT, BSD, or Apache licenses only
Books: the_pile_books3 license and pg19 license
ArXiv Terms of Use
Wikipedia License
StackExchange license on the Internet Archive

Might not suck: "RedPajama is a clean-room, fully open-source implementation of the LLaMa dataset." https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T Lists these licenses for what they scraped: License Please refer to the licenses of the data subsets you use. ``` Common Crawl Foundation Terms of Use C4 license GitHub was limited to MIT, BSD, or Apache licenses only Books: the_pile_books3 license and pg19 license ArXiv Terms of Use Wikipedia License StackExchange license on the Internet Archive

rs-parrot commented

2023-11-15 14:11:26 -07:00

(Migrated from spacecruft.org)

Common Crawl is definitely not open source:

https://commoncrawl.org/terms-of-use

Common Crawl is definitely not open source: https://commoncrawl.org/terms-of-use

rs-parrot commented

2023-11-15 14:11:32 -07:00

(Migrated from spacecruft.org)

2023-11-15 10:54:42 -07:00
Poster
Collaborator

https://huggingface.co/datasets/c4

Under the "odc-by" license:

https://opendatacommons.org/licenses/by/
https://opendatacommons.org/licenses/by/1-0/
https://opendatacommons.org/licenses/by/summary/

OSI doesn't have the license listed.

FSF has comments on the Open Data DB license, from the same organization:

"Open Database license (#ODbl)

This is a free and copyleft license meant for data. It is incompatible with the GNU GPL. Please don't use it for software or documentation, since it is incompatible with the GNU GPL and with the GNU FDL. It makes inconvenient requirements about signing contracts which try to create an effect like copyleft for data that is not copyrightable, so we don't recommend using it; however, there is no reason to avoid using data released this way."

I don't see that any outside group has reviewed/approved odc-by as libre. Punt.

2023-11-15 10:54:42 -07:00 Poster Collaborator https://huggingface.co/datasets/c4 Under the "odc-by" license: https://opendatacommons.org/licenses/by/ https://opendatacommons.org/licenses/by/1-0/ https://opendatacommons.org/licenses/by/summary/ OSI doesn't have the license listed. FSF has comments on the Open Data DB license, from the same organization: "Open Database license (#ODbl) This is a free and copyleft license meant for data. It is incompatible with the GNU GPL. Please don't use it for software or documentation, since it is incompatible with the GNU GPL and with the GNU FDL. It makes inconvenient requirements about signing contracts which try to create an effect like copyleft for data that is not copyrightable, so we don't recommend using it; however, there is no reason to avoid using data released this way." I don't see that any outside group has reviewed/approved odc-by as libre. Punt.

rs-parrot commented

2023-11-15 14:11:37 -07:00

(Migrated from spacecruft.org)

"Books" from the pile:

https://huggingface.co/datasets/the_pile_books3#licensing-information

"Defunct: Dataset "the_pile_books3" is defunct and no longer accessible due to reported copyright infringement."

"Books" from the pile: https://huggingface.co/datasets/the_pile_books3#licensing-information "Defunct: Dataset "the_pile_books3" is defunct and no longer accessible due to reported copyright infringement."

rs-parrot commented

2023-11-15 14:11:44 -07:00

(Migrated from spacecruft.org)

pg19 is project gutenberg. Project Gutenberg is based on public domain works, so is libre.

https://huggingface.co/datasets/pg19

The set is licensed Apache 2.0, so is good to use. Just old texts though, not programming...

pg19 is project gutenberg. Project Gutenberg is based on public domain works, so is libre. https://huggingface.co/datasets/pg19 The set is licensed Apache 2.0, so is good to use. Just old texts though, not programming...

rs-parrot commented

2023-11-15 14:11:50 -07:00

(Migrated from spacecruft.org)

Fastest way to download pg19 is ala:

pip install gsutil
gsutil -m cp -r "gs://deepmind-gutenberg/train" .

Fastest way to download pg19 is ala: ``` pip install gsutil gsutil -m cp -r "gs://deepmind-gutenberg/train" .

rs-parrot commented

2023-11-15 14:12:06 -07:00

(Migrated from spacecruft.org)

The compilers of The Pile have a good table about datasets they included.

https://arxiv.org/abs/2101.00027

The compilers of The Pile have a good table about datasets they included. https://arxiv.org/abs/2101.00027

rs-parrot commented

2023-11-15 14:12:09 -07:00

(Migrated from spacecruft.org)

Probably best to use the Pile, but exclude non-libre sections.

rs-parrot commented

2023-11-15 14:12:15 -07:00

(Migrated from spacecruft.org)

Stackoverflow may actually be free to use, this is in their terms:

"From time to time, Stack Overflow may make available compilations of all the Subscriber Content on the public Network (the “Creative Commons Data Dump”). The Creative Commons Data Dump is licensed under the CC BY-SA license. By downloading the Creative Commons Data Dump, you agree to be bound by the terms of that license."

https://stackoverflow.com/legal/terms-of-service/public

Creative Commons licensing terms (CC BY-SA 4.0)

https://archive.org/download/stackexchange/

https://archive.org/download/stackexchange/stackexchange_archive.torrent

Stackoverflow may actually be free to use, this is in their terms: "From time to time, Stack Overflow may make available compilations of all the Subscriber Content on the public Network (the “Creative Commons Data Dump”). The Creative Commons Data Dump is licensed under the CC BY-SA license. By downloading the Creative Commons Data Dump, you agree to be bound by the terms of that license." https://stackoverflow.com/legal/terms-of-service/public Creative Commons licensing terms (CC BY-SA 4.0) https://archive.org/download/stackexchange/ https://archive.org/download/stackexchange/stackexchange_archive.torrent

rs-parrot commented

2023-11-15 14:12:20 -07:00

(Migrated from spacecruft.org)

Also remove non-libre os questions, etc. from stackoverflow and others.

rs-parrot commented

2023-11-15 17:06:15 -07:00

(Migrated from spacecruft.org)

Perhaps rust cargo mirror for dataset.

rs-parrot commented

2023-11-21 09:38:05 -07:00

(Migrated from spacecruft.org)

Huggingface no_robots is non-libre NC.