***********************************************************************************************************
*
* Multi source locator service suggestion
* Read http://locator.tfr.org/wiki/ for a full feature desciption
*
* By Fredrik Neij - fredrik at tfr dot org
*
*   Everything contained within this document is released into the public domain,
*   including the example data and code at http://locator.tfr.org/
*
***********************************************************************************************************


Using a script to create these files from a existing torrent is only a last resort, to keep this up a locator service 
would have to download and verify alot of torrents to have accurate contents. My hope is that when/if using these 
services becomes a common practice, torrent makers would include one or more hash sum by default when creating new 
torrents, and the service would only have to extract the information from the torrent. Keeping the hash-data in the 
info-dict would also prevent tampering with this data.

Yes it will take a little time longer to make torrents if you calculate one or two more file-hashes per file, but thats 
not really a problem on a modern computer, and (imho) the advantages this gives to find alternative sources to complete 
the download of the content, by far outweighs this slightly longer creation time.

By using standard hash types like sha{1,256,512},md5 and tth to identify the files, the alternative source don't 
necessarily have to be within a BitTorrent swam, but could be another p2p service or a ftp/http service.


For this example I created 4 files with:
 dd if=/dev/urandom bs=1M count=$SIZE of=$FILE

and included them in different torrents to provide an example of the same file existing in several swarms at
the same time. To show how this suggested locator service can help to locate the same file within multiple 
torrents. I also created hash values, and the necessary meta-data to download each file separately 
from each torrent.

 torrent-1.torrent
  torrent-1/file-1.bin
  torrent-1/file-2.bin
 
 torrent-2.torrent
  torrent-2/file-2.bin
  torrent-2/file-3.bin
 
 torrent-3.torrent
  torrent-3/file-1.bin
  torrent-3/file-3.bin
  torrent-3/file-4.bin
 
 file-4.bin.torrent
  file-4.bin

by running the "make_metadata.php" php-script on each torrent-file i created the following data files:

 hashes/4A18DF30601FFEE25BE248D709DF11F8AA5242BA.btih
 hashes/61D9AA673382DB60106BC2B16BBE1FC30CE6E3B2.btih
 hashes/A0FD3C657A1BA7A525CE1DE4F258DF2F2A5D1ED4.btih
 hashes/E85D8135E9D37A3F2947710CF9A2E32198B0A34F.btih
 
 hashes/042CC19DB4CF6998E43770E584543B1C9AD18E46.sha1
 hashes/167372349BA30C089DAFB937B9F0B0BC5E9A4049.sha1
 hashes/2AC7A5E2CACB9387C2A2B02465FF6F94A18244F2.sha1
 hashes/9859B44352BD09AF2CAC936337C89EB80250817E.sha1
 
 hashes/5042BBC97C5847BFCA94A394FA56E042.md5
 hashes/5DB153681AE3725D8C59ED9683DD77FD.md5
 hashes/D418A1227966653A49AC7AFF2DC8FB29.md5
 hashes/F1E37732491E271149E89360CB1C0AB0.md5


With the rewrite rules from /lighttpd.conf i serve the files according to the rules in the /locator.txt file.

With an average of 22.56 files per torrent, and the average torrent-file size of 25.38kb (collected from 2.1 million 
files stored at torrage) The number of files and space required can grow quite a bit, to 100M files and 1TB (compressed) 
of storage, if 100% of the torrent files were indexed with hashes. To optimize a service like this files could be stored 
gzipped, and sent to the client with the header "Content-Encoding: gzip". Storing the files in gzip format would require 
slightly more resources at creation and modification, but would save alot of resources compared to compressing on the 
fly, aswell as saving alot of bandwidth compared to sending the responses uncompressed.

This practice have been tested at torrage, by storing and delivering .torrent files in the same manner, so far no major 
torrent client or webb-browser have failed to receive the gzip encoded data. But to keep this example simple I have left 
everything uncompressed.