DistribNet A global peer-to-peer Internet file system in which anyone can tap into or add content to. http://distribnet.sourceforge.net/ Release Notes So far distribnet has only been tested on my computer which is RedHat 8.0 (Kernel 2.4). Other linux systems should work. Other POSIX systems might work. Forget Win32 for now. Distribnet requires: GNU Libc (I use a few thread safe extensions) Gcc 3.1 or better (Gcc 2.95 may work) POSIX Threads Berkeley DB Version 4 or better OpenSSL (I use version 0.9.6b) Perl (for the scipts) A Web Browser DistribNet has never actually been tested over a network (I launch all the nodes on the same computer). However, it should work provided all the machines are of the same architecture (or at least the use the same endian order) as I do not currently serialize things correctly. Quick Start ./configure make (do not do a "make install") . scripts/setup-path # To add src/ and scripts/ to your path mkdir run # or any other empty directory cd run mk-nodes.sh # to create the environment for 64 nodes doit.sh # to launch 64 nodes To see the routing table in the first node: http://localhost:30001/route.html Any halfway decent web browser will work (mozilla, netscape, w3m, links, etc. but not lynx). To see for any node http://localhost:/route.html Where is an odd number between 30001 and 30127. When the nodes are first starting up the routing table will change frequently so you should refresh the page every minute or so. (The even ports are used for node communication) To submit a file http://localhost:/submit// or http://localhost:/submit/ Where the relative path is relative to run/ or wherever you started doit.sh. For example to submit the file 'lst' http://localhost:30127/submit/lst The key for the file will be returned to you which is the SHA-1 hash of the file's contents. To retrive the file. http://localhost:/data/. The extension is important so that the web browser knows what type of file it is. For example http://localhost:30003/data/922c76ccc3e4680dc0e06936528c966b06bdf405.txt The key used is the key for 'lst'. YOUR KEY WILL BE DIFFERENT. To shutdown all the nodes. killall -w distribnet To shutdown and reset the environment (so for example all submited files will be removed) reset.sh If for some reason killall or reset.sh takes forever than: killall -w -KILL distribnet Each distribnet node will take up around 3 Megs and use 4 threads plus additional "pop-up" threads to handle incoming network connections. Linux-threads will add an additional helper thread making the total number of permanent threads 5. To see what distribnet is doing have a look at the file 'log'. To launch more (or less) nodes modify mk-nodes.sh and doit.sh accordingly. The files are in the scripts directory. See the file docs/design.* for an cource overview of distribnet. Status and Future The routing table is pretty much done. Future versions will use a better selection algorithm for determining which nodes to use. Data keys are implemeted. However, no caching is done. Other key types are not. The protocol is subject to change. Absolutly no garantee will be made that different versions of DistribNet will be compatable with each other. The photocol will stabilize once I get the basics done. Absolutely no security. Once I get all the basics done I work on adding security, which includes encrypting all communication. Integers are not properly converted to network order. This will be done at the same time the photocol is being stabilized. Basically until I get the basics done, don't expect to be able to use DistribNet for anything other than testing. However, please do test it. Since DistribNet is my Masters Theseus I will be working nearly full time on it for the next couple of months. However, I would defiantly appreciate help with the implementation. General ideas, of course, are also welcome. Please post them to DistribNet-devel at lists.sourceforge.net and not to me directly so others can benefit from our discussion. Layout of route.html The file route.html contains a dump of the routing table for DistribNet. The first line is the key for the current node. After that the contents of each row in the routing table. With 64 nodes the first row should be full. While the others are leaf rows. (see design documentation for more detail). For each row the the number of full entries is given. Than an estimated count is given. The count for row 0 is the estimated count for all the rows on the network. For the other rows it is the number of nodes with also have a key with the same x number of digits where x is the row number. The second number on the count line is the estimated accuracy of the count. The remaing entries for the row represents the count of the number of nodes for each entry in the row. The first number is the number of online nodes while the second is the number of offline nodes. After the contents of rows each node in the routing table is listed in lexicographic order. The contents of each entry is as follows: * The node key. * Last Known Online: The time (as given by time(0)) the node was last known to be online. * Checks Since Offline: Number of failed attempts to contact the node. * Position: The position this node is in the routing table. * Num On Row: Number of entries filled in its row x. A -1 means that the node has not been contacted yet. If the row is a "Full Row" than: * Count: The count for its row x+1. * Accuracy: The accuracy estimate for the count Otherwise * The text "Leaf Pos." * Leaf Set Size: The number of nodes in its leaf set. A 0 means that the node has not been contacted yet. * Accuracy: The accuracy estimate. Used to tell if the leaf set size is stable or still growing. Http Commands route.html: Get the reouting table submit/: Submit a file as a data key. data/.: Retrieves a data key node/: Get information on a node skey/.: Get a simple key. HTTP POST: submit-skey/key: Submit a simple key. (Use the distribnet executable) Simple keys are stored based on the 40 digit hex key and are designed to test DistribNet routing. They will eventually be removed. DistribNet exec usage distribnet [options] is one of: -?|help display the help message [dump] config dumps the current configuration to stdout config prints the current value of an option init setup the environment. Use to create a new node. server start distribnet in server mode submit-skey submit a simple key [options] is any of the following: -h,--host= -p,--port= --http-port= -d,--basedir= Scripts Usage doit.sh: Launch the nodes and redirect all output to a log file. By default 64 nodes are launched. To change this modify the variable NUM_NODES at the top of the script. The nodes are launched in the order they are given in the file 'lst' run.sh : Launch a distribnet server. reset.sh: Kill all active distribnet severs and reset all nodes to the orignal settings. mk-nodes.sh: Create the nodes. It will also create the file 'lst' which is the list of the nodes in the order they will created. Do not delete this file, other scripts need it. By deafult 64 nodes are created, to create more or less modify the variable NUM_NODES at the top of the script. clean_db.sh: Remove all database files rand.pl: Setup the initial entries in the routing table to bootstrap the network. Each node's routing table will get 2 nodes already on the network if the nodes are launched in the order of the file 'lst' match.pl: Setup the initial entries in the routing table to bootstrap the network. Each node's routing table will contain the node previous to it according to the file lst'. Node file layout All node information is stored in the directory: s- Where key is the nodes key. The directory is layed out as follows: node.dat: Local Node information conf.dat: Nodes configuration file routing_info.dat: Routing table databases/: Where all files for Berkeley DB is stored data_files/: Where keys are stored if not in the DB -- Kevin Atkinson (kevin at atkinson dhs org)