New streaming.html, more faq entries, other stuff

www.i2p2/pages/api.html

@@ -1,11 +1,11 @@
 {% extends "_layout.html" %}
-{% block title %}Home{% endblock %}
+{% block title %}API{% endblock %}
 {% block content %}
 <p>Application developers should review the <a href="applications">application 
 development guide</a> for ideas about the four basic protocols for writing an
 application to run over I2P:</p>
 <ul>
-<li><a href="sam">SAM</a></li>
+<li><a href="sam">SAM</a></li> and <a href="samv2.html">SAM V2</a></li>
 <li><a href="ministreaming">ministreaming</a></li>
 <li><a href="i2cp">I2CP</a></li>
 <li><a href="datagrams">Datagrams</a></li>

www.i2p2/pages/applications.html

@@ -1,7 +1,7 @@
 {% extends "_layout.html" %}
-{% block title %}Applications{% endblock %}
+{% block title %}Application Development{% endblock %}
 {% block content %}
-<h1>Application development guide</h1>
+<h1>Application Development Guide</h1>
 
 <h2>Why write I2P specific code?</h2>
 
@@ -28,7 +28,7 @@ chat, everyone needs to agree that port 10001 is Alice, port 10002 is Bob, port
 <p>Applications that are designed to work with I2P can take advantage of its 
 built in data security and optional pseudonymous authentication.  All data sent 
 over the network is transparently end to end encrypted (not even the routers
-get the cleartext), and any application using the ministreaming or datagram 
+get the cleartext), and any application using the streaming or datagram 
 functionality has all of that data authenticated by the sending destination's 
 public key.  As an aside, environments where anonymity instead of pseudonymity 
 is required are trivially accomodated by either using the I2CP directly, SAM RAW 
@@ -45,7 +45,7 @@ the traditional application base, but design the application and its communicati
 protocols with I2P's anonymity and security in mind.</p>
 
 <p>There are also efficiency considerations to review when determining how to 
-interact on top of I2P.  The ministreaming library and things built on top of it
+interact on top of I2P.  The streaming library and things built on top of it
 operate with handshakes similar to TCP, while the core I2P protocols (I2NP and I2CP)
 are strictly message based (like UDP or in some instances raw IP).  The important
 distinction is that with I2P, communication is operating over a long fat network - 
@@ -53,13 +53,26 @@ each end to end message will have nontrivial latencies, but may contain payloads
 of up to 32KB.  An application that needs a simple request and response can get rid
 of any state and drop the latency incurred by the startup and teardown handshakes
 by using (best effort) datagrams without having to worry about MTU detection or 
-fragmentation of messages under 32KB.  The ministreaming library itself uses a 
+fragmentation of messages under 32KB. 
+</p>
+
+<p>
+The ministreaming library itself uses a 
 functional but inefficient scheme for dealing with reliable and in order delivery
 by requiring the equivilant of an ACK after each message which must traverse the 
 network end to end again (though there are plans for improving this with a more 
-efficient and robust algorithm).  Given that as the current state, an application
-that uses one of the I2P message oriented protocols can in some situations get 
-substantially better performance.</p>
+efficient and robust algorithm).  With ministreaming, an application
+that uses one of the I2P message oriented protocols could in some situations get 
+substantially better performance.
+However with the full streaming library now the standard interface,
+it isn't clear if that is still the case.
+</p>
+
+<p>
+Applications written in Java and accessible/runnable
+using an HTML interface via the standard webapps/app.war
+may be considered for inclusion in the i2p distribution.
+</p>
 
 <h2>Important ideas</h2>
 
@@ -84,14 +97,16 @@ must stay on a single host.</li>
 public key for encryption, a 1024bit DSA public key for signing, and a variable size 
 certificate (currently this is the null type, but may contain proof of work, blinded
 data, or other information to increase the 'cost' of a destination in an effort to fight
-Sybil).  <br />There are existing ways to refer to these large and ugly destinations by short
+Sybil).  <p>There are existing ways to refer to these large and ugly destinations by short
 and pretty names (e.g. "irc.duck.i2p"), but at the moment those techniques do not guarantee
 globally uniqueness (since they're stored locally at each person's machine as "hosts.txt")
-and the current mechanism is neither scalable nor secure (updates to those hosts files are 
-manually managed within CVS, and as such, anyone with commit rights on the repository can
+and the current mechanism is not especially scalable nor secure (updates to one host file is
+manually managed within Monotone, and as such, anyone with commit rights on the repository can
 change the destinations).  There may be some secure, human readable, scalable, and globally 
 unique, naming system some day, but applications shouldn't depend upon it being in place,
-since there are those who don't think such a beast is possible :)</li>
+since there are those who don't think such a beast is possible.
+<a href="naming.html">Further information on the naming system</a> is available.
+</li>
 </ul>
 
 <h3>Anonymity and confidentiality</h3>
@@ -119,10 +134,20 @@ application without having to write fragmentation, resends, etc.</p>
 
 <h2>Integration techniques</h2>
 
-<p>There are four means of sending data over I2P, each with their own pros and cons.</p>
+<p>There are several means of sending data over I2P, each with their own pros and cons.
+The streaming lib is the recommended interface, used by the majority of I2P applications.
+</p>
 
-<h3>SAM</h3>
+<h3>Streaming Lib</h3>
+<p>
+The <a href="streaming.html">full streaming library</a> is now the standard
+interface. Ministreaming and SAM are not recommended.
+The streaming lib interface is similar to the ministreaming lib interface described below.
+</p>
 
+<h3>SAM and SAM V2</h3>
+
+<p><i>SAM is not recommended. SAM V2 is beta.</i></p>
 <p>SAM is the <a href="sam">Simple Anonymous Messaging</a> protocol, allowing an
 application written in any language to talk to a SAM bridge through a plain TCP socket and have
 that bridge multiplex all of its I2P traffic, transparently coordinating the encryption/decryption
@@ -134,6 +159,12 @@ and event based handling.  SAM supports three styles of operation:</p>
     and Bob doesn't care whether the data's sender is authenticated or not (e.g. the data transferred
     is self authenticating)</li>
 </ul>
+</p>
+
+<p>
+<a href="samv2.html">SAM V2</a> is a new version used by imule
+that fixes some of the problems in <a href="sam.html">SAM</a>.
+</p>
 
 <h3>I2PTunnel</h3>
 <p>The I2PTunnel application allows applications to build specific TCP-like tunnels to peers 
@@ -145,7 +176,8 @@ authenticated and secured through the same streaming library that SAM uses, but
 nontrivial overhead involved with creating multiple unique I2PTunnel instances, since each have
 their own unique I2P destination and their own set of tunnels, keys, etc.</p>
 
-<h3>ministreaming and datagrams</h3>
+<h3>Ministreaming</h3>
+<p><i>Not recommended</i></p>
 <p>For applications written in Java, the simplest way to go is to use the libraries that the SAM
 bridge and I2PTunnel applications use.  The streaming functionality is exposed in the 'ministreaming'
 library, which is centered on the 
@@ -153,6 +185,8 @@ library, which is centered on the
 the <a href="http://www.i2p.net/javadoc/net/i2p/client/streaming/I2PSocket.html">I2PSocket</a>, and the
 <a href="http://www.i2p.net/javadoc/net/i2p/client/streaming/I2PServerSocket.html">I2PServerSocket</a>.</p>
 
+<h3>Datagrams</h3>
+<p><i>Not recommended</i></p>
 <p>For applications that want to use repliable datagrams, they can be built with the 
 <a href="http://www.i2p.net/javadoc/net/i2p/client/datagram/I2PDatagramMaker.html">I2PDatagramMaker</a>
 and parsed on the receiving side by the 
@@ -169,9 +203,45 @@ then fetching those messages by calling
 <a href="http://www.i2p.net/javadoc/net/i2p/client/I2PSession.html#receiveMessage(int)">receiveMessage(...)</a>.</p>
 
 <h3>I2CP</h3>
-
+<p><i>Not easy</i></p>
 <p><a href="i2cp">I2CP</a> itself is a language independent protocol, but to implement an I2CP library 
 in something other than Java there is a significant amount of code to be written (encryption routines, 
 object marshalling, asynchronous message handling, etc).  While someone could write an I2CP library in 
-C or something else, it would most likely be more useful to use the C SAM library instead. </p>
+C or something else, it would most likely be more useful to use the C SAM library instead.
+I2CP also <a href="i2cp">needs documentation</a>.
+</p>
+
+<h3>Web Applications</h3>
+I2P comes with the Jetty webserver, and configuring to use the Apache server instead is straightforward.
+Any standard web app technology should work.
+
+<h2>Existing Applications in Development</h2>
+Contact us if you would like to help.
+<ul>
+<li>
+<a href="http://syndie.i2p2.de/">Syndie</a>
+<li>
+<a href="http://forum.i2p/viewforum.php?f=25">I2Phex</a>
+<a href="http://forum.i2p.net/viewforum.php?f=25">(outside I2P)</a>
+<li>
+<a href="http://www.imule.i2p/">IMule</a>
+</ul>
+
+<h2>Application Ideas</h2>
+<ul>
+<li>NNTP server - there have been some in the past, none at the moment
+<li>Jabber server - there have been some in the past, none at the moment
+<li>PGP Key server and/or proxy
+<li>Download manager / eepget scheduler -
+We use eepget to fetch lots of things reliably over i2p, and there's already an
+implementation of a sequential download manager (net.i2p.util.EepGetScheduler),
+but there isn't any sort of user interface to it.  A web based UI would be
+great.
+<li>Open tracker - use as a backup to the existing trackers, add support for
+a backup tracker to i2psnark
+<li>Web-based applications - The sky is the limit for hosting web-server-based
+applications such as blogs, pastebins, storage, tracking, feeds, etc.
+Any web or cgi technology such as perl, php, python, or ruby will work.
+</ul>
+
 {% endblock %}

www.i2p2/pages/faq.html

@@ -49,6 +49,29 @@
    on the top of your router console for instructions.
 </p>
 
+<h3 id="slow">Why is I2P so slow?
+<span class="permalink">(<a href="#slow">link</a>)</span></h3>
+<p>
+Why are downloads, torrents, web browsing, and everything else so slow on I2P?
+The encryption and routing within the I2P network adds a substantial amount of overhead and limits bandwidth.
+Anonymity isn't free.
+</p>
+<p>
+In addition, you and everybody else probably need to increase your bandwidth limits.
+Two key settings are the inbound and outbund bandwidth limiters on 
+<a href="http://localhost:7657/config.jsp">the configuration page</a>.
+With the default settings of 16KBps you will generally get no better than 5KBps data transfer in I2PSnark.
+Increasing the settings (but keeping within your actual connection limitations)
+will increase the potential transfer rate for I2PSnark and all other applications.
+</p>
+
+<h3 id="snark">Bittorrent / I2PSnark / Azureus I2P Plugin Questions?
+<span class="permalink">(<a href="#snark">link</a>)</span></h3>
+<p>
+See the
+<a href="http://forum.i2p/viewtopic.php?t=2068">I2P Bittorrent FAQ</a>
+<a href="http://forum.i2p.net/viewtopic.php?t=2068">(outside I2P)</a>
+
 <h3 id="remote_webconsole">How can I access the web console from my other machines or password protect it?
 <span class="permalink">(<a href="#remote_webconsole">link</a>)</span></h3>
 <p>

www.i2p2/pages/how_intro.html

@@ -143,5 +143,5 @@ join us on the irc channel #i2p (hosted concurrently on irc.freenode.net, irc.po
 There are currently no scheduled development meetings, however
 <a href="meetings">archives are available</a>.</p>
 
-<p>The current source is available in <a href="monotone.html">monotone</p>.
+<p>The current source is available in <a href="monotone.html">monotone</a></p>.
 {% endblock %}

www.i2p2/pages/i2cp.html

@@ -1,6 +1,7 @@
 {% extends "_layout.html" %}
 {% block title %}I2CP{% endblock %}
-{% block content %}<p>The I2P Client Protocol (I2CP) exposes a strong seperation of concerns between
+{% block content %}
+<p>The I2P Client Protocol (I2CP) exposes a strong seperation of concerns between
 the router and any client that wishes to communicate over the network.  It enables
 secure and asynchronous messaging by sending and receiving messages over a 
 single TCP socket, yet never exposing any private keys and authenticating itself
@@ -21,9 +22,9 @@ Instead, applications can take advantage of the base I2CP plus the
 by using the <a href="sam">Simple Anonymous Messaging</a> protocol (which does not
 require clients to deal with any sort of cryptography).</p>
 
-<p>At the moment, while the I2CP has been quite stable since its inception in
-August of 2003, there have been minor modifications on occation, and until the
-network as a whole is thoroughly peer reviewed (including the 
-<a href="how_cryptography">encryption</a> used), making a hard and fast byte level
-protocol specification does not seem like a wise use of time.</p>
+<p>While the I2CP has been quite stable since its inception in
+August of 2003, there have been minor modifications on occasion.
+Now would be a good time for somebody to write a protocol specification.
+</p>
+
 {% endblock %}

www.i2p2/pages/ntcp.html

@@ -1,5 +1,5 @@
 {% extends "_layout.html" %}
-{% block title %}SSU{% endblock %}
+{% block title %}NTCP{% endblock %}
 {% block content %}
 
 <h1>NTCP (NIO-based TCP)</h1>

www.i2p2/pages/performance.html

@@ -1,6 +1,7 @@
 {% extends "_layout.html" %}
 {% block title %}Performance{% endblock %}
-{% block content %}<p>Probably one of the most frequent things people ask is "how fast is I2P?",
+{% block content %}
+<p>Probably one of the most frequent things people ask is "how fast is I2P?",
 and no one seems to like the answer - "it depends".  After trying out I2P, the
 next thing they ask is "will it get faster?", and the answer to that is a most
 emphatic <b>yes</b>.</p>
@@ -80,6 +81,13 @@ it "feels ok".  Once we've got tunnel healing code and more effective failure
 detection, we'll be able to more safely vary those durations, reducing the
 network and CPU load (due to expensive tunnel creation messages).</p>
 
+<p>
+This is an easy fix for high load on the big-bandwidth routers, but
+we should not resort to it until we've tuned the tunnel building algorithms further.
+A new tunnel build algorithm is planned for release 0.6.1.32.
+</p>
+
+
 <h2>Adjust the timeouts</h2>
 <p>Yet another of the fairly arbitrary but "ok feeling" things we've got are the
 current timeouts for various activities.  Why do we have a 60 second "peer
@@ -123,7 +131,7 @@ consider how often RouterInfo structures are passed across the network, thats
 significant savings - every time a router asks another router for a networkDb
 entry that the peer doesn't have, it sends back 3-10 RouterInfo of them.</p>
 
-<h2>Update the ministreaming protocol</h2>
+<h2>Update the ministreaming protocol</h2> [replaced by full streaming protocol]
 <p>Currently mihi's ministreaming library has a fairly simple stream negotiation
 protocol - Alice sends Bob a SYN message, Bob replies with an ACK message, then
 Alice and Bob send each other some data, until one of them sends the other a
@@ -135,7 +143,7 @@ message, and Bob piggybacked his first reply with the ACK - and perhaps also
 included the CLOSE flag - transient streams such as HTTP requests could be
 reduced to a pair of messages, instead of the SYN+ACK+request+response+CLOSE.</p>
 
-<h2>Implement full streaming protocol</h2>
+<h2>Implement full streaming protocol</h2> [<a href="streaming.html">implemented</a>]
 <p>The ministreaming protocol takes advantage of a poor design decision in the
 I2P client procotol (I2CP) - the exposure of "mode=GUARANTEED", allowing what
 would otherwise be an unreliable, best-effort, message based protocol to be used

www.i2p2/pages/streaming.html

@@ -0,0 +1,136 @@
+{% extends "_layout.html" %}
+{% block title %}Streaming Lib{% endblock %}
+{% block content %}
+<h2>Overview</h2>
+
+<p>
+The streaming library is technically part of the "application" layer,
+as it is not a core router function.
+In practice, however, it provides a vital function for almost all
+existing I2P applications, by providing a TCP-like
+streams over I2P, and allowing exising apps to be easily ported to I2P.
+</p>
+
+<p>The streaming library is a layer on top of the core 
+<a href="i2cp">I2CP</a> that allows reliable, in order, and authenticated streams
+of messages to operate across an unreliable, unordered, and unauthenticated 
+message layer.  Just like the TCP to IP relationship, this streaming 
+functionality has a whole series of tradeoffs and optimizations available, but
+rather than embed that functionality into the base I2P code, it has been factored
+off into its own library both to keep the TCP-esque complexities seperate and to
+allow alternative optimized implemenations.</p>
+
+<h2>History</h2>
+<p><i>copied from the tech intro</i></p>
+<p>
+The streaming library has grown organically for I2P - first mihi implemented the
+"mini streaming library" as part of I2PTunnel, which was limited to a window
+size of 1 message (requiring an ACK before sending the next one), and then it was
+refactored out into a generic streaming interface (mirroring TCP sockets) and the
+full streaming implementation was deployed with a sliding window protocol and 
+optimizations to take into account the high bandwidth x delay product.  Individual
+streams may adjust the maximum packet size and other options, though the default
+of 4KB compressed seems a reasonable tradeoff between the bandwidth costs of 
+retransmitting lost messages and the latency of multiple messages.
+</p>
+
+<p>
+In addition, in consideration of the relatively high cost of subsequent messages, 
+the streaming library's protocol for scheduling and delivering messages has been optimized to
+allow individual messages passed to contain as much information as is available.
+For instance, a small HTTP transaction proxied through the streaming library can
+be completed in a single round trip - the first message bundles a SYN, FIN, and
+the small payload (an HTTP request typically fits) and the reply bundles the SYN,
+FIN, ACK, and the small payload (many HTTP responses fit).  While an additional
+ACK must be transmitted to tell the HTTP server that the SYN/FIN/ACK has been
+received, the local HTTP proxy can deliver the full response to the browser 
+immediately.  
+</p>
+
+<p>
+On the whole, however, the streaming library bears much resemblance to an 
+abstraction of TCP, with its sliding windows, congestion control algorithms
+(both slow start and congestion avoidance), and general packet behavior (ACK,
+SYN, FIN, RST, rto calculation, etc).  
+</p>
+
+
+<h2>Usage</h2>
+<p>
+The standard interface to the streaming lib is for the application to set up
+a I2PSocketManagerFactory from the <a href="ministreaming.html">ministreaming lib</a>.
+Only I2PSocketManagerFactory is used here - everything else is from the full streaming lib
+(I2PSocketManagerFull, not I2PSocketManagerImpl, and I2PSocketFull, not I2PSocketImpl).
+The remainder of the ministreaming lib is not normally used - don't be confused.
+</p>
+
+<p>
+For a good example of usage, see the i2psnark code.
+</p>
+
+
+<h2>Status</h2>
+<p>The streaming lib has many advantages over the ministreaming library was written by mihi as a part of his 
+<a href="i2ptunnel">I2PTunnel</a> application.
+The streaming library is
+a more robust streaming library
+which is further optimized for operation over I2P.  The two main issues with 
+the ministreaming library are its use of the traditional TCP two phase 
+establishment protocol and the current fixed window size of 1.
+jhe streaming lib fixes both of these issues - it has a one-phase setup, and
+it contains a full windowing implementation.
+</p>
+
+<p>
+Significant tuning of the streaming lib parameters,
+greatly increasing outbound performance, was implemented in 0.6.1.28.
+After the changes in that release, the current default values
+in the implementation are listed below.
+These values are tuned for HTTP performance over typical I2P conditions. Other applications such
+as peer-to-peer services are strongly encouraged to
+modify as necessary, by setting the options and passing them via the call to
+I2PSocketManagerFactory.createManager(_i2cpHost, _i2cpPort, opts).
+Time values are in ms.
+<ul>
+<li>MIN_RESEND_DELAY = 2*1000
+<li>MAX_RESEND_DELAY = 45*1000
+<li>DISCONNECT_TIMEOUT = 5*60*1000
+<li>INITIAL_RECEIVE_WINDOW = 1
+<li>INITIAL_WINDOW_SIZE = 12 // i2psnark sets to 1
+<li>MIN_WINDOW_SIZE = 1
+<li>MAX_WINDOW_SIZE = 64
+<li>TREND_COUNT = 3
+<li>DEFAULT_MAX_SENDS = 8
+<li>RTT_DAMPENING = 0.9
+<li>PROFILE_BULK
+<li>MAX_MESSAGE_SIZE = 960 // 960 fits inside a single tunnel message
+<li>INITIAL_RTT = 10*1000
+<li>INITIAL_RESEND_DELAY = 1000
+<li>INITIAL_ACK_DELAY = 2000
+<li>INACTIVITY_TIMEOUT = 90*1000
+<li>CONGESTION_AVOIDANCE_GROWTH_RATE_FACTOR = 1
+<li>SLOW_START_GROWTH_RATE_FACTOR = 1
+</ul>
+</p>
+
+<p>
+The interaction of the routing algorithms with the streaming lib strongly affects performance.
+In particular, random distribtion of messages to multiple tunnels in a pool
+leads to a high degree of out-of-order delivery which results in smaller window
+sizes than would otherwise be the case.
+In release 0.6.1.30, the routing of messages to the outbound tunnels was made
+consistent, with pushback when a tunnel was backlogged.
+This had a significant positive impact on bandwidths.
+The pushback code was reverted in release 0.6.1.31 due to anonymity concerns.
+Consistent message routing to inbound tunnels is more difficult,
+as there is currently no tracking in the code of the message source.
+Perhaps to be implemented in 0.6.1.32.
+</p>
+
+<p>
+Another area for research is the interaction of the streaming lib with the
+NTCP and SSU transport layers.
+See <a href="ntcp.html">the NTCP page</a> for a discussion.
+</p>
+
+{% endblock %}

www.i2p2/pages/techintro.html

@@ -68,6 +68,7 @@ FOR MORE INFORMATION
 
 
   * <a href="udp.html">SSU</a>
+  * <a href="streaming.html">Streaming Lib</a>
   * <a href="package-streaming.html">Streaming Javadoc</a>
   * <a href="http://syndie.i2p2.de/">Syndie</a>
   * <a href="package-tcp.html">Old TCP Javadoc</a>