SH-4492 Create a useful README for llcorehttp.

First edit complete.  Use the library in 15 minutes.  Describe the
code.  Refinements to the initial try.  Describe that code.  Still
to do:  more refinements, how to choose a policy class, FAQ.

indra/llcorehttp/README.Linden

+
+
+
+1.  HTTP fetching in 15 Minutes
+
+    Let's start with a trivial working example.  You'll need a throwaway
+    build of the viewer.  And we'll use indra/newview/llappviewer.cpp as
+    our host.
+
+    Add some needed headers:
+
+
+        #include "httpcommon.h"
+        #include "httprequest.h"
+        #include "httphandler.h"
+
+
+    You'll need to derive a class from HttpHandler (not HttpHandle).
+    This is used to deliver notifications of HTTP completion to your
+    code.  Place it near the top, before LLDeferredTaskList, say:
+
+
+        class MyHandler : public LLCore::HttpHandler
+        {
+        public:
+            MyHandler()
+            : LLCore::HttpHandler()
+            {}
+
+            virtual void onCompleted(LLCore::HttpHandle /* handle */,
+                                     LLCore::HttpResponse * /* response */)
+            {
+                LL_INFOS("Hack") << "It is happening again." << LL_ENDL;
+
+                delete this;    // Last statement
+            }
+        };
+
+
+    Add some statics up there as well:
+
+
+        // Our request object.  Allocate during initialiation.
+        static LLCore::HttpRequest * my_request(NULL);
+
+        // The policy class for HTTP traffic.
+        // Use HttpRequest::DEFAULT_POLICY_ID, but DO NOT SHIP WITH THIS VALUE!!
+        static LLCore::HttpRequest::policy_t my_policy(LLCore::HttpRequest::DEFAULT_POLICY_ID);
+
+        // Priority for HTTP requests.  Use 0U.
+        static LLCore::HttpRequest::priority_t my_priority(0U);
+
+
+    In LLAppViewer::init() after mAppCoreHttp.init(), create a request object:
+
+
+        my_request = new LLCore::HttpRequest();
+
+
+    In LLAppViewer::mainLoop(), just before entering the while loop,
+    we'll kick off one HTTP request:
+
+
+        // Construct a handler object (we'll use the heap this time):
+        MyHandler * my_handler = new MyHandler;
+
+        // Issue a GET request to 'http://www.example.com/' kicking off
+        // all the I/O, retry logic, etc.
+        LLCore::HttpHandle handle;
+        handle = my_request->requestGet(my_policy,
+                                        my_priority,
+                                        "http://www.example.com/",
+                                        NULL,
+                                        NULL,
+                                        my_handler);
+        if (LLCORE_HTTP_HANDLE_INVALID == handle)
+        {
+            LL_WARNS("Hack") << "Failed to launch HTTP request.  Try again."
+                             << LL_ENDL;
+        }
+
+
+    Finally, arrange to periodically call update() on the request object
+    to find out when the request completes.  This will be done by
+    calling the onCompleted() method with status information and
+    response data from the HTTP operation.  Add this to the
+    LLAppViewer::idle() method after the ping:
+
+
+        my_request->update(0);
+
+
+    That's it.  Build it, run it and watch the log file.  You should get
+    the "It is happening again." message indicating that the HTTP
+    operation completed in some manner.
+
+
+2.  What Does All That Mean
+
+    MyHandler/HttpHandler.  This class replaces the Responder-style in
+    legacy code.  One method is currently defined.  It is used for all
+    request completions, successful or failed:
+
+
+        void onCompleted(LLCore::HttpHandle /* handle */,
+                         LLCore::HttpResponse * /* response */);
+
+
+    The onCompleted() method is invoked as a callback during calls to
+    HttpRequest::update().  All I/O is completed asynchronously in
+    another thread.  But notifications are polled by calling update()
+    and invoking a handler for completed requests.
+
+    In this example, the invocation also deletes the handler (which is
+    never referenced by the llcorehttp code again).  But other
+    allocation models are possible including handlers shared by many
+    requests, stack-based handlers and handlers mixed in with other,
+    unrelated classes.
+
+    LLCore::HttpRequest().  Instances of this class are used to request
+    all major functions of the library.  Initialization, starting
+    requests, delivering final notification of completion and various
+    utility operations are all done via instances.  There is one very
+    important rule for instances:
+
+        Request objects may NOT be shared between threads.
+
+    my_priority.  The APIs support the idea of priority ordering of
+    requests but it hasn't been implemented and the hope is that this
+    will become useless and removed from the interface.  Use 0U except
+    as noted.
+
+    my_policy.  This is an important one.  This library attempts to
+    manage TCP connection usage more rigorously than in the past.  This
+    is done by issuing requests to a queue that has various settable
+    properties.  These establish connection usage for the queue as well
+    as how queues compete with one another.  (This is patterned after
+    class-based queueing used in various networking stacks.)  Several
+    classes are pre-defined.  Deciding when to use an existing class and
+    when to create a new one will determine what kind of experience
+    users have.  We'll pick up this question in detail below.
+
+    requestGet().  Issues an ordinary HTTP GET request to a given URL
+    and associating the request with a policy class, a priority and an
+    response handler.  Two additional arguments, not used here, allow
+    for additional headers on the request and for per-request options.
+    If successful, the call returns a handle whose value is other than
+    LLCORE_HTTP_HANDLE_INVALID.  The HTTP operation is then performed
+    asynchronously by another thread without any additional work by the
+    caller.  If the handle returned is invalid, you can get the status
+    code by calling my_request->getStatus().
+
+    update().  To get notification that the request has completed, a
+    call to update() will invoke onCompleted() methods.
+
+
+3.  Refinements, Necessary and Otherwise
+
+    MyHandler::onCompleted().  You'll want to do something useful with
+    your response.  Distinguish errors from successes and getting the
+    response body back in some form.
+
+    Add a new header:
+
+
+        #include "bufferarray.h"
+
+
+    Replace the existing MyHandler::onCompleted() definition with:
+
+
+        virtual void onCompleted(LLCore::HttpHandle /* handle */,
+                                 LLCore::HttpResponse * response)
+        {
+            LLCore::HttpStatus status = response->getStatus();
+            if (status)
+            {
+                // Successful request.  Try to fetch the data
+                LLCore::BufferArray * data = response->getBody();
+
+                if (data && data->size())
+                {
+                    // There's some data.  A BufferArray is a linked list
+                    // of buckets.  We'll create a linear buffer and copy
+                    // it into it.
+                    size_t data_len = data->size();
+                    char * data_blob = new char [data_len + 1];
+                    data->read(0, data_blob, data_len);
+                    data_blob[data_len] = '\0';
+
+                    // Process the data now in NUL-terminated string.
+                    // Needs more scrubbing but this will do.
+                    LL_INFOS("Hack") << "Received:  " << data_blob << LL_ENDL;
+
+                    // Free the temporary data
+                    delete [] data_blob;
+                }
+            }
+            else
+            {
+                // Something went wrong.  Translate the status to
+                // a meaningful message.
+                LL_WARNS("Hack") << "HTTP GET failed.  Status:  "
+                                 << status.toTerseString()
+                                 << ", Reason:  " << status.toString()
+                                 << LL_ENDL;
+            }           
+
+            delete this;    // Last statement
+        }
+
+
+    HttpHeaders.  The header file "httprequest.h" documents the expected
+    important headers that will go out with the request.  You can add to
+    these by including an HttpHeaders object with the requestGet() call.
+    These are typically setup once as part of init rather than
+    dynamically created.
+
+    Add another header:
+
+
+        #include "httpheaders.h"
+
+
+    In LLAppViewer::mainLoop(), add this alongside the allocation of
+    my_handler:
+
+
+        // Additional headers for all requests
+        LLCore::HttpHeaders * my_headers = new LLCore::HttpHeaders();
+        my_headers->append("Accept", "text/html, application/llsd+xml");
+
+
+    HttpOptions.  Options are similar and include a mix of value types.
+    One interesting per-request option is the trace setting.  This
+    enables various debug-type messages in the log file that show the
+    progress of the request through the library.  It takes values from
+    zero to three with higher values giving more verbose logging.  We'll
+    use '2' and this will also give us a chance to verify that
+    HttpHeaders works as expected.
+
+    Same as above, a new header:
+
+
+        #include "httpoptions.h"
+
+
+    And in LLAppView::mainLoop():
+
+
+        // Special options for requests
+        LLCore::HttpOptions * my_options = new LLCore::HttpOptions();
+        my_options->setTrace(2);
+
+
+    Now let's put that all together into a more complete requesting
+    sequence.  Replace the existing invocation of requestGet() with this
+    slightly more elaborate block:
+
+
+        LLCore::HttpHandle handle;
+        handle = my_request->requestGet(my_policy,
+                                        my_priority,
+                                        "http://www.example.com/",
+                                        my_options,
+                                        my_headers,
+                                        my_handler);
+        if (LLCORE_HTTP_HANDLE_INVALID == handle)
+        {
+             LLCore::HttpStatus status = my_request->getStatus();
+
+             LL_WARNS("Hack") << "Failed to request HTTP GET.  Status:  "
+                              << status.toTerseString()
+                              << ", Reason:  " << status.toString()
+                              << LL_ENDL;
+
+             delete my_handler;    // No longer needed.
+             my_handler = NULL;
+        }
+
+
+    Build, run and examine the log file.  You'll get some new data with
+    this run.  First, you should get the www.example.com home page
+    content:
+
+
+----------------------------------------------------------------------------
+2013-09-17T20:26:51Z INFO: MyHandler::onCompleted: Received:  <!doctype html>
+<html>
+<head>
+    <title>Example Domain</title>
+
+    <meta charset="utf-8" />
+    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <style type="text/css">
+    body {
+        background-color: #f0f0f2;
+        margin: 0;
+        padding: 0;
+        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
+        
+    }
+    div {
+        width: 600px;
+        margin: 5em auto;
+        padding: 50px;
+        background-color: #fff;
+        border-radius: 1em;
+    }
+    a:link, a:visited {
+        color: #38488f;
+        text-decoration: none;
+    }
+    @media (max-width: 700px) {
+        body {
+            background-color: #fff;
+        }
+        div {
+            width: auto;
+            margin: 0 auto;
+            border-radius: 0;
+            padding: 1em;
+        }
+    }
+    </style>    
+</head>
+
+<body>
+<div>
+    <h1>Example Domain</h1>
+    <p>This domain is established to be used for illustrative examples in documents. You may use this
+    domain in examples without prior coordination or asking for permission.</p>
+    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
+</div>
+</body>
+</html>
+----------------------------------------------------------------------------
+
+
+    You'll also get a detailed trace of the HTTP operation itself.  Note
+    the HEADEROUT line which shows the additional header added to the
+    request.
+
+
+----------------------------------------------------------------------------
+HttpService::processRequestQueue: TRACE, FromRequestQueue, Handle:  086D3148
+HttpLibcurl::addOp: TRACE, ToActiveQueue, Handle:  086D3148, Actives:  0, Readies:  0
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  About to connect() to www.example.com port 80 (#0) 
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:    Trying 93.184.216.119... 
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connected to www.example.com (93.184.216.119) port 80 (#0) 
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connected to www.example.com (93.184.216.119) port 80 (#0) 
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADEROUT, Data:  GET / HTTP/1.1  Host: www.example.com  Accept-Encoding: deflate, gzip  Connection: keep-alive  Keep-alive: 300  Accept: text/html, application/llsd+xml
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  HTTP/1.1 200 OK  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Accept-Ranges: bytes  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Cache-Control: max-age=604800  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Content-Type: text/html  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Date: Tue, 17 Sep 2013 20:26:56 GMT  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Etag: "3012602696"  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Expires: Tue, 24 Sep 2013 20:26:56 GMT  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Server: ECS (ewr/1590)  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  X-Cache: HIT  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  x-ec-custom-error: 1  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:  Content-Length: 1270  
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  HEADERIN, Data:    
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  DATAIN, Data:  256 Bytes
+HttpOpRequest::debugCallback: TRACE, LibcurlDebug, Handle:  086D3148, Type:  TEXT, Data:  Connection #0 to host www.example.com left intact 
+HttpLibcurl::completeRequest: TRACE, RequestComplete, Handle:  086D3148, Status:  Http_200
+HttpOperation::addAsReply: TRACE, ToReplyQueue, Handle:  086D3148
+----------------------------------------------------------------------------
+
+
+4.  What Does All That Mean, Part 2
+
+    HttpStatus.  The HttpStatus object encodes errors from libcurl, the
+    library itself and HTTP status values.  It does this to avoid
+    collapsing all non-HTTP error into a single '499' HTTP status and to
+    make errors distinct.
+
+    To aid programming, the usual bool conversions are available so that
+    you can write 'if (status)' and the expected thing will happen
+    whether it's an HTTP, libcurl or library error.  There's also
+    provision to override the treatment of HTTP errors (making 404 a
+    success, say).
+
+    Share data, don't copy it.  The library was started with the goal of
+    avoiding data copies as much as possible.  Instead, read-only data
+    sharing across threads with atomic reference counts is used for a
+    number of data types.  These currently are:
+
+        * BufferArray.  Linked list of data blocks/HTTP bodies.
+        * HttpHeaders.  Shared headers for both requests and responses.
+        * HttpOptions.  Request-only data modifying HTTP behavior.
+        * HttpResponse.  HTTP response description given to onCompleted.
+
+    Using objects of these types requires a few rules:
+
+        * Constructor always gives a reference to caller.
+        * References are dropped with release() not delete.
+        * Additional references may be taken out with addRef().
+        * Unless otherwise stated, once an object is shared with another
+          thread it should be treated as read-only.  There's no
+          synchronization on the objects themselves.
+
+    HttpResponse.  You'll encounter this mainly in onCompleted() methods.
+    Commonly-used interfaces on this object:
+
+        * getStatus() to return the final status of the request.
+        * getBody() to retrieve the response body which may be NULL or
+          zero-length.
+        * getContentType() to return the value of the 'Content-Type'
+          header or an empty string if none was sent.
+
+    This is a reference-counted object so you can call addRef() on it
+    and hold onto the response for an arbitrary time.  But you'll
+    usually just call a few methods and return from onCompleted() whose
+    caller will release the object.
+
+    BufferArray.  The core data representation for request and response
+    bodies.  In HTTP responses, it's fetched with the getBody() method
+    and may be NULL or non-NULL but zero length.  All successful data
+    handling should check both conditions before attempting to fetch
+    data from the object.  Data access model uses simple read/write
+    semantics:
+
+        * append()
+        * size()
+        * read()
+        * write()
+
+    There is a more sophisticated stream adapter that extends these
+    methods and will be covered below.  So, one way to retrieve data
+    from a request is as follows:
+
+
+        LLCore::BufferArray * data = response->getBody();
+        if (data && data->size())
+        {
+            size_t data_len = data->size();
+            char * data_blob = new char [data_len + 1];
+            data->read(0, data_blob, data_len);
+
+
+    HttpOptions and HttpResponse.  Really just simple containers of POD
+    and std::string pairs.  But reference counted and the rule about not
+    modifying after sharing must be followed.  You'll have the urge to
+    change options dynamically at some point.  And you'll try to do that
+    by just writing new values to the shared object.  And in tests
+    everything will appear to work.  Then you ship and people in the
+    real world start hitting read/write races in strings and then crash.
+
+    HttpHandle.  Uniquely identifies a request and can be used to
+    identify it in an onCompleted() method or cancel it if it's still
+    queued.  But as soon as a request's onCompleted() invocation
+    returns, the handle becomes invalid and may be reused immediately
+    for new requests.  Don't hold on to handles after notification.
+
+
+5.  And Still More Refinements
+
+
+6.  Choosing a Policy Class
+
+
+7.  FAQ
+