Skip to content
Snippets Groups Projects
  1. Apr 11, 2019
  2. Sep 30, 2017
    • Nat Goodspeed's avatar
      MAINT-7081: Eliminate unused variable errors after new refactoring. · 6b508cd9
      Nat Goodspeed authored
      The new helper functions check_curl_easy_setopt() and
      check_curl_multi_setopt() encapsulate the pervasive idiom:
      
          code = curl_{easy,multi}_setopt(handle, option, arg);
          check_curl_{easy,multi}_code(code, option);
      
      But since each of these helper functions contains its own local CURL{,M}code
      variable 'code', having a caller-scope variable reused for every such call is
      no longer necessary -- in fact is no longer used at all. That produces a fatal
      warning with MSVC. Get rid of those now-unused variables.
      6b508cd9
  3. Sep 29, 2017
  4. Feb 19, 2016
  5. Jan 21, 2016
  6. Nov 10, 2015
  7. Oct 16, 2015
  8. Jul 08, 2015
  9. Sep 19, 2014
  10. Sep 04, 2014
    • Monty Brandenberg's avatar
      Pipelining work. Extend transfer timeout by the pipeline depth · 0c20beda
      Monty Brandenberg authored
      as transfers can appear delayed with deep pipelining and more
      requests in the pool.  Added bad HTTP status error (typically
      getting a 0 back as HTTP status from libcurl) to the list of
      retryable errors.  There's a response stream problem with libcurl
      and pipelining that induces this problem.  Retrying helps but
      may not be entirely safe.  Watch bug 1420 on the libcurl sourceforge
      bug tracker.  Extend options of test/example program to include
      un-ranged requests.  Document the excessive data transfer induced
      when ranged requests are disabled.  This is an abnormal mode for
      very rare users so we'll just eat that for now.
      0c20beda
  11. Aug 12, 2014
    • Monty Brandenberg's avatar
      Better support for dynamic option changes in llcorehttp. Libcurl has · e79a88c8
      Monty Brandenberg authored
      some problems disabling pipelining on a multi handle with outstanding
      requests so build a more conservative system that allows requests
      to drain before setting curl multi options.  Would rather not have
      this but it is significantly safer.  "HttpPipelining" debug setting
      is now fully dynamic.  Connection limits can also be made dynamic
      in the near future.  Upped the default connection count back to 8 for
      now but will revisit this in the tuning phase.  It might be time to
      combine mesh and textures into a single asset class.  For normal
      server operations that would be a clear path, but for server under
      load, the current scheme may be better.  Minor cleanup in logging
      to elminate some redundant strings.  Might add some more tracing to the
      stall logic 'just in case'.
      e79a88c8
  12. Jun 23, 2014
    • Monty Brandenberg's avatar
      First HTTP pipelining viewer. Enable pipelining for · 5cca78e7
      Monty Brandenberg authored
      GetTexture and GetMesh2 at a pipeline depth of 5.  Create
      global debug option, HttpPipelining, to enable and disable
      HTTP pipelining (defaults to true).  Tweak texture and
      mesh low- and high-water request levels based on pipelining
      status and depth.  Fixup texture console which was damaged
      in a recent release.  Split logging of the no-request
      HTTP error case into two cases:  one for missing URL in
      HTTP request, one for HTTP request not created.  A refactor
      in llcorehttp is coming:  I will be moving all libcurl-
      using code into libcurl-specific modules.
      5cca78e7
  13. Sep 11, 2013
  14. Aug 16, 2013
  15. Jul 29, 2013
    • Monty Brandenberg's avatar
      SH-4371 Reduce 22mS inter-conenction latency on HTTP operations. · eb2869bd
      Monty Brandenberg authored
      Simple change dropped this value by 7-10mS or so.  Any time we
      complete an operation on a transport pass, arrange to skip the
      run-loop sleep so that we fill a possible empty slot as quickly
      as possible.  With pipelining, this kind of thing should become
      unnecessary but for now, we'll do this to increase throughput.
      eb2869bd
  16. Apr 15, 2013
    • Monty Brandenberg's avatar
      SH-4106 Significantly upgrade the HttpHeaders interface for SSB. · 4eef1c8a
      Monty Brandenberg authored
      Header container moves from a vector of raw lines to a vector
      of string pairs representing name/value pairs in headers.  For
      incoming headers, we normalize the name to lowercase and trim
      it.  Values are only left-trimmed.  Outgoing headers are left
      as-is.  Simple find() method for the common case, forward and
      reverse iterators for those few who need to do it themselves.
      The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated
      as a header to be returned to caller.  Unit tests, as usual,
      were a bear but they absolutely ensured outgoing HTTP header
      conformance after the change.  Grunt work paid off.
      
      LLTextureFetch was also given a second options structure
      for texture fetches.  Same as the original but with header return
      to caller requested.  Baked textures should use this, the other
      20,000 texture fetch requests should continue to use the original.
      4eef1c8a
    • Monty Brandenberg's avatar
      SH-4106 Significantly upgrade the HttpHeaders interface for SSB. · 8868964b
      Monty Brandenberg authored
      Header container moves from a vector of raw lines to a vector
      of string pairs representing name/value pairs in headers.  For
      incoming headers, we normalize the name to lowercase and trim
      it.  Values are only left-trimmed.  Outgoing headers are left
      as-is.  Simple find() method for the common case, forward and
      reverse iterators for those few who need to do it themselves.
      The HTTP status line (e.g. 'HTTP/1.1 200 Ok') is no longer treated
      as a header to be returned to caller.  Unit tests, as usual,
      were a bear but they absolutely ensured outgoing HTTP header
      conformance after the change.  Grunt work paid off.
      
      LLTextureFetch was also given a second options structure
      for texture fetches.  Same as the original but with header return
      to caller requested.  Baked textures should use this, the other
      20,000 texture fetch requests should continue to use the original.
      8868964b
  17. Mar 29, 2013
  18. Mar 13, 2013
    • Don Kjer's avatar
      Large changes to the LLCurl::Responder API, as well as pulling in some changes... · f9454152
      Don Kjer authored
      Large changes to the LLCurl::Responder API, as well as pulling in some changes to common libraries from the server codebase:
      
      * Additional error checking in http handlers.
      * Uniform log spam for http errors.
      * Switch to using constants for http heads and status codes.
      * Fixed bugs in incorrectly checking if parsing LLSD xml resulted in an error.
      * Reduced spam regarding LLSD parsing errors in the default completedRaw http handler.  It should not longer be necessary to short-circuit completedRaw to avoid spam.
      * Ported over a few bug fixes from the server code.
      * Switch mode http status codes to use S32 instead of U32.
      * Ported LLSD::asStringRef from server code; avoids copying strings all over the place.
      * Ported server change to LLSD::asBinary; this always returns a reference now instead of copying the entire binary blob.
      * Ported server pretty notation format (and pretty binary format) to llsd serialization.
      
      * The new LLCurl::Responder API no longer has two error handlers to choose from.  Overriding the following methods have been deprecated:
      ** error - use httpFailure
      ** errorWithContent - use httpFailure
      ** result - use httpSuccess
      ** completed - use httpCompleted
      ** completedHeader - no longer necessary; call getResponseHeaders() from a completion method to obtain these headers.
      
      * In order to 'catch' a completed http request, override one of these methods:
      ** httpSuccess - Called for any 2xx status code.
      ** httpFailure - Called for any non-2xx status code.
      ** httpComplete - Called for all status codes.  Default implementation is to call either httpSuccess or httpFailure.
      * It is recommended to keep these methods protected/private in order to avoid triggering of these methods without using a 'push' method (see below).
      
      * Uniform error handling should followed whenever possible by calling a variant of this during httpFailure:
      ** llwarns << dumpResponse() << llendl;
      * Be sure to include LOG_CLASS(your_class_name) in your class in order for the log entry to give more context.
      
      * In order to 'push' a result into the responder, you should no longer call error, errorWithContent, result, or completed.
      * Nor should you directly call httpSuccess/Failure/Completed (unless passing a message up to a parent class).
      * Instead, you can set the internal content of a responder and trigger a corresponding method using the following methods:
      ** successResult - Sets results and calls httpSuccess
      ** failureResult - Sets results and calls httpFailure
      ** completedResult - Sets results and calls httpCompleted
      
      * To obtain information about a the response from a reponder method, use the following getters:
      ** getStatus - HTTP status code
      ** getReason - Reason string
      ** getContent - Content (Parsed body LLSD)
      ** getResponseHeaders - Response Headers (LLSD map)
      ** getHTTPMethod - HTTP method of the request
      ** getURL - URL of the request
      
      * It is still possible to override completeRaw if you want to manipulate data directly out of LLPumpIO.
      
      * See indra/llmessage/llcurl.h for more information.
      f9454152
  19. Aug 01, 2012
    • Monty Brandenberg's avatar
      SH-3308 Beef up retry messaging. · bf004be1
      Monty Brandenberg authored
      Reformatted messages around request retry.  Successfully retried requests
      also message so you can see the cycle closed.  Added additional retryable
      error codes (timeout, other libcurl failures).  Commenting and removed some
      unnecessary std::min logic.
      bf004be1
  20. Jul 23, 2012
  21. Jul 16, 2012
    • Monty Brandenberg's avatar
      SH-3189 Remove/improve naive data structures · d238341a
      Monty Brandenberg authored
      When releasing HTTP waiters, avoid unnecessary sort activity.
      For Content-Type in responses, let libcurl do the work and removed
      my parsing of headers.  Drop Content-Encoding as libcurl will deal
      with that.  If anyone is interested, they can parse.
      d238341a
  22. Jul 03, 2012
    • Monty Brandenberg's avatar
      More integration work for texture fetch timeouts. · 2d7b7de2
      Monty Brandenberg authored
      The fetch state machine received a new timeout during the WAIT_HTTP_REQ
      state.  For the integration, rather than jump the state to done, we issue
      a request cancel and let the notification plumbing do the rest without
      any race conditions or special-case logic.
      2d7b7de2
  23. Jun 26, 2012
    • Monty Brandenberg's avatar
      SH-3184/SH-3221 More work on cleanup with better unit test work and more... · e8b0088d
      Monty Brandenberg authored
      SH-3184/SH-3221  More work on cleanup with better unit test work and more aggressive shutdown of a thread.
      Some additional work let me enable a memory check for the clean shutdown case and
      generally do a better job on other interfaces.  Request queue waiters now awake
      on shutdown and don't sleep once the queue is turned off.  Much better semantically
      for how this will be used.
      e8b0088d
  24. Jun 23, 2012
    • Monty Brandenberg's avatar
      SH-3184/SH-3221 Improve cleanup, destructor, thread termination, etc. logic in library. · e172ec84
      Monty Brandenberg authored
      With this commit, the cleanup paths should be production quality.  Unit tests have been
      expanded to include cases requiring thread termination and cleanup by the worker thread.
      Special operation/request added to support the unit tests.  Thread interface expanded
      to include a very aggressive cancel() method that does not do cleanup but prevents the
      thread from accessing objects that will be destroyed.
      e172ec84
  25. Jun 22, 2012
    • Monty Brandenberg's avatar
      This sets down the groundwork for dynamic policy classes. · bc7d5b24
      Monty Brandenberg authored
      Groundwork is used for the default class which currently represents
      texture fetching.  Class options implemented from API user into
      HttpLibcurl.  Policy layer is going to start doing some traffic
      shaping like work to solve problems with consumer-grade gear.
      Need to have dynamic aspects to policies and that starts now...
      bc7d5b24
  26. Jun 19, 2012
  27. Jun 14, 2012
    • Monty Brandenberg's avatar
      LLMutex recursive lock, global & per-request tracing, simple GET request,... · b08125a5
      Monty Brandenberg authored
      LLMutex recursive lock, global & per-request tracing, simple GET request, LLProxy support, HttpOptions starting to work, HTTP resource waiting fixed.
      Non-LLThread-based threads need to do some registration or LLMutex locks taken out in these
      threads will not work as expected (SH-3154).  We'll get a better solution later, this fixes
      some things for now.  Tracing of operations now supported.  Global and per-request (via
      HttpOptions) tracing levels of [0..3].  The 2 and 3 levels use libcurl's VERBOSE mode
      combined with CURLOPT_DEBUGFUNCTION to stream high levels of detail into the log.  *Very*
      laggy but useful.  Simple GET request supported (no Range: header).  Really just a
      degenrate case of a ranged get but supplied an API anyway.  Global option to use the
      LLProxy interface to setup CURL handles for either socks5 or http proxy usage.  This
      isn't really the most encapsulated way to do this but a better solution will have to
      come later.  The wantHeaders and tracing options are now supported in HttpOptions giving
      per-request controls.  Big refactoring of the HTTP resource waiter in lltexturefetch.
      What I was doing before wasn't correct.  Instead, I'm implementing the resource wait
      after the Semaphore model (though not using system semaphores).  So instead of having
      a sequence like:  SEND_HTTP_REQ -> WAIT_HTTP_RESOURCE -> SEND_HTTP_REQ, we now
      do WAIT_HTTP_RESOURCE -> WAIT_HTTP_RESOURCE2 (actual wait) -> SEND_HTTP_REQ.  Works
      well but the prioritized filling of the corehttp library needs some performance
      work later.
      b08125a5
  28. Jun 11, 2012
  29. Jun 08, 2012
    • Monty Brandenberg's avatar
      Implemented HTTP retry for requests. Went in rather easily which · 28a04400
      Monty Brandenberg authored
      surprised me.  Added a retry queue similar to ready queue to the
      policy object which is sorted by retry time.  Currently do five
      retries (after the initial try) delayed by .25, .5, 1, 2 and 5
      seconds.  Removed the retry logic from the lltexturefetch module.
      Upped the waiting time in the unit test for the retries.  People
      won't like this but tough, need tests.
      28a04400
  30. Jun 06, 2012
    • Monty Brandenberg's avatar
      Policy + caching fixes + https support + POST working · 05af16a2
      Monty Brandenberg authored
      Implemented first global policy definitions to support SSL CA certificate configuration
      to support https: operations.  Fixed HTTP 206 status handling to match what is currently
      being done by grid services and to lay a foundation for fixes that will be a response
      to ER-1824.  More libcurl CURLOPT options set on easy handles to do peer verification
      in the traditional way.  HTTP POST working and now reporting asset metrics back to
      grid for the viewer's asset system.  This uses LLSD so that is also showing as compatible
      with the new library.
      05af16a2
  31. Jun 01, 2012
    • Monty Brandenberg's avatar
      Major steps towards implementing the policy component. · b8edacd0
      Monty Brandenberg authored
      Identified and reacted to the priority inversion problem we
      have in texturefetch.  Includes the introduction of a priority_queue
      for the requests that are ready.  Start some parameterization in
      anticipation of having policy_class everywhere.  Removed _assert.h
      which isn't really needed in indra codebase.  Implemented async
      setPriority request (which I hope I can get rid of eventually along
      with all priorities in this library).  Converted to using unsigned
      int for priority rather than float.  Implemented POST and did
      groundwork for PUT.
      b8edacd0
  32. May 23, 2012
    • Monty Brandenberg's avatar
      Integrate llcorehttp library into lltexturefetch design. · 8fc35012
      Monty Brandenberg authored
      This is the first functional viewer pass with the HTTP work of the texture fetch
      code performed by the llcorehttp library.  Not exactly a 'drop-in' replacement
      but a work-alike with some changes (e.g. handler notification in consumer
      thread versus responder notification in worker thread).
      
      This also includes some temporary changes in the priority scheme to prevent
      the kind of priority inversion found in VWR-28996.  Scheme used here does
      provide liveness if not optimal responsiveness or order-of-operation.
      
      The llcorehttp library at this point is far from optimally performing.
      Its worker thread is making relatively poor use of cycles it gets and
      it doesn't idle or sleep intelligently yet.  This early integration step
      helps shake out the interfaces, implementation niceties will be covered
      soon.
      8fc35012
  33. Apr 23, 2012
Loading