This is an important differentiator between getTokens() and the present
LLCommandLineParser::parseCommandLineString() logic: you cannot currently
--set SomeVar to an empty string value because parseCommandLineString()
discards empty strings.

We didn't have any tokenizer suitable for scanning something like a bash
command line. We do have a couple hacks, e.g. LLExternalEditor::tokenize() and
LLCommandLineParser::parseCommandLineString(). Both try to work around
boost::tokenizer limitations; but existing boost::tokenizer support just
doesn't address this case. Neither of the above is available as a general
scanner anyway, and parseCommandLineString() fails outright when passed "".
New getTokens() also distinguishes between "drop delimiters" (e.g. space,
return, newline) to be discarded from the token stream, versus "keep
delimiters" (e.g. "+-*/") to be returned as tokens in their own right.
There's an overload that honors escapes and a more efficient one that doesn't;
each has a convenience overload that returns the scanned string vector rather
than requiring a separate declaration.
Tweak and comment older getTokens() implementation.
Add unit tests for both old and new getTokens() implementations.
Break out StringVec and std::ostream << StringVec from
indra/llcommon/tests/listener.h to StringVec.h: that's coming in handy for a
number of different TUT test sources.

string replacement, e.g. [[FOO]]

Migrate to the .cpp files where it's needed.

DEV-41081 - better than using the msvc kludge for suppressing warnings about memcpy, don't actually use memcpy - use our own safe 'copy' method.  also stripNonprintable() is probably a little faster now...

Extend automated string tests to cover the formerly-crashing "[]" substitution case (EXT-1282, DEV-41081)

Also 'fix' string tests to match the new expected behaviour of LLStringUtil::format().  They must have been failing dreadfully before, but the legacy tut tests aren't run in a viewer-only build, so...