/* $DOC$
   $NAME$
      TokenInit()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Initializes a token environment
   $SYNTAX$
      TokenInit( <[@]cString>], [<cTokenizer>], [<nSkipWidth>],
                 [<@cTokenEnvironment>] ) -> lState
   $ARGUMENTS$
      <[@]cString>          is the processed string
      <cTokenizer>          is a list of characters separating the tokens
                            in <cString>
                            Default: Chr( 0 ) + Chr( 9 ) + Chr( 10 ) + Chr( 13 ) +
                                     Chr( 26 ) + hb_BChar( 138 ) + hb_BChar( 141 ) +
                                     Chr( 32 ) + ",.;:!\?/\\<>()#&%+-*"
      <nSkipWidth>          specifies the maximum number of successive
                            tokenizing characters that are combined as
                            ONE token stop, e.g. specifying 1 can
                            yield to empty token
                            Default: 0, any number of successive tokenizing
                            characters are combined as ONE token stop
      <@cTokenEnvironment>  is a token environment stored in a binary
                            encoded string
   $RETURNS$
      <lState>              success of the initialization
   $DESCRIPTION$
      The TokenInit() function initializes a token environment. A token
      environment is the information about how a string is to be tokenized.
      This information is created in the process of tokenization of the
      string <cString> - equal to the one used in the Token() function
      with the help of the <cTokenizer> and <nSkipWidth> parameters.

      This token environment can be very useful when large strings have
      to be tokenized since the tokenization has to take place only once
      whereas the Token() function must always start the tokenizing process
      from scratch.

      Unlike CT3, this function provides two mechanisms of storing the
      resulting token environment. If a variable is passed by reference
      as 4th parameter, the token environment is stored in this variable,
      otherwise the global token environment is used. Do not modify the
      token environment string directly !

      Additionally, a counter is stored in the token environment, so that
      the tokens can successivly be obtained. This counter is first set to 1.
      When the TokenInit() function is called without a string a tokenize,
      the counter of either the global environment or the environment given
      by reference in the 4th parameter is rewind to 1.

      Additionally, unlike CT3, TokenInit() does not need the string
      <cString> to be passed by reference, since one must provide the
      string in calls to TokenNext() again.
   $EXAMPLES$
      TokenInit( cString )             // tokenize the string <cString> with default
                                       // rules and store the token environment globally
                                       // and eventually delete an old global TE
      TokenInit( @cString )            // no difference in result, but eventually faster,
                                       // since the string must not be copied
      TokenInit()                      // rewind counter of global TE to 1
      TokenInit( "1,2,3", "," , 1 )    // tokenize constant string, store in global TE
      TokenInit( cString, , 1, @cTE1 ) // tokenize cString and store TE in
                                       // cTE1 only without overriding global TE
      TokenInit( cString, , 1, cTE1 )  // tokenize cString and store TE in
                                       // GLOBAL TE since 4th parameter is
                                       // not given by reference !!!
      TokenInit( ,,, @cTE1 )           // set counter in TE stored in cTE1 to 1
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenInit() is compatible with CT3's TokenInit(),
      but there is an additional parameter featuring local token environments.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      Token(), TokenExit(), TokenNext(), TokenNum(), TokenAt(), SaveToken(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      TokenNext()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Successivly obtains tokens from a string
   $SYNTAX$
      TokenNext( <[@]cString>, [<nToken>],
                 [<@cTokenEnvironment>] ) -> cToken
   $ARGUMENTS$
      <[@]cString>             the processed string
      <nToken>                 a token number
      <@cTokenEnvironment>     a token environment
   $RETURNS$
      <cToken>                 a token from <cString>
   $DESCRIPTION$
      With TokenNext(), the tokens determined with the TokenInit() functions
      can be retrieved. To do this, TokenNext() uses the information stored
      in either the global token environment or the local one supplied by
      <cTokenEnvironment>. Note that, is supplied, this 3rd parameter has
      always to be passed by reference.

      If the 2nd parameter, <nToken> is given, TokenNext() simply returns
      the <nToken>th token without manipulating the TE counter. Otherwise
      the token pointed to by the TE counter is returned and the counter
      is incremented by one. Like this, a simple loop with TokenEnd() can
      be used to retrieve all tokens of a string successivly.

      Note that <cString> does not have to be the same used in TokenInit(),
      so that one can do a "correlational tokenization", i.e. tokenize a string
      as if it was another! E.G. using TokenInit() with the string
      "AA, BBB" but calling TokenNext() with "CCCEE" would
      give first "CC" and then "EE" (because "CCCEE" is not long enough).
   $EXAMPLES$
      // default behavhiour
      TokenInit( cString ) // initialize a TE
      DO WHILE ! TokenEnd()
         ? TokenNext( cString )  // get all tokens successivly
      ENDDO
      ? TokenNext( cString, 3 )  // get the 3rd token, counter will remain the same
      TokenExit()                // free the memory used for the global TE
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenNext() is compatible with CT3's TokenNext(),
      but there are two additional parameters featuring local token
      environments and optional access to tokens.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNum(), TokenAt(), SaveToken(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      TokenNum()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Get the total number of tokens in a token environment
   $SYNTAX$
      TokenNum( [<@cTokenEnvironment>] ) -> nNumberofTokens
   $ARGUMENTS$
      <@cTokenEnvironment>     a token environment
   $RETURNS$
      <nNumberofTokens>        number of tokens in the token environment
   $DESCRIPTION$
      The TokenNum() function can be used to retrieve the total number
      of tokens in a token environment.
      If the parameter <@cTokenEnvironment> is supplied (must be by
      reference), the information from this token environment is used,
      otherwise the global TE is used.
   $EXAMPLES$
      TokenInit( "a.b.c.d", ".", 1 )  // initialize global TE
      ? TokenNum()  // --> 4
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenNum() is a new function in Harbour's CT3 library.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNext(), TokenAt(), SaveToken(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      TokenEnd()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Check whether additional tokens are available with TokenNext()
   $SYNTAX$
      TokenEnd( [<@cTokenEnvironment>] ) -> lTokenEnd
   $ARGUMENTS$
      <@cTokenEnvironment>     a token environment
   $RETURNS$
      <lTokenEnd>              .T., if additional tokens are available
   $DESCRIPTION$
      The TokenEnd() function can be used to check whether the next
      call to TokenNext() would return a new token. This can not be
      decided with TokenNext() alone, since an empty token cannot be
      distinguished from a "no more" tokens.
      If the parameter <@cTokenEnvironment> is supplied (must be by
      reference), the information from this token environment is used,
      otherwise the global TE is used.
      With a combination of TokenEnd() and TokenNext(), all tokens from a
      string can be retrieved successivly (see example).
   $EXAMPLES$
      TokenInit( "a.b.c.d", ".", 1 )  // initialize global TE
      DO WHILE ! TokenEnd()
         ? TokenNext( "a.b.c.d" )     // get all tokens successivly
      ENDDO
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenEnd() is compatible with CT3's TokenEnd(),
      but there are is an additional parameter featuring local token environments.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNext(), TokenNum(), TokenAt(), SaveToken(), RestToken()
   $END$
 */

/* $DOC$
   $NAME$
      TokenExit()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Release global token environment
   $SYNTAX$
      TokenExit() -> lStaticEnvironmentReleased
   $ARGUMENTS$

   $RETURNS$
      <lStaticEnvironmentReleased>   .T., if global token environment is successfully released
   $DESCRIPTION$
      The TokenExit() function releases the memory associated with the
      global token environment. One should use it for every TokenInit()
      using the global TE. Additionally, TokenExit() is implicitly called
      when the thread or application ends.
   $EXAMPLES$
      TokenInit( cString ) // initialize a TE
      DO WHILE ! TokenEnd()
         ? TokenNext( cString )  // get all tokens successivly
      ENDDO
      ? TokenNext( cString, 3 )  // get the 3rd token, counter will remain the same
      TokenExit()                // free the memory used for the global TE
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenExit() is a new function in Harbour's CT3 library.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenNext(), TokenNum(), TokenAt(), SaveToken(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      TokenAt()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Get start and end positions of tokens in a token environment
   $SYNTAX$
      TokenAt( [<lSeparatorPositionBehindToken>], [<nToken>],
               [<@cTokenEnvironment>] ) -> nPosition
   $ARGUMENTS$
      <lSeparatorPositionBehindToken>   .T., if TokenAt() should return
                                        the position of the separator character
                                        BEHIND the token.
                                        Default: .F., return start position of a token.
      <nToken>                          a token number
      <@cTokenEnvironment>              a token environment
   $RETURNS$
      <nPosition>
   $DESCRIPTION$
      The TokenAt() function is used to retrieve the start and end position
      of the tokens in a token environment. Note however that the position of
      last character of a token is given by tokenat (.T.)-1 !!

      If the 2nd parameter, <nToken> is given, TokenAt() returns the
      positions of the <nToken>th token. Otherwise
      the token pointed to by the TE counter, i.e. the token that will
      be retrieved by TokenNext() _NEXT_ is used.

      If the parameter <@cTokenEnvironment> is supplied (must be by
      reference), the information from this token environment is used,
      otherwise the global TE is used.
   $EXAMPLES$
      TokenInit( cString ) // initialize a TE
      DO WHILE ! TokenEnd()
         ? "From", TokenAt(), "to", TokenAt( .T. ) - 1
         ? TokenNext( cString )  // get all tokens successivly
      ENDDO
      ? TokenNext( cString, 3 )  // get the 3rd token, counter will remain the same
      TokenExit()                // free the memory used for the global TE
   $STATUS$
      Ready
   $COMPLIANCE$
      TokenAt() is compatible with CT3's TokenAt(),
      but there are two additional parameters featuring local token
      environments and optional access to tokens.
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNext(), TokenNum(), SaveToken(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      SaveToken()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Save the global token environment
   $SYNTAX$
      SaveToken() -> cStaticTokenEnvironment
   $ARGUMENTS$

   $RETURNS$
      <cStaticTokenEnvironment>   a binary string encoding the global TE
   $DESCRIPTION$
      The SaveToken() function can be used to store the global TE for future
      use or when two or more incremental tokenizers must the nested.
      Note however that the latter can now be solved with locally stored
      token environments.
   $EXAMPLES$

   $STATUS$
      Ready
   $COMPLIANCE$
      SaveToken() is compatible with CT3's SaveToken(),
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNext(), TokenNum(), TokenAt(), RestToken(), TokenEnd()
   $END$
 */

/* $DOC$
   $NAME$
      RestToken()
   $CATEGORY$
      CT3 string functions
   $ONELINER$
      Restore global token environment
   $SYNTAX$
      RestToken( <cStaticTokenEnvironment> ) -> cOldStaticEnvironment
   $ARGUMENTS$
      <cStaticTokenEnvironment>     a binary string encoding a TE
   $RETURNS$
      <cOldStaticEnvironment>       a string encoding the old global TE
   $DESCRIPTION$
      The RestToken() function restores the global TE to the one encoded
      in <cStaticTokenEnvironment>. This can either be the return value
      of SaveToken() or the value stored in the 4th parameter in a
      TokenInit() call.
   $EXAMPLES$

   $STATUS$
      Ready
   $COMPLIANCE$
      RestToken() is compatible with CT3's RestToken(),
   $PLATFORMS$
      All
   $FILES$
      Library is hbct.
   $SEEALSO$
      TokenInit(), TokenExit(), TokenNext(), TokenNum(), TokenAt(), SaveToken(), TokenEnd()
   $END$
 */
