Tokenize string

Str::Tokenize

Str Tokenize (const STRCHAR* tokens, CPOS& startat) const

By calling this method one or more times, you can extract information (substrings) from a string, separated by specified single-character tokens. The method is similar to the C RTL function strtok but it is fully thread safe and does not destroy the string object during the extraction.

The tokens parameter should be a string containing one or more characters that will be accepted as tokens. These will typically be spaces, tab characters, colons, semicolons, but you can also pass any other non-null characters. If you use letters as tokens, be aware that the tokenizer is case-sensitive.

The startat parameter must initially contain the index of the first character at which tokenization must begin (usually, zero). After each call, this will be updated with the index of the first character after the extracted substring.

When there are no more substrings to be extracted, the returned string will be empty, and the startat parameter will contain the special value -1.

Example:

   const STRCHAR* tokens = _T(" #~");
   Str sfull (_T("~Alpha Beta#Gama~"));
   CPOS nextp = 0;
   do {
      Str res = sfull.Tokenize(tokens, nextp);
      // res will contain the values Alpha, Beta and Gama here
   } while (nextp > 0);