RWWTokenizer(3C++) RWWTokenizer(3C++)
NameRWWTokenizer - Rogue Wave library class
Synopsis
#include <rw/wtoken.h>
RWWString str("a string of tokens", RWWString::ascii);
RWWTokenizer(str); // Lex the above string
Description
Class RWWTokenizer is designed to break a string up into separate tokens,
delimited by arbitrary "white space." It can be thought of as an
iterator for strings and as an alternative to the C library function
wstok() which has the unfortunate side effect of changing the string
being tokenized.
Persistence
None
Example
#include <rw/wtoken.h>
#include <rw/rstream.h>
main(){
RWWString a(L"Something is rotten in the state of Denmark");
RWWTokenizernext(a); // Tokenize the string a
RWWString token; // Will receive each token
// Advance until the null string is returned:
while (!(token=next()).isNull())
cout << token << "0;
}
Program output:
Something
is
rotten
in
the
state
of
Denmark
Public Constructor
RWWTokenizer(const RWWString& s);
Page 1
RWWTokenizer(3C++) RWWTokenizer(3C++)
Construct a tokenizer to lex the string s.
Public Member Function
RWWSubString
operator();
Advance to the next token and return it as a substring. The tokens are
delimited by any of the four wide characters in L" 0. (space, tab,
newline and null).
RWWSubString
operator()(const wchar_t* s);
Advance to the next token and return it as a widesubstring. The tokens
are delimited by any wide character in s, or any embedded wide null.
RWWSubString
operator()(const wchar_t* s,size_t num);
Advance to the next token and return it as a substring. The tokens are
delimited by any of the first num wide characters in s. Buffer s may
contain embedded nulls, and must contain at least num wide characters.
Tokens will not be delimited by nulls unless s contains nulls.
Page 2
