RWCTokenizer(3C++) RWCTokenizer(3C++)
NameRWCTokenizer - Rogue Wave library class
Synopsis
#include <rw/ctoken.h>
RWCString str("a string of tokens");
RWCTokenizer(str); // Lex the above string
Description
Class RWCTokenizer is designed to break a string up into separate tokens,
delimited by an arbitrary "white space." It can be thought of as an
iterator for strings and as an alternative to the ANSI C function
strtok() which has the unfortunate side effect of changing the string
being tokenized.
Persistence
None
Example
#include <rw/ctoken.h>
#include <rw/rstream.h>
main(){
RWCString a("Something is rotten in the state of Denmark");
RWCTokenizernext(a); // Tokenize the string a
RWCString token; // Will receive each token
// Advance until the null string is returned:
while (!(token=next()).isNull())
cout << token << "0;
}
Program output:
Something
is
rotten
in
the
state
of
Denmark
Public Constructor
RWCTokenizer(const RWCString& s);
Page 1
RWCTokenizer(3C++) RWCTokenizer(3C++)
Construct a tokenizer to lex the string s.
Public Member Operators
RWCSubString
operator();
Advance to the next token and return it as a substring. The tokens are
delimited by any of the four characters in " 0. (space, tab, newline and
null).
RWCSubString
operator()(const char* s);
Advance to the next token and return it as a substring. The tokens are
delimited by any character in s, or any embedded null.
RWCSubString
operator()(const char* s,size_t num);
Advance to the next token and return it as a substring. The tokens are
delimited by any of the first num characters in s. Buffer s may contain
nulls, and must contain at least num characters. Tokens will not be
delimited by nulls unless s contains nulls.
Page 2
