StringTokenizer

The processing of String often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimitersare characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

StringTokenizer Example

import java.util.*;
class StringTokenizerDemo
{
 public static void main(String args[])
 {

 String st1="this is a test";
 String st2="this,is,a,test";
 String st3="this.is.a.test";

 // Create a StringTokenizer with default delimeter i.e. space
 StringTokenizer s1=new StringTokenizer(st1);
  // Create a StringTokenizer with ',' as delimeter
 StringTokenizer s2=new StringTokenizer(st2,",");

 // Create a StringTokenizer with default delimeter i.e. space
 StringTokenizer s3=new StringTokenizer(st3);

  // Print the no.of tokens split by delimeter above
 System.out.println("No.of tokens for s1 are "+s1.countTokens());
 System.out.println("No.of tokens for s2 are "+s2.countTokens());
 System.out.println("No.of tokens for s3 are "+s3.countTokens());

  // Loop till hasMoreElements return false
  while(s1.hasMoreElements())
  {
  System.out.println(s1.nextToken());
  }  
  System.out.println();  
  // Loop till hasMoreTokens return false
  while(s2.hasMoreTokens())
  {
  System.out.println(s2.nextElement());
  }
  System.out.println();

  // Loop till hasMoreElements return false
  while(s3.hasMoreTokens())
  {
  System.out.println(s3.nextToken("."));
  }
 }
}

 OUTPUT

No.of tokens for s1 are 4
No.of tokens for s2 are 4
No.of tokens for s3 are 1
this
is
a
test

this
is
a
test

this
is
a
test

 

Explanation

s1=new StringTokenizer(st1): Create StringTokenizer for the first string st1. The default delimeter is a blank space. A delimeter is nothing but a string that is used for separating another string.

s2=new StringTokenizer(s2,","): Here, we have specified ',' as the delimeter. So this is used to split the string s2 into parts called tokens separated by a comma. This acts as s2.split(",").

s3=new StringTokenizer(st3): Create StringTokenizer with default delimeter pointing st3.

s1.nextToken(): Get's the next token in the string nothing but the word that is after the delimeter (here word after the space). You'll have to note that this also includes the first token though it doesn't have the specified delimeter (space) before it i.e. the first word this

 

s2.nextElement(): Same as the above, but not the set of characters between a space, but the set of characters between a comma also including the first one.

s3.nextToken("."): This returns the token enclosed between two dots (.) also including the first word (this) but not space. Though the delimeter was not specified in the constructor, it was specified here (i.e. in the nextToken(".") method) to get the tokens enclosed between dots.

Note: The s3.countTokens() method returns only one i.e. only the first token is returned as the default delimeter was space and there was no space found in the string st3 so the first one (from starting till ending) was returned.