tokens

Arguments

str

A single string to be split. It can include extended UTF-8 international characters.

separators

Vector of characters. Default value = [" ", ascii(9)], ascii(9) being the horizontal tab.

Chunks

Column vector: Pieces of the split string.

Description

tokens(…) searches the string for given separator(s), and splits it into chunks. Chunks are free of separators. Consecutive separators are merged.

Examples

tokens("The given   text")

tokens("SCI/demos/scicos", "/")'

tokens("Επιστήμη και καινοτομία", ["α"," "])'

nbsp = ascii(160); // non-breakable space
t = "the" + nbsp + "given   text"
tokens(t)

--> tokens('The given   text')
 ans  =
  "The"
  "given"
  "text"


--> tokens('SCI/demos/scicos', '/')'
 ans  =
  "SCI"  "demos"  "scicos"


--> tokens("Επιστήμη και καινοτομία", ["α"," "])'
 ans  =
  "Επιστήμη"  "κ"  "ι"  "κ"  "ινοτομί"


--> nbsp = ascii(160); // non-breakable space
--> t = "the" + nbsp + "given   text"
 t  =
  "the given   text"

--> tokens(t)
 ans  =
  "the given"
  "text"

See also

• strsplit — split a single string at some given positions or patterns
• regexp — in a string, locate (and extract) substrings matching a regular expression
• strindex — search position of a character string in another string
• tokenpos — returns the tokens positions in a character string