Scilab Home page | Wiki | Bug tracker | Forge | Mailing list archives | ATOMS | File exchange
Please login or create an account
Change language to: Français - Português - 日本語 - Русский
Scilab Help >> Strings > tokens


Splits a string using separators and gives its chunks


Chunks = tokens(str)
Chunks = tokens(str, separators)


A single string to be split. It can include extended UTF-8 international characters.

Vector of characters. Default value = [" ", ascii(9)], ascii(9) being the horizontal tab.

Column vector: Pieces of the split string.


tokens(…) searches the string for given separator(s), and splits it into chunks. Chunks are free of separators. Consecutive separators are merged.


tokens("The given   text")

tokens("SCI/demos/scicos", "/")'

tokens("Επιστήμη και καινοτομία", ["α"," "])'

nbsp = ascii(160); // non-breakable space
t = "the" + nbsp + "given   text"
--> tokens('The given   text')
 ans  =

--> tokens('SCI/demos/scicos', '/')'
 ans  =
  "SCI"  "demos"  "scicos"

--> tokens("Επιστήμη και καινοτομία", ["α"," "])'
 ans  =
  "Επιστήμη"  "κ"  "ι"  "κ"  "ινοτομί"

--> nbsp = ascii(160); // non-breakable space
--> t = "the" + nbsp + "given   text"
 t  =
  "the given   text"

--> tokens(t)
 ans  =
  "the given"

See also

  • strsplit — split a single string at some given positions or patterns
  • regexp — find a substring that matches the regular expression string
  • strindex — search position of a character string in another string
  • tokenpos — returns the tokens positions in a character string
Scilab Enterprises
Copyright (c) 2011-2017 (Scilab Enterprises)
Copyright (c) 1989-2012 (INRIA)
Copyright (c) 1989-2007 (ENPC)
with contributors
Last updated:
Mon Jan 03 14:23:29 CET 2022