Scilab Website | Contribute with GitLab | Mailing list archives | ATOMS toolboxes
Scilab Online Help
6.1.1 - Français

Change language to:
English - 日本語 - Português - Русский

Please note that the recommended version of Scilab is 2024.0.0. This page might be outdated.
See the recommended documentation of this function

Aide de Scilab >> Chaînes de caractères > tokens

tokens

Splits a string using separators and gives its chunks

Syntax

Chunks = tokens(str)
Chunks = tokens(str, separators)

Arguments

str
A single string to be split. It can include extended UTF-8 international characters.

separators
Vector of characters. Default value = [" ", ascii(9)], ascii(9) being the horizontal tab.

Chunks
Column vector: Pieces of the split string.

Description

tokens(…) searches the string for given separator(s), and splits it into chunks. Chunks are free of separators. Consecutive separators are merged.

Examples

tokens("The given   text")

tokens("SCI/demos/scicos", "/")'

tokens("Επιστήμη και καινοτομία", ["α"," "])'

nbsp = ascii(160); // non-breakable space
t = "the" + nbsp + "given   text"
tokens(t)
--> tokens('The given   text')
 ans  =
  "The"
  "given"
  "text"


--> tokens('SCI/demos/scicos', '/')'
 ans  =
  "SCI"  "demos"  "scicos"


--> tokens("Επιστήμη και καινοτομία", ["α"," "])'
 ans  =
  "Επιστήμη"  "κ"  "ι"  "κ"  "ινοτομί"


--> nbsp = ascii(160); // non-breakable space
--> t = "the" + nbsp + "given   text"
 t  =
  "the given   text"

--> tokens(t)
 ans  =
  "the given"
  "text"

See also

  • strsplit — split a single string at some given positions or patterns
  • regexp — find a substring that matches the regular expression string
  • strindex — search position of a character string in another string
  • tokenpos — returns the tokens positions in a character string
Report an issue
<< tokenpos Chaînes de caractères Sons - fichiers audio >>

Copyright (c) 2022-2023 (Dassault Systèmes)
Copyright (c) 2017-2022 (ESI Group)
Copyright (c) 2011-2017 (Scilab Enterprises)
Copyright (c) 1989-2012 (INRIA)
Copyright (c) 1989-2007 (ENPC)
with contributors
Last updated:
Mon Jan 03 14:33:08 CET 2022