Tokenization, at its core , is the act of separating a larger piece of data into smaller units called tokens . Think of it like segmenting a paragraph into items . These elements can then be analyzed https://poppiehjkf195843.webbuzzfeed.com/42273112/tokenization-explained-a-beginner-s-guide