I want to tokenize Twitter messages including hash- and cash-tags. A correct example for tokenization would be like this:
"Bought $AAPL today,because of the new #iphone".match(...);>>>> ['Bought', '$AAPL', 'today', 'because', 'of', 'the', 'new', '#iphone']
I tried several regexes for this task, ie:
"Bought $AAPL today,because of the new #iphone".match(/\b([\w]+?)\b/g);>>>> ['Bought', 'AAPL', 'today', 'because', 'of', 'the', 'new', 'iphone']
and
"Bought $AAPL today,because of the new #iphone".match(/\b([\$#\w]+?)\b/g);>>>> ['Bought', 'AAPL', 'today', 'because', 'of', 'the', 'new', 'iphone']
and
"Bought $AAPL today,because of the new #iphone".match(/[\b^#\$]([\w]+?)\b/g);>>>> ['$AAPL', '#iphone']
Which regex could I use, to include the leading sharp or dollar sign in the tokens?