RetireJapan

I was bored and run my webcrawler on retirejapan.com, specifically limited to this forum. Sorry, Ben!

I crawled 1514 unique pages, extracted all words from those paged and collected their frequencies after removing so-called stop words (e.g. “the”, “a”, “I”, etc) and lemmatization. This is what we are talking about:



It is a wordcloud generated for the 1000 most frequently found words

It is far very far from perfect (e.g. many words that can be found on each page of the forum such as "post" and date related words are at the top of list). Anyway, anything interesting standing out for you? I noticed:

• nisa is more frequently found than ideco
• stocks more than bonds
• buy more than sell

Ha, ha, that is great! I wonder if there is any way to exclude the site operational words...

Very interesting

Ah, it’s a rhetorical question! That is a productive waste of time indeed!

RetireJapan wrote: ↑Sat Nov 21, 2020 5:19 am I wonder if there is any way to exclude the site operational words...

Yes, now I simply extracted all words from a page, but if I will take a deeper look at the HTML structure I can probably narrow it down to only extract words from the actual posts, the unique contents. I will give that a try next time.

Yeah as it stands the thread should really be named "What are the most common words used by templates on this forum?"

The word "post" appears on a typical full page 34 times even it nobody wrote it once, for example.