I know I said I'll restrain from posting for a short while but … it's not an easy task. It's some stuff you need to talk about :) and at least here I know there's some who actually understand it.

Going after the sentences!

A short while back I published a tutorial on how to easily find and extract text blocks within HTML. It is time to evolve. I'm sure many have put that to good use and applications are only limited by imagination but I'd like to take this further.

So we have the text blocks from content. Those text chunks are separate slices of content. They are not part of the same phrase or sentence. They are different. So why don't we extract actual text sentences from that text? I bet sentences will be much more useful than those text chunks in terms of blackhat content generation and so on.

Locating sentences. What makes them special?

In order to find a sentence we need to understand the structure of the sentence, the way it looks and feels, the way it moves and the way it smells :) Of course I will not teach you semantics here. Semantics is like freestyle. You have the elements but your creativity in how you combine them is the limit.

There are 4 simple steps to follow in order to break text blocks into sentences:

Weld them using one of the obvious separators: .?!|
Split them using the obvious separators: .?!|
Remove between ( … ) { … } [ … ] - (usually brackets contain extra comments which are not part of sentence flow)
Validate each chunk against a set of rules I will show below!
Weed out the invalid and Keep the hot and pretty ones that make them wobots horny.

Sentence 101 - Looks and Feels!

I will enunciate the first rule you need to understand about sentences. If you understand this one everything else will flow:

Sentences are like women. There are so many you can afford to be picky about the ones you choose!

After I said that I feel feminine anger. But I don't think girls read this blog so I don't worry about that.

There is one basic rule for a sentence. It starts with a Letter (Capital would be best) and ends with letters too. Last words should have over 3 characters unless you aim for sex. And let me get one thing clear. These two sentence features already ruled out a lot but think about the women. We're gonna be picky here!

Sentence model varies - but there are outlines!

First thing we need to understand is that each topic has a different sentence model. Some are slim(many short words), some are fat(many long words), some are short(not summer short), some are tall, some contain many numbers and some contain a lot of punctuation signs. Sentence model varies based on topic. E.g.: You will find a certain type of rather short and concise sentences in an ecommerce store, and long ones with lots of words in a content site.

But no matter how much the models vary there are rules that no viable sentence can break (many will but those are not viable). I made up these rules by analyzing several sentences that pass my judgment filters in term of viability. The tests I ran them against were:

Upper and Lower and all letters count.
Words count (count of words in sentence).
Words length (maximum, minimum and average length of words).
Numbers count (number words or digits).
Consecutive punctuations (a , followed by . followed by , does not make a lot of sense).
… and several others I'll let you brain-storm about!

By checking some sample sentences I came to some rules of common sense-tence (get it … sensetence :)).

Upper-Case characters should not exceed 5% of letter count
Digits should not exceed 5% of letter count
Punctuation signs should not exceed 5% of letter count
Percentage of Unique Words out of Word Count should not be less then 50%
Should start with Upper-Case
Should have more then 5 words and less then … fill this in yourself!
etc …

Add to the above you own rules to filter sentences or change the above to fit your needs. (When searching for cellphone models you need to allow more digits and so on ….) These rules may seem strict but there's so many ~~women~~ sentences out there you can and will be picky!

Putting it all together!

You extract text chunks. You weld, split, verify each sentence against your own set of rules and be picky about them! Dump duplicates.

Stay tuned as there's more to come in future text scraping and content generation episodes. We'll talk some word stemming and more juicy stuff. And … don't forget the comment form :)

13 Comments Posted By Readers :

Add your comment

#1 Garcia from Bolivia web

Posted on Saturday, 09 February, 2008

I was just thinking about this a couple of days ago. Not for content generation but for a short description of a post to show in the home page. You've come up with really good ideas... good job.

#2 5ubliminal web

Posted on Saturday, 09 February, 2008

Thanks :) Glad to see you're still alive.
This can be used for yourself to index content on site or extract sentences that contain some keywords and so on.
Like creating snippets for your articles.

#3 Paul D from United States

Posted on Sunday, 10 February, 2008

It would take significantly more time but shouldn't you also create rules on language using...
I don't know..perhaps a database you gave our a few months ago ;) Just make your life that much easier when you start to spin the content using that same database.

#4 5ubliminal web

Posted on Sunday, 10 February, 2008

I'm not really sure I got your comment but this article is one in a longer series. One bit at a time. This is about finding sentences.
These sentences that come out with this algo are 99% quality slices of text. Are material to work on.
What you (I) do with them is a whole different thing … :)

We'll go over parts of speech, keyphrase searching and so on.

#5 Gab from Canada web

Posted on Sunday, 10 February, 2008

So ... looking to rank for text chunks, and word stemming huh? Why not try Eli on for size and try and rank for "spammy text" and "spammy spam spam text" :D? Anyways, nice tutorial and clever point abut the brackets.

#6 5ubliminal web