5ubliminal@twitter

Scraping At The Speed Of Light | Breaking TCP/IP Limitations : 5ubliminal's TellinYa

<a href="http://www.tellinya.com/art2/325/">Scraping At The Speed Of Light | Breaking TCP/IP Limitations : 5ubliminal's TellinYa</a>
Must Reads: Web Scraping | Link Farming | Code Snippets | SEO Freeware » I'm back to work … sorting my shit now :)
Reveal More!
TCP/IP Limit
Who should read this Post?

First of all this won't affect the average user. All those who use multi threaded downloads in areas such as: scraping, file sharing, port scanning, … And this applies to all of you who have large bandwidth and need many concurrent connections established simultaneously.

What is the TCP/IP limitation?

Fighting worm spreading at the speed of light, Microsoft decided it's time to TCP/IP limit their non-server editions. So instead of fixing the problem the decided to limit the speed of the worm spread. They have a limit present in XP SP2 (not in SP1) and XP x64 editions (maybe Vista too) that will not allow you to have more than 10 half-open connections during one second. And this limits a lot of legitimate uses of TCP/IP:)

What is a half-open connection?

When you use the Sockets connect() function there are several ICMP requests that travel between sides before your function actually returns and gives you the success or failure notification. In that small delay between connect() is called and connect() succeeds a socket is called half-connected. In that short interval Microsoft imposed the limit on sockets in Windows.

So if you start a multi threaded downloader with over 10 threads, threads number 10-19 will only connect in second two giving you a one second delay. It may seem small but it does matter and the difference is huge for me as I connect 100s of sockets at same time.

Half-Open connections' limit in plain English!

Let's say you are in a rock throwing contest. You are the best, a freaking cannon and you can throw 100s of rocks in one second. But there's a rule which will never allow you to show your true skill. The rule claims: no more then 10 stones can be in the air during one second. So if you throw 10 rocks in 1/10 of a second you will have to wait the rest of 9/10 of a second. But you could have thrown 100 rocks per second yet they limit you to 10.

How can I fix this dreaded limitation?

I found out about the limitation here and I found my XP x64 Pro TCP/IP Patch (the 1000 limit) here. These sites will show you where to get the patch and how to apply it to tcpip.sys. Do not use tools to patch. Get a patched tcpip.sys file, start Windows in Safe Mode without networking support and backup the old file in several places (and rename it to tcpip.old) :) and overwrite. Patching executables can't patch it as it's open at all times in a non safe-mode environment and it's better to backup and overwrite it by hand.

This has been the difference between heaven and hell for me. I'm consuming my entire bandwidth now when doing the nasty things! Do let me know how it feels…

16 Comments Posted By Readers :

Add your comment
#1 Dan from United States
Posted on Tuesday, 11 March, 2008
Other Solution - > Don't use windows :)
#2 5ubliminal web
Posted on Tuesday, 11 March, 2008
@Dan: Unacceptable :)
#3 Papa Rage from United States
Posted on Tuesday, 11 March, 2008
A) don't use windows
B) if you have to use windows, definitely fix the max connects per second issue
C) do reconsider dropping windows
D) still unacceptable? might want to tweak your mtu and enable tcp window scaling (unless you are behind a router that doesn't support it)

FYI, I develop and test stuff on windows XP at home and then deploy to linux servers.
#4 5ubliminal web
Posted on Tuesday, 11 March, 2008
@PR: This entire post was comment-bait for you:) Always looking for ideas to make whole thing faster and better and you got some of'em.

I've been coding on Windows for so long I'm not ready to start learning the internals of other Operating Systems.
I also depend on Windows software and such, and no time to decypher other OSes right now.

I'll think about it in the near future … who knows.

PS: Don't tell me yo'll blackhats use ****x operating systems.
#5 Dan from United States
Posted on Tuesday, 11 March, 2008
PHP scripts run best on *nix :D
#6 5ubliminal web
Posted on Tuesday, 11 March, 2008
@Dan: I guessed you use PHP but I code in C++ the heavy stuff where I need 100% CPU usage for hours.
#7 Papa Rage from United States
Posted on Tuesday, 11 March, 2008
Smart guy like you? You'd pick it up faster than you realize. You don't have to be a guru, you just have to be able to launch your code. I'm not an expert at all, but in the end figuring stuff out is the same as windows, you google your problem, read/post on forums and tinker until you have your solution.

It only requires one thing: portable code FTW !

I'm not a linux zealot or a MS hater, and I'm not trying to convert anyone. If what you're doing works then you're golden. But you do learn a lot when going to linux if you can find the time. And a lot more hosting options AFAIK.
#8 5ubliminal web
Posted on Tuesday, 11 March, 2008
Thanks:) Berkley socket functions are the same (Win and UNIX), file access functions are pretty much the same but … I also rely on GUI programming and more OS system related tricks.

The truth is my code runs very well, stable and fast on Windows. I got tools that pull GIGs from the web (MEGs per second), parse and process them, keep (the quad-core) CPU usage to 100% for hours or days and no crashes or memory problems. I have no software using less than 10 threads:) But I'm always trying to speed things up and the TCP/IP limit patch helped very much and, as I'm fully using bandwith I'd say there's no need to improve the TCP/IP speed anymore.
My next target is stripping RegExp away as much as possible from my code.

UNIX network coding is cool I know but the TCP/IP stuff is just a tiny part of what I do.
#9 Papa Rage from United States
Posted on Tuesday, 11 March, 2008
Regex can be expensive on large pages, but it is obviously very powerful.

Boyer-Moore is a good string searching algorithm. When it comes to cpu usage, profiling is your friend.
#10 5ubliminal web
Posted on Tuesday, 11 March, 2008
Could you please define profiling?

RegExp is awesome but in most cases it can be replaced with custom-algos written for a specific purpose for hyper-fast parsing.
#11 Papa Rage from United States
Posted on Wednesday, 12 March, 2008
You have jobs that run for hours/days with 100% CPU, your code is screaming to get profiled. The modern CPU profiler is a tool that will run side by side with your code for as long as you want and then after, it will display for you a visual graph of what functions are taking the most time.

It will let you drill down into the expensive functions and see how much time your code is taking and how much time is spent on the functions you call.

You have to know for sure what parts of you code are slowing you down, before you can make it faster. Otherwise you are just guessing.

Running a profiler can be an eye-opening experience. Recently I profiled a web app of mine because I was impressed with how efficient it was and I wanted to fine tune it. Profiling showed that 80% of the time was spent on disk access. A bug on 2 lines of heavily used code was causing hundreds of xml files to be read fully from disk for every page load instead of being cached. After that I found that a lot of time was being wasted in string functions and in a specific 3rd party library.

A couple days profiling and I changed the load time of those web app pages from about 480ms to about 25ms. So much for fine tuning, I hadn't even gotten the major kinks out before profiling. Cutting the run time down to 1/20 of what it originally was is an extreme example, but that's why it's memorable.

You could say that 480ms to load a page is fast enough, but you need that speed when you are under heavy load. You make small decisions at every turn while coding, and it’s not obvious which ones will have huge performance impact.

This link is to a C# profiler, I only post it because it has couple good screen shots, not as a recommendation (I've never used it). http://www.jetbrains.com/profiler/features/newfeatures.html#Performance_profiling_enhancements
#12 5ubliminal web
Posted on Wednesday, 12 March, 2008
Funny, few hours ago I finished ditching some of my RegExp for the scrapers I use and have cut down execution time to less then 30% with less load on CPU. I profiled my string functions and noticed a RegExp was taking 3-5 seconds to run on some large files and I replaced it with a C++ equivalent hand-jobed for the purpose and cut execution time from 3-5 seconds to an average of 300 miliseconds.

Profiling is quite a fancy term but I know realize I've been doing this using my own stone-age tools: a class combining QueryPerformanceFrequency and QueryPerformanceCounter (or GetTickCount() - for miliseconds) and using macros with __LINE__, __FILE__, __FUNCTION__ to record execution time between lines of code in terms of microseconds separated on threads. And it works rather well. I usually avoid disk reads and I cache virtually everything that can fit the memory.

I'm curious, do these tools need to integrate with code or they can just blend in with your applications with no special arrangements? I'll be looking for a C++ profiler, the .NET versions are not good for me but, meanwhile, my 'code lines heart-beat listenting' with the two great functions MSVC++ provides do the magic. I try at all costs to avoid 3rd party stuff. If they built it I can do it too:)

PS: Thanks for the long and elaborate reply:) I've been building my code for many years now but I'm currently in the place where I think in terms of microseconds and not miliseconds about execution times and a huge overhaul especially on string parsing is taking place:)
#13 Papa Rage from United States
Posted on Wednesday, 12 March, 2008
The profilers I've used require a special launch configuration where the profiler actually launches your code and it doesn't blend well with your application. I usually use 30 day trials. And since it never takes me that long to fix the slow spots, I never want to spend the money when the trial is up.

The big benefit I've seen of a profiler over hand coded timings is it lets you find things that you never thought would be slow to begin with. The bugs that don't break your app but do slow it down. And memory leaks.
#14 5ubliminal web
Posted on Wednesday, 12 March, 2008
I'll look into this as I see VC++ has an integrated profiler which I gotta see how it's supposed to work.
I'm just curious how it would blend in with an application that runs packs of threads with different purposes, running at same time usually not less than 50 - 500 :)

I'll get one of these trials next time I run into a slow-down which is not cause by the only slow type of code I use: advanced RegExp.

PS: I don't worry about memory leaks as my tools usually run in terms of 5MB - 200MB - 500MB RAM and then go back to 10MB :) My memory management is quite reliable with the use of classes and self freeing pointer lists that do all the hard work of keeping count of pointers and freeing them when they are no longer required.
#15 Autodidact from United States
Posted on Thursday, 03 April, 2008
@Subliminal

The profiler I use at work handles profiling multiple threads very well. It just adds up the total time spent on each line and then shows a line by line and function by function break down of the time spent executing (inline with the source code). The overhead of running a profiler seems (in my individual and completely bias opinion) to add almost 200-300% execution time.

The built-in profiler for VC++ might only be for the 'Managed' version of C++, so watch out for that, unless you are running CLR code.

I'm spoiled, I use vb.net and get memory management for (almost) free.
#16 5ubliminal web
Posted on Thursday, 03 April, 2008
@Autodidact: Never have I touched CLR:) I like it the C++ traditional hard-core way. I love speed and I love to handle memory myself:)
I finally found a method to profile. The C++ one didn't do.
As mentioned before, I use a combination of Performance Counters and I place them strategically where I know I may have problems.
I get a log with duration between lines of code measured in nanoseconds. And it works very well.
I had performance improved on some functions (like concatenating 250k+ strings) with over 2500%.
Some functions went from 5 seconds to 0.02 seconds.

It's much easier for the .NET coders regarding memory and … everything … but I chose the hard way knowing what I was getting into.
PS: Not to mention I started coding in '97 and no managed code was in sight:) And it was a good thing for me.
Post Feedback 
Name *
Mail *
URL
« Anti-Spam
» URL will only go live after a review. Comments are moderated. «
5ubliminal's TellinYa.com SEM & SEO Blog © 2007 - All rights reserved unless mentioned otherwise .
Rendered On : [Tuesday, 07 October, 2008 - 12:51:26 GMT]   No Ajax / Flash Used Here
" Scraping At The Speed Of Light | Breaking TCP/IP Limitations : 5ubliminal's TellinYa "