Clustering time

[WoolyHat]

Hola,

So, I downloaded beta 10, deleted my previous db and started a new import. With previous betas, I just imported a few hands to fiddle with the product. This time, I decided to push it a bit and imported 13.5m hands. The import took about 34 hours, but it's been clustering for a further 12 hours since then. Is this typical? Any idea how much longer it might take?

Thanks.

[APerfect10]

:shock:

With that many hands, 12+ hours of clustering sounds about right.

I am going to warn you though...if it took you 34 hours to import those hands then your computer is more than likely not fast enough to handle that and stat retrieval is going to be slow.

Best regards,

Derek

[WoolyHat]

Yeah, I was getting worried about that. What spec do you guys recommend for PT3? The machine currently importing is an Intel 3Ghz Core Duo, with 2Gb of memory and 200Gb of free disk space. Running no other processes, but PT3.

I was really just testing it with this import, and will probably split my hands down into multiple DBs in future. What kind of DB size do you think I can get to before I start running into serious performance problems?

[APerfect10]

A decent Core2Duo should be sufficient and 2GB of RAM is fine. Free space is not as important on your computer as the speed of your hard drive. If you want to spend any money to upgrade your computer, buy two (2) 10K RPM hard drives and put them in Raid 0. You will notice improvements up to 10x.

To be honest, there are so many variables in determining what database size your computer can handle there is no way for me to even guesstimate. I would think you should be able to handle a few million under your current setup. If you upgraded to two 10K RPM hard drives in Raid 0; I think 10M+ hands would be no issue.

Best regards,

Derek

[oracle3001]

I am having an issue with clustering.

I imported about 2-3 million hands and did the clustering which obviously took quite a while (few hours). Then, while this was happening I had been datamining so more, so I manually imported them (another 14k hands), and at the end of the import the clustering algorithm ran again. As I write this post, the clustering has been going on for over 30mins, should this be expected? Should I not be clustering if I only import a few thousand new hands?

Another issue I have found is that while I was doing the import I was also configuring the stats for my HUD. When the clustering began, the window focus was lost and now I can no longer edit my HUD settings, even though the configure stats window is still open.

[WhiteRider]

I think the recommendation is to cluster about every 50,000 hands imported.

[oracle3001]

It is set on by default for all manual imports. Maybe some further consideration given to this, if it is going to take a very long time to cluster even after importing just a few hands.

I am now very frustrated as I am unable to stop the clustering, and it is likely I will now have to wait several hours after importing just a few thousand hands. I can imagine lots of other people will have the same issues and frustrations.

[asdfg]

A decent Core2Duo should be sufficient and 2GB of RAM is fine. Free space is not as important on your computer as the speed of your hard drive. If you want to spend any money to upgrade your computer, buy two (2) 10K RPM hard drives and put them in Raid 0. You will notice improvements up to 10x.

To be honest, there are so many variables in determining what database size your computer can handle there is no way for me to even guesstimate. I would think you should be able to handle a few million under your current setup. If you upgraded to two 10K RPM hard drives in Raid 0; I think 10M+ hands would be no issue.

Best regards,

Derek

Hello Derek,

because I am new to this forum and I am a first time user of your product let me first of all say you did an excellent job!

But I have to say that your respond to WoolyGat's post, well, could be misinterpreted ...

If I understand WoolyHat's posting correctly he says that he imported 13500000 hands in 34 hours (without clustering, etc.). That means the import speed was about 400000 hands / hour, about 6600 hands / minute or about 110 hands / second.

And you are saying that a more powerful system could improve the import speed by a factor of 10.

Well, I do not judge this number but I give you some numbers from my tests.

I imported 3000000 hands with an import speed of 240 hands / second on a Quad Core Xeon 2,33 Ghz with 4 GByte RAM and on 2 146 GByte SAS HD (15k RPM) without RAID 0 configuration running on XP SP 2 - no additional software running (Anti-Virus software) and a tuned (seperated data and xlog folders, etc.) PostgreSQL installation. Even with a RAID 0 set up - which could not more then double this performance by definition - I am still far of your numbers ...

So I think it would be better to take a different approach in answering such postings. For example to describe you test system (hardware, software, configuration, etc.) and test data and what results you were seeing ...

With regards,

Gerhard

[mofo]

Yes the clustering is annoying when you accidentally cause it to start when you don't need it. What exactly is it doing? It can't be done while importing hands/using the db? Would it be possible to add a feature where you could schedule a time for clustering, for example every night 05:00 AM, and PT3 would automatically pause any auto-importing that is happening at the time, begin clustering, and resume autoimporting?

[Josh]

I would definitely like to implement something like that.