PT3 DB Conversion Issue?

[oracle3001]

I am trying to convert a PT3, it extracted quickly, and now is trying to import those hands at an incredibly slow rate of 5 h/s. These are all iPoker hands.

I have brought up the "View Import Status" and it keeps saying "Adjusting session stakes" followed by "Updating the Cache"...over and over again. Hands are being imported as this is going on, but the whole process is taking so long it is completely pointless. I have a million hands to import and after 10mins it has only done 3k of them. This is into a nearly empty db (has a few 1k hands).

When I first started PT4 I did the same with my main DB and it all seemed to progress at an ok, albeit a bit disappointing h/s rate, I think 350k hands in 90 minutes or so.

Edit:- It seems to be grinding to a complete standstill now. Down to 1 h/s.

[kraada]

This appears to be an issue with Automatically Apply Treat As (which in PT3 used to be Adjust Qualifying Table Limits) - we're going to look into what we can do to optimize import speed with that function enabled. It is going to make everything slower though because we have to constantly check to see if the table is "really" a table of another limit.

[oracle3001]

:-(

That sounds bad all round. I have to say, the only real thing that I am disappointed with so far with PT4 is the import speed of existing hands. Even with hands that aren't using that function I have only getting about a 1/3 of the speed of your competitors product on the same hardware etc, and appears even slower than PT3 (even when you include the house keeping).

My hardware is a bit old and looking to upgrade fairly soon, but neither is it really clapped out / under powered. Maybe my postgres settings could be further tweaked, but as mentioned above I was getting a decent to good rate with the other products.

[Josh]

From my understanding, our competitor decided to essentially write their own database server. They store a lot of your data in files and in memory instead of in PostgreSQL. It's faster to manipulate the data that way, but it also uses a lot of memory and could be a potential time bomb for your data (in regards to losing information and/or corruption).

We decided to stick with trusting PostgreSQL to handle concurrency, data integrity and everything else a database needs. PostgreSQL has been under development for years and many database experts have spent thousands of hours perfecting it. There's no way anyone could duplicate that effort and still have the same level of stability that PostgreSQL offers without years of dedicated development.

Our imports might take a little longer (although this should only be an issue for the occasional large manual import), but our memory use is lower and PT4 should be more responsive. Not to mention, your data is safe and sound in the hands of the experts developing PostgreSQL. :)

All that being said, we've been much more focused on features than optimizing. Once PT4 is stable and the features are where we want them, we will turn our efforts to performance tuning.

[oracle3001]

To be honest, I am not choosing PT4 vs HEM2 based simply on how fast the imports are. I can live with 100-150 h/s, but with iPoker hands I have got a max of 20 h/s for a manual import and as mentioned above gridlock with a DB conversion, and in general from the various imports I have done so far today it seems at the moment PT4 is slower than PT3 (even when you include the housekeeping) which is disappointing. I was hoping for a bit of a speed bump with all the talk of no longer needing to do housekeeping.

[Josh]

Yes, in some cases PT4 is slower at imports than PT3. A lot of this is contributed to the fact that we handle database housekeeping automatically, but we're also doing more than just that. PT4 is calculating adjusted equity at import time, it's handling NoteTracker rules as well as adding more data to the database.

The main slowdown you are experiencing looks like it's coming from AATA. This had to be changed a bit for PT4, so there might be some problems with it that are causing delays. We will look into it and hopefully make it a bit faster.

[StevenM]

To be honest, I am not choosing PT4 vs HEM2 based simply on how fast the imports are. I can live with 100-150 h/s, but with iPoker hands I have got a max of 20 h/s for a manual import and as mentioned above gridlock with a DB conversion, and in general from the various imports I have done so far today it seems at the moment PT4 is slower than PT3 (even when you include the housekeeping) which is disappointing. I was hoping for a bit of a speed bump with all the talk of no longer needing to do housekeeping.

The vast majority of imports in PokerTracker 4 are conducted during live in-game player, this is the #1 priority, and it works flawlessly from testing on all the major sites. Manual importing is of course a different animal, many things have changed since PT3 as Josh mentioned. We now perform maintenance during import to build the cache, conduct all-in equity simulations to populate the database, and NoteTracker must scrub each hand history to populate automated note collection. Although it is not included in the Beta Preview, soon we will add more features which will take place during import such as Auto-Tagging.

You may be surprised to know that PokerTracker 4 manual imports have been timed verses PT3 (which is faster than HEM1 in all the tests we have performed) and in most situations PT4 is faster than PT3 when including the manual maintenance that was required. So in average PT4 may appear to be slower because it is giving an accurate hands per second report, but in reality it is faster than PT3 in the majority of situations when factoring in the maintenance variable which was necessary before.

As Josh also pointed out our dev team is still working on refining the AATA functionality which slows down imports in scenarios where we must estimate the table size, this is not an issue on networks where te table size is included in the hand history such as Merge or PokerStars. As an example, say you sat down at a 10 handed table but there were only 2 people playing heads up. We cannot determine if this is a full ring game, a six max game, or a heads up table when this information is not provided to PokerTracker via the hand histories - and in turn we may not be able to determine where to place the HUD for each player. AATA is necessary due to preferred seating preferences (and other issues), as an experiment if you wish to disable AATA and then run an import to compare times that would be very helpful. Good luck, and thanks for the feedback!

[oracle3001]

I appreciate what you are saying, but there is a minimum level of import speed that is required. WIthout it, being able to make large imports of stored hh's just won't be feasible (and to most serious players it is really important to be able to get this data into the system, often on a daily basis).

Just done a bit more testing.

With the AATA turned off, a PT3 DB conversion of only iPoker hands appears to run more like 50-60h/s, which results in a 1 million hand db being estimated to take anywhere between 6 and 10hrs to import (definitely longer than I am used to with PT3, including all the housekeeping). However, a manual import of ipoker hands still is very slow, ~30 h/s, and as a result to all intent and purpose making PT4 currently unusable for my playing requirements.

As stated previously my hardware is a little old, but I have a top of the line traditional style hard-drive and plenty of RAM, which means that with PT3 / HEM2 I get decent import rates. Could there be an issue with the input functionality with ipoker hh's (as it appears they are much slower to import than for instance stars hh's)? Or some further tweaking / changes that could be made in my postgres config?

[StevenM]

Could there be an issue with the input functionality with ipoker hh's (as it appears they are much slower to import than for instance stars hh's)?

Yes, we know that iPoker will require further optimisation, Josh alluded to that in his post earlier. Additionally you will find that hands which are imported in order will result in the fastest import time because we are building the cache during import.