Very odd: PT2 hangs during middle of "export HHs to file"

[domino66]

I consider myself something of a PT2 power-user, but having an impossible time figuring out what's going on here.

By way of background: I like to import a ton of Raw HHs and then export ONLY those that have 5+ players in them, and then re-import THOSE into a 'streamlined' DB. It's tedious, but I've done it this way for ages and i think it helps keep my DB not so bloated.

Anyway, the strangest thing is happening. I imported 1.8 Million raw HHs into a postgres PT2 DB, and then tried to import only those with 5+ players using PT2's "export HHs to file" feature like I always have.

The counter zips along as it tells me how many HHs have been exported to .txt files, at a rate of about 500 HHs / second, but then at around the 250,000-hand mark, it inexplicably hangs. It does not COMPLETELY freeze, but rather the rate of import slows to about one hand every 30 seconds. Yes, you read that right. The counter is showing one hand exported to HH roughly every 30 seconds, so it might as well have completely frozen, and I have to ctrl-alt-delete the PT2 process.

I have tried all sorts of troubleshooting to try and narrow down the cause of the problem. I have done a reboot, a complete vacuum/analyze of the DB in question, I've even tried ONLY exporting HHs from a single site to .txt files (i.e. I tried telling PT2 to ONLY export the ~500,000 Party HHs to .txt files, and alternatively tried telling it to only export the ~375,000 Stars hands. It doesn't matter what way I try to filter the export process, the "magic number" seems to be about 248,000 HHs. As soon as it has exported close to 248K HHs, the entire export hangs / slows to a crawl. Cannot figure this out for the life of me.

[WhiteRider]

I'm going to move this to the PT2 section where you'll get better help for PT2 issues.

[ptrack pat]

Are you able to use any date filters to try to restrict the number of hands being exported so it's less than this number? There is probably a memory issue with PT that is causing this.

[domino66]

Are you able to use any date filters to try to restrict the number of hands being exported so it's less than this number? There is probably a memory issue with PT that is causing this.

Yes, I tried that, and I'm essentially able to do smaller batches (like 150K at a time)...but this never ever used to happen at all, and I've been doing things this way for years. Could it be something that was introduced in the newest PT2 patch...? I saw that there was a new release of PT2 for the first time in a long time on 3/31/09...any way I can get my hands on the previous PT2 version to see if it's something that was introduced in this latest release...?

p.s. I even RESTORED an old PT2 DB that I had lying around from a different computer, which was all Ongame HHs and tried a big export, and once again, PT2 hung around this 'magic' number of 248,000 HHs...so strange!

[ptrack pat]

I haven't touched the export code in years. Have you done a FULL vacuum/analyze on the database?

[domino66]

I haven't touched the export code in years. Have you done a FULL vacuum/analyze on the database?

Yes, as I wrote in the OP, i did a full vac/analyze on the DB in question. Not only that, but as I wrote in the post above, I even restored an OLD DB and tried an export on that DB, and it ALSO hung around the 248K mark.

I did recently do a Windows reinstall and installed postgres 8.3 (with the Stackbuilder fix) and not 8.2 that I used to have...could that be the problem...?

[ptrack pat]

That could be a problem I guess. I know PokerAce had some issues with 8.3 but I haven't had any PT2 users have issues...but I doubt anyone has been trying to export this many hands. Can you reinstall 8.2 and then restore from your backup and try the export again. If that works then we'll know it's an issue with 8.3.

[domino66]

Uninstalled 8.3, reinstalled 8.2

Tried an export HHs to file of 1.1M hands, and sure enough, right at the 248,500-HH mark, the process slowed down to about 1 new HH exported every 10 seconds!!

This is so weird. Is there any way I can get the installation file for the version of PT2 prior to the 3/31/09 upgrade?

Either that, or if ur willing to give me another reg code, I can install the current version of PT2 on a dif machine and see if it's specific to my desktop...(or u could try a big export yourself and see if it also hangs for you at around the 248K hand mark)

[ptrack pat]

Go ahead and install on another computer and send me the product ID (using the online support system) and I'll send you a code. I'll see if I can find an old EXE but I'm certain nothing in the export code has changed in a very long time.

[domino66]

Well, add another pretty critical piece of evidence:

I did as you suggested, DL'd PT2 to a different computer (also one I'd done a recent XP re-install on) running postgres 8.2
Restored a DB on it, and tried to export 1.1M HHs to txt files. It started out zipping along, as usual. I checked in on it 30 minutes later, and it had slowed to importing one HH every 30 seconds or so...and guess how many HHs had been exported up to that point: 248,300.

So this was a completely different computer and a completely new installation of PT2, and it hung at exactly the same place! I think we have our smoking gun. I guess the last piece of evidence would be finding an .exe of the previous PT2 version and verifying that it could finish such an import without hanging, but at this point it's pretty academic that that's the case (if you believe me that I have been running huge exports like this for years with PT2, which is true.) I believe you when you say you haven't touched the export code, of course...well what else could be the issue? FWIW, on the other computer, the super-slow import hang was accompanied by a message from windows telling me that the machine's virtual memory was running low...so could it be a general memory leak causing the problem? (i.e. as opposed to an actual bug in the export code).

Also, note it's not *always* just some magical # around 248,000 when it starts hanging. When I reboot my computer, open PT and try a big export it always hangs around 248K, BUT if I first do a smaller export (say, 120,000 HHs), it will frequently complete this smaller export without hanging, but then if I try running another export immediately thereafter, it will often hang at a much lower #...like, 90,000 or even lower. In other words, I'm trying to say that the workaround is not just a matter of keeping your exports under some magical # around 248,000...rather, the fact that subsequent exports will hang at a much lower # of hands is more evidence of a memory leak of some kind, is it not...? (p.s. i have never coded anything in my life, so I don't even know if I'm using the term 'memory leak' properly...)

Anyway, if you are so inclined, why don't you try doing an export of 300,000+ HHs yourself and I'll bet it hangs for you too...and while it's true that I consider myself something of a PT2 power user...a 250,000 HH export really isn't THAT crazy a number of HHs to be exporting...