I’ve been working with moving large databases in PostgreSQL and have found that PostgreSQL is a true memory hog when inserting millions of rows in one statement. If the target table has a number of foreign key references PostgreSQL seems to choke horribly and use insane amounts of memory, 1GB per million rows in my original case.
We have a rather simple table that is composed of six integer columns, two timestamps, and one boolean. I first ran into this issue when inserting with a select statement that had a simple join and where clause. Initially I thought the select statement was the one causing the out of memory issues. But I found running the select alone worked and changing the statement to a select into piped the data into a new table without any issues.

I created a simple test case to reproduce this with the following schema,

CREATE TABLE table1 (
table1_id integer NOT NULL,
name character varying NOT NULL,
PRIMARY KEY(table1_id)
);
CREATE TABLE table2 (
table2_id integer NOT NULL,
name character varying NOT NULL,
PRIMARY KEY(table2_id)
);
CREATE TABLE table3 (
table3_id integer NOT NULL,
name character varying NOT NULL,
PRIMARY KEY(table3_id)
);
CREATE TABLE table4 (
table4_id integer NOT NULL,
name character varying NOT NULL,
PRIMARY KEY(table4_id)
);
CREATE TABLE test_target (
table1_id integer NOT NULL,
table2_id integer NOT NULL,
visible boolean NOT NULL,
date_added timestamp with time zone NOT NULL,
date_updated timestamp with time zone NOT NULL,
table3_id1 integer NOT NULL,
table3_id2 integer NOT NULL,
table4_id1 integer NOT NULL,
table4_id2 integer NOT NULL,
FOREIGN KEY (table1_id) REFERENCES table1(table1_id),
FOREIGN KEY (table2_id) REFERENCES table2(table2_id),
FOREIGN KEY (table3_id1) REFERENCES table3(table3_id),
FOREIGN KEY (table3_id2) REFERENCES table3(table3_id),
FOREIGN KEY (table4_id1) REFERENCES table4(table4_id),
FOREIGN KEY (table4_id2) REFERENCES table4(table4_id)
);

Running this INSERT / SELECT below can cause the PostgreSQL to keep expanding in memory usage. On a 32-bit machine it aborts due to an out of memory error around 2GB, but on a 64-bit Linux machine it keeps using memory until all of the main memory and swap is full and then the oom-killer process is spawned and starts killing processes.

INSERT INTO test_target
(
table1_id, table2_id, visible,
date_added, date_updated,
table3_id1, table3_id2,
table4_id1, table4_id2
)
SELECT
0 as table1_id, 0 as table2_id,
TRUE as visible,
now() as date_added,
now() as date_updated,
0 as table3_id1, 0 as table3_id2,
0 as table4_id1, 0 as table4_id2
FROM generate_series(1, 13000000);

I've tried this on PostgreSQL 8.3.7 under Linux and I found the same behavior in PostgreSQL 8.4 Beta for Windows.

A simple workaround I found is to simply drop the foreign keys before the insert, and re-add the foreign keys after the data is loaded into the table.

I have registered a project for Munin Node for Windows on SourceForge and am awaiting approval.
http://sourceforge.net/projects/munin-nodewin32/

Once it is approved I'll upload the source code to the Subversion repo for public access.

I've posted v1.5 of Munin Node for Windows.

Summary
Munin Node for Windows AKA munin-node-win32, is a Windows client for the Munin monitoring system.

It is written in C++ with most plugins built into the executable. Which is different from the standard munin-node client, which only uses external plugins written as shell and Perl scripts.

Change Log for v1.5
+ Added readme documentation
+ Updated -install / -uninstall switches to handle Window Firewall exceptions
+ Added SpeedFan plugin
+ Added External Plugins plugin, external plugins are now supported
+ Added Generic Performance Counter plugin
+ Removed Uptime, Disktime plugins. Replaced with the Generic Performance Counter plugin
+ Improved error handling in Cpu plugin
+ Now Unicode compatible
+ Fixed memory leaks
+ Improved multi-thread safety
+ Added better Event Log support

Download
MSI Installer (225KB)
Binary (135KB)
Source Code (423KB)

I've created a plugin for foobar2000 v0.9, it stores all your plays in a SQLite database.

In addition to storing details on every play, the time and duration, this plugin automatically scores every played track. It uses a formula similar to AmroK scoring with a range of 0-100.
You can access the score for each track via the %score% title formatting string.

I'm hoping to create a viewer for the database so that you can see detailed trends as every play is stored in the database.

Here are the downloads,
foo_playcount_db v0.4 Binary (253KB)
foo_playcount_db v0.4 Source Code + Binary (1.31MB)

I've released a new version of munin-node-win32 today.
It is an Windows client for the Munin monitoring system. It supports getting disk usage, HDD temperatures, memory usage, and uptime. If you have Motherboard Monitor installed and running, munin-node-win32 can query it for system temperatures, voltages, and fan speeds.

The major new feature for this release is the addition of a cpu monitoring plugin and a configuration file that allows disabling and enabling plugins.

Binary + Source Code ZIP (141KB)
Beta MSI Installer (168KB)

I'm current working on embedding Python so Python scripts can be used to write new plugins for munin-node-win32.

I've moved my blog to my primary domain, www.jory.info, with hosting by 1and1, a fairly decent host with good prices.

munin-node-win32 v1.1 (30KB)

An Windows client for the Munin monitoring system. Supports getting disk usage, HDD temperatures, memory usage, and uptime. If you have Motherboard Monitor installed and running, munin-node-win32 can query it for system temperatures, voltages, and fan speeds.

Binary and Source Code included.

Use the -install command line switch to install as
a service, -uninstall will remove the service.

munin-node-win32 v0.1 (30KB)

An Windows client for the Munin
monitoring system. Supports getting disk usage, memory usage, and
uptime. If you have Motherboard
Monitor installed and running, munin-node-win32 can query it
for system temperatures, voltages, and fan speeds.

Binary and Source Code included.

Use the -install command line switch to install as
a service, -uninstall will remove the service.

I've setup yet another website.
My previous host Dreamhost deleted my host account without warning and hasn't responded to my questions yet.

Now I've moved my site to silenceisdefeat.org, which is quite an awesome free service ($1 donation required to get an account). They've been stable for over 2yrs and provide 50MB of free space.

My site is also now in blog format, hopefully it will make it easier to update

Well, I've redone my website. Gone is the Mambo CMS PHP and back to nice lean XHTML and CSS. The Mambo CMS was easy to use, but required being online to make any changes. It also was a pain to backup, you had to backup both the files and the MySQL database. Also it was exploitable, being PHP, I really should have keep it updated to the latest Mambo version but that was too much work for a website as small as mine.

Also called The Advancement of Jory Stone

---

Cicra Prehistoric times: An Amstrad, believed to have a 2x86 CPU.
<No Picture>

Cicra 1996: An IBM 486 50Mhz with 20MB RAM and a 2x CD-ROM.


Cicra 1998: Then on a 6x86 133Mhz with 96MB of RAM.


Cicra 2000: Now with a M2 250Mhz with 160MB of RAM.


2003: Then a T-Bird 650Mhz with 320MB of RAM.
<No Picture>

2005: After that a Athlon 64 2800+ with 512MB of RAM.
<No Picture>

2006: Currently with a Intel Core 2 E6300 with 1GB of RAM.
<No Picture>
Continue reading "The Life of Jory Stone in Pictures"

Dad had new tires put on the van today. The old tires were well worn, one was balding so bad you could see the metal mesh underneath. Another slowly leaked air and had to be aired up everyday before driving anywhere.

I spent most of today outside cleaning Mom's AC and playing with Lily. I didn't plan on playing in the pool but I slipped and fell into the pool, and so since I was already wet I went ahead and played with Lily for a while. Took me quite some time to dry even though it was scorching outside today. I think a have a light sunburn, it doesn't hurt but my skin is visibly red.

In the computer side of things, I setup my local Linux box to act as a caching http proxy using squid. It was very easy to configure and setup and it actually does seem to help. I expect with all four computers using the proxy we should benefit from visiting the same sites.

After setting up the proxy I thought about RAID-5, the Linux box is also our backup server. Every night all our systems do a daily backup via ftp to the Linux box. RAID-5 would make me feel much safer and would actually be very easy to setup.

After googling a little bit I found this interesting article on Tom's Hardware about using Software RAID-5 in Windows XP Pro, http://www.tomshardware.com/storage/20041119/.

After reading that I think software RAID is much better than hardware RAID for the recovery aspect, as with hardware if your RAID controller fails you will need to buy another one of the same model if you still can. With software RAID all you need is a system that has enough IDE (or SATA) connections.

The cost of software RAID isn't high either, just the cost of the hard drives. Buying three 160GB hard drives for a total of 320GB of storage in RAID-5 comes out to almost $200.

I compiled a version of the Welcome Back Plugin for XChat for Windows users.

Welcome Back Plugin 0.5 Win32 Build (07/25/2005, 4KB)

Homepage: http://tuxserve.sourceforge.net/wb.php

Wow, I haven't written a blog entry for quite sometime.

I've been busy looking for my own car and working to have enough money to actually pay for the car

I recently bought a laptop and have been using the wireless lan. I must say a notebook is one of the coolest things I've messed with. Being able to move from room to room without any hassle. Using it in the car it sweet to, of course I only use it when I'm not driving... One thing that amazes me is hot the laptop gets whenever you really start using the cpu. I don't know how it compares to other laptops, but it gets hot enough that it actually burns my bare legs. You have to wear some jeans to keep your legs intact while doing heavy processing. Light processing like surfing the internet, listening to mp3s, chatting, and compiling doesn't heat it up very much so those tasks work very well on it. However I'm afraid to do any video encoding for fear that it would melt itself.
Continue reading "Long time no see"

Only a week left of college this semester. It has has just flown by.

I managed to apply for FA early enough so that this Fall I will be getting some aid. But I had still missed the deadline for the extra scholarships offered through the college. I had only missed the deadline by two weeks

I've been taking my younger brother to praise team practice at the church and have been thinking about getting a laptop so I could work while there. During the summer I could really use it, since right now I'm just using that 2-3 hours we stay there to study. But when college is over I'm not going have anything useful to do.

I looked at Walmart's $500 laptop, quite nice for the price. Though for me the cpu is too slow, not enough memory or hard drive space. I've read the 1Ghz VIA C3 is roughly equal to a Celeron 633Mhz which isn't too bad but it would be too much of a slow down from my current A64 1.8Ghz. I think 1Ghz should be useful. The 128MB of RAM is also much to small, 512MB would be the ideal amount as I multi-task like crazy. 20GB would barely hold all my mp3's much less my development tools and code. I think I could manage with a 40GB, of course my dream laptop would have a 120GB+. Hey I said dream laptop