![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Well, it's tax day
Apr. 17th, 2007 06:48 pmLong night tonight.
I've been juggling the processing queues between 6 machines all afternoon trying to keep up. The maddening thing is, my machines are not at fault. The guys running the NetApps have overloaded the ones I rely on. Ah well. The nice thing about the architecture I built a few years ago is that it doesn't crash, it just piles the work in a queue and just keeps hacking at it until it catches up.
It is kind of irritating though because when you're I/O bound, it's hard to tell how the rest of the system would have fared if you weren't. So yeah, we'll have to upgrade our I/O capacity, but we may have other limitations that are being masked just below the surface. We can load test but it's almost impossible to build a realistic test; this is a complex system involving tens of thousands of users making millions of web requests and lots of cron jobs triggering hundreds of times a day. Previous attempts at simulating tax day haven't been realistic at all though they do help us tune individual components, but we're never sure if we're tuning the right ones, or whether they won't interact in weird ways in real life.
I'm really very happy with the system, but having a basic limitation that I can't control is quite irritating. But it won't cause a disaster, it just doesn't run as fast as I wish it did.
We blew by our previous record for returns processed in one day yesterday, and today looks to be way busier.
The last processing window is 2AM in Fresno. Yeah, that's 5 AM here. I'll be napping on the couch and setting alarms tonight.
I've been juggling the processing queues between 6 machines all afternoon trying to keep up. The maddening thing is, my machines are not at fault. The guys running the NetApps have overloaded the ones I rely on. Ah well. The nice thing about the architecture I built a few years ago is that it doesn't crash, it just piles the work in a queue and just keeps hacking at it until it catches up.
It is kind of irritating though because when you're I/O bound, it's hard to tell how the rest of the system would have fared if you weren't. So yeah, we'll have to upgrade our I/O capacity, but we may have other limitations that are being masked just below the surface. We can load test but it's almost impossible to build a realistic test; this is a complex system involving tens of thousands of users making millions of web requests and lots of cron jobs triggering hundreds of times a day. Previous attempts at simulating tax day haven't been realistic at all though they do help us tune individual components, but we're never sure if we're tuning the right ones, or whether they won't interact in weird ways in real life.
I'm really very happy with the system, but having a basic limitation that I can't control is quite irritating. But it won't cause a disaster, it just doesn't run as fast as I wish it did.
We blew by our previous record for returns processed in one day yesterday, and today looks to be way busier.
The last processing window is 2AM in Fresno. Yeah, that's 5 AM here. I'll be napping on the couch and setting alarms tonight.
