Story time again I guess.
Although this is not a pleasant tale, even remembering some of the details make me squirm in discomfort still.
I remembered it was a Tuesday night, 10 pm, I was wrapping up my photo editing work and getting ready for bed, when a call came to me. It was the support guy from work, he calls to inform me that the prices for some ETFs have not came through. Since there was a new production deployment of the ETL software responsible for sending the prices through the day before, he assumed that the ETL software must be to blame.
Unfortunately the particular colleague that deployed the changes yesterday had turned off her phone, “clever girl” I whispered to myself. Since I was the next person with knowledge on the software, it was up to me to debug the situation.
The first thing I did was to bring up a diff of the code been deployed, I traced through all the configuration changes, everything seem innocent enough. Certainly nothing that will stop the file been picked up from source ftp location to the destination.
Just to be safe, I reversed the changes, most of which are superficial, and redeployed the ETL software, since I had no production access, I IM the support guy to drop the file and try again. The support guy regret to inform me that, the file was again not picked up, completed ignored by the software.
By that point it was close to midnight, and my memory became hazy, but the loop went on something like this, I try something fix the issue, deployed it, support guy tell me it didn’t work. And on and on it went, at one point, even the programming cat that lived next door came over to help debug, he too was stumped. This cycle continued to 3 am.
I was tired and broken by then, my mind was a mess of tangled wires and burnt fuses. But at last, something came to me, something ridiculous in its simplicity.
I asked the support guy, “hey, which FTP are you putting the file in?”
“FTP a of course”
My rage burned with the heat of a thousand suns.
“NO! NO! NO! NEIN! NEIN! NEIN! NEIN! FTP a is the destination FTP, you have to put the file in FTP b, this is an outgoing file not an incoming file!” I might have been more polite had this not been common knowledge and detailed in the support document they made us write.
So another 30 minutes was wasted restoring all the changes back, and redeploy the latest version, and waiting for the support guy to put the file on the correct FTP server, and of course, everything flowed after that.
Lesson to take away from this? If nothing makes sense, question your/their assumptions.