Network Problem Tonight

We had a network problem tonight which affected some of our servers and applications.

It was a strange problem. I could get to our applications from home. In fact, eLearning, Skylight and the Wiki farm were all working properly. Our legacy applications showed an error which pointed to the direction of AD authentication failure. 

My colleague went in to the office and said he could not get to any of our applications at all. So, for a while, we thought there were DNS resolution problems instead. I called central ITS. They checked and confirmed that everything was working properly on their side.

We performed further tests on various part of the network. Finally we concluded that it was our firewall cluster not routing traffic properly. We rebooted them one by one, and services resumed right away.

Fortunately, eLearning (which is by far our highest usage application), Skylight Matrix Survey System, and the Wiki farm were not affected. But the rest of our legacy applications were down for about 1 hours and 20 minutes. If it happens during the finals week, and the students cannot access their materials in their last hours of revising, the impact is much more severe. 

The firewalls have been performing reliably well all along. I almost believe that they are infallible. Nothing is! We need to better prepare ourselves, have better procedure to diagnosis network problems quickly.  

Or look at it in a different angle, firewall is just one example of many possible single point of failures. We need to ask ourselves what should our operational strategy (or institutional strategy?) be to prevent single point of failures as we move forward?

Moving into Web2.0 - Delicious

Just finished our departmental retreat today about "Moving into Web2.0". I did a short story on how I am using Delicious social bookmarking service as a vehicle to help me do my job more efficiently.



I felt that the message was very well received during the presentation and demonstration. But yet during the "What are the obstacles in moving into Web2.0?" activity, many said they did not have time to experiment with various Web2.0 tools.

At least, the process of doing the presentation helps me reflect on what I'm doing and how it has been working. I feel very good about this.

Now... Challenge yourself to change some old habits, you can learn to work more efficiently. Yes, you can!

Wrapper for Microsoft SQL Injection Source Code Analyzer Tool

Microsoft has released a SQL Injection Source Code Analyzer for ASP code. Refer to KB 954476 for more information about the tool. However the analyzer can only check one ASP page at a time. It does not automatically recursive scan all ASP files in a folder. Therefore I wrote a simple wrapper around it. Here is the source code:

@echo off
setlocal
set source=\\servername\applicationname
set include=%source%\include
set logfile=check_applicationname.log
IF EXIST %logfile% DEL %logfile%
FOR /F "usebackq tokens=1 delims=?" %%i IN (`dir /S /B %source%\*.asp`) DO (
msscasi_asp.exe /NoLogo /GlobalAsaPath=%source% /input="%%i" /IncludePaths=%include% >> %logfile%
)
endlocal

F5 iRule Competiton Honorable Mention Finalist

I attended the Load Balancer iRule (scripting) competition organized by F5 Networks. I didn't win the first prize. In fact, they never even contacted me afterwards.

I just found this out today that I was on their Honorable Mention Finalist.

SQL Mirroring Manual Failover for eLearning (Blackboard Learning System CE8)

In my previous SQL Mirroring for eLearning post, I listed the detail on how to setup SQL database mirroring for Blackboard Learning System CE8. Since the application (Weblogic) does not support automatic failover, we need to have a procedure and scripts ready to perform manual failover in the event of a diaster (when the production database server is no longer available).

Here is how:

• Stop WebCT service on all frontend nodes.
• If you use asynchronous mirroring, you can execute the following commands on the mirror (business continutiy) server to force failover. The databases will then be in "In Recovery" status. It will take a minute for it to change to "Principal, Disconnected" status. The Disconnected status means that there is no mirrored session going on for this new principal server which is correct.

ALTER DATABASE webctdatabase SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS

ALTER DATABASE webctdatabase$PF SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS

• If you use synchronous mirroring instead, execute the following commands to failover:

ALTER DATABASE webctdatabase SET PARTNER FAILOVER

ALTER DATABASE webctdatabase$PF SET PARTNER FAILOVER

• After failover, you need to execute the following in the new principal server to set the webct and webct$PF users' SID.

Use webctdatabase

go

sp_change_users_login 'report'

Use webctdatabase

go

sp_change_users_login 'update_one', 'webct', 'webct'

Use webctdatabase

go

sp_change_users_login 'update_one', 'webct$PF', 'webct$PF'

Use webctdatabase$PF

go

sp_change_users_login 'report'

Use webctdatabase$PF

go

sp_change_users_login 'update_one', 'webct', 'webct'

Use webctdatabase$PF

go

sp_change_users_login 'update_one', 'webct$PF', 'webct$PF'

• Then you will need to change the jdbc connection string on the admin node and all managed nodes to point the database to the new principal server.

cd \bea92\weblogic92\config\WebCTDomain\config\jdbc\

copy /y CP-WebCTConnectionPool$PF-jdbc.xml CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml

copy /y CP-WebCTConnectionPool-jdbc.xml CP-WebCTConnectionPool-jdbc.before_db_migraton.xml

copy /y CP-WebCTJMSConnectionPool-jdbc.xml CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml

cat CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTConnectionPool$PF-jdbc.xml

cat CP-WebCTConnectionPool-jdbc.before_db_migraton.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTConnectionPool-jdbc.xml

cat CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTJMSConnectionPool-jdbc.xml

diff CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml CP-WebCTConnectionPool$PF-jdbc.xml

diff CP-WebCTConnectionPool-jdbc.before_db_migraton.xml CP-WebCTConnectionPool-jdbc.xml

diff CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml CP-WebCTJMSConnectionPool-jdbc.xml

• Bring up the Blackboard CE8 cluster.
  
  

SQL Mirroring for eLearning (Blackboard Learning System CE8)

We finally have a set of servers in another building designated for business continuity of our eLearning Learning Management System (powered by Blackboard Learning System CE8). We have a procedure to failover to that location in the event of a disaster. We even ran a contingency drill back in May to verify the process.

One of the outstanding problems we have is : eLearning's database is so hugh (and is getting larger every day) that in the event of a disaster, it takes a very long time to restore it from backup. In order to minimize the potential down time, we decided to use SQL database mirroring to mirror eLearning's database from our current production to the business continuity server.

Here is how:

• Backup the eLearning's databases on the principal (production) server and restore them to the mirror (business continuity) server using the NORECOVERY option to ensure the log files on both servers match exactly.

restore database webctdatabase from DISK='D:\folder\webctdatabase.bak' WITH NORECOVERY, REPLACE

restore database webctdatabase$PF from DISK='D:\folder\webctdatabase$PF.bak' WITH NORECOVERY, REPLACE

• On both the principal (production) and mirror (business continuity) servers, create database mirroring partner endpoint. (TCP port 5022 is used in this example since that the the default port number number. But that can be changed to any TCP port number available).

CREATE ENDPOINT mirroring_end_point
STATE=STARTED

AS TCP(LISTENER_PORT=5022)

FOR DATABASE_MIRRORING (ROLE=PARTNER)

• On the mirror server, specify the principal server as the mirror partner.

ALTER DATABASE webctdatabase SET PARTNER = 'TCP://BusContServerName:5022'

ALTER DATABASE webctdatabase$PF SET PARTNER = 'TCP://BusContServerName:5022'

• On the principal server, specify the mirror server as the mirror partner.

ALTER DATABASE webctdatabase SET PARTNER = 'TCP://ProdServerName:5022'

ALTER DATABASE webctdatabase$PF SET PARTNER = 'TCP://ProdServerName:5022'

• Now the principal server will start sending transactions to the mirror server.
• Initially, the principal server and the mirror server will be in a synchronizing state. After all transactions from the principal server have been reapplied on the mirror server, their states will change to synchronized.

By default, a mirror session operates in synchronous mode. But you may prefer the session to be asychronous instead due to performance reason (e.g. high latency between the principal and mirror server). You can execute the following command on either the principal or the mirror server to change the mirror session to asychronous mode.

ALTER DATABASE webctdatabase SET PARTNER SAFETY OFF

ALTER DATABASE webctdatabase$PF SET PARTNER SAFETY OFF

Also worth noting that one can set up a SQL server as a witness server for automatic failover. But the application has to support it. In our case, Blackboard CE8 does not. Therefore, we do not pursue this path. We have a procedure, and a script to perform manual failover instead.

Visit from our Spokane Campus Colleagues about LMS Issues

Our colleagues in Spokane campus visited us to talk about the university learning management system. This is our presentation about the current state of eLearning (powered by Blackboard Learning System CE) which is used by the main Pullman campus.

They also maintain a learning Management system, which is powered by Blackboard Academic Suite, for the urban campuses and Distance Degree Program. Somewhere down the road, we are going to need to migrate, and merge into a single campus-wide solution. We compared notes, and exchanged ideas. It was good to start this conversation with the people who run a LMS day to day.