Blackboard CE8 SP2 Upgrade and Scholar Powerlink

We upgraded our Blackboard Learning System CE8 to Service Pack 2 level on production yesterday afternoon. After the upgrade, the Scholar powerlink was no longer working.

The error message was:

An error occured in the communication between your Blackboard system and Scholar.
The Blackboard system clock may be incorrect.
If you continue to see this error, contact your Blackboard system administrator.

Multiple of us tested it, and we consistently received the above error.

This was really upsetting because this was one piece of the upgrade we could not test on our development cluster. We had a lot of troubles when we initially registered our development cluster onto the Blackboard's Scholar system for configuration testing, which then took away the ability of our production system to be registered. It took Blackboard technical support and their Beyond team over three months to resolve this (basically deleted the registration record of our development cluster in their database). As it turns out, Blackboard does not support both development and production clusters to be registered onto their Scholar system. Therefore we cannot test Scholar powerlink when we tested SP2 upgrade on development.

To my surprise, when I logged on our Blackboard CE8 SP2 and clicked on the Scholar link today, it worked. I immediately checked with my colleagues who helped in testing yesterday. They confirmed that it was also working for them. None of us encountered the above error any more.

It's good that it's working.

But I'm uncomfortable of not being able to understand was why it worked after we left the upgraded system overnight. Is it because some background job needed to run first so that the Scholar Powerlink would work after an upgrade? Mystery.

MMR Vaccine and Autism

A friend of mine asked me for advice on whether she should give her 1-year-old daughter the MMR vaccine since there has been so much controversy about this vaccine causing autism.  

I have done a lot of readings a year ago about vaccine and have created my own spreadsheet to organize what I have learned.  But I'm glad that I decided to do more searching tonight to see if there was any new information.  And I found this New Study Shows No Link Between MMR Vaccine and Autism article in Dr. Sears' web site.

I sent her all the information I had.  I hope she can make an educated decision for her own daughter.

Vaccine Schedule

Given so much controversy about vaccine safety, I decided I need to do my own reading, and come up with my own decision about the vaccine schedule for my baby.

My rationales are how common, and how severe the disease is, what are the ingredients contained in the vaccine and their potential side effects, whether the vaccine contains live viruses.  Also, I would try to space out all the shots, and not take any all-in-one combined shot as much as possible so that the little body does not have to handle that much drugs all at once.

Here is the planned vaccine schedule I come up with for my baby last year:

Planned Age	Disease	Brand
2 months	DTaP	Daptacel
2 months 1 week	HIB	ActHIB
2 months 1 week	HIB	ActHIB
2 months 2 weeks	Pc	Prevnar
2 months 3 weeks	Polio	IPOL
4 months	DTaP	Daptacel
4 months 1 week	HIB	ActHIB
4 months 2 weeks	Pc	Prevnar
4 months 3 weeks	Polio	IPOL
6 months	DTaP	Daptacel
6 months 1 week	HIB	ActHIB
6 months 2 weeks	Pc	Prevnar
9 months	Polio	IPOL
10 months	Flu	Fluzone (half of the single dose vial)
11 months	Flu	Fluzone (half of the single dose vial)
15 months	DTaP	Daptacel
15 months 1 week	HIB	ActHIB
15 months 2 weeks	Pc	Prevnar
17 months	Varicella	Varivax
18 months (likely later)	MMR	MMR
2 years 6 months	Hepatitus B	Recombivax HB
2 years 7 months	Hepatitus A	Harvix
3 years 6 months	Hepatitus B	Recombivax HB
3 years 7 months	Hepatitus A	Harvix
4 years 4 months	DTaP	Daptacel
4 years 4 months 1 week	Polio	IPOL
4 years 6 months	Hepatitus B	Recombivax HB
5 years 6 monhts	Varicella	Varivax
5 years 7 months (likely later)	MMR	MMR
~12 years	Tdap	Adacel
~12 years	Meningococcal	Menactra

He is almost one year old now.  I have been following this schedule most of the time with a few delays.  For example, when he got sick (from a cold), I would delay the scheduled shot until he recovers.  I have been pretty happy with it.  Therefore I'm posting it online so others can learn about it.

The majority of the information I gathered is from Dr Sears' The Vaccine Book. Other than that, it is from various web sites Google directed me to.

I urge all parents to do their own readings and come up with their own decisions on the subject.

Using Google Alerts to Check your Web Sites

I recently found Google Alerts very useful as a quick way to detect whether any of our web sites are hosting spam.  (For those who do not know, Google Alerts gives you updates of the latest relevant Google results based on your choice of query or topic. They can be in the form of email, or news feed).

What I did was, I defined several search terms such as the followings in Google Alerts.

• viagra site:yourdomain.com
  
• mortgage site:yourdomain.com
  

I chosed comprehensive search type, delivered to my email address, and once a day in the alert settings.

When Google found new entries, it would email me the summary once a day.  I can then glanced through the summary to verify if any of those entries could possibly be legitimate.  Or else, I would send that to the appropriate party to follow-up.

SQL Injection Attacks by Country of Origin

Here is a world map of SQL injection attacks by their country of origin from January to October this year. The darker the color, the higher the number of hits or percentage.



Here is a pie chart representation of the information.

Network Problem Tonight

We had a network problem tonight which affected some of our servers and applications.

It was a strange problem. I could get to our applications from home. In fact, eLearning, Skylight and the Wiki farm were all working properly. Our legacy applications showed an error which pointed to the direction of AD authentication failure. 

My colleague went in to the office and said he could not get to any of our applications at all. So, for a while, we thought there were DNS resolution problems instead. I called central ITS. They checked and confirmed that everything was working properly on their side.

We performed further tests on various part of the network. Finally we concluded that it was our firewall cluster not routing traffic properly. We rebooted them one by one, and services resumed right away.

Fortunately, eLearning (which is by far our highest usage application), Skylight Matrix Survey System, and the Wiki farm were not affected. But the rest of our legacy applications were down for about 1 hours and 20 minutes. If it happens during the finals week, and the students cannot access their materials in their last hours of revising, the impact is much more severe. 

The firewalls have been performing reliably well all along. I almost believe that they are infallible. Nothing is! We need to better prepare ourselves, have better procedure to diagnosis network problems quickly.  

Or look at it in a different angle, firewall is just one example of many possible single point of failures. We need to ask ourselves what should our operational strategy (or institutional strategy?) be to prevent single point of failures as we move forward?

Moving into Web2.0 - Delicious

Just finished our departmental retreat today about "Moving into Web2.0". I did a short story on how I am using Delicious social bookmarking service as a vehicle to help me do my job more efficiently.



I felt that the message was very well received during the presentation and demonstration. But yet during the "What are the obstacles in moving into Web2.0?" activity, many said they did not have time to experiment with various Web2.0 tools.

At least, the process of doing the presentation helps me reflect on what I'm doing and how it has been working. I feel very good about this.

Now... Challenge yourself to change some old habits, you can learn to work more efficiently. Yes, you can!

Wrapper for Microsoft SQL Injection Source Code Analyzer Tool

Microsoft has released a SQL Injection Source Code Analyzer for ASP code. Refer to KB 954476 for more information about the tool. However the analyzer can only check one ASP page at a time. It does not automatically recursive scan all ASP files in a folder. Therefore I wrote a simple wrapper around it. Here is the source code:

@echo off
setlocal
set source=\\servername\applicationname
set include=%source%\include
set logfile=check_applicationname.log
IF EXIST %logfile% DEL %logfile%
FOR /F "usebackq tokens=1 delims=?" %%i IN (`dir /S /B %source%\*.asp`) DO (
msscasi_asp.exe /NoLogo /GlobalAsaPath=%source% /input="%%i" /IncludePaths=%include% >> %logfile%
)
endlocal

F5 iRule Competiton Honorable Mention Finalist

I attended the Load Balancer iRule (scripting) competition organized by F5 Networks. I didn't win the first prize. In fact, they never even contacted me afterwards.

I just found this out today that I was on their Honorable Mention Finalist.

SQL Mirroring Manual Failover for eLearning (Blackboard Learning System CE8)

In my previous SQL Mirroring for eLearning post, I listed the detail on how to setup SQL database mirroring for Blackboard Learning System CE8. Since the application (Weblogic) does not support automatic failover, we need to have a procedure and scripts ready to perform manual failover in the event of a diaster (when the production database server is no longer available).

Here is how:

• Stop WebCT service on all frontend nodes.
• If you use asynchronous mirroring, you can execute the following commands on the mirror (business continutiy) server to force failover. The databases will then be in "In Recovery" status. It will take a minute for it to change to "Principal, Disconnected" status. The Disconnected status means that there is no mirrored session going on for this new principal server which is correct.

ALTER DATABASE webctdatabase SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS

ALTER DATABASE webctdatabase$PF SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS

• If you use synchronous mirroring instead, execute the following commands to failover:

ALTER DATABASE webctdatabase SET PARTNER FAILOVER

ALTER DATABASE webctdatabase$PF SET PARTNER FAILOVER

• After failover, you need to execute the following in the new principal server to set the webct and webct$PF users' SID.

Use webctdatabase

go

sp_change_users_login 'report'

Use webctdatabase

go

sp_change_users_login 'update_one', 'webct', 'webct'

Use webctdatabase

go

sp_change_users_login 'update_one', 'webct$PF', 'webct$PF'

Use webctdatabase$PF

go

sp_change_users_login 'report'

Use webctdatabase$PF

go

sp_change_users_login 'update_one', 'webct', 'webct'

Use webctdatabase$PF

go

sp_change_users_login 'update_one', 'webct$PF', 'webct$PF'

• Then you will need to change the jdbc connection string on the admin node and all managed nodes to point the database to the new principal server.

cd \bea92\weblogic92\config\WebCTDomain\config\jdbc\

copy /y CP-WebCTConnectionPool$PF-jdbc.xml CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml

copy /y CP-WebCTConnectionPool-jdbc.xml CP-WebCTConnectionPool-jdbc.before_db_migraton.xml

copy /y CP-WebCTJMSConnectionPool-jdbc.xml CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml

cat CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTConnectionPool$PF-jdbc.xml

cat CP-WebCTConnectionPool-jdbc.before_db_migraton.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTConnectionPool-jdbc.xml

cat CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml | sed 's/ProdServer/BusContServer/' > CP-WebCTJMSConnectionPool-jdbc.xml

diff CP-WebCTConnectionPool$PF-jdbc.before_db_migration.xml CP-WebCTConnectionPool$PF-jdbc.xml

diff CP-WebCTConnectionPool-jdbc.before_db_migraton.xml CP-WebCTConnectionPool-jdbc.xml

diff CP-WebCTJMSConnectionPool-jdbc.before_db_migration.xml CP-WebCTJMSConnectionPool-jdbc.xml

• Bring up the Blackboard CE8 cluster.
  
  

SQL Mirroring for eLearning (Blackboard Learning System CE8)

We finally have a set of servers in another building designated for business continuity of our eLearning Learning Management System (powered by Blackboard Learning System CE8). We have a procedure to failover to that location in the event of a disaster. We even ran a contingency drill back in May to verify the process.

One of the outstanding problems we have is : eLearning's database is so hugh (and is getting larger every day) that in the event of a disaster, it takes a very long time to restore it from backup. In order to minimize the potential down time, we decided to use SQL database mirroring to mirror eLearning's database from our current production to the business continuity server.

Here is how:

• Backup the eLearning's databases on the principal (production) server and restore them to the mirror (business continuity) server using the NORECOVERY option to ensure the log files on both servers match exactly.

restore database webctdatabase from DISK='D:\folder\webctdatabase.bak' WITH NORECOVERY, REPLACE

restore database webctdatabase$PF from DISK='D:\folder\webctdatabase$PF.bak' WITH NORECOVERY, REPLACE

• On both the principal (production) and mirror (business continuity) servers, create database mirroring partner endpoint. (TCP port 5022 is used in this example since that the the default port number number. But that can be changed to any TCP port number available).

CREATE ENDPOINT mirroring_end_point
STATE=STARTED

AS TCP(LISTENER_PORT=5022)

FOR DATABASE_MIRRORING (ROLE=PARTNER)

• On the mirror server, specify the principal server as the mirror partner.

ALTER DATABASE webctdatabase SET PARTNER = 'TCP://BusContServerName:5022'

ALTER DATABASE webctdatabase$PF SET PARTNER = 'TCP://BusContServerName:5022'

• On the principal server, specify the mirror server as the mirror partner.

ALTER DATABASE webctdatabase SET PARTNER = 'TCP://ProdServerName:5022'

ALTER DATABASE webctdatabase$PF SET PARTNER = 'TCP://ProdServerName:5022'

• Now the principal server will start sending transactions to the mirror server.
• Initially, the principal server and the mirror server will be in a synchronizing state. After all transactions from the principal server have been reapplied on the mirror server, their states will change to synchronized.

By default, a mirror session operates in synchronous mode. But you may prefer the session to be asychronous instead due to performance reason (e.g. high latency between the principal and mirror server). You can execute the following command on either the principal or the mirror server to change the mirror session to asychronous mode.

ALTER DATABASE webctdatabase SET PARTNER SAFETY OFF

ALTER DATABASE webctdatabase$PF SET PARTNER SAFETY OFF

Also worth noting that one can set up a SQL server as a witness server for automatic failover. But the application has to support it. In our case, Blackboard CE8 does not. Therefore, we do not pursue this path. We have a procedure, and a script to perform manual failover instead.

Visit from our Spokane Campus Colleagues about LMS Issues

Our colleagues in Spokane campus visited us to talk about the university learning management system. This is our presentation about the current state of eLearning (powered by Blackboard Learning System CE) which is used by the main Pullman campus.

They also maintain a learning Management system, which is powered by Blackboard Academic Suite, for the urban campuses and Distance Degree Program. Somewhere down the road, we are going to need to migrate, and merge into a single campus-wide solution. We compared notes, and exchanged ideas. It was good to start this conversation with the people who run a LMS day to day.

Juniper NetScreen Firewalls Monitoring using SNMP

I just finished configuring MRTG to generate graphs of the CPU usage and the number of active sessions of our Juniper NetScreen Firewalls (via SNMP). Here are the OIDs I find useful:

OID	Description	Date Type
.1.3.6.1.4.1.3224.16.1.1	Average System CPU utilization in percentage.	integer
.1.3.6.1.4.1.3224.16.1.2	Last one minute CPU utilization in percentage.	integer
.1.3.6.1.4.1.3224.16.1.3	Last five minutes CPU utilization in percentage.	integer
.1.3.6.1.4.1.3224.16.1.4	Last fifteen minutes CPU utilization in percentage.	integer
.1.3.6.1.4.1.3224.16.2.1	Memory allocated.	integer
.1.3.6.1.4.1.3224.16.2.2	Memory left.	integer
.1.3.6.1.4.1.3224.16.3.2	Allocate session number.	integer