--------------------------------
TSG Report 6

TSG Success Rate -- Revisited

by 'oscar51'

February 2003

Introduction

About a year ago, I attempted to determine what I called the "TSG Success Rate," that is, the proportion of the requests for help posted to the TSG help forums that were replied to in a way that resolved the issue. The results of that study were reported in tsgrpt1. During December 2002 and January 2003 I took another look at the TSG success rate to see if there was any change since last year. The results of this study are given below.

Data Set

The data set for this study consisted of all the threads that started and ended during the month of November 2002 in all 12 TSG help forums. Thus, for a thread to be included, the initial post had to be made on or after November 1 and the last post made on or before November 30. I found 3,553 threads that met this criterion as of the dates I downloaded the data. These download dates and other details about this data set are shown in Table IV in the Appendix to this report.

Analysis and Results

Definition of 'Resolved'

A thread was considered 'Resolved' if:

1. The thread was so marked by a moderator; or

2. Either the member who made the initial post, or a member who replied to the initial post with the intent to help, explicitly reported the thread 'solved' or 'resolved' in a reply even though the thread itself was not so marked; or

3. The initial request for help was resolved even if an additional request made later in the same thread and was not resolved; or

4. The initial post in a thread asked for advice or an opinion and at least one relevant and helpful reply was posted; or

5. I judged by inference that a successful conclusion had been reached, though no explicit reply to that effect was posted by anyone.


Note: I realize that criterion 5 sounds subjective, but it really wasn't that difficult to apply. If anything, I probably let many of these threads go as unresolved because I was scanning the posts so rapidly I often did not take enough time to stop and reflect on a thread's status.

Threads that contained partial solutions, work-arounds, and the occasional "I give up" were counted as 'not resolved'. User-solved threads, i.e., threads where the person who made the initial post replied to the effect, "I solved it myself," were counted as 'resolved'.

Threads that contained no reply or only one reply to the initial post were ignored in the calculation of success rate, initial response time, and thread duration. These no-reply and one-reply threads were, however, included in the day-of-week and time-of-day post distribution tables and in the tabulation of the number of posts made by each TSG member participating in the help forums.

Closed threads were considered according to their merit on the same basis as open threads subject to the guidelines given in this section.

Success Rate

Table I shows the success rate in percent for this study under the heading 2002 and for last year's study under the heading 2001. Note that the forum "Games" did not exist in 2001.

Table I - Success Rate

  Success Rate
Forum: 2002   2001
  Business Applications 60.0%   41.9%
  Games 35.7%   ---
  All Other Software 45.5%   45.2%
  Hardware 44.6%   40.6%
  Development 67.4%   43.7%
  Windows 95/98/Me 47.8%   53.1%
  Windows NT/2000/XP 44.8%   25.0%
  UNIX/Linux 52.9%   42.8%
  DOS/Mac/PDA/Other 50.0%   25.0%
  Web & Email 47.1%   40.6%
  Networking 38.6%   43.7%
  Security 50.9%   62.5%
Overall: 46.7%   45.3%

The overall TSG success rate for 2002 of 46.7% compares favorably with the 45.3% success rate reported for 2001. Be aware that threads with no reply or only one reply were not included in the data shown in this table. If these threads were included and all of them were assumed to be 'not resolved', the overall success rate would be 33.1% for 2002 and 32.8% for 2001.

Note: I think it quite likely that many one-reply threads were actually resolved, but without a confirming reply from the original poster there was no way to be sure. The phenomenon of not reporting when a suggested action resolved an issue is a big irritation at TSG. A search through the 3,553 threads downloaded for this study found 227 threads that contained the phrase "post back" or "get back" in the context of "I will let you know if it worked or not." Rare was the thread where this promise was fulfilled. I think it very likely that in threads where this promise was the last posted reply, the thread was resolved. As a conservative estimate, if only half of these 227 threads were resolved the overall TSG success rate would be over 50%. And if only half of the one-reply threads were resolved, the overall success rate would be significantly higher yet, maybe approaching 60%.

Initial Response Time and Thread Duration for Resolved Threads

Table II shows the initial response time (the elapsed time from the initial post to the first reply), and the thread duration (the elapsed time from the initial post to the last reply), for all resolved threads in this study. For example, the first number in column A shows there were 692 resolved threads -- 58.8% of the total number of resolved threads -- with an initial response time of less than one hour. (The quickest initial response time was under 2 minutes.) Also, the first seven numbers in column B added together show that 665 resolved threads (56.5% of the total) were concluded in 24 hours or less. Other entries in this table may be interpreted similarly.

Table II - Initial Response Time and
Thread Duration for Resolved Threads
Column A: Initial response time
Column B: Thread duration

Note the change in time scale from
hours to days between rows 7 and 8
Elapsed time: A B
  Under 1 hour 692 78
  1 thru 4 hours 276 181
  5 thru 8 hours 78 108
  9 thru 12 hours 35 67
  13 thru 16 hours 26 60
  17 thru 20 hours 16 75
  21 thru 24 hours 10 96
  during 2nd day 23 206
  during 3rd day 9 91
  during 4th day 6 58
  during 5th day 4 53
  during 6th day 0 25
  during 7th day 0 21
  over 7 days 3 59
Totals: 1,178 1,178

Two threads tied at 5 minutes each for the quickest resolved thread; one with 2 replies and the other with 3 replies. The first reply in each of these two threads was posted less than 2 minutes after the initial post. The slowest resolved thread lasted 27.7 days with 8 replies. There was one 25-day, 18-hour gap between replies in this thread. Twenty-four resolved threads contained a gap of more than one week. One of these threads had two such gaps yet lasted only 17-days, 19-hours.

Note: While screening the TSG data base for threads that both started and ended in November 2002, I came across several threads that had gaps of more than a year before someone replied. Gaps of several months were not uncommon.

Two resolved threads in this study had more than 100 replies -- one had 123 and the other 109. The overall average number of replies per resolved thread was about 7. One unresolved thread had 113 replies.

The total number of words in all 3,553 threads in the data set was 1,470,872, excluding signatures. The total number of posts in the data set was 19,920. Thus, the average number of words in a post was about 74. The maximum number of words in a post was 5175. (This post included a 580-line startup list where, in addition to 'normal' words, each file name in a path was counted as one word.)

Day-of-Week and Time-of-Day Post Distribution

No unusual trends were perceived in the day-of-week and time-of-day distribution of posts. See Tables V and VI in the Appendix if you are interested in these data.

TSG Member Participation

A total of 2,376 TSG members contributed one or more of the 19,920 posts made to the 3,553 threads included in this study. The "Top Ten Posters" among these members are shown in Table III.

Table III - Top Ten Posters

Member: Posts   Member: Posts
  AcaCandy 875     TonyKlein 216
  pvc9 562     Byteman 211
  Jtoast 460     eddie5659 210
  Rollin' Rog 406     john1 207
  Rockn 340     davey7549 204

729 members made only one post. Included in this group were several otherwise prolific posters, such as angel, LadyLisa, rhettman5, tantap, and our very own TechGuy. Contributions from other high-overall posters were: Bryan, 53; hewee, 60; brianF, 115; Mulder, 23; and, in case you're wondering, the "Queen of Random", 26. To keep things in perspective, however, it is necessary to point out that yours truly contributed zero helpful posts. Member eddie5659 (one of the "Top Ten" in Table III) posted 44 of his informative no-reply "notices", all of which were submitted on a Thursday, a Friday, or a Saturday. (See Table V in the Appendix.)

Concluding Remark

I can think of no better concluding remark than to quote my comment at the end of last year's Success Rate report:

"Reading through many threads I would not normally read increased my appreciation for the high quality and friendly tone of the help offered by TSG members. I also discovered several items that were helpful to me in my computer use. These findings reinforced what I already knew: By far the most help I've gotten from the TSG site is by reading threads others have started that pose questions similar to mine."

--------------------------------

Appendix

The following tables are inserted here for archival purposes only. Their perusal is not necessary to understand this report.

Table IV - Data Set Properties
Column A: Date threads were downloaded (mm-dd-yy)
Column B: Total size of downloaded threads, MB
Column C: Number of threads downloaded
Column D: Number of threads in C with no replies
Column E: Number of threads in C with one reply
Column F: Number of threads used to find success rate (C - D - E)
Column G: Number of threads in F found to be resolved
Column H: Number of threads in G marked Resolved/Solved
Column I: Number of threads in C marked Closed

Forum: A B C D E F G H I
  Business Applications 12-15-02 7.7 221 45 61 115 69 1 2
  Games 12-15-02 4.6 100 16 14 70 25 7 1
  All Other Software 12-16-02 14.4 350 16 88 246 112 38 10
  Hardware 12-29-02 37.9 733 55 99 579 258 74 14
  Development 01-06-03 3.4 87 24 17 46 31 0 4
  Windows 95/98/Me 01-26-03 40.5 741 25 120 596 285 141 22
  Windows NT/2000/XP 01-27-03 23.7 573 80 118 375 168 36 10
  UNIX/Linux 12-17-02 2.1 45 3 8 34 18 2 0
  DOS/Mac/PDA/Other 01-05-03 0.5 17 9 2 6 3 0 0
  Web & Email 01-06-03 15.2 379 54 81 244 115 33 2
  Networking 12-20-02 6.1 154 28 25 101 39 2 2
  Security 12-19-02 7.0 153 14 31 108 55 25 1
Totals: --- 163.1 3,553 369 664 2,520 1,178 359 68


Table V - Day-of-Week Post Distribution

      no-reply threads
Day-of-Week: EST GMT EST GMT eddie
  Sunday 2,702 2,736 59 58 0
  Monday 2,865 2,908 35 34 0
  Tuesday 2,913 2,836 53 49 0
  Wednesday 2,855 2,832 47 52 0
  Thursday 2,959 3,006 69 64 15
  Friday 2,852 2,919 51 58 16
  Saturday 2,774 2,683 55 54 13
Totals posts: 19,920 19,920 369 369 44


Table VI - Time-of-Day Post Distribution

Time of day: EST GMT
  0000 to 0100 744 1,044
  0100 to 0200 477 922
  0200 to 0300 281 884
  0300 to 0400 293 840
  0400 to 0500 299 797
  0500 to 0600 286 744
  0600 to 0700 421 477
  0700 to 0800 440 281
  0800 to 0900 582 293
  0900 to 1000 783 299
  1000 to 1100 1,011 286
  1100 to 1200 1,215 421
  1200 to 1300 1,208 440
  1300 to 1400 1,200 582
  1400 to 1500 1,161 783
  1500 to 1600 1,332 1,011
  1600 to 1700 1,346 1,215
  1700 to 1800 1,169 1,208
  1800 to 1900 1,185 1,200
  1900 to 2000 1,044 1,161
  2000 to 2100 922 1,332
  2100 to 2200 884 1,346
  2200 to 2300 840 1,169
  2300 to 2400 797 1,185
Total posts: 19,920 19,920
--------------------------------