--------------------------------
TSG Report 1
TSG Success Rate
by 'oscar51'
Original release
February 2002
Minor revisions
March 2002
Summary
From middle November 2001 to middle January 2002, 332 threads containing 2780 posts by 417 TSG members asking for or giving help were downloaded. After identifying those threads that had been resolved, the success rate in each of the eleven TSG help forums was calculated. These forum success rates were then applied to the total number of threads in each forum and an overall success rate was calculated.
Based on these calculations, it was determined that the overall TSG success rate was 45.3%. In other words, 45.3% of those threads containing two or more replies in response to the initial post were resolved. If threads with no reply or only one reply were included in the analysis, the overall success rate dropped to 32.8%.
It was also determined that in 92% of the threads, the first response to the initial post occurred in 24 hours or less.
There was no correlation between number of replies, elapsed time between replies, or duration of a thread and whether or not that thread was resolved.
Introduction
I became interested in doing this study while I was reading through the thread "Suggestion to improve this site" by Alex Ethridge in the TSG "Site Comments & Suggestions" forum. An ongoing issue discussed in that thread is how to get those members who ask for help to respond with a post stating whether or not the initial issue was resolved and, if so, how. It seems that "too many" members who pose problems do not report back to let everyone know what happened.
One of my first reactions to that discussion was to wonder if there might be something hidden in the statistics of threads and replies that would give a clue as to who reports back and who does not and why. After some reflection on that question, my next reaction was to say, "Of course not. That idea is so farfetched as to be not worth persuing." But then my curiosity took over and I decided to take a look at the TSG help forums to get a feel for what the actual "no response" situation was like. Along the way I also wanted to find out what the actual 'success rate' was. So here is what I did -- and found.
Data Set
In this type of study, the results achieved are almost totally a function of the subset of data chosen to represent the full data base, i.e., the complete record of all the threads in all the help forums of TSG. I say "almost totally" because the method of analysis has some small effect on the results as does the interpretative skill of the investigator. But the choices made in selecting the data set to be studied dominate. Here are the choices I made.
To assemble the data set, I selected from each of the help forums 32 threads that met the following criteria:
1. Contained two or more replies;
2. Was started and concluded during November 2001, December 2001, or January 2002; and
3. (a) Had been inactive for at least three weeks or
(b) was marked in some way as 'Resolved' or 'Solved'.
Criterion 1 was based on the observation that before a request for help can be reported as resolved there must be at least one exchange between the member who made the initial post (whom I shall refer to herein as the "client") and the member who responded with help (whom I shall refer to as the "helper"). In diagram form, this exchange can be represented as:
Initial post by client (reply 0)
|
Response by helper (reply 1)
|
Success reported by client (reply 2)
While not common, this sequence was found to occur more than several times.
Also, because of criterion 1, all threads with zero or one reply were ignored. Such threads are, by definition, unresolved and account for an average of about one-fourth of the total number of threads in each forum. Because no meaningful dialog had been established in these threads, I decided that it was not fair to count them against the overall success rate. Threads with zero replies are, of course, an embarrassment, but that is a topic beyond the scope of this study.
Criterion 2, though somewhat arbitrary, limited the data set to the more recent threads.
Criterion 3(a) was a judgment call based on the assumption that after three weeks with no reply posted, a thread was very unlikely to be resolved. It so happened that there were a few threads marked 'Resolved' that had gaps between the replies of more than three weeks, but these threads were quite rare.
Criterion 3(b) is self-explanatory.
Why 32 threads in each forum? Because I had to stop somewhere and 32 is a nice binary number. :-) In the 'UNIX/Linux' forum I could find only 14 threads that met the above criteria. Also, after the analysis was well under way, it was discovered that two threads -- one in the 'Business Applications' forum and one in the 'All Other Software' forum -- had to be dropped from consideration because they failed to meet criterion 2. I assumed that the threads chosen for study in each forum were representative of the totality of the threads in that forum. This is a standard assumption in statistical analyses and its validity in this study was taken for granted.
An alternative way to select a data set would be to choose the number of threads from each forum in proportion to the total number of threads in that forum. Thus, the 'larger' forums would be represented in the data set by more threads than would be the 'smaller' forums. This selection method would tend to increase confidence in the results for those forums with a larger number of total threads. Another alternative way to select a data set would be to just pick a large number of threads, say 1024, at random from throughout the forums. If done properly this method would also result in the larger forums being represented by more threads than the smaller forums. I did not use either of these alternative methods because their use would have required too much downloading. Both these methods would, however, be considered viable by someone with direct access to the TSG data base.
I started downloading threads during the last week of December 2001. But because of holiday disruptions (a houseful of company for three weeks), I could not get all the threads downloaded and ready for analysis at that time. So I repeated the entire effort one month later during the last week of January 2002. However, in January three of the threads first downloaded in December now contained errors* and two threads could not be found at all. In these five cases I used the original threads downloaded in December. I assumed that any extra posts that might have been added to these threads during the month's delay would have no significant effect on the overall results.
* A typical error message looked like this: "Warning: Unknown modifier 'c' in /home/httpd/vhosts/helponthe.net/forums/admin/functions.php on line 272" where the Unknown modifier was either 'c' or '1' or 'r'. In these cases a portion of the web page was displayed on the screen but the text of the posts was missing.
The above choices and procedure resulted in a 17Mb data set (subsequently reduced to 0.8Mb by filtering) containing 332 threads with 2448 replies, a total of 2780 posts by 417 TSG members.
I realize that this data set is not nearly as large or as random as a statistical purist would require, but it was the one I used. So be it.
Analysis and Results
Definition of 'Resolved'
It became apparent as I began to look through the data set that many threads not marked 'Resolved' or 'Solved' were, in fact, resolved. Therefore, the first issue to be confronted in analyzing the data was how to decide whether or not a thread was resolved successfully. Based on what I observed as I read through the threads, I decided to count a thread as 'Resolved' if:
1. It was so marked by a moderator; or
2. Either the client or the helper explicitly reported the thread 'solved' or 'resolved' in a reply even though the thread itself was not so marked; or
3. I judged that a successful conclusion had been reached by inference, though no explicit reply to that effect was posted by anyone. This was not difficult to do and was not as arbitrary as it might sound.
Partial solutions and the occasional "I give up" were not counted as resolved. Work-arounds and requests for advice were counted either way depending on the issue in question and the flow of the discussion. The number of threads in these four categories was very small compared to the total number of threads in the data set.
After the 'Resolved' threads had been identified and suitably marked, I proceeded with the statistical calculations. For this purpose I used my own programs written in SNOBOL4P. (Yes, I know: SNOBOL is old and slow, but it works and has never given me a BSOD or a hang. Besides, I never took the time to learn 'C'. FORTRAN was all I needed at NASA.)
Success Rate
Success rate in percent, shown for each forum in Table I, column A, was calculated by dividing the number of threads judged to be resolved (col. B) by the number of threads in the data set (col. C). For example, in the "Business Applications" forum, 13 threads were resolved out of 31 threads in the data set for a success rate of 41.9%. Similarly for the other forums. (See below Table I for an explanation of columns D and E.)
Table I - Success Rate
| Column A: Success rate |
| Column B: Number of resolved threads in data set |
| Column C: Total number of threads in data set |
| Column D: Number of resolved threads in data base |
| Column E: Total number of threads in data base |
| Forum: |
A |
B |
C |
D |
E |
| Business Applications |
41.9% |
13 |
31 |
509 |
1,214 |
| All Other Software |
45.2% |
14 |
31 |
1,769 |
3,918 |
| Hardware |
40.6% |
13 |
32 |
2,625 |
6,461 |
| Development |
43.7% |
14 |
32 |
112 |
255 |
| Windows 95/98/Me |
53.1% |
17 |
32 |
6,897 |
12,982 |
| Windows NT/2000/XP |
25.0% |
8 |
32 |
619 |
2,476 |
| UNIX/Linux |
42.8% |
6 |
14 |
78 |
183 |
| DOS/Mac/PDA/Other |
25.0% |
8 |
32 |
68 |
272 |
| Web & Email |
40.6% |
13 |
32 |
1,527 |
3,759 |
| Networking |
43.7% |
14 |
32 |
166 |
379 |
| Security |
62.5% |
20 |
32 |
237 |
379 |
| Totals: |
--- |
140 |
332 |
14,607 |
32,278 |
The number of threads resolved in each forum in the full TSG data base (col. D) was calculated by multiplying the number of threads in the data base (col. E) by the success rate (col. A). For example, in the "Business Applications" forum, out of 1,214 threads in the full data base, 509 were calculated to be resolved.
Adding up the numbers in column C and dividing by the sum of the numbers in column D gives an overall TSG site success rate of 45.3%. A very cursory least-squares estimation indicates that a random change of + or -1 thread in column B in each forum would cause a + or -1.5% change in the 45.0% overall success rate.
Note: The astute reader will remember at this point that all threads with zero or one reply have been excluded from these calculations (as per criterion 1 under the heading "Data Set" above). If these threads had been included, the overall success rate would have been only 32.8%. The difference in magnitude between 32.8% and 45.3% illustrates the significance of the decision to exclude the zero- and one-reply threads.
Initial Response Time and Thread Duration
To look a little more deeply into what was going on in the data set, I calculated a set of 'frequency tables', i.e., tables of data that show how often a given parameter value occurs within a specified interval of time. (I realize this type of data is more easily visualized and interpreted when shown in figures called 'histograms' rather than in tables, but I don't happen to have a plotting program handy right now.)
Table II (a frequency table) shows in column A the number of threads with the indicated initial response time, i.e., the elapsed time from the initial post to the first reply. Column B shows the number of threads with the indicated duration, i.e., the elapsed time from the initial post to the last reply. For example, the first number in column A shows there were 168 threads with an initial response time of less than one hour. Adding together the first 7 numbers in Column A (168 + 67 + 36 + 11 + 12 + 7 + 3) shows that in 304 out of 332 threads, i.e., in 92% of the threads, the initial response time was 24 hours or less. That's a pretty good response record, it seems to me. Also, for example, column B shows there were 13 threads with a duration of less than one hour and 60 threads with a duration of more than 7 days.
Table II - Initial Response Time and Thread Duration
| Column A: Initial response time |
| Column B: Thread duration |
Note the change in time scale from
hours to days between rows 7 and 8
| Elapsed time: |
A |
B |
| Under 1 hour |
168 |
13 |
| 1 thru 4 hours |
67 |
24 |
| 5 thru 8 hours |
36 |
22 |
| 9 thru 12 hours |
11 |
12 |
| 13 thru 16 hours |
12 |
7 |
| 17 thru 20 hours |
7 |
17 |
| 21 thru 24 hours |
3 |
21 |
| during 2nd day |
12 |
62 |
| during 3rd day |
4 |
40 |
| during 4th day |
3 |
20 |
| during 5th day |
1 |
13 |
| during 6th day |
0 |
15 |
| during 7th day |
1 |
6 |
| over 7 days |
7 |
60 |
| Totals: |
332 |
332 |
Elapsed Time Between Replies
Table III shows the elapsed time between replies for various categories of members. Column A represents all members. For example, for all members slightly more than one-half of the replies (1233 out of 2448) occurred in less than one hour from the previous reply and about 90% (2199) occurred within one day. (See below Table III for an explanation of columns B, C, and D.)
Table III - Elapsed Time Between Replies
| Column A: All members |
| Column B: Members with <1000 posts |
| Column C: Members with =>1000 posts |
| Column D: Normalized ratio (C / B) |
Note the change in time scale from
hours to days between rows 7 and 8
| Elapsed time: |
A |
B |
C |
D |
| Under 1 hour |
1233 |
601 |
632 |
1.354 |
| 1 thru 4 hours |
421 |
235 |
186 |
1.019 |
| 5 thru 8 hours |
215 |
118 |
97 |
1.058 |
| 9 thru 12 hours |
113 |
67 |
46 |
0.884 |
| 13 thru 16 hours |
109 |
73 |
36 |
0.635 |
| 17 thru 20 hours |
60 |
45 |
15 |
0.429 |
| 21 thru 24 hours |
48 |
37 |
11 |
0.382 |
| during 2nd day |
98 |
77 |
21 |
0.351 |
| during 3rd day |
34 |
21 |
13 |
0.797 |
| during 4th day |
32 |
28 |
4 |
0.183 |
| during 5th day |
18 |
14 |
4 |
0.367 |
| during 6th day |
8 |
8 |
0 |
0.000 |
| during 7th day |
8 |
7 |
1 |
0.183 |
| over 7 days |
51 |
47 |
4 |
0.109 |
| Totals: |
2448 |
1378 |
1070 |
--- |
Columns B and C in Table III show, respectively, the elapsed time between replies for members with less than 1000 posts and members with 1000 or more posts. Column D shows the ratio of column C to column B normalized to the ratio of the total number of replies in each column. Thus, for example, the first number in column D shows that members with 1000 posts or more replied 1.354 times more frequently in less than one hour than did members with less than 1000 posts.
Let me hasten to say here -- lest I be accused of promoting 'post-count ego' -- that these data are really not that important and are not intended to make anyone feel either good or bad. They illustrate only that members who submit posts more frequently tend to reply somewhat more rapidly. Actually this is no great surprise because the more frequent posters are the helpers who, like teachers, tend to give assignments to those who ask for help. Those who receive the help are then, like students, expected to do the assignment as homework and report back. Nothing wrong with this scheme; it's the essence of TSG.
Thread Resolution Correlation
In an attempt to discover if there was any correlation between the duration of a thread and whether or not the thread was resolved, I compared the durations of resolved and unresolved threads as shown in Table IV. Columns A and B show, respectively, the number of threads with the duration indicated for resolved threads and for unresolved threads. Column C is the normalized ratio of column B to column A.
Table IV - Thread Duration Correlation
| Column A: Resolved threads |
| Column B: Unresolved threads |
| Column C: Normalized ratio (B / A) |
Note the change in time scale from
hours to days between rows 7 and 8
| Elapsed time: |
A |
B |
C |
| Under 1 hour |
5 |
8 |
1.154 |
| 1 thru 4 hours |
15 |
9 |
0.433 |
| 5 thru 8 hours |
10 |
12 |
0.866 |
| 9 thru 12 hours |
3 |
9 |
2.165 |
| 13 thru 16 hours |
3 |
4 |
0.962 |
| 17 thru 20 hours |
8 |
9 |
0.812 |
| 21 thru 24 hours |
9 |
12 |
0.962 |
| during 2nd day |
30 |
32 |
0.770 |
| during 3rd day |
22 |
18 |
0.590 |
| during 4th day |
5 |
15 |
2.165 |
| during 5th day |
3 |
10 |
2.405 |
| during 6th day |
5 |
10 |
1.443 |
| during 7th day |
5 |
1 |
0.144 |
| over 7 days |
17 |
43 |
1.844 |
| Totals: |
140 |
192 |
--- |
There is no correlation whatsoever between the duration of a thread and whether or not it was resolved. A significant correlation would have been indicated -- in my opinion -- if the numbers in column C had been consistently either 2 or larger (meaning that the duration of unresolved threads was twice or more the duration of resolved threads), or 0.5 and smaller (meaning that the duration of unresolved threads was half or less the duration of resolved threads). No trend in either direction was indicated.
Note, however, from column A in Table IV that 38% of the resolved threads were resolved in 24 hours or less. That's pretty good. It is also interesting to note from near the bottom in column A that it took more than one week to resolve 17 of the 140 threads. Moral: Hang in there, people.
Time-of-Day Post Distribution
As a bonus for those who have read this far, Table V shows the number of posts submitted during each one-hour block of time in a 24-hour 'day'. Two values of time are shown: Eastern Standard Time (EST) and Pacific Standard Time (PST) where PST = EST - 3 hours. For example, the first row shows that 60 posts were made during the period from 0000 to 0100 (midnight to 1:00 a.m.) EST. This time block is equivalent to 2100 to 2200 (9:00 to 10:00 p.m.) PST, 3 hours behind GMT.
The distribution of posts throughout the day indicates that most members log off about midnight EST and start to log back on about 8:00 a.m. EST. This time span corresponds to 9:00 p.m. to 5:00 a.m. PST. Because this is highly unrealistic behavior for USA west coast people (except yours truly for whom it's very realistic), the distribution more likely indicates (a) that more TSG members are located in the Eastern Standard Time (EST) time zone on or near the USA east coast than in other time zones, or (b) that east-coast members contribute more posts than other members, or both. Following this logic, the peak at 1100 to 1200 EST may reflect lunch-hour activity on the east coast (11:00 a.m. to noon) and the other peak at 1900 to 2000 EST may reflect dinner time on the east coast (7:00 to 8:00 p.m.). But who knows? A map showing the geographical distribution of TSG members and the number of posts contributed by each member could possibly shed some light on this conjecture.
Table V - Time-of-Day Post Distribution
| EST |
Posts |
PST |
| 0000 to 0100 |
60 |
2100 to 2200 |
| 0100 to 0200 |
62 |
2200 to 2300 |
| 0200 to 0300 |
43 |
2300 to 2400 |
| 0300 to 0400 |
47 |
0000 to 0100 |
| 0400 to 0500 |
28 |
0100 to 0200 |
| 0500 to 0600 |
42 |
0200 to 0300 |
| 0600 to 0700 |
49 |
0300 to 0400 |
| 0700 to 0800 |
54 |
0400 to 0500 |
| 0800 to 0900 |
113 |
0500 to 0600 |
| 0900 to 1000 |
122 |
0600 to 0700 |
| 1000 to 1100 |
176 |
0700 to 0800 |
| 1100 to 1200 |
185 |
0800 to 0900 |
| 1200 to 1300 |
144 |
0900 to 1000 |
| 1300 to 1400 |
156 |
1000 to 1100 |
| 1400 to 1500 |
150 |
1100 to 1200 |
| 1500 to 1600 |
161 |
1200 to 1300 |
| 1600 to 1700 |
178 |
1300 to 1400 |
| 1700 to 1800 |
158 |
1400 to 1500 |
| 1800 to 1900 |
144 |
1500 to 1600 |
| 1900 to 2000 |
184 |
1600 to 1700 |
| 2000 to 2100 |
169 |
1700 to 1800 |
| 2100 to 2200 |
132 |
1800 to 1900 |
| 2200 to 2300 |
120 |
1900 to 2000 |
| 2300 to 2400 |
103 |
2000 to 2100 |
| Total posts: |
2780 |
---- |
Day-of-Week Post Distribution
Table VI shows the number of posts made during each day of a typical week based on EST (col. A) and on PST (col. B). Note that Monday's contribution of posts far exceeds those for any other day of the week. I find this curious, but have not a clue as to why. After Mondays, the days with a high number of posts are Thursdays and Sundays. Tuesdays have the lowest number of posts and, again, I have not a clue why.
Also note that the trend in number of posts per day of the week is the same whether based on EST or PST. Daylight Savings Time was not a factor because these posts were made from the middle of November 2001 to the middle January 2002 when all of North America was on standard time.
Table VI - Day-of-Week Post Distribution
| Column A: Number of posts based on EST |
| Column B: Number of posts based on PST |
| Day-of-Week: | A | B |
| Sunday | 396 | 423 |
| Monday | 565 | 552 |
| Tuesday | 323 | 315 |
| Wednesday | 370 | 369 |
| Thursday | 434 | 444 |
| Friday | 366 | 358 |
| Saturday | 326 | 319 |
Miscellaneous Trivia
To end this discussion, here's some miscellaneous trivia collected from the data:
The shortest thread among those analyzed lasted 14 minutes with 5 replies. (The first reply in this thread was posted 3 minutes after the initial post.) The longest thread took 47.1 days with 22 replies. Neither of these two threads was resolved.
The quickest 'Resolved' thread took 40 minutes with 4 replies; the slowest, 43 days with 7 replies. (Did I say, "Hang in there," or what?)
The average number of replies for 'Resolved' threads was 8.1; for all threads, 7.4; for unresolved threads, 6.9. Is this a trend -- finally?
Concluding Remarks
The overall success rate of 45.3% was higher than I had anticipated. (Oh me of little faith.)
I did not find -- nor did I expect to find -- anything statistically significant that would apply to the question of how to motivate members to signal success at the end of a thread.
Reading through many threads I would not normally read increased my appreciation for the high quality and friendly tone of the help offered by TSG members. I also discovered several items that were helpful to me in my computer use. These findings reinforced what I already knew: By far the most help I've gotten from the TSG site is by reading threads others have started that pose questions similar to mine.
This effort may not mean much in the grand scheme of things, but it was fun.
--------------------------------