The evaluation below (downloaded from the MERI web site August 14, 2006) was apparently contracted for by MERI with Professor Hodgkin. Comments on the various statements in the evaluation are in underlined boldface type. These have been added here directly, rather than as a separate document, to make it easier to follow which parts of the original document are being commented on.
 
EVALUATION OF MERI
LEGISLATOR RATINGS METHODOLOGY

by Douglas I. Hodgkin, Professor Emeritus of Political Science Bates College

            The methodology that MERI uses to rate Maine legislators is a sophisticated process that comes much closer to the goals that such ratings attempt to achieve than do most such efforts.   This is probably misleading since when MERI questioned businessmen they must have produced a questionnaire and selected various issues to include on the questionnaire.  This is a quick and cheap method because roll calls are so readily accessible.  However, the roll call is only one dimension of the legislative process and of the record of a legislator.  It also is problematic to rely solely upon the judgment of staff, for there is evidence that staff may be out of touch with their membership and may have ideological agendas of their own.  They may select roll calls to achieve a particular result. They may also select items to include or leave out of the questionnaire to achieve the same end They may include roll calls that have little to do with the central purpose of their organization.

            MERI is to be commended for a methodology that relies upon the judgment of those whom they represent.  Moreover, the ratings are based in significant part upon a “qualitative” assessment of the attitudes and activities of legislators.  While these are difficult to measure and to convert into a rating, these elements of legislators’ records are just as important, if not more important than the roll calls. Important to note that this strong use of “qualitative” assessment is actually the off-the-cuff subjective assessment of a unknown number of lobbyists whose names and employers are not available about an unknown number of legislators.

            It is impossible to develop a methodology that is immune to criticism.  At every step in the process, judgment is required.  If different judgments are possible, critics who dislike the outcomes will challenge the objectivity of the ratings.

            At least three central questions guide the critique of a methodology:

            1.  Does a method measure what the user says is being measured?

            2.  Does a method measure accurately?  Is there a better measure of the same thing?

            3.  Once a measure is selected, is there a chance for bias in the collection of data?

  It should also be added that the methodology of such a process should be “transparent” so that persons can judge the value of the result. Over and over again this evaluator notes that such basic information (particularly in the area of the “qualitative” assessments) is not available.

I.  Senior Management Survey.  This is a survey of business leaders to determine the issue areas relevant to the business community.  There seem to be two overlapping, but possibly distinct criteria for the selection:  (1) issues “essential to their success” and (2) issues “which are fundamental to a strong economy.”  (The criterion of “their ability to create quality jobs” is possibly a distinct goal, but may be a component of the latter.)  Are both criteria used equally by the business leaders?  Are the criteria always consistent with one another?  Which are you really measuring: issues essential to business success (how defined?) or issues “fundamental to a strong economy”? If this evaluator cannot answer these questions how is anyone else going to answer them? He asks exactly what is really being measured and is unable to answer his own question. On this basis alone the survey fails his first key question of whether or not the survey measures what it says is being measured.

            The use of business leaders to judge what is essential to their businesses’ success is probably “as close as you can get to knowing what” business leaders consider best public policy of benefit to their business.  The selection of a particular issue to include in the rating undoubtedly depends upon business consensus on this issue.  Undoubtedly there are some issues of disagreement, as when small businesses compete with large businesses or when businesses who are customers of other businesses have different interests. Again the question is who selected the issues to ask the businessmen about in the first place? Secondly, there is nothing available about how many businesses actually replied – only data on how many employees they have. Neither is there data on the geographic location of the businesses and the type of business. Obviously, if this is not available it is impossible to make any judgment at all on whether or not the so called “businessmen” replying are all in one county or all in one or two lines of business.

            Even if businesses are in consensus on an issue as helping business, we have the stickier question of what is “fundamental to a strong economy.”  Economists disagree on this.  Are large tax cuts good or bad for the economy?  Is NAFTA good or bad for Maine’s economy?  As long as you have these disagreements, you can be challenged on whether the issues selected are good or bad for the economy.  The challengers essentially will say that you are not measuring what you say you are measuring.  The bottom line is that you are measuring what the business community says is “fundamental to a strong economy.” Since we don’t know who,  or where they are it is a fundamental misstatement to say that you are measuring what the “business community” thinks. Eight gift shops, two manufacturers and three lumber yards in Cumberland and York counties would hardly be a cross section of the “business community.” We just don’t know and cannot answer this question.

            The overlap between these criteria comes with the assumption that if businesses are healthy, the economy is healthy.  Policies that promote healthy businesses thereby promote a healthy economy.

            For purposes of influencing general public opinion, it could be difficult to argue that this is without bias.  However, I understand that a major purpose of the rating is to address the business community, and only secondarily the general public. If this was the real purpose why spend thousands of dollars to print several hundred thousand  full-color supplements and pay to have it inserted in all the major Maine newspapers?  Why stop the presses to print shorter press runs with the names of specific Democratic legislators at the top of the front page?  Why conceal the fact that half or more of the so-called “economic” rating of legislators is in fact based, not on roll calls, but the subjective ratings of unnamed lobbyists?  Most business people do not have the time to investigate the record of relevant legislators.  The MERI rating is an excellent service to business people.  It tells them how legislators perform on issues that their own colleagues judge are the relevant pieces of legislation. What colleagues? What businesses? Where located?

            A breakdown of the respondents by number of employees is provided.  Since there is probably a degree of self-selection in the decision to participate, it would be helpful to know whether the distribution is similar to that of the business sector as a whole.  It would also be helpful, but probably impossible to determine, to know whether the survey participants had the same party distribution as the business sector as a whole.  These are questions concerning the representative ness of the participants. These are good basic questions. If the evaluator hired by MERI is unable to answer them how can anyone else be expected to do so? He raises the same important questions – who are these “businessmen” whose answers form the foundation of the entire survey? Are some Democrats or are they all Republicans? We simply don’t know and neither does the evaluator.

II.  Key Legislation.  The staff reviews all bills introduced in the Legislature to identify those germane to the issues identified in the management survey.  This is an appropriate and essential function of staff. Again while the evaluator indicates in the opening that while judgment of staff may be faulty, he now declares that it is all right for the staff to select the legislation (out of thousands of measures) that is the best example of something indicated by businessmen as being important such as “lower taxes.” Although staff can have influence, MERI has built in some checks with the participation of an Advisory Committee and final approval of the legislation selected by MERI’s board.  The composition of the two bodies might be examined for bias. This time the evaluator is correct “the two bodies might be examined for bias.” We learn from the Brunswick Times Record (4/19/06) that the “advisory committee” which controls the compiling and “weighting” of records is actually a “smaller group of 14 lobbyists” that meets every two weeks, but we don’t know the names of these 14 lobbyists.

III.  Legislator Ratings.  As noted, the inclusion of both roll calls and a measure of other forms of legislative participation is a major strength of this methodology. Actually, as indicated below, this is a major weakness not a major strength. The heavy weight given to the subjective opinions of a small group of unnamed business lobbyists means that somewhere been half and two thirds and possibly more (impossible to determine exactly) of  the final rating of each legislator depends on whether these lobbyists consider him/her to be pro-business or anti-business. Any rating based entirely on roll call votes, irrespective of how the roll calls were selected, can easily be judged for importance, validity, bias, relevance etc. because it is a matter of public record. Such judgment here is impossible because the strong use of the subjective opinions of an unnumbered, unnamed group of lobbyists by 14 other unnamed lobbyists makes the process secret. Unfortunately the use of lobbyists by MERI to rate legislators is also kept secret. In a July 24, 2006 press release the MERI President stated that “key economic legislation is identified and used to guide all MERI research – including MERI’s Economic Index ratings for each state legislator. Ratings are based on performance.” Here as well as in the hundreds of thousands of MERI “roll call 2006” tabloids there is no mention at all of using the opinions of lobbyists to rate legislators.

MERI Voting Record Evaluation:  Governmental affairs representatives recommend roll calls for inclusion in the voting record evaluation.  They are undoubtedly the best judges of what they consider to be crucial roll calls for the business they represent.  MERI’s Board apparently determines who is eligible to participate.  Is there any bias in the selection of these participants?  Are they representative of business representatives? Again, if the evaluator hired by MERI can’t answer these questions of built-in bias and representation who can?

            If the business representatives are representative of the pool from which they are chosen, strict staff deference to the governmental affairs representatives regarding selection of roll calls and determination of the correct position ensures that the composition of this portion of the rating adheres to the intent that the rating reflect the judgment of business people on these issues. Please note the word “if” here. Again we do not know “if the business representatives are representative….”

            Most ratings weight each roll call or bill equally.  The MERI weighting of each bill according to frequency of submission is a further sophistication of this system.  Actually, apparently, the businessmen responding had nothing to do with selecting the actual bills that exemplified the issues they considered important. The legislation to represent each issue or as stated above the staff reviews “all” the legislative bills to identify those “germane” to the issues identified in the basic business survey. Those roll calls of broader importance to business and the economy should carry greater weight.  MERI’s Board rather than staff make the final check on whether a roll call is included.

MERI Qualitative Evaluation.  By definition, a qualitative evaluation is more subjective.  However, it is essential to get at behavior that is not easily quantifiable from documents or other records.  It is indicated that the governmental affairs representatives evaluate “behavior in committees, issue advocacy, and general support for important economic and business issues.”  Apparently, the evaluator provides a general rating as a total impression of the legislator.  It is hoped that the evaluator is given a brief checklist of behaviors to consider so that the one doing the ratings is encouraged to think of a variety of behaviors and contexts. Note the words “apparently” and “It is hoped” clearly indicating that the evaluator does not know exactly what kind of rating “general” or otherwise is provided by the unnamed lobbyists – the only “hopes” that they have been given some kind of checklist to follow – but he doesn’t know that either.

            The ten-point scale which ranges from 10 to 100 has a bias toward raising an anti-business legislator’s pro-business score.  There is no 0 (zero) permitted even though a legislator might be rabidly anti-business in philosophy and behavior.

            It is an excellent feature that the evaluators are to by-pass rating legislators not known well enough.  This enhances the accuracy of the assessment.  What happens when a legislator has such a low profile that there are no ratings?  It could also be problematic if there are only one or two ratings; the rating could be skewed by the judgment of one or two people. Here the evaluator is correct. But there is the unanswered question of “at what point does a lobbyist know a legislator well enough to evaluate him?” If there are very few of these subjective ratings, obviously they could be badly “skewed” as he says by the subjective opinion of one or two lobbyists. The skewed opinion could be created also by a situation in which a particular legislator did well on roll call votes but was a committee chair and a  strong opponent on legislation favored by a few lobbyists who, after all are paid by the business interest for which they lobby.

MERI Legislator Rating Calculation.  The combination of the two scores is always an arbitrary process. Right again! In fact if this combination is “arbitrary” then the entire final result is arbitrary as well. The combination can be weighted and changed in any fashion by those in control of the survey to reach any result they wish.  Critics can always question their weighting in the formula for the calculation of the summary score.  However, a rationale for the formula needs to be thought through. The comment that “a rationale for the formula needs to be thought through” clearly indicates that since 1999 in rating four Maine Legislatures there never has been such a rationale.

            As indicated above, the roll calls may be less important than other dimensions of the legislative process.  It is more accurate to weight the “qualitative” dimension more than the roll call score.  MERI’s system does this, but it does so in a cautious manner. This indicates that the subjective opinions of a few lobbyists count for more than the publicly available roll call votes – sometimes a lot more. If two-thirds or more of the roll calls are decided by less than two-thirds of the vote, the qualitative and voting record components are weighted equally.  The apparent rationale is that closely contested roll calls reflect highly contentious issues that test most strongly the legislator’s position.  One-sided roll calls reflect consensus and votes on which a legislator’s philosophy is less challenged.  If most of the roll calls are highly contentious, then it is less necessary to weight the qualitative dimension more highly; the more objective roll calls do an adequate job of sorting out the legislators.  However, when more of the roll calls are less contentious, it is more important to get at the behind-the-scenes activity to distinguish among legislators.  Hence the qualitative rating then receives greater weight, namely, a two-to-one rating when roughly half the roll calls are closely divided or a three-to-one rating when less than a third of the roll calls are contentious. This indicates that the lobbyist’s subjective evaluation of any legislator may be given twice as much or even three times as much importance as the legislator’s actual roll call votes on issues. The extent of this “weighting” by the 14 lobbyists in the “advisory” group is not available but it does clearly indicate the foundation of the entire survey results rests on the subjective opinion of a handful of lobbyists. We don’t know now, but it would be valuable to know, which lobbyists, paid how much by what clients were the ones used in publicly branding (in hundreds of thousands of so-called “impartial scientific” “roll call” publications) some legislators as voting to “hurt” the Maine economy.
 

            The prime criticism for such a system is that it enhances the likelihood of a partisan division in the ratings.  Which is exactly what happened in the MERI survey. All Republicans received good ratings and all Democrats bad ratings. When the roll calls are closely divided, there has been a failure to achieve consensus.  A prime factor that undercuts legislative consensus is the party divide.  Nevertheless, this is simply a reflection of a basic underlying fact that there are real differences between the parties on many economic issues.

            Another criticism is that a roll call that reflects some consensus could be on a bill that is highly significant for the Maine economy and businesses.  In choosing their roll call nominations, governmental affairs representatives may choose those on which they had to fight the hardest to win.  Other, “more important” bills are passed over because they were not such a tough fight.

            It is not specified how the two ratings are combined.  Presumably when the one-to-one ratio is used, it is a simple average. “It is not specified” and “presumably” – if the MERI paid evaluator doesn’t know how the figures were arrived at – how can anyone else know and how can anyone actually “evaluate” the accuracy, value, use, impartiality of the survey? In mathematics one cannot “average averages” to reach an average, particularly when it involves two entirely different rating systems – one actual votes and the other opinions of lobbyists – but, apparently, this is exactly what is being done.

Conclusion.  This rating system is one of the most sophisticated, most realistic and most accurate I have seen in decades of studying interest-group ratings of legislators.  It measures what business leaders consider the important legislative issues.  It includes objective and easily quantifiable roll call data, but with consideration of the meaningfulness of these votes.  It makes a good attempt to include other dimensions of the legislative process than roll calls. Based on the questions raised above one has to conclude that there is no factual justification for this conclusion. Two other Maine professors (Michael Hillard, professor of economics at the University of Southern Maine, and David Vail, professor of economics at Bowdoin College) have publicly (op-ed column Maine Sunday Telegram August 13, 2006) called the MERI survey and its ratings of legislators “flawed and one sided.”

            I have critiqued each component of the rating system with the intent to alert a reader to the points at which a critic can challenge the rating.  There are elements of the system that have a potential for inaccuracy and bias, This is certainly true, one might note for instance, the omission of votes on legislation proposed by Democrats and passed with bipartisan support.  but the MERI system does well at eliminating or checking against many problems found in other interest-group ratings.  Fundamentally, the attempt to calculate a summary rating does oversimplify the complexities of political philosophy.  A challenge to the rating need only point at some complexities that might not be captured.

            Nevertheless, for the purposes of providing information to a business audience that does not have the time or other resources to research the records of their legislators, the MERI system does a first-rate job.